Concepts
Typed entities
Typed extraction and canonical normalization, and why they make a small model behave.
A row is flat. It has a sender, a title, a body, some fields. A graph needs things: this person, that organization, this channel. Extraction rules are what turn the first into the second.
What a rule declares
An extraction rule points at a field in a row and says what that field is.
| Part | What it says | Values |
|---|---|---|
type | What kind of thing the field names. | person, organization, location, channel |
role | What the row is to that thing. | from, owned_by, attended_by |
weight | How confident the rule is, in [0,1]. | a number |
canonical | How to normalize the value before it becomes a node. | email, phone, url, lower, raw |
A mail row with a from address produces a person node, with the row related to
it in the from role, at whatever weight the rule carries.
Canonical normalization is the whole trick
canonical is the small part that does the heavy lifting. Two rows can name the
same person with different text, and without normalization they become two nodes
that never meet. Normalize first, and [email protected] and [email protected]
land on one node, so a fact learned from either row is a fact about the same
person.
This is the difference between a graph and a pile of strings that happen to be similar. See why not files.
Shared values become edges
The second kind of rule works between rows rather than inside one. It connects rows that share a common value at the same path: the same thread id, the same document reference, the same channel. That is the mechanism that turns a table of independent rows into a connected graph.
Retrieval then uses those edges. expand_graph is on by default in
answer(), and it pulls in one-hop
neighbours at a discounted score.
Which rung a rule lands on
A rule reading a declared field produces structural evidence, the top rung. A
rule matching two normalized values produces lexical_canonical. Neither of them
involves a model. That matters, because the alternative, asking an LLM to read
the text and assert the relationship, sits at the bottom rung as
llm_extraction, and the graph marks it as such.
The full ladder and what each rung means is in how Cortex re-fits data.
What you can declare today
Be precise about what that means for a feed of your own. Your rows are embedded, searchable by vector and full text, returned with citations, and they become nodes in the graph. What they do not get is entities and edges: a feed registered without rules produces neither, so there is nothing for one-hop expansion to add on top of those rows. A built-in integration's rules apply to that integration's feed, not to yours.
cursorCol is the field worth your attention here: it drives incremental sync,
so each run picks up where the last one stopped rather than re-reading
everything, which is what keeps a real database in step.
For pushing rows without standing up a feed table of your own, see
ingest and query. Any extra top-level
key on a row is stored verbatim in raw_data and stays resolvable.
Why this makes a small model behave
Two reasons, and they are both structural.
The expensive work already happened
There is no LLM at query time. Embedding and extraction were paid once at ingest, so the model you run at query time is not the model doing the semantic work.
The context carries answers, not a haystack
Retrieval returns a small set of scored, typed, cited results rather than a pile of documents. A small model asked to pick between four cited facts behaves far better than the same model asked to read forty pages and find one.
Context and inference cost works through what that means for what you pay and where the model runs.