The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

A data agent can write valid SQL and still send the wrong number into a board deck. In this Typedef workshop, co-founder Yoni Michael starts with the context needed to trust those agents; Brandon Callender then builds a smaller version for a coding agent. The recurring problem is that reading code or rows does not automatically reveal what the system is supposed to mean.

Three ordinary data-engineering tasks expose the gap: identifying what a proposed change could break, choosing safe joins, and editing transformation logic without changing a metric’s definition. Their context lives across warehouse rows, transformation code, BI definitions, documentation and people’s working knowledge. An agent needs to connect those pieces before it can judge whether an operation is safe.

Grain means what one row represents. A customer-by-month table and a customer-by-week table can contain familiar columns while requiring different downstream calculations. Change the grain without updating those calculations and the SQL may compile, execute and quietly return misleading sums.

The concrete failure is a pipeline whose row count jumps 40% overnight. An agent queries the data and concludes that more records arrived. But an upstream change actually turned a one-to-one join into a one-to-many join: existing records now match several rows and appear repeatedly in the result. Sampling the output establishes that the extra rows exist; it does not, by itself, establish whether they represent new activity or accidental duplication. The missing comparison is between the intended grain and the grain the join now produces.

Giving the agent a dbt or Airflow project makes it reconstruct relationships within a limited context window, then repeat that work in the next session. Giving it database access lets it inspect values, but does not supply their intended meaning. The proposed context layer preserves a model of system behavior that either form of access alone leaves the agent to rediscover.

The context layer makes four kinds of information queryable: row grain, join safety, metric meaning and relationships between assets. Join syntax shows why these must be connected. An equality join can preserve rows or multiply them, depending on the matching keys. A BETWEEN condition can support an as-of lookup or match enough dates to expand the result’s grain. The SQL expression alone does not settle which behavior the underlying data will produce.

Relationships also help recover business groupings. Table and column lineage, shared keys and shared naming provide signals for clustering models into domains such as revenue or product. A question like which models belong to the revenue domain can then become a query over stored relationships, rather than a fresh conversation about folder conventions and tribal knowledge.

The extraction work has distinct sources of evidence:

• Transformation structure: Parse SQL into abstract syntax trees, or ASTs, to derive lineage and compute grain from the implemented transformations.
• Observed data properties: Profile warehouse tables to check assumptions such as whether a join key is actually unique. This grounds cardinality decisions in the data.
• Human-readable meaning: Use an LLM for business names, summaries and ambiguity that remains after structural analysis and profiling. Generated descriptions sit on top of the implementation-derived facts.

Why store this in a graph? The useful relationship is between several evidence sources and one queryable model: transformation logic explains how assets are built, while profiles check properties of the resulting data. The diagram separates those inputs so that profiling does not appear to be a consequence of parsing, or an LLM summary the source of structural facts.

Michael describes a graph refreshed atomically on commits through incremental updates. Typed nodes distinguish assets such as semantic views and physical tables; typed relationships carry provenance, including materialization and semantic-definition relationships. At query time, the agent retrieves the relevant subgraph instead of rereading the entire project. That access pattern is the reason for the graph: dependencies become traversable information rather than connections the model must infer again from a pile of text.

Impact analysis returns to the grain example. Before changing a monthly model to weekly rows, the agent walks its downstream dependencies. A monthly ARR measure repeated on weekly rows can then be summed several times by an unchanged downstream calculation. Michael’s example produces roughly four times the intended value and carries it into a board-level dashboard. The important timing is before execution: the graph lets the agent inspect the proposed change’s consequences before they become a production incident.

Join recommendations apply the same discipline to attributing ARR by marketing channel. The agent checks key uniqueness and the profiles behind candidate paths, rather than choosing columns whose names look compatible. The example distinguishes a safe path from alternatives that cause sevenfold row fan-out or silently drop 12% of rows. The useful comparison is what each path does to the represented population: multiplication and loss can both corrupt the attribution.

Safe refactoring requires preserving business predicates too. Removing a condition that restricts a metric to paid customers leaves valid SQL, but changes the definition to include trial churn. In the example, that altered definition reaches three dashboards. Treating the definition as a constraint gives the agent something to check when a seemingly harmless cleanup changes who is counted.

Typedef evaluates these tasks through repeated runs on isolated clones of data environments, combining output pass/fail checks with LLM judging for some impact-analysis work. Michael reports an almost 35-point increase in task success when the same models and tasks receive access to the context layer. This is a reported result for Typedef’s evolving benchmark; the presentation does not supply enough protocol detail to turn that figure into a general performance guarantee.

The implementation lessons follow directly: derive structure deterministically where possible; “store evidence, not conclusion”; retrieve through traversal; curate tools for the actual tasks; and maintain evaluations continuously. Even warehouse MCP tools needed adaptation to fit the intended operations. A persistent graph supplies useful context, but the tool interface and the tests determine whether an agent can put it to work.

Callender makes the same problem small enough to build in a notebook. The workshop indexes selected Rust code from Qdrant and stores relationships in Neo4j. Local setup uses Docker or Docker Compose for the database; live LLM calls can use OpenRouter, while the supplied material also supports following along without live model calls.

The starting query is PostingList. Ripgrep finds two types with that name, plus lookalikes such as a posting-list view and iterator. A matching string does not identify which type the user means, which crate defines it or which methods depend on it. The agent can read around the matches and reconstruct those relationships, but its exploration order can change between runs. A plausible first match can send it down the wrong branch.

The goal is greater consistency across attempts. Extract each symbol’s identity and location once, preserve the relationships as edges, and query them repeatedly. Callender distinguishes raw string search, ranked lexical or semantic retrieval, and a traversable graph. Ranked retrieval improves which locations the agent sees first; explicit relationships reduce how much connecting work it must perform after retrieving them.

Building that graph requires another distinction: syntax, resolution and meaning. Tree-sitter supplies syntax trees, but parsing two declarations does not resolve their identities or references. Rustdoc JSON provides compiler-derived information that separates the two types. Only after those deterministic layers does an LLM enter to classify and summarize. Rust is a deliberate shortcut here: the compiler has already done much of the resolution work that would make another language harder.

Notebook one works over a selection of Qdrant crates rather than the entire repository. Extraction helpers run Tree-sitter and process Rustdoc information, producing both syntactic symbols and resolved versions. The parsing machinery is supplied in the workshop code rather than developed line by line during the session.

A re-export makes the difference inspectable. Syntax extraction can recognize that a PostingList is exported without establishing where it was originally defined. Resolution connects that exported name to its canonical identity, source path and line. The same item can therefore appear through another crate without becoming a new, unrelated entity in the agent’s understanding.

A full-text index over the extracted symbols still returns separate entries for re-exports and structs. The agent could infer which belong together, but then it must repeat that inference whenever it searches. Loading the resolved symbols and relationships into Neo4j preserves the connection directly. Search finds candidate entities; the graph supplies relationships between identified entities.

The first consequential tool is blast radius. It accepts a canonical qualified name and traverses the stored relationships to find dependent symbols. That changes the question from which files mention this string to which identified symbols depend on this entity. The result supplies candidates to inspect when assessing a change; dependency alone does not establish that every possible edit will break every dependent.

After a live-demo stumble, the returned relationships include methods and places where the struct is returned. Querying blast radius separately for the two PostingList types produces different result sets. This is the payoff from canonical resolution: the shared spelling no longer merges their separate consequences.

A small Pydantic agent receives a system prompt and tools backed by Cypher queries, including blast radius and implementer-related queries. Asked for dependent symbols with file and line locations, it searches first, then makes two blast-radius calls. More tools can expose the same graph: list all symbols in a file, or find structs and the traits they implement. Supplying the node-and-edge schema also lets an agent translate a natural-language request into Cypher.

A short semantic-enrichment example uses Fenic, Typedef’s dataframe library combining traditional operations with LLM calls. Each row supplies an item’s kind, qualified name and docstring; the model answers what that Rust item is for in one concise sentence. Persisting such summaries in the graph is a proposed next step. Their immediate advantage is inspectability: a wrong summary can be noticed and corrected before an agent relies on it while changing code.

The first questions concern keeping the graph current. For a maintained coding tool, Callender proposes a cached canonical graph based on main, with a local copy updated as code changes. Incremental indexing avoids repeating extraction for unchanged files. The tradeoff becomes more attractive as the corpus grows and reconstructing its relationships during every agent run becomes more expensive.

Dynamic languages complicate resolution. Python does not offer the same ready-made compiler information as this Rust example; static inspection and type-analysis tools can recover some information, but runtime-dependent behavior remains harder. The workshop is consequently an educational example of agent context and tooling, not a recommendation that every developer build and maintain a code-analysis product from scratch.

The data-engineering benchmark uses a larger setting: Mattermost’s open-source dbt project, described as roughly 300–350 models, populated with simulated data. Scenarios include introduced failures, repairs and new work, evaluated for correctness, completeness and efficiency. The context is queried while the agent runs, so its value comes from avoiding reconstruction during inference. The intended scope is maintaining data architecture and making operations trustworthy, beyond producing analytical SQL alone.

A direct dependency query is useful, but giving it to an agent alongside familiar tools exposes another failure. The agent continues to glob, grep and read files. Sometimes it tries the graph, becomes confused and returns to its existing approach. Callender attributes that preference to learned tool-use habits: many familiar coding situations reward searching text and reading the matches. A better underlying operation does not automatically become the preferred operation.

The response is to make graph access fit those habits:

• Use a familiar entry point: Name the tool search and accommodate the multi-keyword queries Callender observes models sending to it.
• Return familiar evidence: Format results like Ripgrep output so the model receives information in a form it already handles well.
• Make the first call useful: An empty or confusing response can make the agent conclude that the tool is broken and switch away. Early results need to help it continue the task.

Notebook two compares agents using the same model: one receives graph tools, the control receives Pydantic-maintained filesystem probing tools. The cached evaluation contains 30 questions with five repeated attempts, including tasks suited to graphs, tasks suited to grep and controls useful for both. One example asks which public types are affected by changing a PostingList, requiring the agent to handle the name collision.

Expected answers begin with graph-derived data but are cross-checked against source code. That check matters: a graph should not become its own unquestioned answer key. Questions are chosen so their answers can be scored deterministically where possible, reducing the cost and variance of asking another LLM to judge an open-ended opinion.

The scorer checks whether an answer cites the expected evidence. Pydantic Evals supplies the cases, agent runner and evaluator, but the decisive work is defining what evidence a correct answer must contain. That produces a comparison the developer can inspect and improve, rather than a score whose meaning depends entirely on another model’s judgment.

With one attempt per task, the graph agent edges ahead only slightly. The more revealing comparison repeats each task five times and asks two separate questions: did the agent succeed at least once, and did it succeed on every attempt? Those measure different qualities. Occasional success shows that an agent can find an answer; repeated success shows how reliably it follows a successful route.

On success at least once, the approaches remain similar and grep has a slight edge. The graph covers public items, leaving some private-code control questions better served by filesystem search. Callender reports roughly twice the all-five-runs success rate for the graph approach and about 3.1 times fewer tokens on average. These results describe the cached workshop comparison over its selected corpus and tools; they do not establish superiority across repositories or task types.

Tool adoption needs its own measurement. Inspecting tool calls in traces can reveal whether the graph is being used at all. Callender illustrates a prompt change moving usage from 34% to 72%, then contrasts a naive prompt with one that nudges the agent toward search and getSymbol. Adoption is a diagnostic alongside correctness: it explains whether the new capability actually participated in producing the answer.

The practical evaluation recipe is to mine tasks from the corpus, anchor their scoring in evidence, test the scorer itself and repeat the runs. Include difficult cases and weaknesses instead of selecting only questions the new tools already answer well. An evaluation that exposes no failures gives little guidance about where the system needs work.

Could the graph be distilled into files that grep would naturally search? Callender considers that reasonable, though it was not implemented in the workshop. A Markdown corpus could preserve useful context; the difficult part is preserving explicit relationships and their provenance. For integration into existing coding workflows, he proposes exposing graph tools through MCP and promoting successful prompting guidance into skills.

Further extraction could add call relationships using SCIP-based indexing. That is an extension beyond the demonstrated symbol graph: knowing a symbol’s identity and relationships does not automatically supply every caller. In data systems, a catalog supplies a different complementary input. Parsed dbt logic identifies the model and its output table; catalog information adds the physical table’s columns and profiles. The graph connects those logical and physical descriptions.

Data-engineering evaluations can vary how much diagnosis the prompt supplies. An easy task names the suspect model. A harder task says a dashboard number has doubled and leaves the agent to trace upstream dependencies to the responsible transformation. Scoring must verify that the agent fixes the cause in the right model, rather than compensating for the bad value elsewhere.

The harness can accidentally reveal the solution. An agent may compare the faulty branch with main to discover the injected change, or use a revealing branch name as a clue. Those shortcuts bypass the intended diagnosis. ADEBench is mentioned as an existing framework using repository changes and warehouse inspection, but realistic fixtures still require a running database, the intended faulty data and internally consistent records. A dataset generated only for 2024 should not send the agent chasing the absence of newer records as though it were the target bug.

The uncomfortable result is that grep works very well. Building a graph and discovering a near tie can tempt its developer to keep adding machinery until the expected win appears. Callender describes that as a sunk-cost trap: the agent has a preferred approach, and the developer has one too. Understanding why the existing approach works may be more useful than repeatedly trying to displace it.

Trace inspection also changed the apparent conclusions. A tool returned file paths that were wrong, so a search result could send the agent trying to read the wrong location. Aggregate results had looked favorable to graphs while concealing the defect. Reading the actual calls and responses exposed a concrete failure that a scoreboard alone did not explain.

“Never trust your own scoreboard” is the resulting warning. Read the traces, identify the misleading artifact, harden the harness and rerun. Monitoring must establish both that the agent uses the intended tools and that those tools return usable evidence. More evaluation runs cannot compensate for misunderstanding what the harness is measuring.

Rust made the lesson easier because compiler-backed resolution was available. In Python, a common name such as client can return a mass of unrelated matches, while finding implementers is less direct than searching Rust’s implementation syntax. Callender expects stronger value from structural context in such settings, but the workshop does not demonstrate a Python performance comparison. The proposed design keeps ingestion modular so a harder language need not force a new graph or tool interface.

Meaning should be added through small, auditable operations. Comments and docstrings help explain intent, while parsed structure establishes what is present and connected. Instead of sending a large SQL model to an LLM and asking for everything it can infer, parse it first and give the model one column, its type and its comments. Then ask a focused question, such as whether it is a measure or dimension.

If that question needs a better answer, add relevant context to that question instead of adding more questions to the same prompt. Fenic supports these focused classification and summarization operations over structured rows. The larger system can later incorporate call relationships, code comments, GitHub issues and tickets, building semantic understanding on top of extracted structure rather than asking one model call to invent the whole map.

Cross-repository and cross-language dependencies remain additional work. The coding example is centered on one repository; connecting Rust and Python would require recognizing their calls across the language boundary and adding the linking relationships. A graph cannot traverse a connection that its ingestion process never identifies.

The comparison with semantic retrieval clarifies the intended experience. Retrieval finds likely locations that the agent must then examine. A resolved graph aims to provide the equivalent of an IDE’s navigation from a known symbol to its references within the agent’s tools. Reaching that experience at scale depends on synchronization: avoid querying the warehouse hundreds of times or reprocessing all SQL whenever a small part changes. Callender prioritizes incremental indexing before storage optimization, while also noting the compute cost of graph analysis.

Evaluation data has a different requirement. To compare agent versions, freeze the repository snapshot and the questions. Otherwise, a changed score may reflect a changed task rather than an improved agent. Testing the current codebase remains useful, but it answers a separate question from a stable benchmark.

When a refactor makes an old buggy function obsolete, the original benchmark can still remain valid against its original commit. Callender defines the case through a specific commit, an injected diff and the intended resulting changes for the prompt. Repeating that circumstance tests the same capability over time; it does not require pretending that the old bug still exists in today’s repository.

The closing discussion widens the graph into an ontology: organized concepts describing where assets live, what they mean and how they relate, available through agent tools. An audience member poses the operational need—a small team wants versioned semantic relationships across SaaS systems, internal software, warehouse data and BI, useful both to agents and to people learning the systems.

The intended coverage extends beyond a warehouse catalog to source systems and transfer mechanisms, including Salesforce, Fivetran, Airbyte and Airflow. The point is to trace a metric through transformations and joins back to its origin—for example, Stripe data joined with Salesforce lead data—while reducing the manual upkeep that makes semantic layers burdensome as new models arrive.

The business user’s question is simpler: a revenue-dashboard number looks wrong, so who should fix it? Today that can become a ticket passed through triage and several teams. The proposed agent starts with the named dashboard, traces upstream to a faulty selection several layers back, identifies the owning team and connects the error to a recent change and the Airflow run that populated the affected data. It can then route a ticket with the diagnosis already attached.

That end-to-end workflow is explicitly a product ambition, with Typedef described as partway there. In the ideal version, the responsible team reviews the proposed resolution and approves its implementation. The destination gives the technical layers a practical purpose: connect a business symptom to the data, code, runtime event and owner needed to resolve it.