The Data Context Layer for Agents — Yoni Michael & Brandon Callender, Typedef

We'll get going. Uh, my name's Yoni Michael. I'm one of the co-founders of a startup called Typedef. Uh, we're here to do, along with, uh, our founding engineer, Brandon Callender, this workshop. Um, we have a two-hour time slot. We're gonna try and keep it as interesting and fun as possible. Uh, I know it's very tempting right now. You guys all have your laptops out, and Germany's down, uh, one nothing in the World Cup game. But it's our promise to try and make it as fun as possible, so you don't feel the need to be streaming that. Um,

so, um, what we're gonna talk about today, uh, is we're gonna do... Like, before we actually get into the fun, roll up your sleeves, do more of a technical workshop, uh, I wanted to walk through a little bit around the problem set that we're trying to solve here at Typedef, um, and kinda set the stage and give some context for concepts and principles, uh, that will later be applied to the actual workshop that Brandon will lead us on. Um, so the main idea behind this talk, um, is the background on Typedef. We're building,

uh, the enter-- the trust layer for data agents to be able to access and perform, uh, high complexity tasks on data platforms and data systems. Um, and so the title of the talk is The Data Context Layer, which is a core piece of infrastructure, um, that we've actually built to help us with that mission. Um, and so, uh, a lot of it, uh, comes from the idea that perhaps maybe you guys aren't as much into the data space, but once you try and unleash, uh, coding agents or software engineering agents on the data,

it's actually a lot more of a nuanced, complex environment, and there's a lot of things that you need to think about when building the right harness, when context engineering for these agents to be really effective. That's what we'll be talking about. The punchline is, like, the, the model isn't the bottleneck. The model is gonna continue to grow in capabilities, um, but the missing context to make these agents actually effective is the important part in this. Um, so this is what we call the data context layer.

Um, all right, so here's the bet that we're taking as a startup called Typedef. Um, all of us have spent, let's say, like, uh, the last two years generally in the industry, um, going and thinking about context engineering for coding agents, right? So we have, like, the, the Anthropic's, the Codex's of the world, the Cursors, um, where you're able to understand how to map repositories effectively, do retrieval, and then tool design for these coding agents to be really effective. Um, where we're falling short and where we're starting to see there's more of a need for is that the

same thing needs to be done for agents that are able to access and touch the data. And the data is exactly where, uh, if a-an agent is confidently wrong, which it oftentimes is, it can actually end up costing a lot of money, right? Because a lot of these metrics and calculations that we're asking, uh, agentic analytics to actually do wind up on things like board decks, um, or their metrics in a model or their pipelines that fail quietly that have really, really important features and, and, um, and metrics and measures that they're trying to calculate.

Um, so the core bet that we're saying is that agents are gonna be core operators of these data systems. In order for them to be really effective, we need to be able to ensure that when they do read and write actions, they're able to do these actions safely and accurately and be able to trust these results at scale. And that only really works if you're able to give the agent the reasoning layer so that it can actually understand the data system end to end, very much like how a data engineering or a data team that actually has working

knowledge that was able to build the system from end to end is able to also reason about it. So that's the goal of the data context layer here.

Um, so where I'm gonna say and what we found over, let's say, the last year and a half of us building in this area is that there are three general concrete areas where today's, let's say, coding agents fail over when you're trying to get them to be effective in data. The first thing is impact analysis. So essentially, like, what breaks if I actually make this change? Um, the next thing is join recommendations. So if the-- i-is this actually-- is this join actually safe for me to make across my data system? And then safe code changes generally as well, right?

Safe transformation logic changes. So does this edit still mean the same thing? Is it preserving the meaning of this metric, or is this metric definition now gonna change because I'm making this data commit and this data change that's happening? These aren't fringe tasks, right? These are, like, the day-to-day, uh, tasks and workflows of, like, data teams, um, as they're thinking of how to actually get enterprise data to be working, um, with AI. And if your agent can't reason and do these tasks reliably, it's very hard to actually trust it, and very

hard to actually trust it to give it write and even read access many times. Um, and that's what actually matters, is be able to get them to be effective core operators of our data systems and not just, like, question and answers types of agents.

So here's the gap, the core gap that we've been able to define and, like, really discover through our work here. Um, this context that the data needs is, uh, ag-is, uh, distributed across many different systems, right? So your warehouse can have the store. It stores the, the rows and the actual data itself. Your transformation code has the actual business logic that's really important. Your BI tool stores the metrics, the dimensions, some business definitions in there. And then you have your docs and Slack channels and, um, and, uh, Slack threads and, like, the tribal knowledge that

oftentimes lives in, like, people's heads as well, right? But the agent actually needs something in between all of that. It ag-needs to actually understand what a model means, how it relates to other models, what breaks if I make this certain change, um, and then which operations are actually safe to make, right? Um, and for the next, let's say, fifteen minutes while I'm going over this, like, we'll be using, like, um, a term that's called grain. So for all of you, like, non-data folks in the room, what grain actually means from, like, a data team

perspective is, um, is, uh- Um, sorry, is like what does each row of the table actually mean? What does each row actually represent? Um, and so if the grain is customer by month, uh, and I accidentally turn it into customer by week, then every downstream sum after that, uh, can become wrong even though the SQL compiles and everything runs correctly, right? So that's what I mean when I talk about grain change. Um,

and so let's make this a little bit more concrete, like in terms of reliability, what does that actually mean? If someone asks an agent why a pipeline's row cou- row count suddenly spiked and it jumped forty percent overnight, um, the agent can go and write valid SQL. It runs it, it could render a beautiful chart, but at the end, it's gonna report the wrong root cause, the ri- the wrong analysis of what actually went wrong. It says, "Looks like there's more data that came in," um, but what actually happened is there was an upstream column that,

uh, turned a one-to-one join into a one-to-many join, right? So that means the grain actually changed. Uh, and the agent literally cannot tell a real spike, like a real data spike, and differentiate between that and like an actual grain change because the grain isn't in the data itself, right? Um, and so the agent can run the query, it could get some sample rows, it could sample the data, it could read the SQL very well to understand, but it didn't know that what one row was actually supposed to mean, and that now it means

something completely different, right? So this is why a valid SQL isn't enough, right? The semantics are actually very, very important when you're trying to get an agent to be effective.

So then let's say like the, like thinking on it, there are maybe two obvious fixes for what you can do for this type of scenario. And this is why n-neither obvious of-- like neither of these obvious fixes actually works, right? So you can give the agent the code, right? But the problem with that is if you go and give it, let's say like your dbt project or your Airflow project, it's rederiving the context every time from scratch, right? On every single prompt, which means it's probabilistic, um, it's bounded by the context window, which doesn't always work very well. And then every

time you start a new session, you have to go and rebuild that context from scratch. Then let's say, okay, let's try and give it database access. So I'm gonna connect my Snowflake or Databricks MCP to it. Um, and instead it can see the rows, and then it can pro-- it, it can underst-- it can see, it sample some data, but it doesn't actually have an understanding of what each row actually means. So it can read the numbers, but not necessarily explain it, right? And so the first situation is one can read, uh, uh, one can read the code, but doesn't actually have

memory to be able to persist the knowledge of the context that you spent a lot of time going back and forth and helping build, build up. And the other sees the data, but doesn't actually associate the meaning behind it. So neither has like a persistent model of how the system actually behaves, and that's the mission that we set out to try and solve.

So let's like take a step back and name exactly what's missing from the context that you need to actually give the agent to be able to reason effectively across these, uh, uh, complex data changes. There are four facts, let's say, that decide whether an answer is right, and none of them are actually written down anywhere that the agent can read or infer. So the first thing is what, what one row actually re-represents in a table. This is what we talked about, the grain. The next thing is whether a join is safe or not. And then there's what a metric actually

means and what relates to what, right? The dependencies are very important, um, in data systems. And so an agent can read your SQL just fine, um, but it can't see these four things very easily j-- at query time when it's the most important time for it to gather the, the exact context that it needs. So the goal of what we did was, let's see how we can make them computable and queryable so that the agent is effectively able to grab just the context it needs, just at the time it needs, and not have to rebuild it during every single

session. So the next few slides are about how we actually went about doing that. Um, and so let's take, for example, the join idea. This is like the, the second fact that we talked about in the previous slide. And this is because let-- This is a good example because it's a cleanest example of like why syntax isn't actually enough to be able to get the agent to be effective. So there's one join, we call it like an, an equi join that's happening here. That's written a certain way, but it could have four completely different outcomes depending on the underlying data,

right? So it can either preserve your rows, it can look up a dimension, or it can silently replicate every single row when it goes from like a one to n type of relationship. The same thing is for this between clause that happens oftentimes in SQL. So it could be a safe as of lookup or a date spine that actually explodes your grain, right? So the agent can see this syntax, but it can't actually understand which of these scenarios is gonna result because of this syntax, syntax. And that's not really a model or capability problem, that's like a context

problem that we're actually trying to arm the, the, the, the agent with.

The fourth fact in here, and this is one of the more fun ones that I like to, um, to show off, is that, um, understanding of like what models, what data assets relate to other ones, right? So this is which models are in the revenue domain, for example, which are in the product domain. And those oftentimes correlate to the personas of the people that are actually trying to access or run the data to access these, uh, the core metrics or numbers that are in these, uh, in these domains. Oftentimes today, like teams will organize it based on folder names, and a lot of times it's

tribal knowledge to understand which models relate to which, um, business units. But you can actually discover it from the, uh, you can actually do more and actually discover it from the structure of the data. So for example, you can take a look at how the models actually connect, looking at lineage, for example. Table level lineage, column level lineage. Looking at shared keys, looking at shared naming. So the system clusters them automatically for you into the domains, so that when you go and ask the agent, "Okay, which models are in my revenue

domain for the rev ops team?" Um, it becomes a query that the agent can actually run and not a question it has to ask, like, you know, that has to go back and forth with the human to be able to understand which are in the same domain. So this is kinda the same principle for context engineering that you see over and over again. It's you take something that is tribal knowledge that lives in someone's heads or it lives in some, uh, documentation elsewhere, and you try and actually compute it and then make it queryable for the agent.

Um, so how do you actually produce these facts, right? And the, the key, uh, principle here that we talk about a lot is, like, you wanna always be as deterministic as possible first, and then only use LLMs when you're trying to resolve ambiguity or do things that models are actually really good at, right? And so you're able to compute grain and lineage straight from the actual SQL itself. So what we do is we actually take all the transformation logic, and we parse it all the way down to the ASTs to actually understand how the data is actually structured, right? And that way,

the agent can't drift from the actual implementation itself because the, the basis of the graph and the context we're giving it is actually derived from the implementation itself. Um, and then, uh, and then you can do things like verify join safety against the actual data, right? So is the join key really unique? This is something that you can do by profiling the tables, right? So you're able to go and profile the, the tables that you have in your warehouse. Um, so that allows the agent to not really guess about things like cardinality. And then you can use the

LLM for things where the human-readable parts of it, right? So if you're trying to understand business names or summarize the utility of models, or resolve any ambiguity from, like, the first two layers in there, that's where the LLM can really come into the picture. Um, these facts are computed and then checked against your data so that the LLM, um, uh, just kind of names these data assets at the end on the graph that we're building. And that allows it to make sure that we're not actually drifting from the core implementation.

And that separation matters because, um, it keeps the core facts tied to the actual implementation, kinda like we talked about, and to the data itself.

So that's all good and great. Where does this all live? What we call it, we call it the data context layer. At the end of the day, it's a graph, right? And it's computed once, and then on every commit, it's refreshed atomically, right? So we're incrementally modifying the, uh, this context graph as data teams are making more data commits to it, so it's always representative of the, the freshest state of, uh, your production enterprise data. Um, and the reason is a graph, the reason it's a graph and not just a pile of text or the

robust set of markdown files is because of the access pattern, right? So the agent's able to pull exactly the, the subgraph it needs in one traversal, um, and it avoids things like prompt stuffing or context stuffing, and it scales way past five hundred models because you're not, like, rereading the project every time you're trying to build the context layer, right? Over the last few years, we've obviously seen a huge resurgence of, like, graph databases, and I've always, like, wondered even, like, before AI, like, what the main utility of

it is. But it turns out that models and AI is very, very good at being able to, to grab context and, uh, be able to write Cypher, be able to write a query against it, and then traverse. Um, and the nature of how data assets are, it's very dependency-based, so it yields really nicely to be able to create these, uh, typed objects and nodes and edges in this graph. Um.

Sorry, we can't see the slides.

Oh.

want my

You guys can't see the slides? Oh.

Turn off the lights.

Can you guys turn off the lights? I don't know. No?

Oh, okay. Um, yeah. Oh, that's better. Getting a little better. We dim them a little bit more.

Sorry, guys. I should have, uh, thought of the, the black background and other colored, um... But I can, happy to send this out afterwards also, the deck.

All set.

Okay. Um, apologies for that. Um,

so, um, yes. Let's see. So you can see, or maybe you can't see, I guess. Um, but there, there's, like, a sample of, like, a subgraph in here, and you can... And what you're, is actually represented is that the edges carry a lot of provenance, right? So the edges are defining different types of relationships amongst the, the data assets themselves. We have, like, for example, um, uh, is, is materialized by, uh, uh, has a semantic definition, and these are all nodes that are representing different, uh, types of

data assets in the actual graph itself. So there'll be, like, a node that represents a semantic view, for example. Um, and then there will node, be another node that will present, like, a physical table, which is, like, the actual physical representia- representation of the logical code, um, of the table itself. Um, I'll send this to you guys afterwards so you can take a closer look at it, or we'll figure out how to get it to you guys. Um, and, uh, and so yeah. And so this is, like, essentially, like, what a, a subgraph will look like. Um, and so if we go back and think about the use

cases that are relevant that the agent can now solve because of this context layer that we have, um, is, is, is impact analysis is the first one, right? So the agent proposes a change, um, for modifying, like, a model's grain, for example, right? Before it actually does that, before it writes a line of SQL to make this change, it can actually walk downstream the graph and ask each node that is a downstream dependency of this node it's trying to change- Does this actually break the grain that you have represented in this node here? So here

it's able to catch that, um, uh, a monthly ARR measure would get repeated across weekly rows if I'm trying to go from the monthly grain change to the weekly grain change, and that downstream there's some ARR that's actually summing all of the different, um, values for this on a monthly basis, which would result in, let's say, the metric having like a four X, um, of the actual value amount that it's actually accurate, right? And then it could land on, let's say, everything downstream on like a board-level dashboard that you have because the context

graph is inclusive of your entire data system, everything from like source systems all the way down to the BI layer and the dashboards layer. So the agent has visibility and able to walk the lineage all the way across the, the data life cycle, if you will. So the key part is this all happens pre-flight, right? It's not at build time, it's not in a production incident. It knows a blast radius before it actually even acts on it. And then the second use case here is join recommendations, right? So this is fact two. Um, so this is like the agent wants to attribute the, the exercise here is like the agent

wants to attribute ARR by marketing channel, right? So it doesn't pick a path because of like column names that it's able to understand. It actually checks whether each join key is actually unique, and this is done because we're profiling the actual tables themselves. Um, and it's verified against the data, right? So then it has the recommendation of what the actual safe join path is to make this happen while flagging the ones that will actually cause like a fan out or like a seven X, um, in, in rows or one that will suddenly, like

silently drop twelve percent of the rows, um, downstream as well. So we're making recommendations with actual reasoning and evidence is the key here, and it's grounded in the data itself. And then safe code changes, right? This is the third fact. Oftentimes you'll find like business definitions. The reality is they're oftentimes nested in the SQL itself, right? And so if you're asking an agent to, let's say, tidy up some measure or some metric that you have and it removes a predicate, um, syntactically that will work, right? The, the SQL will still compile.

Um, but what it's actually happening and it, it can-- could look harmless, but the predicate is, was actually a very important part of the business definition itself, right? So here in this example, it's dropping the definition of, um, of paid, right? So it, it's si- it's silently redefining what the metric, uh, the metric to include trial churn, uh, across all the way downstream three dashboards that are actually happening representing this metric. Um, so this context layer actually catches this, uh, j- uh, change in, uh,

business definitions and business metrics before it's actually implemented. Um, and so this is like if you're making the agent ser-- like have these as like first class constraints, it's not just a comment that is in some YAML file or something like that. That's the key here, and that's the line between like a, a syntactic refactor and a change that a human can actually trust.

So all good and great, but does it actually work, right? Um, and so you have to, you have to measure this. Anytime you're building anything agentic, you need to be able to measure it. Um, and so we're, we practice like eval-driven development, which is a very important part of it. We're essentially going and creating a bunch of like real production data engineering use cases and tasks and workflows that range in difficulty from like easy to blind, and they each run multiple times and they're is-- they're each running on isolated clones of the actual data environment, which is a very important part

so that there's no actual leakage that happens across these environments. And then we have pass-fail that's running, checking the actual output of it. We have some LLM as judge doing like impact analysis things as well. Um, but the key findings here is that although it's the same models, it's the same task, the only variable that's changing is whether the agent has the access to the data context layer. And so you see a material lift, um, when it has this right context, it's able to reason across these really complex changes, and the task success rate goes up tremendously. It's almost thirty-five points

higher when it's armed with the data context layer. And this exact benchmark is obviously gonna keep growing and keep evolving as we're working with more teams to figure out their kind of, um, bespoke use cases in there. Um, but the signal is consistent, right? The better context, um, the more rich context you're able to give the agent, the outcome changes tremendously, um, than just let's say like swapping models or adding more generic like tools, MCP tools, um, to, to the model itself.

So, uh, I'll wrap up here, but before I wrap up the, the five patterns, and I'll hand it off to Brandon obviously, but the five patterns that are even if you're not building this exact system, these are some key takeaways that we've really learned the hard way actually over the last like year of us building this, right? The first thing is like deterministic always when you can, right? Don't ever try and take like your entire DBT project or your Airflow logic and just throw it to the model to be able to try and do these tough complex tasks. That, that won't work, right? Uh, um, what you really wanna do is be able to get a really deep and detailed

understanding of the transformation logic in a deterministic way. So parsing it all the way down to the ASTs to understand the structure, the relationship, the dependencies are very, very important. And then things like being able to compute grain, right? Like that's something that you can use. We spent a lot of time refining and like using li- like relational algebra to actually calculate and compute for each stage of the data life cycle. Um, and then use the model when you need to, when it's like summarizing models or, or summarizing the different data

assets or categorizing or classifying. Models are really great for that. The second thing is store evidence, not conclusion. So it's oftentimes good to just keep the signals and then let the agent be able to reason, 'cause these models are very capable, they're very smart, they can reason at query time effectively. And then the third thing is that traversal always beats prompt stuffing or context stuffing, right? Um, it's gonna want to let it pull the subgraph that it needs. Don't dump the entire repository. At runtime, it'll pull the subgraph and be able to traverse to

figure out how to solve the task at hand because all the dependencies and everything is cle- clearly mapped out there. And the context when-- Naturally, you'll have, be running more efficient AI because the context is gonna be a lot smaller every time, um, the task is trying to be solved by these agents. And then the fourth thing is like own the tool surface. So like what we found is like curated, a curated type tool set always beats like a buffet of just like off-the-shelf MCPs. Um, even like for the MCPs that we do have that give our a-

agent access to Snowflake MCP or Databricks MCP, we found ourselves like rewriting a lot of it to kind of match the capabilities and functionality of what we're trying to accomplish. Um, and then, um, let's say the fifth thing is we talked about like evals, evals, evals, right? This is, uh, the most important thing. You can't really build a reliable agent regardless of what the agent is doing if you can't measure it. You can't learn from anything you can't measure. Um, and so invest in like the eval infrastructure early and often, and it's a continuous,

like living, breathing, uh, piece of your software that you have to always maintain and up, and, and update, um, and run continuously. So to close, uh, this portion of the workshop, this hasn't been a real workshop so far, it's just me kind of rambling, but we'll get to the fun part now. Um, the-- what we found is the agents don't really fail at data engineering or data task because they can't read or write SQL. They do that. They do a very good job of that. They fail because they don't have, or they don't understand what the data actually means, how it flows, or

what breaks when it changes, right? So if you make that cl-- that context really a first-class citizen, um, you extract it, you compute it, you persist it, and then the agent's able to query it in a really efficient way because you have a graph. Uh, then you're able to cross the line between from an agent that only reads your code to one that's actually ready to like operate your data system. And that's like what we've built with this like data context layer. All right, um, so thank you for that like, uh, little brief talk, um,

that we've given here. I wanna hand it over to Brandon, who's actually gonna like start walking you guys through more of like a hands-on workshop that carries over some of the concepts we talked about in here. Um, I won't, uh, take any questions now 'cause I wanna make sure that we have enough time for Brandon to go through his entire workshop, but I'll be around like after this and whenever, like feel free to reach out to me. I'm happy to chat more about this. Thank you.

Good job.

Okie dokie.

How is everybody doing today? We're doing good? Fantastic.

We'll get going.

Sorry, give me one moment.

All righty, so the topic that I'm gonna talk about today, um, is, uh, definitely related to what Yoni was talking about. We're gonna try to bring these concepts together in a form that we can actually do in, uh, the rest of our time here. Um, so my name's Brandon. I'm at Typedef. Um, and you're gonna hear a lot of talks here at the conference this week about how an agent is only as good as the context you give it, and that building that context is in fact the actual job. And so this is basically a hands-on version of that exact same lesson.

Um, over the next ninety minutes, we're gonna build a context layer for a coding agent together, a real one over a real example code base. We're gonna be using the Qdrant code base today. Um, and then I'm gonna show you, uh, the last part, which is that even if you build-- when you build these amazing tools, um, the agent's mostly going to ignore it unless you help it along the way. Um, getting it to actually use the tool is going to be, uh, where the work really is. Um, so

before we go ahead and dive in, um, if you wanna go ahead and follow along, this QR code links to my GitHub repository, which does have all of the relevant code. Um, that repository has a few Jupyter notebooks for people to follow along with. Again, don't worry if you don't wanna have-- set that-- set everything up. The dependencies there are, uh, Docker Compose or Docker for running Neo4j. Um, and if you do wanna try out the LLM portions of it live, you can use an OpenRouter key.

However, that's not required. It'll all run offline if you desire. Or you can just follow along with me, uh, as I build it up here. Um, so, uh, I'm gonna leave this up for just a little bit longer and then, uh, I'm gonna start doing a little bit of talking for people to be able to set things up if they want to, and then we'll dive into the notebooks themselves. Okay? Everyone got the scan in, ready to go? Give it one more minute.

All right. Perfect. So let's start where every coding agent starts, which is text search. Um, I'm gonna ask about this type for this struct called posting list. Uh, it's a real type in Qdrant. Um, so when the agent goes, it runs Ripgrep, um, and you see, great, it found that we have these two different, um, answers that are exact matches. They're both exactly posting list. But we see it has a dozen more lookalikes. You know, we have a posting list view, a posting list iterator,

et cetera, et cetera. Your agent's gonna do the same thing. It's going to grep, it's gonna get this wall back, and it's gonna pick one, right? It may pick the right one, it may pick the wrong one. It's gonna explore that one first. Maybe it has to go back, backtrack, free the next one to actually figure out what you want, right? But it doesn't really have any means of knowing which posting list you want, right? It, it, it can't. The text itself doesn't have any concept of identity or, you know, which posting list is actually, you know, meaningful. Um, this-- the, the things you care about,

you know, what methods use it, where it's used, what crate that it's in, that's not necessarily just in the text. It's in the actual structure of the code itself. Um, so we just watched Yoni make this, you know, a very similar case for data, and it's, it's problematic to have the, the agent rederive the entire system- for every run with no memory. Um, because for one thing, it's going to be expensive, right? It's going to go through, it's gonna have to re-read the same set of files every time. If you ask it the same question

five times, it's gonna go back and read all those same files every time, right? But the important thing is that it's fragile, right? When it goes back and it reads those five files, it might read them in a different order next time, or it might read one file first that gives it the wrong answer and sends it on a goose chase to another file, right? So it's that issue of actually having this thing be deterministic and not as fragile, uh, that we're really trying to accomplish when we're trying to engineer context for these kinds of agents, right? The goal isn't necessarily to get the agent to be a

lot more correct or a lot more efficient, even though that's sometimes the case. The real goal is to have it be a lot more deterministic, right? Because we're trying to apply as much determinism as we can to this very non-deterministic system. Um, without, you know, this persistent, trustworthy model of the code, it's going to guess, um, every time. And, you know, what we see is that over time, the models have been getting better and better and better at guessing, right? You know, we can all see what happened when we ran Claude Code a year ago versus running it now, right? It's a completely different

experience. So, like, if we wanna build a context layer for code, we basically want to follow this exact same recipe, um, but, um, just at a slightly different, uh, level. We're gonna scale this down so that we can build a workshop, and you can leave with something that you can actually run today, um, instead of trying to replicate all of our, uh, our entire product up here. So we're basically going to take, um, that, that concept, this exact, this, uh, extract, persist, and query, um, to basically

go ahead and start extracting information out of our code base. So we're gonna pull out information about the symbols, you know, every tr- every type, every trait, every function, uh, with its real identity and location. Um, we're going to-- the relationships between these different symbols get, uh, edges inside of our graph that we can traverse. These get computed once, and then they end up being, uh, queried many times. This is the same, uh, overall approach as what we do when we're tackling a sprawling data architecture, just with a small enough example to run in this room.

Um, and so, yeah, that's basically what I was just saying. Uh, so we're gonna stand it up today. So, uh, y- this is gonna be a concept that you're gonna hear about a lot during the conference. Um, I thought it was, I thought it was really cool. I, I brought up-- I made this slide of three levels of context, and I look at the schedule, and someone has a talk about four levels of context tomorrow. So I feel a bit preemptively one-upped. But we're still gonna go through this today because I think it's still, uh, pretty useful for this context here. Um, so you can think about context of a coding

agent in sort of these terms of levels. Uh, that's how I like to think about it. So level one is sort of what you just saw. It's the grep, it's the glob, raw strings, no structure, right? Level two is where you get, you know, most of the code RAG solutions that we have today, right? You get, um, semantic search, you index the code, you embed it, right? You do some nice lexical search, semantic search. You get ranked results, right, of all the, you know, the locations that this, this thing was mentioned in. But at the end of the day, that's still going to be a flat list, right? It's going to be a flat list that gives you more relevant results.

It's going to be still a flat list. It's not gonna show you the relationships between these different parts of the code, right? So the, the top level or, or the highest level at level three is, in our opinion, a traversable graph, um, something that links the symbols with real identity and adds typed edges between them that you can actually traverse. Um, so I think for the next part of this workshop, we're gonna go through and build up through these different levels, and right by the end, we'll get to an agent that can answer questions about the code base,

um, using this traversable graph. Um, and, uh, yeah. Let's, uh, let's, let's lay it out. So as we're building this, there's gonna be a few different, uh, stepping stones or rungs to get to that level three understanding of how we do things. So rung zero is sort of the syntactic level. First, you need to actually parse the code that you're actually trying to, trying to understand. And so the easiest way to do that is with something that's language-agnostic like tree-sitter, and that gets you a very basic

high-level AST breakdown of the different, um, parts of your code base and how things interact. Um, however, that still doesn't really have any idea how to deal with collisions, right? Posting list the symbol that's in posting list.rs and the posting list the symbol that's in, you know, quant.rs are two different posting lists. They're still gonna collide at rung zero. So the name of the game is going to be rung one. So resolution is the one thing that grep can't really do, right? It's

the thing that turns those two ambiguous posting lists into two distinct identifiable types, right? So with Rust, it's a little bit cheating. You kinda get rung one for free because Rust compiler will do a lot of this work for you. And, uh, you know, we'll basically be using rustdoc JSON to give us all that information basically out of the box. Now, is this cheating? Yes. Um, I picked Rust on purpose because I needed to build this workshop and, you know, building an entire parser for a whole other language or trying to build something out

is a little bit harder. However, at the end, I will talk about a more useful case of this on Python, and there is an example notebook in the repo for you guys to peruse on that if you don't get to it, uh, during this workshop today.

Um, the next layer, uh, that we're-- th- there is a w- there is a notebook for it, I'm not sure we'll get to today, is rung two, which is the actual meaning behind, um, these different things that we're extracting, right? So these first two layers are solely deterministic. We're basically always trying to be-- ground our understanding in as deterministic a base approach as possible. So we build out these first two layers using tree-sitter and rustdoc JSON, and then if we still have some time, we'll explore, um, how we build out that rung two layer. Like, how do we take that information, feed it into an LLM to

classify and summarize and do the things that the LLMs are good at, as opposed to just trying to throw everything in there and asking it to make sense of it.

Um, fantastic. So now we're gonna go ahead and open up our, uh, notebook 01.

And going to be confused.

Let me move this over here. Oh, actually, it's already here. Just have to navigate over to it. Okay.

All right. Can folks see this okay? Or did I bring them up on your side? Great. So this, uh, this notebook... What this notebook is gonna do is it's just going to walk through the process of building up those, uh, those levels of different context for our, uh, our Qdrant project. We're not gonna be indexing the entire code base today. It's a little bit big. Um, we're taking a look at, uh, a sampling of, I think, five crates. Um, but it gives us a good corpus of code to look over, uh, for our testing here.

Um, so we can start going ahead and running these cells. And so what we do first, we just do a bunch of setup. Um, and now that the setup is good, um, we can go ahead and start looking at the, um... Yes. How do we access it? Ah. Is it in the GitHub? It is in the GitHub, yes. I can put up the link again to that real quick.

Bring that up. Um, so the link is up here.

Uh, if you don't wanna follow along locally, there are Colab links in each of the different, um, things there. Yes. Uh, for the Colab, for the notebook one, if you do want to use the graph, you will need a Neo4j Aura login, but that's just a, that's just a special thing for that.

Okie dokie. So we can see here, um, there's a bunch of additional code in here to actually do all of the, um, parsing inside of, uh, Rust itself. I'm not gonna necessarily go over that today, but it's all available in the repo for you at your leisure. Um, we just have a magical function here today, uh, extract Rust crate, that will, uh, pull out the information from running tree-sitter across the Rust code base. Uh, and then another one that will go ahead and do the same thing, but for our information from, uh, Rustdoc.

So we can see here that from the first couple of crates, we pulled out, um, a bunch of symbols from tree-sitter, and then a bunch of symbols, uh, resolved versions of those symbols from, uh, Rustdoc.

So now we can go ahead and start running some code that's gonna start trying to understand why we're differentiating between rung one and rung zero. So we can see when we ask about something like posting list, in rung zero, we see that, okay, we see this exported version of posting list. We don't know actually where that originates from, though. We can't resolve that backward into the actual, um, location where it was defined. Um, so that's what rung one is for, right? It's gonna be able to say, "This is posting list. Its canonical location is here at posting list, posting list,

posting list," and that actual location is at this path, at this line, right? So this is what's gonna be able to let us figure out when things get e- re-exported outside of a crate, link it back to the original crate. The agent's gonna be able to traverse all the relationships between those different things, um, once we actually build out the graph itself. Um, so now we'll just build very quickly the level two that I was talking about before, um, where we have a kind of index, where we build a, uh, very,

you know, where we basically take all the symbols, and then we let, um, a function go that just does a full text search, right? So we do a search, and we get the same problem that we saw in graph, right? We can search for posting list, and we can get very useful information. We can see this is the re-export, this is the struct, this is the struct. But it's still not gonna tell us the linkages between those different things, right? And the agent can infer that, and it can, you know, make the, make the figure out, oh, probably that if the posting list, posting list is a struct, it's

probably re-exported up here, so that's probably the base index of it. But you don't want it to have to make that logical leap every single time, because it's gonna be that one time where it gets it wrong, and then it's gonna end up on a completely different wild goose chase.

So in our next step here, we build out the... We start building out the actual graph, um, itself. So we take that, all those different, um, uh, symbols and, uh, information that we just extracted. We transform that into a bunch of nodes and edges that we can populate into our Neo4j graph. Um, and then we go ahead and actually, um, load all of that in. So once that's in the graph, right, we can even go into Neo4j. Um, we can see we

have, you know, a beautiful little graph of, uh, references to all the different parts of our, uh, of our, of our code base here. Um, so now that we have this sort of graph, we can build tools that specifically harness the capabilities of the graph to give our agent access to something that's gonna make it easier for it to do its job. So if we build a tool, and we wanna say, "Okay, I wanna look at a function, and I wanna see

what's going to change, what's going to break if I change this function," right? So we can build this function, blast radius, where we say, "Okay, you pass in the qualified name of something, the canonical name. It's going to go in, and it's going to try to figure out, walk down the graph through all those different edges that we defined, and try to figure out exactly, um- What depends on this thing? What will break if this thing gets changed, right? And so in that sense, you can ask the agent to answer a question that would've taken multiple graphs across multiple different files, and

in theory, um, the agent can just answer it in one, in one go. So, uh, of course, that didn't work because why would it, why would it work? Live demos, everybody.

Okay. So we can see here when we say we're gonna pass the blast radius to this struct, we can say, "Okay, cool. It's linked here via ... It gets returned by this. Um, there's a method that does this. Um, it gets returned here." So we can see all the linkages that we're able to establish, um, just by having the content inside of the graph.

So finally, um, we get to resolve our collision problem. Um, now if we say we have two different posting lists and we wanna see the difference between them, um, we can call our new tools blast radius for this posting list and this posting list, and we get a different set of results. Changing one posting list is not gonna impact the same level of changes as you impact- change the other posting list. So even though these things have the same underlying name, the actual canonical resolution of them results in us being able to

more definitively say what's going to change when you break this thing, more than just going to graph, reading all the files, and then trying to guess. Um, so now we can actually take what we built so far and build a very small, limited function agent to actually use this new tooling. Um, now that we have, uh, this blast radius tool, um, we can wire up a very quick little agent. Um, this agent is very simple. It's just a Pydantic agent, um,

that, uh, it's just configured with a system prompt that says you have the ability to, uh, run these different tools. And we wrote some very basic tools for it to be able to use blast radius, the implementers methods of, um, that work with very basic Cypher queries in order to get the information out. So once you have this information in the graph, writing the queries to actually pull it out becomes quite simple. Um, if you can express the idea that you want to, uh, do, then it's pretty easy to write Cypher or the equivalent language in whatever

graph you're using, um, to actually pull that data out, and that's a sign that you have created the right schema for the job. So now if you go ahead and run this agent, this agent will actually spend a few turns, um, calling the tools and trying to figure things out. Uh, and then it will go ahead and, uh, try to answer our question. So for now, if you can say what breaks if I change posting list, tell me each dependent symbol with its file line, and then it will tell me exactly, uh, what will break, um, based on the

blast radius finding. And we can see here what tools it called. It started with search, did a couple of searches before calling the blast radius, uh, and then doing another blast radius call. So we can see that if you have an agent that is, uh, skilled enough in tool usage, um, it can use these new tools to, uh, build out its understanding, uh, of the code base. Now, this part is a, uh, more of an interactive section where you can build a,

whatever tool that you would like. Um, how this basically works is we can write a tool that basically executes arbitrary s- arbitrary Cypher. Um, so here we're write, writing some Cypher that says, "Find me all of the, um, symbols within this file path, um, and then just lay them out." So this tool will do just that. It will find all of the symbols in this file path and lay them out for me.

Um,

yes. So then if I wanted to write a new tool, I wouldn't even necessarily have to know Cypher, right? Um, I could give the agent access to a system prompt that has a schema in it, right? That tells the agent, "Okay, these are the different nodes. These are the different edges." And I can write, um, I can write my, um, my requests in natural language. Like, I wanna see, uh, every struct that implements a trait along with the trait that it implements, right? And if I

run, um, if I run that, uh, it'll take a second 'cause the agent's gonna actually try to figure it out, and it does. It actually generates this Cypher query that it can run, and it can go ahead and pull that information, um, out of the, uh, out of the graph. Um, we're not gonna get to, uh, rung two today, but we can get a little taste of rung two. So, so far everything has been very deterministic. Um, you get your structure and you get your resolution. Um, that third rung I was talking about was the actual semantic meaning. Um, you can actually

have an LLM summarize these symbols and, uh, embed them, and, uh, Typedef actually has an open source library for this called fenic. It's a data frame library for, uh, marrying traditional data frame operations and LLMs. Um, so here we're just doing a quick configuration of fenic where we say we're gonna have it talk to OpenRouter. Um, we're gonna pull in some information. We're gonna say, uh, we're gonna try to classify this specific Rust, or we're gonna pull in the kind and the qualified name and the doc string,

and then we're gonna ask in one concise sentence, "What is this Rust item for?" And so when we run fenic over this, over this corpus, over these rows, we get back, um, the summarized outputs from the LLM. So in the long run, right, the idea would be to make this information up into your graph so that this information is also available, um, for the agent. And the nice thing about this is that this is auditable when you generate it. So if you're looking at this and you say, "Hey, this explanation of what this function does is wrong," we can go ahead and actually get that fixed.

If the agent on the fly figures out what that function does and it's wrong, you're not gonna know that it's wrong until much later down the line when it writes code that doesn't work.

Um, awesome.

Yeah, sure. Um, that was, uh, notebook one. We have a little bit more stuff to talk about, but I'm happy to take any questions about the content that was there, and I'll put it back up real quick.

Yeah, I see two in the back. Yoni will come back with a, with a microphone.

Microphone testing. Oh, we're good.

Yeah. Is the, um, is the graph reestablished at runtime when whenever you're, uh, building these ASTs and, uh, traversing them?

Define runtime.

Runtime as in, uh, when the PR or the CICD comes through or-

Yeah. In an ideal world... I mean, we don't build this product for a living, but, like, in an ideal world, that would be, uh, cached and saved in the... a canonical form as like, say, the gold version based on main, for example, right? Um, and then as you tweak stuff and modify things locally, you need to probably maintain a local copy of that. That would ideally be incrementally refreshed, so you wouldn't have to do it all upfront.

Yeah. Yeah. I guess my question is similar. How do you,

uh, think about the cost of keeping these structures up to date versus, you know, whatever the increase in performance and fidelity? Um, because at least for me, you know, I'm using AI to make, uh, large changes, so-

Yeah

... um, so it's hard for me to know, um, offhand, you know, whether it would really be better if you have to maintain these structures.

Yeah, absolutely. I think, I think the key to that is, um, a lot of these problems can be boiled down and incrementalized because a lot of the things that you're doing is you, you don't necessarily have to re-index the things that you've already indexed if they're not changing, right? And when you're making broad sweeping changes across your code base, like, you are changing a lot of files, but there also are a lot of files that are not changing, right? So if we can be smart about incremental re-indexing, making sure we're only keeping track of what's changed, only re-indexing that

part, then what we've seen so far is that the trade-off is in fact worth it, especially as the context size of what you need to keep indexed grows. It actually becomes more useful because the cost of doing it live every time just becomes even more expensive.

Um, thank you very much. That was super useful. Um, I, I'm, I'm wondering how it works on kinda like a real code base in a, in a messy language where, uh, for example, you have dynamic dispatch or, like, not, not all types are available at, uh, you know, at compile time or before. Sorry, uh, when you just analyze code. So have you tried? Like, what are the limitations? Any, any, any tips there?

Yes. Um, the... I do have a third notebook here, which is talking about that in particular, um, which is basically how you do this thing on a language like Python, which doesn't have the same, uh, uh, easy access to the types as gruf- as, as Rust. Um, we use... You can use other tooling. We can use tools like Griffe and same typesitter. Uh, typesitter will still give you AST, and Griffe will give you static resolution of some of the other stuff like the types. You can use something like Ty to do type checking as well, um, to gain more index there. But

yes, it does become exponentially harder with different languages. Um, so the... And I will talk about that a little bit later on. Um, but yes. The answer to your question is yes, I did think about it. It is, uh, a much harder problem, which is why I chose to solve a slightly easier problem today.

Yeah.

Mm. I don't think you should be building this, to be clear. This is, this is, this is a, this is a worked example for educational purposes. Um, people build these kind of coding, uh, graph things, like, on the reg- on the regular, and I wouldn't try to build one from scratch, um, just for doing code analysis. Uh, d- I don't know if that answers your, your, your question. A- a- and I think they are worth it to build. I don't know if it's worth it for everyone to build their own. I think there is probably a... And there are a bunch

of folks in this game already that are, are doing this sort of, you know, analysis just on the code layer. Um, so I, I would say it's probably not worth it for a single person to build if for the purpose of being a product. It is probably worth it for someone to build for the purpose of learning about how to build an agent and how to build tooling and how to build context graphs. Um, I think the scale of this problem is, like, where it... That's where it's more useful. That answer your question? Awesome.

So, uh, my question is about the data context layer-

Mm-hmm

... uh, that you talked about earlier.

Mm-hmm.

Uh, so, uh, I'm assuming that that's useful in a, for, like, a developer.

Yes.

So you develop this, this context layer, and it helps, uh, produce, uh, better SQL procedures, whatnot. Um, so, uh, two questions. Number one, um, does it... Have you seen if it helps at runtime as well, uh, like inference time? Uh, that's number one. And number two, um, do you have any benchmark results to demonstrate?

Yeah, absolutely. So what Yoni was talking about in his talk, um, we do have benchmark results for that. Um, what we do is we've, uh, w- we've tried to find, uh, representative data architectures, which is kind of hard because you can build a, a, an easy tour example, like in your, in your, in your... And say, "Oh, my agent-" Aces 100% of my tests on my 30-model corpus. And it's like, well, okay, that's not very impressive. Um, so what we tried to do is find, uh, open source examples of data architecture that was larger. Um, so we ended up settling on using... Uh, I don't know if

anyone's heard of Mattermost. They're a incident response and chat company. Um, they actually open sourced their entire DBT project with, like, several three hundred, three hundred and fifty models in it. Um, so we were actually able to pull that down, simulate data in there, and then run evaluations against specified scenarios inside of that. Um, asking it to do a specific task, break something in there, and then ask it to fix it, ask it to build out a set of, you know, new tasks for it to build new things. And we're able to see that, uh, by building out our evaluation

layer and layering out, you know, evaluations of how correct it is, how complete it is, um, how ef- efficient it is, right? We are able to see that, uh, with the appropriate context, our agents are able to do a better job, uh, overall and in certain particular categories, uh, to, uh, like, accomplish those same tasks better than just Claude code running alone. Um...

And what about the earlier question, or the other question about inference time? Uh...

So, uh, it does-- So you're talking about, like, when the agent is actually running?

Uh, yes. When the agent is actually running.

That, that, that, that's when we find the most use out of it because when the agent is running, it's going to be querying all of the tools that we built, and that's where the real value is coming into play, right? If the agent's able to query all this stuff at runtime, it doesn't have to re-derive it all at one time, right?

So, um, so then have you tried this with, like, NL to SQL benchmarks like, uh, Bard or Spider or one of those?

Yeah. So I mean, yes and no. Like, yes, we have tried it on some of those benchmarks. I think the, the tricky bit is that, like, the goal of the product isn't necessarily to write analytical SQL. The goal of what we're trying to build for people is more on the data engineering side. So it's-- I think, you know, it's, it's sort of a very common thing to say, "Oh, we're gonna build text to SQL, and we're gonna build that." I think everyone's trying to build that, right? But it's-- I think it's more important to say, "We want to build the tools that are going to make it possible for

you to actually trust text to SQL," not necessarily that we're gonna be able to, you know, come in and write better analytics queries for you. Like, we wanna make sure that when you are actually going in and doing your day-to-day work of maintaining your data architecture, that we're gonna be able to help you with that. Does that answer your question?

Yes. Thank you.

All righty.

So popping back into-- If I can find my mouse.

Popping back into the slides.

So

what we saw there at the end, right, is that when we-- uh, we can actually go ahead and tell these different posting lists apart, um, because we're able to, uh, query our new graph context. Um, and we're able to actually figure out all the dependencies of a given symbol with one hop, uh, without any regrapping. Um, and so how the query works, you know, it matches on the qualif-- the fully qualified name, and then it goes and finds all the dependencies.

And that's it, guys. Great workshop. We're done for the day. No, no, I'm just kidding. So the thing is, like, this is cool. This is real. This tool is really great, right? This tool can do in one call what would certainly take multiple graphs to do. I don't think you could accomplish this in one graph even if you did the perfect graph, right? But, uh, the problem is the agent, you have to actually go and give it to an agent. So when you give this agent this tool, and you give it this agent this tool, and you give it alongside all the existing

tools the agent already has, and, you know, we have this really cool tool, we've seen it do this amazing stuff, and it ran grep anyway. Um, if you give it access to the graph tools and grep, it's gonna run grep anyway. And we gave it amazing tools, a precomputed traversable model of the code base, and it just ran grep anyway. It just globbed and read files and, and ran grep, and it would sometimes, you know, poke the graph and get confused, and then it'd get back to, you know, wading through the graph stuff. And this is where-- what no one really tells you.

Everyone's like, "Oh, we built these cool, sexy tools, and they do this amazing stuff." But the agent has to actually use them, right? You have to do the prompt engineering and the rest of the hard work and the evaluations. That's the rest of-- That's the hard part, right? You can build that notebook. You can build the-- your amazing graph context in an afternoon. The hard part is actually making the agent use it. Um, like the, uh... Yes. Um, and why does it do this? Well, it's because of the post-training. The models are trained and designed to call these tools, right?

There are so many circumstances in their training where the answer is to grep and read, right? And you're not gonna be able to necessarily fight that, um, just by making a better tool. You can make a better tool, but the agent's muscle memory isn't gonna let it use it. It's very similar to humans, right? You can build humans a better tool, and if you try to force them to use it, you're gonna get-- they're gonna not be happy about it. This may be a little bit topical. I don't know. Um, but so the, the important thing is that, like, you're not necessarily competing

on the merits of your tool being better than all the tools inside of the tool base, the, the tool set of the agent, right? You're trying to fight against its own reflexes. So how do you actually do that, right? Because I don't think there's a question anymore that, yes, if it could query the graph, it would be more efficient and better. We'll prove that later. But how do we actually get it to do that? And the answer is to not really fight the, the prior experience of the model, but to kind of lean into it, right? We, we make the graph feel more like the

model is using grep, right? You know, you, you, you take this-- you take a, a tool that may be called query search or query graph or query whatever, and you just call it search, right? The agent will use it in a certain kind of way. What I found is that if you call a tool search, the agent will basically always do multi-keyword search into it with- No training. So you have to, you have to kinda meet the model halfway, right? You take the model, you name it search, you i- account for it to adjust for how the model is gonna query it, and then you just, you know, you get back the results from the graph. And you return the results from the graph

in the same form as ripgrep, right? You take the same tool structure and response as ripgrep, and you return it back to the model, right? Because that's what the model's been trained on. It knows how to handle this stuff, and you're not gonna get it to, to completely change its own self, um, just by prompting, right? And the last thing is that you need to have, uh, your tools pay off in the first call, uh, every time. Because when something, when the agent goes down a, a path and it finds a, a, a dead end or something that it thinks is

a dead end, it called search, didn't get any results. Well, sometimes it's gonna think search is broken. I- if I'm getting no results and, like, I, there should be results, maybe search is broken. I should try another tool, right? It's not gonna go back and use that tool again, and if it does, maybe it tries to use a different way. You can't really necessarily predict that, right? So you have to do your best to make sure that your tools are returning something that's super useful, uh, right away. Uh, otherwise, they're just gonna go back to grep, and they are, uh, never gonna come back. So

the second notebook is, uh, shorter. Um, it's a little bit of a more

focused example, and this is all about evals, right? How do you actually show that the agent is doing what you think it is supposed to be doing, right? And how do you actually, um, prove that your tool is better than the existing tooling that may already exist, right? So, um, all of this stuff is already pre-cached. I did this run. If you do wanna run the notebook locally, you can absolutely set sample size to, um, something greater than zero and run the eval live.

It will take a long time. Don't say I didn't warn you. Um, but, um, I did run this overnight with, uh, five retries and, uh, thirty questions in my eval set, so that's where these results are coming from for this, uh, for this notebook. So the first thing that we do, um, after we do this setup is, um, we basically, uh, set up our, our test cases, right? So we're gonna have an agent with, um, our new graph tools, so we're gonna give the agent access to

the same model and our graph tools, and we're gonna have our control agent that's gonna have access to the same model and a set of, uh, file system probing tools that's maintained by Pydantic. Um, so, uh, that's gonna be our two test agents, and then we're gonna build up a corpus of questions. Um, each question is a real task. We're gonna figure out, okay, questions like, "What public types break if I change posting list?" There are two posting lists. What do I use? You know, this kinds of thing, um, where we try to, you know, say, "Okay, this, this should be a question

that should be really good for the graph. This is a question that's probably great for, for, for grep." Um, you know, there's a control questions that are probably gonna be good for both, right? So the gold data for this is actually, we do pull the main data from the graph. We do cross-reference it back to the source code. So it's not that we're polluting the context of the gold with the output of the graph. We do have to cross-reference it to make sure that the gold isn't cheating. Um, so now we can go ahead and load up our, uh, our gold, and we can basically start

trying to set up the actual evaluation structure itself. Um, and so this is what lays out what's basically a good eval question versus a poor eval question. Um, you want an eval question that can be... In an ideal world, your eval answers should be able to be deterministically scored. I think LLM as judge is really fantastic for a lot of use cases, but it tends to be overused when you really should be trying as hard as possible to have your scoring be as deterministic as possible. It's gonna reduce variance in your runs, and it's going to reduce cost in your runs, and it's gonna make you happier in the long term. Um,

so you wanna create questions where the answers are pullable from the-- Y- you're able to structure an answer to them that's gold, that works. You're able to check the answer against the gold. Um, you're not just asking the agent to provide its opinion and then asking another agent to counterman that opinion with its opinion. Those questions don't really tend to ever work out as evals. Um,

all right, so this is, this is some cruft text. You don't need to worry about this too much. This is just basically building out the, um, the, the test cases that I was saying. This is the deterministic part where we check to see if it cites all of the evidence that we expect it to cite. Um, we're basically just creating a list of examples, creating a list of, uh, uh, things it should be citing, and then checking to see if it's citing all of them. For the evals themselves, we're using a configuration in Pydantic evals just because it's really easy to set up. Um, how this works is we

build out, uh, our two different agents, and then we build out, um, the agent runner itself, and then we build out an evaluator, right? So this is what actually will evaluate the agent based on, um, the given tasks that we have in front of it. Uh, we'll basically compare what the agent has said to, uh, what we expect it to for each given task, right? And that way, we're able to compare the responses and the results between our grep agent and our graph agent, right? So that's how we're able to establish, um,

one, you know, what is the actual utility of our skills, and two, um, how do we, uh, what, what can we improve or, or what can we do better, uh, as time goes on? So, um, go ahead and run. Right, so I'm not gonna go ahead and run the eval again because that would take about thirty minutes. But, um, we can see that we run the eval the first time, and we run it with, uh, K equals one. So we just say, for

every task in our dataset, one shot, grep agent, graph agent. And when we do that, the graph agent doesn't do very well. You can see that the, the graph agent, it does edge out the, the grep agent a little bit, but it's not really consistent, and it's not really, like, very- It's not what we wanted, right? But what we have to understand is that the goal wasn't ever necessarily to build something that's going to be more correct than grep. The goal... Because all of us that use grep on a day-to-day basis when we use our

coding agents know that it's very good at what it does. Like, it uses grep for a reason. It's not gonna be wrong a lot of the time, right? But the important part is not that it's right th- this one time I tried it, but that when I try it 10, 5, 10, 15, 20 times in a row, that it gets it right every time, right? So that's how we need to actually start sampling these tests. Now when we say we run these same set of tests over the same corpus five times to get our, our, our pass@K metrics, now we get a slightly different, um, answer, right? So we have two different statistics, our

pass@K and our pass Hat K. Our pass@K says, across these five runs, was this ever correct, right? In that case, they're still... They're very similar. Actually, grep has a little bit of an edge there. Um, the graph doesn't sample any of the private variables, only the, the private part of the crates, only the public, so there's some control questions that the gre- grep will always do better on. Um, but what's really interesting is when you change the metric and say, "Is it doing the right job on every run?" And when you do that, the numbers completely flip, right?

Now, all of a sudden, the graph agent is twice as good at doing the correct thing all five runs than the, than the graph agent, right? Um, and we can see that within that as it's doing that stuff and it's doing it more consistently, it's also doing it more efficiently, right? We can see that over the course of all our different runs, on average, the graph approach took about three and a... 3.1 times fewer tokens. The last part is, okay, how do we actually get the agent to use the graph? This is where your evals come into place, 'cause you basically have to write an eval that

says, "Hey, examine all the tools in my trace. How often is my agent using the tools that I want it to be using?" And then you can get your sort of adoption numbers, right? So that's how you can get to a place where you can say, "Oh, I wrote a prompt that didn't have any instructions in it to use the tools. I got 34% of tool usage. I went back, I changed my prompt. Now I have 72% tool usage," right? So the evals are the only way that you're going to be able to actually understand that the agent is using the tools that you want it to use. Um, this is just a test of that

exact concept. We have, uh, two different prompts that are both trying to answer the same question with the same tool set. Um, if you, if you nudge the prompt, uh, if you're just naive, it will just go ahead and grep, um, and then it'll start doing some other stuff. If you nudge it, the ideal world is that it will go and actually run the search and use getSymbol to go and figure out the actual answer. And so the recipe for building, you know, really solid, honest evals is to mine real task candidates out of your

corpus, uh, build a deterministic as much as you can, an anchored scorer, uh, and unit test the scorer itself first. Um, I like Pydantic evals. There's many, many other eval k- things out there, but the core structure of them is all basically gonna be the same. You have a dataset, you have cases, you have an evaluator. Um, make sure you run your evals many times over and report pass@K, pass at K, uh, and all your different metrics and, uh, make sure that you are,

um, putting in examples of things that your agent may not necessarily be good at, 'cause you don't want it to pass 100%. It's not a useful... If your ev- if your agent's passing your eval 100%, your eval's too easy. Your agent should be close to failing your evals. Um, like, your evals are your chance to see where the metal hits the road and what your agent's gonna do in its hardest possible circumstances. So it doesn't really do a good anyone any service by just, uh, giving it very easy and very simple evals.

All right. Anyone have any questions on this eval notebook before I jump back into the slides for the last part of the talk here?

Uh, I w- I was just wondering, in the example when... Or when you were discussing how you, the agent would often use grep instead of graph, so you had to retool it to be this search tool, uh, was there ever any, uh, uh, like, attempt to take the graph and distill it into a file structure, maybe, like, with some loss, but a file structure that then grep would naturally search over so you didn't have to define any separate tool at all?

Yeah, not today, um, just because, uh, there was a bit of a time crunch. But I think that's definitely a reasonable approach. I mean, I think that's what, you know, that's core base, LLM Wiki, you know, there's all these different things where, like, you can get a lot of these different benefits just from the markdown corpus. But, like, I... And I think that that's, that's definitely true, and I think that you would still get a lot of the same, like, benefit out of it, but I don't... Like, in the markdown, like the LLM Wiki, that kind of stuff, right?

I still think that the, your problem is gonna be, like, you want real, like, edges and provenance between them, right? And, like, those connections are where it's gonna be really hard. 'Cause even when you do something like DRain or something like that, right? It... Yeah, it builds all this up in markdown, but it does build a graph on top of it for you to see the timeline and the connections and stuff, right? So I think at some level, you know, if your task is based in needing to understand the relationships between these different things, then at some level you're, you, you will want some kind of

graph, even though I do think that you can get a very similar level of, of, of, of success out of, out of an approach like that. Yeah.

Um, do you think there's anything missing from a setup like this for letting non-technical, non-data engineer folks, like, kick off a PR with this as, like, the guide through a code base if you tried that? Or, like, is there a something you'd add before you empower people to do it that way?

Oh, like, for this as a, as a tool?

Yeah.

Oh, yeah. I think, um... That's a good question. Yes. I, I think the,

the, the easiest part, I mean, would probably be, like, starting to hook it in as an MCP to the tools that you're already using. So you would have, you know, this code graph in addition to your, your Claude Code, and you would Ask your Claude Code to call the graph and d- do those other things, and then you would write skills in order for it to actually use that, that, that properly. I think that would be the, the best approach there. Um, but I would think that, yeah, if you're less technical or non-technical, I, I, I would think that, you know, the, the, the goal would be that you would...

If you want to engineer the agent to be using these things in the context of actual coding, you would p- promote these prompt engineering things I'm talking about here into skills and give those to your coding agent.

Yeah, like that PR lands at the engineer for the first time with, like, a half decent first pass as a result of being more deterministically guided by this.

Yes. Yeah, and, and I think the, the, the deterministic part's helpful because you can, like, like we were talking about before for the, uh, the question in the back, like, you can incrementally just diff over what's changed, right? Analyze that deterministically and get a really, really fast conclusion back about what's, what's happening and what's been changing since the last time, as opposed to waiting for the agent to go and read everything and look at all the changes and, and give you back that, that same diff.

This one back there, I think.

Um, did you, uh, experiment or think through, like, different, uh, levels of resolution? Like, one thing I was thinking of, um, in some of the earlier talks is, like, you do next token prediction, and you will have the prefix, I think people said-

Cool

... which is just like, you know, it's all at the level of, like, this is the whole string of everything. In the code base here, you have the symbols, but, like, the symbols is a certain gra- granularity. There would be higher levels of granularity that are imminent in the code base, and making those things searchable and accessible to an agent, you know, might have payoffs. And I was wondering if you thought about that or if you have thoughts on how you'd approach that.

That's a fantastic question. I did think about this. There are, uh... and I will actually talk about it at the end of the, at the end of the, the presentation. Um, there are definitely some follow-ups I would do here. Um, if you-- If we're actually trying to build this, um, you would, uh, uh, use something like SCIP to actually trace the call graphs inside of the code base, so you can actually see what is calling what and how that interacts. Because you can track, um, like, symbols and how they move across the code base, but if you wanna actually track, you know, what is calling what and how it's all connected, that is the

higher level that you're talking about, and you need a, you need a slightly augmented approach, another rung of, of, of deterministic parsing. And you can do that, um, with, uh, there's, like, a Rust or SCIP. Um, there's similar SCIP outputs for, like, TypeScript or, or Python. And I was gonna do all of that too, but I, I ran out of time.

But to answer your question, yes. There's, uh, there's definitely more structure we can extract from a given code base, and that's sort of what we try to do at Typedef is find all the different ways that we can pull, uh, as much context as possible, uh, out of the data that we have.

Um, I have a quick question for how would you say, like, this concept of the data context layer, um, combines with, like, something like a data catalog? Like, is it the data catalog would help feed a graph? Are they kind of redundant to both have in an enterprise?

I don't think they're redundant. I think they answer different questions. Um, your data catalog is a,

a sort of fixed understanding of exactly what is present in your data warehouse, which I think is kind of a different ask than what the context graph is trying to extract, right? Um, I think they, they coexist. We read information from the catalog to form the, the context graph. Um, so what you'll try to do is basically say, "Okay, I have my," say my DBT project that has, you know, a bunch of models in it. Those models build out, uh, uh, tables and views in

Snowflake, right? So we may say, "We're gonna ingest that DBT project. We're gonna pull in all that SQL, and we're gonna parse it, link everything together." And then we're gonna say, "Oh, okay, this model is corresponding to this table. Let me go query the catalog, get information about that table, pull it into the graph as well." So we can say, "Okay, this model is, uh, producing this physical table, and this physical table has these columns, and these columns have these, these profiles." And that information will come from the catalog, right? So that catalog of information, the information

schema, the profiling information, all that stuff, that's essential to what we do, and that's all in the catalog. So I would say they coexist more than it's a replacement.

Nice.

Yeah.

Thank you.

To try to bring this back to, like, data engineering and this concept of, like, evals, do you think about it the same way in terms of, like, asking a set of, like, data questions and then seeing how those react, uh, to using the tool or not using the tool?

Yes.

Um, yeah.

Yeah. Yeah, precisely. Oh, so do you have more? Sorry.

Sorry?

Oh, I didn't know if you were done. Um, yes, absolutely. Um, we do use, uh, like I was saying, we basically try to find, like, as, as real a data corpus as we can, and then we try to find multiple different kinds of scenarios, right? So you have your basic fix-a-bug- Um, or the scenario is we'll inject a diff into the code base that introduces a bug, right? And then you'll ask the agent to go and try and fix this bug. Um, and then you can es- you can test,

you know, that it looked at the right places, that it checked the right things, that when it fixed the bug, it didn't fix it upstream or fix it downstream, right, by putting a Band-Aid on it, right? It actually found the right model and fixed the right issue. And you can make those tests easier or harder, right? Like, you can, you can change the prompt language to say, "Oh, I think there's a problem with model XYZ," or you can say, for the harder example, "My down- my dashboard Y is having this issue where this number is doubled. I don't know why." So forcing the agent to work backward from what that depends on upward through the tree to

understand, like, where the actual root cause of the issue is coming from, and then asserting that it actually goes in and fixes that problem instead of leaving it be. Um, so that's what's really hard, and the problem is that the agents are always going to cheat. Um, so in your eval harness, you have to be really careful about hardening it. Um, like for example, your agent, if you, if you, if you say that, "Oh, my test case is the agent will check out a branch," and then it will go and try to fix the problem. The agent will very happily cheat and use Git to go back to main and try to figure out

how you introduced the problem in this branch from, from there. Um, and it'll go and very happily find the branch name that says this is a test and be like, "Oh, this is a test. Oh, I can cheat on this." Um, so you have to be very careful about hardening your, your, your harness to actually, uh, be representative of the tasks that, uh, you want your agent to do, um, because that's always gonna be the danger there.

For these evals, do you find people, like, encoding them from scratch for the most part? Or like-

There are, there are some existing frameworks for this. Um, ADEBench is, is a great one. Um, it's probably the current industry standard for this kind of like data engineering benchmarks. They have a very similar approach, um, for, uh, uh, creating problems in repos using diffs, and then having the agent go and read data from the warehouse and read data from the code and try to fix the problem. Um, so those do exist. People are wiring them up, and I think it's becoming more and more of a thing. I think the hard part is always the scaffolding around it, because

for code, you just have your code base, but you need a running database. You need it to be, you know, populated with the right buggy data. You need that data to be internally consistent, so you don't want the agent to be spinning around trying to figure out, "Oh, oh, why are the numbers from twenty twenty-four?" It's like, "We only generated twenty twenty-four data in our sample set." So, um, I think you run into those kind of issues a lot, um, which is why people are trying to standardize it in ADE. I think the hard part is building out a benchmark like that, that other people can use that actually has a big enough data set. That's

gonna be the tricky part, and that's not something that has been done yet, uh, at least to me, to my satisfaction. Um, but yeah. The, uh, you know, we get back the, the-- This is just a, a summarization of the same results we just saw. Um, we see that as we, uh, increase the amount of ability, as we tell-- as we get the agent to use the tools more, um, the agent is able to be more consistent in its

ability to answer the same set of questions and is able to answer those same set of questions, um, more efficiently. Um,

an example of why it's quite important to, uh, run these evals multiple times. It's not about your per-shot accuracy, it's about your determinism. Um, and the number that actually matters was how do you actually make sure that your agent is using the new tooling? And so you really wanna make sure that you're getting your adoption numbers up, and that you're measuring your adoption of the tools, because that's really what's gonna get you, um, the right, the, the last part of the way, right? You know, an elegant tool that gets reached for a third of the time is just a third of a tool. Um, and every

capability win that you can get from better tools is gated by this number.

Um, so this is why we built the eval. Not to prove that the tool is good, but to prove that it's not going unused. So the hard part about this whole talk was basically admitting that grep works really, really well. Um, I did all this work, and I built this context graph, and I ran the evals, and it was a tie. And it was like, it was kind of frustrating, and you kinda wanna keep optimizing and building the thing and trying to get more and more out of it. And it's kind of a trap, right? That instinct is like to build more, I think can sometimes lead to a sunk cost, right? Like, the agent has its own

reflex. I have mine, what I think it should be doing, and, like, we're, we're trying to meet in the middle, right, and actually figure out a way to move forward with this, right? And it's like, oh, you think that maybe it's just, you just need more testing or more evals or, or, or more something, right? And you end up with this, this false feeling of, "I just need more rigor and more tests, and then the agent will be better at using the tools." But you really-- Maybe you need to ask, "Hey, why is grep working so well?"

As opposed to saying, "Why isn't it using my tools?" Well, maybe the answer is that grep actually does work really well already, right? So maybe you're not going to win by fighting it. Um, maybe you're gonna have to lean into it just a little bit. So the agent is going to grep anyway. Uh, so when you keep trying to out-engineer grep, um, you're just gonna have a kind of frustrating time. Um, so if you can design as with the grain as much as possible, help the agents lean into what they wanna do anyway, uh, it will help you out quite a bit.

Um, I've now done multiple talks where I, I, I had a conclusion, finished the talk. I said, "Let me rerun one set of evals," and then I had to redo the whole talk because the conclusions were completely different. Um, what had happened was I wrote my evals and I built this whole harness, and I built it wrong, and I got answers that were directionally correct. They said that graph beat grep, but I wanted to look into it a little deeper. And when you look into it a little deeper, you'll find that, oh, that tool that we made, it returns the path to the file, but it's not, like, the right

path. So sometimes when they call search, it would find the wrong path, and it would go down a wrong path of trying to read these files. So you actually have to go in and you read all of your different traces, right? Go into LogFire, make sure, or go into wherever you're, you're, you're, you're sending your traces in, right, and make sure that you're actually reading them. Because you're not supposed to just-- If you just look at your scoreboard, you might get answers that look directionally correct or, or that look great for you, but they may be burying underlying issues that are actually much larger and, and a much bigger problem inside of your, inside of your

architecture. Um, so the fix is never just to make the number- Higher or nicer is to build a harder eval. Uh, read the traces, find the artifact, harden the harness, and then rerun. Never trust your own scoreboard. Um, and yeah, these are just some of the other thoughts I had. The, you know, the, the models are only getting better, um, so, you know, you can, you- we can continue to believe that the models are gonna have less variance and more drift towards being able to, uh, do the work that we want them to do. So

the easy part may actually be building the context, building the tools. The hard part is really gonna be building out your eval loop, building out your monitoring and observability, and making sure that your agent is actually doing the work that you want it to do with the tools you want it to be using. Um, and so all this stuff is there. The, the score, the loop, uh, it's all in the repo if anyone wants to play with that, um, on their own time. And, uh, yeah. So this is what I was talking about before. You know, we did-- I did cheat by using Rust. Um, the Rust compiler already

did rung one. I, for its homework, uh, I picked the language that was free because the lesson was you can build a well-engineered version of something, uh, on a relatively easy instance of a problem. Uh, and in general, if you can get things to work on a slightly easier instance of a problem, it's slightly easier to go and scale that back up to a larger issue. Um, so we now can say, "Oh, wow, we have this really easy thing that we did for Rust. Oh, let's do it for Python." And Python is just harder, right? Like, you can't really... If you look at

this, these lines of code, you're not gonna get anywhere, right? If you grep client, you're gonna get stuck in a land of fifty thousand clients. Uh, you can use static analysis to, to load this stuff and things like that, but, you know, you're gonna have, like, a, a difficult time. And so this is where a code graph really earns its keep, right? In a case like this, where the code bases are larger and the languages are, are less well-structured, a code graph is actually even more effective than grep in that case. Because for Rust, you know, I wanna see what implements this

trait. Okay, cool. I just search impl for, right? Like, it's really easy. But for Python, that becomes a lot harder, right? So that's where, um, you really would need to build something out that's more robust. Um, but the structure of the graph and the structure of the tools doesn't necessarily have to change, right? It's the structure of the way that we ingest and load the data. And if you make that modular, then you can build that out with, for as many languages or as many projects as you want, right? So that's kind of the underlying structure for, like, how Typedef works,

is we try to build out as many of these different ingest processes as we can to pull data from many different places and then sort of synthesize it all into one. Um, there is another notebook for this. I'm not gonna go through it now because we've, we've all been here for a little while. And, um, but feel free to go through it yourselves and, uh, try to learn more if you want. Um, last thing I was talking about was, uh, basically the meaning rung. Um, the structure is what's telling you what's connecting to what, but the meaning is telling you sort of why. Uh,

so there's comments in code and docstrings and all the other information. And folding all of that information in helps us understand the intent that the coders had when they wrote the code, in addition to the actual structure of the code that currently exists. So, um, we would basically want to add in small, uh, passes like this to say, basically summarize the information from the code, um, and, uh, understand it at a, at a row-by-row basis so that search can now answer what the code does in addition to, um, its

location. Um, and

the important part when you're building something like that, um, you're building this sort of auditable semantic analysis that's gonna go on top of your deterministic analysis. Um, the important part is to ask many small questions and not one big question. The answer isn't writing, you know, one really... So when we do analysis in SQL, right, we don't just write one really, really good prompt and throw your SQL at it and say, "Hey, uh, here's a gigantic model with ten fields. Like, give me all the information you can out of this SQL," right? It's not gonna work. Um, the

model is gonna hallucinate. It's gonna invent information. It's gonna not collect as many, as much information as it can. You want to basically, instead of asking the model ten things about a chunk of code at once, you ask the question, you ask the model ten different questions about ten different smaller chunks of the code that answer specific questions, right? You parse the structure deterministically and then ask many small focused questions that play to the model's strengths, right? Instead of asking the model to look at an entire block of SQL and tell you what all the

columns mean, you parse the SQL, and you say, "Hey, here's one column. Here's the type. Here's the comments. You know, is this a measure? Is this a dimension? Is this blah, blah, blah?" Then at that point, you can ask the agent to do classification and summarization, and those are the things that it's good at, right? And if you want to weigh it more, um, you give it a more, you give it more, um, prompt input to that single focus question. You don't try to cram more questions in. You give it more context for it to be able to answer that single question with. So small, grounded, auditable semantic operations over parse

structure. Um, that's what our library fenic is for, and that's what we use it for as part of Typedef. Um, so that last ladder, the last rung in the ladder, which is basically we have all this stuff layered in. We know how everything links together. Now, what does it actually mean? Now we can start pulling some of that semantic understanding of what things actually mean by using the LLMs in the last layer. Um, yeah, so as I was talking about before, there's a lot more that you can do here if you actually want to build a coding, uh, code-based graph analysis agent. Um, you

can use, you can leverage call graphs with SCIP to understand who's calling this and who references this, right? And, you know, it can read the comments inside and outside of the code, and it can fold in prose from outside of the code. It can fold in GitHub issues and, and, and your, your, your linear issues and your ticket and stuff. So that's how you build this into something that's actually more useful and, and, and grounded, is you start small. You start with the smallest possible thing you can do, which is how do I extract structure out of this code base? And then you move on to how do I take that structure

and layer into it the semantic understanding that's really going to help, um, build out w- ... a substrate that's useful for agents to actually work inside of the space and make changes. Um, and that's sort of what the, uh-- where the end goal is there. So anyway, thanks so much, guys. Uh, this has been my talk, The Agent Will Grep Anyway. Um, I'm Brandon Callender from Typedef. Here is the link again for the workshop and my LinkedIn. So, uh, thanks. I can take any questions or, or any other things, but thanks so

much, everybody.

Any, any questions? We can pass around the mic for one last round.

Have you tried this kind of workflow, like, across repositories where things might be either microservices or, like, going across HDP boundaries, um, with any success?

Uh, not particularly, no. I, I think where, where this would, where this would come into play is on a single repository. Now, we do that at scale for our, like, larger data architecture product, so I think it can work. Um, you basically just need to... The hard part there is obviously figuring out the boundaries between the different things. You basically need to write boundary classifiers that basically try to understand, um... I was trying to do an example with Python and Rust, like with Maturin, so you could have

a similar example of that, but I didn't have time to do that. But you basically would need to write some way of understanding when Rust called code is calling Python or vice versa, right? To try to, like, build out that linking layer between them. Um, so yes, you could do it. I just think it would, it would require quite a bit more work.

Any other questions?

Oh, sorry.

Yeah, no problem.

Yeah, I got you.

Um, how does this compare to existing harnesses like Cursor?

Cursor does a lot of the, um, level two stuff. So, like, when you see, like, Cursor doing s- uh, searching through your code base, it's usually doing semantic search, like vector embedding search. Um, and that's, that's-- That, that does work really well. Like, I think that there's a reason why there's all these, like, code RAG products that do, um, that do semantic retrieval, right? And I think that that's, that's really great. It's just that, you know, that approach does get you the ability to find all the places in the code that you want to then examine with the agent.

But you still have to go and examine all those places with the agent, right? The goal of this approach is if I want to be able to, in one shot, say, "I know that I'm looking at this function right here," right? I basically wanna be able to do the IDE click on the thing to see all its call sites, but within my agent, without my agent needing to grep everywhere. That make sense?

Any other questions? All right, one over here. One sec.

Mm-hmm.

I imagine half the fun of building this is that you end up with, uh, I don't know, sort of a, a vector of the graph that you can recall.

Yeah.

Um, I'm really curious about caching and storage as you really start thinking about scaling this-

Mm-hmm

... given that, uh, making sure the agent gets it right on the first call is so key in steering.

Yeah, that makes sense. I think, um, as you st- as you start to scale out your, um, your graph database, you're gonna start finding that it's, uh, it can get very hairy very fast. And you're, you're absolutely correct. Um, so I think storage is less of a concern, I think, because, like, these... Like, yes, these graphs do tend to, uh, get a little sprawly, especially as they grow, and they're harder to kinda garbage collect for. But they do, um...

I'm less concerned about storage and more concerned about, like, uh, the amount of compute that it takes to actually run, like, graph analysis across all these different things. Um, so I think the-- A-as you try to scale it up, I think the main thing you need to focus on, like I was saying earlier, is the incrementalism. Like, figuring out how to rebuild this thing without regenerating all of it, I think is probably the most important part. And that's something we fight with at Typedef for trying to build this, you know, at a larger scale across a large data

architecture. Because you don't wanna have to go and query th- your warehouse, you know, five hundred times again, right? You don't wanna have to go check, you know, compile and all the SQL code again, right? You wanna be able to update the stuff as it's coming in. So I think the most important way to get around that is design for incremental indexing first, as a first-class citizen. Um, that's kinda the only way you'll be able to get to a point where you have to deal with the other scaling problems. I, I think you have to deal with the problem of re-syncing the information long before

you have to deal with the actual scaling problem of how to store it or how to compute across it all.

Yeah.

So, uh, for the golden data set, how do you, how do you keep up to date? Because for us, I found that...

I found that for us, the building of the golden data set and maintaining it is always the biggest problem.

Yeah. I,

I agree with you one hundred percent to that, and I, I... That's, that is one I don't have a, a great answer for, I don't think. 'Cause I, I think...

I think-- So I have two answers to that, actually. One of them is that, like, the, the gold data set isn't necessarily...

If I was, if I was actually building this and I was testing it on something, I would pick a specific snapshot of that code base, right? So I could say that this is my snapshot, and that when I run my evals, I have a deterministically set of my gold questions, right? That's, that's kind of a cop-out to your question, right? But that's what I would do if I was actually designing this, right, is I would, I would desi- I would design this against a snapshot, so my eval questions are deterministic and aren't changing over time, right? Because the- In an ideal world, right, your agent

is going to get better and better at answering the same questions. But changing the questions based on changing the corpus doesn't actually help you evaluate if your agents are getting better, right? The best way to evaluate if your agents are getting better is have them do the exact same task again, right? So sort of the answer is you don't, because you-- if-- for the purpose of evals, your goals should be constant against a constant corpus. Does that make sense? Now, if you're testing live, yes, like you need to be able to figure out new ways of testing it against your code base. But

I think that and like your fixed eval set are different things. Does that make sense?

Yeah, but like... Yeah, but I mean like if you-- So if we go with the code base example, and let's say you have a kind of like a complicated bug that is like a function that is calling a different function, and there's like a-- in one of those functions, there's like a bug or there's like a if, like you said. And then you have an eval that tests that if this bug was introduced, then the agent went to this and this function and fixed it. But then you do a refactoring, and

then this function is actually not even used anymore, so then the eval breaks.

Oh, I see what you're saying. So in that case, I, I think it's still-- I think my answer is still the same because what, what, what I'm saying is that I-- when I'm building my eval, I would say my eval is a combination of a specific commit that like my code is at, right? The specific diff that I want to introduce to, to see if the agent can fix it, and then the resultant set of changes from that prompt, right? That should always stay constant. Like I, I, I see what you're saying because like, yeah, the code's gonna change. It's not gonna need

to fix that issue anymore. But that's not what you're testing, right? You're not testing the agent's ability to react to the new code. You're testing the agent's ability to make that change based on that circumstance that it was in when you wrote the eval, right? Like that task I don't think has changed even if like it's not relevant anymore. It's, it's still testing something that's useful. Does that make sense?

Yeah.

Next question.

Great talk.

Thank you.

Loved it. Uh, um, this kind of reminds me of ontology, um-

Yes

... classes, and this is what Palantir is making a killing out of, actually.

Yeah.

So, um, once you build the ontology and the dependency, um, can we talk with the LLM in these higher constructs now?

Yes.

With these classes?

Yes.

And is this is essentially what you were kind of insinuating at this point in time?

Yes. So that-

Is that right?

... that's, that's basically our product, right, is, is, is that concept of we, we build, um, we build the context graph, we build the ontology layer on top of it, uh, where we try to organize all the information around your data architecture, where things are pointed, where things live, what things mean. Uh, and then we give the agent access to that tool set to be able to answer questions and, and make changes within your data architecture.

Excuse me.

All right, so this may be a little naive, but so going back to kind of the product you guys are building, what you demonstrated here. So a lot of the stuff that you showed here is very similar to what, uh, Neo4j showed at an earlier workshop-

Mm-hmm

... which is like, okay, here's the knowledge graph. You, as the developer with domain expertise, are going to build a knowledge graph in this area, and then you're gonna provide it out to an agent, and then make that agent query Cypher to get some semantic information.

Mm-hmm.

I'm responsible for, I'll call it a sprawling data estate, where there's a lot of, you know, third-party software as a service. We've got other things that are internal. My team is responsible for, you know, bringing in a lot of information from those things. Some of them are highly configurable, like a Salesforce. You bring the information in, you put it in the data warehouse, et cetera, et cetera. And eventually, you have business intelligence on the other side, and also interactions. So the goal is: How can I get my pretty small team of pretty good people-

Mm-hmm

... to define the semantic relationships between all of this stuff in one place, one time, with good version control, so that then I can both provide context and tools to AI agents for mostly analytic work, but also write back occasionally, and documentation for human beings that have to use these systems, whether that's onboarding or in one place. I haven't figured out the right way to do that yet.

Let's chat.

So are, are you gonna, are you gonna give me that?

I am.

'Cause you also said, like, you're not gonna do the data catalog, right?

No, we're not gonna-

And you said-

Well, the thing is, I think, I think what, what I'm, what I mean to say is we're not going to be limited to the data catalog because we're not gonna be reading just your Snowflake catalog, just your, you know, just your warehouse information. The goal is to be reading from your Salesforce, from your, from your, from your Fivetran, from your, your, um, your Airbyte, from your whatever, from all the transfer mechanisms that you have, uh, your Airflow, your whatever. Incorporate all of that into a single understood context tree, uh, context graph that we can actually traverse. And then from there, the

goal is to actually, um, distill out of that the things that you were talking about. The, the main problem with semantic layers is no one wants to maintain a semantic layer, right? It's kind of a dirty word because once you build one, it's all really cool and, and sick and sexy, but then you have to actually keep updating it every time you add models, and it's a, it's a nightmare. So the goal of our-- of Typedef is you really just-- It was great. Thank you for, like... Gave a great understanding of exactly what we want the product to be able to do, which is exactly what you said, is to be able to reason across all those different

layers that you have in your data architecture, build off a single graph that you can actually trace how data is flowing from Salesforce all the way down to your BI layer, right? And then when your agent asks-- when you ask your agent a question like, "Hey, um, where is my... Like where is AR being coming from?" It's able to say, "Oh yeah, great. That originates up here in the Stripe data, actually, and then it gets joined into your Salesforce lead data down here, um, to like figure out, you know, this, this thing down here, which is where the actual data is being calculated."

Yeah, like that, that lineage is very useful, but... And, and for us, it's, you know, partially for the data team that's doing this development, great. But I want that to eventually be accessible to the business, right? So they can look at their report and then we're like, "That's not right. What the hell?" And then rather than coming to my team and to yell at me, they can say, "Oh, well, that's actually owned by these people on this system and this record," and like, "This is the person I have to talk to."

So-

Keep it all together

... I agree 100% with you, and I think that what we're- Trying to, like, one thing that the LLM sort of let us do at a high level that wasn't necessarily possible before is be that kind of translation layer that sits between the business folks, the, the accountants, the analysts, right? The... And then the data analysts, the, the engineers or whatever, things like that, right? Because what we're saying is, what we want to say is, "Okay, that business analyst says, 'Hey, that number in that revenue dashboard is wrong.'" Right? So the process for them to fix that would

be, "Okay, I need to go file a ticket somewhere." That gets sat in some system, someone picks it up, triages it, tries to figure out who owns that thing, right? Sends it over to them. They had to talk to the original... So I... It's crazy, right? I, I worked at Salesforce before this, so I, I get a little bit of that on, on a high level. Um, so the goal with, with Typedef, right, is for the able for that, that, that business analyst to be able to go in and say, "Hey, my dashboard is broken." Right? They can put in the name of the dashboard. We can pull up the dashboard, you know, in that session where they

are. We can trace back and say, "Oh, yes, I think I see what's the problem here. It's probably that, you know, on multiple layers back this is happening. I can detect that this team owns this, this model up here. I can go ahead and file that ticket for you with the resolution information already encaptured in it, because I already can see what the root cause is, because I looked back three layers and saw that this select was wrong." Right? And I can trace that back to a change that landed last week, you know. Oh, that's why this broke here, right? And oh, we can see when that happened because there was an Airflow run over here that actually populated that data, right? That's

what we're trying to get to, is that unified vision of how everything works from runtime through lineage, and being able for the agent to actually answer those kind of questions and sit between those different layers and m-make that process more efficient. 'Cause if, if instead of that ticket sitting around and bouncing between five different teams, if it's immediately understood, understood what the problem is and that gets routed to the right team right away, that team can even just, in an ideal world where our product is ubiquitous and works perfectly, that team can just say, "Yeah, that solution sounds good. Check plus,"

and then it goes and implements it, right? So that's, that's the dream, right? So but that, that's kind of what we're trying to, to get to, and we're, we are definitely part of the way there. So be happy to chat.

All right. Any final questions?

Cool. We'll be around if you guys wanna connect with us on LinkedIn or anytime. We love chatting about context graphs generally. So thank you guys so much for coming through, and we'll catch you guys later.

Thanks, everyone.