AI Engineer World's Fair 2026
Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked
Read the talk
Beyond RAG: Build a Relational Context Engine from Scratch
Peter Werry builds a natural-language query engine over GitHub pull requests, showing how schema discovery, identity resolution, procedural validation and retries turn plausible queries into useful retrieval—and where semantic search still belongs.
From a talk by Peter Werry
At a glance
Ideas worth remembering
Semantic relevance does not enforce authorship, state or time. Let the model write a structured query when those conditions determine which records answer the question.
Query planning needs the database schema, resolved identities and a current time reference; natural language alone leaves crucial values unspecified.
Validate generated queries procedurally with permitted operators. Keep internal tenant constraints under application control rather than accepting model-supplied metadata.
Return validation and execution errors for bounded retries, but distinguish a bad plan from a missing capability: the authentication example needed full-text search, not merely another exact-match query.
Dynamic schema sampling supports exploration, but 300 documents is an empirical choice that can miss rare structures. Controlled production schemas can instead be maintained with code and CI.
A relevant pull request can still be three years too old
Engineering context lives across pull requests, review comments, tickets, wikis and chat. Those records contain meaning, but they also contain authors, dates, states, labels and relationships. Peter Werry, who builds context engines at Unblocked, uses that distinction to frame the workshop: retrieve the discussion that sounds relevant, and you may still miss the records the question actually asks for.
Here, “RAG” has a deliberately narrow meaning: semantic similarity search or vector retrieval. Embed the question, find documents with similar meaning, rerank them and put the selected content into a model’s prompt. That works well as a starting point for finding code, a PR that changed it or a Slack thread that debated it. An agent can then inspect files and follow leads through GitHub.
“PRs I merged last week” exposes the missing mechanism. Nearest-vector search can find text about merging or about last week without enforcing who performed the action, whether the PR was merged or when that happened. A semantically convincing result could have been merged three years ago. Reranking for relevance does not by itself enforce those structural and temporal conditions.
The other half of retrieval is a database query that expresses those conditions. The model’s job changes: it writes the query that accesses the records. The database supplies the matching data. This separation is the foundation for the six stacked pull requests in the workshop, which progressively replace a stubbed engine with a working one.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The query needs more than natural language
A request such as “Show me Peter’s open PRs” looks simple until it reaches the database. Peter may not be the value stored in the author field. A generated query may also use an unsafe operator, or fail even when its intent is sensible. Four supporting mechanisms make query generation useful:
- Schema awareness. Give the model the fields and types it can query.
- Identity resolution. Map a person’s name to the identifier stored in the records.
- Validation. Treat the generated query as untrusted output and restrict what it can execute.
- Retries. Return validation and execution errors to the model so it can revise its plan.
The starting application already has a query playground and a chat tab, but the backing query engine is a stub. Ingestion loads repository content first; the workshop then builds around the existing entry point. Keeping the interface in place makes the changes observable: the playground begins unable to answer anything and becomes usable when query planning is connected in step three.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reconstruct a schema from 300 documents
The ingestion script knows enough about its records to download and store them. A separate query service may not have that knowledge. Unlike reading table definitions from a relational database, this MongoDB example reconstructs a description from the documents themselves.
Schema discovery samples 300 documents and merges their nodes into one tree. Null values are skipped because they do not constrain the type of a field. Procedural rules infer types and recognize enumeration patterns. An LLM could perform extraction, but its variable output would still need structural checks; this implementation puts the extraction itself in ordinary code.
What information will the planner actually receive? The discovery run produces a Markdown description of the GitHub pull-request schema and also creates some indexes. Markdown is an intentional output format: the resulting field description becomes part of the prompt used to synthesize queries. Discovery therefore turns sampled document structure into an input the model can consult, rather than asking it to guess the database’s shape.
state: closed|open · draft: boolean · reviews: [{state: APPROVED|COMMENTED}]
indexes: (user.login) (state) (created_at) (text)
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Peter is a name; pwerry is the stored identity
Knowing that an author field exists does not tell the model which value belongs there. A literal match for Peter can return nothing when GitHub stores a login instead. Identity resolution supplies that missing connection between the person named in the request and the identifier used by the data.
The workshop uses simple fuzzy matching; Werry names Levenshtein distance as a suitable technique. This approach couples the resolver to some knowledge of the schema. Dynamically discovering the mappings with an LLM is another possibility, but the example chooses a smaller procedural solution. The resolver alone does not yet make the playground work: it must be connected to the planning step that follows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the planner a schema, a resolver and a clock
Query planning is the LLM portion of the engine. The prompt receives the schema, and a required tool call performs name extraction as part of producing the plan. Temporal requests need another input: the current date and time. Without that reference, a phrase such as “last week” lacks the anchor needed to generate a useful date condition.
How does a human request become a database condition? With “Open PRs by Peter,” the connected engine generates a match clause and resolves Peter to pwerry, Werry’s GitHub login. The important change is the value used for matching: the request can use a familiar name while the query uses the identity present in the records.
json
[
{
"$match": {
"user.login": "pwerry",
"state": "open"
}
},
{
"$sort": {
"created_at": -1
}
},
{
"$limit": 50
}
]
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A generated query is still untrusted code
The first working query is a milestone, not permission to execute whatever the model produces. MongoDB’s where or function clauses can run JavaScript on the server. Passing generated queries through unchecked would make the model’s output an execution path for unwanted behavior. Internal ID or metadata fields create a separate risk: the query can expose information the model should not control.
Validation uses “deny by default.” An allowlist defines the permitted operators, so an unfamiliar operator does not become executable merely because nobody remembered to prohibit it. Fields can also be allowlisted, although this dynamically discovered schema example does not implement a general field allowlist. Known internal metadata still needs explicit handling.
Tenant isolation makes the distinction concrete. Werry recommends separate collections for separate tenants. If a collection does contain multiple tenants, hiding its tenant-ID metadata from the model is only part of the job. Application code must also remove any model-supplied version of that metadata and inject the application’s own tenant constraint. The trusted tenant selection comes from code outside the model, and this sanitization should be procedural.
The next demonstration searches for authentication in a PR title. Regular expressions are disallowed, so the validator rejects the generated query. At this stage the error is simply returned. The engine has successfully enforced a restriction, but it still needs a way to recover from the rejection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retries fix a query; full-text search fixes a missing capability
The retry loop handles two different failure points. Validation can reject an operator before execution. Execution can reject a query that passed the validator—for example, a projection that improperly mixes inclusion and exclusion. In either case, the engine returns the error to the model and asks for a revised plan, up to a maximum number of attempts.
Where does a failed plan go, and what is allowed to reach the database? The branches below separate validation failure from execution failure. Both can send diagnostic information back to planning; only a validated query proceeds to execution. A retry therefore gives the planner another chance without removing the checks that rejected its earlier output.
Rerunning the authentication query reveals a subtler problem. By the third attempt, the planner has worked around the unavailable regular expressions and full-text search by using an exact match. That query tries to match authentication exactly and fails to find the intended content. What changed across the attempts? The query became acceptable to the available operators, but the remaining operation still did not express the search the user needed.
The correction is to add full-text search and create its index. The engine can then combine text search with author and state filters in the same match. The demonstration succeeds after that addition. This is a useful limit on self-correction: retries can help choose among available operations, but they cannot supply a search capability the database interface does not offer.
Validation and execution have separate failure paths
Read the diagram as text
- Generate or revise the plan. The model writes a query and can use returned errors to revise it.
- Validate. Procedural rules allow only permitted query operations.
- Execute. Database execution can still fail, such as for an invalid projection.
- Return the error. Feed validation or execution diagnostics to the model while attempts remain.
- Generate or revise the plan → Validate: Generated query.
- Validate → Execute: Validation passes.
- Validate → Return the error: Validation fails.
- Execute → Return the error: Execution fails.
- Return the error → Generate or revise the plan: Revise while attempts remain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The chat agent reasons over the same retrieval engine
The final application step moves to the chat tab and asks, “Did we work on anything authentication related last week?” The agent generates a query, retrieves the data and reasons over it. The query playground and chat agent share the same backing engine; the chat interface adds reasoning over the retrieved records.
The resulting engine supports filtering, aggregations, time-scoped queries and full-text search, with validation and error-driven retries. Werry singles out the error loop as especially important, but also recommends strengthening the validation harness before internal deployment. The workshop establishes an end-to-end implementation; it does not make the sample validator a complete deployment safeguard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale schema selection separately from ingestion
The first audience question widens the example from one pull-request collection to many collections and tables. Putting every schema into every prompt would enlarge the planner’s context. The proposed extra LLM step works with small collection summaries and descriptions, selects the relevant collection, and loads the corresponding schema for the query.
Keeping the records current is a different problem. Unblocked continuously ingests changes: a PR event triggers a webhook, the content is ingested and re-embedded, and the result is stored in the database. Schema selection reduces what the planner must read; the ingestion pipeline keeps its searchable data current. The answer describes that architecture without quantifying processing latency or cost.
Choosing sources depends on the questions people want answered. Two sources carry especially useful decision history:
- Slack conversations. Werry prioritizes Slack for “why” questions. A recollection such as a conversation with a particular person roughly three weeks ago already contains participant and time constraints that structured retrieval can use.
- PR comments. Review discussions hold organizational history that is otherwise easy to lose. Unblocked processes past PR comments into “best practice memories,” distilling useful lessons before serving them as context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
MCP can expose the engine; semantic search can guide it
Could a restricted MCP interface replace generated database queries? MCP can expose constrained operations, and its tools can be updated when the schema changes. It can also accept a natural-language request and run the workshop’s translation engine behind the call. The interface choice determines what the client may ask for; the query engine explains how the backend can fulfill that request.
The next question returns to the opening claim: should structured retrieval replace similarity search? “They’re absolutely complementary.” Full-text search finds keywords, while semantic search can match an idea expressed in different words. One useful sequence starts with vector retrieval, gives the agent relevant starting data, and then lets it form structured queries to fetch additional records.
A schema-discovery refresher also narrows what the sample algorithm guarantees. It merges JSON-like document structures and uses token and pattern heuristics to infer types and enumerations. Werry calls this iteration “pretty hacky,” describing a character-count check between two and eight as one heuristic. Procedural extraction makes the rules inspectable; it does not make those rules universally correct.
“Hybrid search” needs its own definition. In the later exchange, it means running semantic and lexical search and combining their rankings through rank fusion. An attendee describes full-text and semantic stages in an Atlas aggregation pipeline, and Werry confirms that Unblocked uses this kind of combination. That ranking combination serves a different purpose from enforcing author, state and date constraints: both can contribute to retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The ending sharpens the limits: sampling, tool calls and schema change
Why sample 300 documents? The number came from empirical trial and error, including customer reports that retrieval had missed something they expected to find. It is not a completeness guarantee. Document variety affects the sample size a use case needs, and rare shapes can still be missed. Unblocked uses many static schemas to prioritize correctness; dynamic discovery is useful for exploring unfamiliar data and connecting arbitrary databases.
The discussion also clarifies what the required tool call does: schema discovery happens at startup; the enforced call is for name extraction. Two enforcement techniques are described:
- Validate the query tool’s username argument. Make the username an input to the query tool and reject an incorrect or malformed value there.
- Monitor the extraction call. Watch the agent’s actions. If it skips the required extraction tool, insert a message telling it to make that call before continuing.
Finally, schema change requires an explicit maintenance path. An engine discovering an unknown schema can rerun its sampling process periodically as documents arrive. Unblocked’s controlled, static schemas instead change with the codebase: upstream changes are adopted in code and deployed through CI. These choices leave a practical decision for an implementation: use discovery to learn an unfamiliar database, or maintain a schema deliberately when the application controls its shape.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:12
Okay. So I'm, I'm Peter. I'm from Unblocked. Um, over here is my colleague, Brandon. And today, uh... Well, first of all, I just want to talk a little bit about what we do. So, uh, we build a context engine. Our company's name is Unblocked, and, uh, we deliver organizational context to humans and increasingly, of course, agents. Um, so the session was originally a couple hours. It's, it's now one hour, so it's quite condensed, uh, which means that I'm gonna step through the workshop pretty quickly. If you
- 0:42
don't get, uh, you know, all the steps nailed down as we go, don't worry about it because the repo's up online, um, and you can run through it at your leisure, um, afterwards. Um, so the thesis here is that RAG, and I'm just gonna say that RAG in our parlance today is gonna be semantic similarity search or vector retrieval, okay? RAG is only half of retrieval, and today we're gonna build the other half, which is turning nat-natural language into real
- 1:12
database queries. So here's the, again, the QR code for setup, um, if people haven't already done so. I'm gonna now skip, um, onto the next thing, and I'm just gonna briefly bring up, hopefully that renders. No, it won't, of course. Give me one second to switch.
- 1:32
Whoops.
- 1:37
And hopefully people can see this. Uh, this is the repo here. So setup steps are pretty simple. Um, there are s- a few sort of, uh, dependencies up here, prereqs that, that, uh, you'll need to run through. Um, and the, the course-- throughout the course of the workshop, we're gonna be running through a series of pull requests, uh, and the network is basically gone
- 2:04
or at least isn't running properly. Let's try this one instead.
- 2:16
Well, the good thing is we don't need network for this talk. Um, so hopefully you got everything before we started, um, 'cause this one's obviously not gonna work. But at any rate, uh, we're gonna be s- following a series of six pull requests, and they stack on top of each other. So I'll be... There we go. I'll be referencing these as we go. Um, and I'll just go back to here so that everyone can see that.
- 2:41
And back to the keynote. Now I'm gonna have to extend it again. Sorry.
- 2:55
Yeah. There we go.
- 3:02
All right. Hopefully people can see that. Great. So, um, our agents need context, and context is scattered across every tool that engineering teams live in. So you're gonna find it in PRs, review comments, um, issues and tickets, like in Linear and Jira, docs and wikis, Notion, Confluence, and of course, in conversations in places like Slack and Teams. And all of that is semantically queryable, but it's, it also has structure. It has authors,
- 3:32
dates, uh, states, labels, relationships, et cetera. And you know, the, the instinct, I, I love to say this, like even two years ago is ancient history in the realm of AI, but the instinct even two years ago was let's just dump this all in a vector store and retrieve it and re-rank it, and hopefully we get the right thing. Um, but that's, that's not gonna work. So I'm just gonna step back and, and kinda relate this to the larger discussion about context. Um, in the, in the context of a context
- 4:01
engine that serves organizational context, um, this is, this is made up of more than just code, as I said. We're connecting everything together. So the scattered context, uh, on the left in all your code and PRs, docs, et cetera, goes in, and the idea is to get, you know, grounded context that relates to each other out the other side that's specifically tailored to the task at hand. Um, and in that model, uh, we have to care not just about the semantic relationships, but of course the structural
- 4:31
relationships that connect all the data.
- 4:35
So again, RAG is about embedding the query, finding docs with similar meaning, and then we take that stuff, and we stuff it into the prompt, right? Well, um, that shines for content questions like find the code, find the PR that changed it, find the Slack thread that debated it. But, um, and if you wrap it in an agent loop, you can get pretty far with that, and you can also wire this thing up to like, um, you know, tools that can recurse files, and it acts
- 5:05
as a good jump-off point. You get a good semantic head start, and then from there, your agent can recurse over the actual file system in GitHub, um, and you can get pretty far with that. But in our experience, um, our, our-- this is not really the way things go in practice. Engineers often ask questions like this, PRs that I merged last week or my team merged last week. Um, they're, they're basically, you know, hunting for something in particular, and they know where the starting point is, but they don't know where the
- 5:35
destination is. But they do know the relationships.
- 5:42
So as an example, PRs I merged last week, let's think about this in a RAG context. You'd, you'd, you know, embed that sentence. You'd find the nearest vectors. You'd re-rank it, pick top K. Uh, you get PRs that mention merging about a talk last week, but, uh, y-you wouldn't get the actual relationships like- ... the author, the merge state, um, and whether it was merged last week actually. So what you could end up with is the wrong answer where you pull a PR
- 6:12
that's semantically related, but it was, uh, merged three years ago. Um, and so that doesn't actually answer the question 'cause there's a structural, there's a temporal aspect to that question. And so retrieval has two halves here, semantic, and again, today we're gonna be talking about this, structured queries.
- 6:34
And here's the key. We're gonna take, uh... The model doesn't actually answer the question in this paradigm, it writes the query that accesses the database.
- 6:45
Okay, so, um, it would be naive to say that generating the query is the only thing that we have to do. Um, in order to generate the query and to do it reliably, um, and, and to take care of maybe even some security concerns, we need to look at all four of these points here. So the-- first of all, the LLM needs to be aware of what the schema is before, uh, creating the query. It needs to understand, uh, some of the relationships in the data. So for example, if you pass a natural language
- 7:15
query that says, "Show me Peter's open PRs," um, then it's not gonna know what to do 'cause it'll synthesize a query that says Peter, when really the GitHub login name might be different. So it needs to be able to map that across. Um, and of course, you're gonna have some, uh, untrusted queries coming out the other end, um, maybe using the function operator in, in, in Mongo parlance which might execute JavaScript directly on the server, and we, we probably don't want that.
- 7:45
Um, and then finally, if there are errors the first time around, we wanna be able to self-correct those things. So here's, here's what this looks like in one s-single shot. What we want is something that looks like this. Um, it, it maps, uh... My colleague's name is Rasheen. His user login name is, is rashrab. Um, and then we want to be able to map things like the state and get maybe, uh, operators like limit and sort.
- 8:16
So, uh, this is what we're going to build. We're gonna start with a full UX, a stubbed engine, um, that is not capable to start with, and we're gonna gradually build this up. And then at step three, the thing will come alive, and so we'll actually be able to issue queries to it and see what it does.
- 8:36
And, uh, if you hadn't seen that before, now is another good time to scan that and, and get it in.
- 8:44
Count 30. If anyone needs help, I'm here to help you. Put your hand up, and I will walk over to your desk and help you get set up.
- 9:00
Yep.
- 9:00
Do we connect it to any repo right now or vcheck?
- 9:03
Yes, any repo's fine. Doesn't matter which one. Um, and if, if you wanna take a shortcut, you can just point an agent at the thing and let it set everything up for you. That works.
- 9:16
Okay. Let's move. So this is what we're gonna do to start with. Um, by now you, you probably should have run this ingestion thing. Don't worry if you haven't. Um, again, like, you can revisit this at the end. I'm just gonna quickly show what the, the kind of bare bones interface looks like for this thing.
- 9:36
So now this thing is running. Now, if I type anything in here, this is obviously not gonna work, um, and what you should see is something like that. Um, there's also a tab for chat. This is a fully implemented agent, uh, that can, uh, do work for you. So we'll come back to that in a sec.
- 9:58
And so this, this is the contract and the code if you wanted to read. Um, we're gonna-- Don't worry about this. We're gonna skip through this. Um, but basically, like, that's the entry point, and we're gonna be building around that. So the first thing we wanna talk about is schema discovery. Um, during ingestion, if you, if you look at the way that the ingestion script work, it, it actually had to build out a schema in order to download the content and store it. Um, but you can imagine that you might not know the schema. So the
- 10:27
ingestion may be happening in a service elsewhere, and you wanna build, like, a fully dynamic schema discovery engine. Um, Mongo's schemaless, so there's no schema to read, um, unlike, you know, PG or something where we could read the table definitions. Um, so what we'll do is we'll sample the documents to infer the schema.
- 10:51
And that's the whole trick. So the-- We're just gonna reconstruct it. We're gonna take, um, all the nodes and merge them down. So this is gonna sample three hundred documents, collapse it into a single tree. Uh, nulls are skipped because missing fields won't be able to constrain the type. And then, uh, we're gonna do something kind of interesting here with enum types.
- 11:17
And so now, uh, with, with enum types, we can do an interesting sort of procedural trick. This is just one thing. I would suggest using procedural tricks to, to do this type of thing because you could pass it through an LLM and do extraction, and that works, um, but it's fairly non-deterministic. So you always need some kind of structural, um, harness around it that is running procedural code to either validate the things that you pull out or to actually do the
- 11:47
extraction itself. And in this sample project, um, it's the procedural code that's doing the extraction.
- 11:55
And let's go and show that.
- 12:00
So we'll do-
- 12:05
And then run it again.
- 12:09
And now what we get is this thing here. So the schema ran, uh, the, the discovery process ran and it created a schema for us that outlines what the pull request schema looks like in GitHub. It even created some indexes for us. Now, this is, like, Markdown on purpose because we're going to take this schema and pass it to the LLM when we synthesize the actual, uh, query.
- 12:40
Now, the second step is the thing that I referred to a few slides ago about identity resolution, and this is because, like, my lo- GitHub login, um, is similar, but it's not quite the same. And so if I were to ask a question like, um, "Hey, find all the PRs by Peter," um, it ... if without doing some sort of identity mapping, it would just say, you know, it would just match across Peter and it's not gonna find anything. So we have to be able to solve that.
- 13:11
And again, um, we're gonna do this ... At, at first blush, this seems like we're coupling to the schema a little bit here. Um, but, you know, and it is absolutely possible to determine this dynamically, and you can have, uh, your LLM essentially synthesize the mappings for you.
- 13:32
So, but in, in our case, we're just gonna do these simple little tricks. Um, very simple sort of fuzzy matching. Um, Levenshtein distance is fine. You can just, like, match across the names and, uh, there you go.
- 13:50
Oh, actually, I wanna show you one thing before we get onto that one. So let's do the same thing again. We can pop up- Make this one bigger. Oh, yeah, I'm sorry. Absolutely. Tell me when it's big. Yeah. Good? Yeah. Thanks. Good call out.
- 14:11
Okay, so let's run our thing again. And now, um ... Oh, actually, w- I have to, I have to wait till the next step. I apologize. That's not gonna be wired up just yet. So we'll go on to the thing that will really give us the payoff here, which is step three.
- 14:36
So this is the actual query plan step, and it's, it- it's arguably sort of the simplest part. It's the LLM part. Um, and what we do here is we pass the schema, give it a tool, and force it to call that tool for name extraction and synthesis. Um, and then it will execute the plan. There's one other thing, which is the temporal aspect. Uh, LLMs are actually pretty good about generating temporal queries if you pass them the
- 15:06
current date and time. So that's another important thing to do in the, in the prompt.
- 15:18
And so here, this is the name extraction part. We're gonna force that tool call to happen in this case.
- 15:29
And at this point, I am going to show you the, the payoff for this. So,
- 15:36
whoops.
- 15:46
There we go. So now I can do things like this, "Open PRs by Peter."
- 15:57
And bam. So what we'll see here is that it, uh, generated this, um, this match clause and, uh, you can't see it from here, but it, it mapped Peter to my GitHub user login, pwerry, and generated a query, and that's the outcome.
- 16:22
So that's cool, but, um, we need to get onto the next step because even though we are able now to query the data, we have some safety concerns. Um, it could be the case that it produces untrusted output. And so, you know, it could use the where or function clauses and literally run JavaScript on the server, which means that this is now a prompt injection point. Um, you could also have hidden fields
- 16:52
in your schema, like ID or metadata fields that leak internals. So we need to sanitize, um... And we don't deny list these things. We allow list them. So it's deny by default. You want to create basically an allow list of allowed operators and, uh, you know, potentially an allow list of allowed fields. In our case, this is dynamic schema discovery, so we're not gonna do that. Um, but we could have some fields that we know about, like metadata fields, and this
- 17:22
is a, a useful thing for, like, tenant, uh, tenant isolation. So if you have a, a Mongo, uh, collection that houses multiple tenants, I don't recommend that. You should probably separate your collections by tenant. But if you did, then you could have a metadata field that contains the tenant ID, and you would want to hide that, kn- that, um, information from the LLM but also validate the output. So if it outputs that metadata field, you need
- 17:52
to wipe it out, and then you need to inject your own metadata field underneath.
- 18:05
And again, like, technically this is, it's not actually that difficult to do. You just have to write the code. It's all procedural. Um, I, I wouldn't recommend using LLMs for this part.
- 18:19
And so again, allow list, don't deny. Or deny everything, I should say.
- 18:33
Now the last part is, um, we're going to... This kinda dovetails into the next step. At the end of validation, we're obviously gonna return an error, and it's gonna get dumped out. Right now, that error is just, it's just there. We're not gonna do anything with it just yet, but I will show you now what this looks like, um...
- 19:03
Oops. Let's stay on that one and run our service. So now,
- 19:12
I'm gonna write, run another query that checks for authentication in the title, and we've disallowed regular expressions. So now what we can see is that it rejects the query.
- 19:36
And finally, this is the thing that kinda ties it all together, is the retry loop.
- 19:47
The plan can fail in a couple of different ways, right? Um, it can fail validation, so maybe the, uh, the LLM actually outputs things that are illegal according to our validator, or it could actually fail during execution. Um, so for example, let's imagine that it, it mixed an inclusion, exclusion projection, um, then that's obviously gonna fail, and then, but we can self-correct. We can take that error and
- 20:17
pass it back into the LLM and say, "Hey, that was wrong. Um, maybe you can do better." Up to some number of max attempts, of course.
- 20:30
So I'm going to show now what this looks like back over to our project so we can punch up a level. Whoops, sorry.
- 20:45
You have to forgive me. I actually have to read this off of the screen out here.
- 20:56
Okay. So now if we run that same query...
- 21:09
Perfect. So what we can see is that it took several attempts to get it right. Um, by the third attempt though, there's a bit of a problem here because we said no regular expressions, no FTS, um, so now it said we're gonna just use match. But now it's an exact match, so it tries to find authentication, fails, which means there's one more thing that we can do,
- 21:39
which is this.
- 21:42
We'll add full text search.
- 21:49
It's pretty basic. Um, we need to create an index obviously, and we'll combine it with author and state filters, uh, in the same match. And so now we should be able to answer a question like this.
- 22:08
And let's see what happens and see if it, it's actually gonna do what I said it's gonna do.
- 22:29
Bam. There we go.
- 22:36
And so now the, the final payoff here is if we go into the chat tab, we can ask some sophistic- more sophisticated questions that, um, you know, an agent can actually reason about behind the scenes. So we can say things like, um, "Did we work on anything authentication related last week?"
- 23:10
So here we can see that it generated the query. Now it's pulling the data and kinda reasoning about it. There we go.
- 23:22
And that's it.
- 23:35
So just a quick summary of what we built. We're basically, like, reaching the end. This is gonna leave lots of time for questions. So just to recap here, um, there are questions that, that RAG can't touch. All those structural questions around things like state or temporality, um, you don't get the right data out when you ask those types of questions with RAG, traditional RAG. And so the same engine behind the query playground and the chat agent, uh,
- 24:05
they're, they're the same, so if you guys wanna play with that after, go ahead. Um, it's, it's the same backing thing. The only difference is, of course, that, uh, the chat agent actually does reasoning, whereas the query interface is just for queries. So just to, to recount on this, here are the reusable ideas here. Schema-aware prompting, dynamic schema discovery, identity resolution, validation, and error retries.
- 24:36
The error loop is actually one of the most important things.
- 24:45
And so we built something that's capable of filtering, providing aggregations, it's time-scoped, has full text search, um, it's validated, it self-corrects, and it powers a full-blown chat agent. So you could take this into your organization and just deploy... I, I would recommend maybe working a little bit more on the validation harnesses before deploying this internally, but, um, yeah, it'll, it'll answer all your questions about your PRs.
- 25:14
And that's it. So we've left lots of time for questions now. Did, did everyone, uh... Was everyone able to actually set it up? Just curious. No? Okay. If, if you want, uh, you can come and fetch Brandon and I afterwards. We'll just kinda linger outside here. If... Oh, at the, and at the P16 booth as well. Yeah, but that, that starts later today at around four o'clock, and we'll be there all week. Cool. Any questions?
- 25:44
Yes.
- 25:45
I have a question. I think you did a great job breaking down, like, the, the basics.
- 25:49
Yeah.
- 25:49
A, a lot of my mind goes to, like, okay, so at scale, how does this work? For example, you, you generate the schema, and you put it into the system prompts. How does that work at scale? It's like, also, depending on the size of your company, how are you getting all of this data? How do you choose what data? And, like, how long does it take to process all this? What's the cost? And, like, that's where my mind is going.
- 26:08
Yeah, it's a good question. So, you know, um, I'm just gonna repeat the question so everyone can, can hear it. The demo showed obviously a fairly contrived example where you're, you're just focusing on pull requests. Um, and so the question is, how do you scale this? Like, how do you do this across multiple different collections, multiple different tables, et cetera? Um, so you, you can add, uh, another LLM step on top where, uh, the s- the schema collections have tiny little summaries and descriptions. And then, uh, depending on what the
- 26:38
query is, you can load the right schema for that query. LLMs are surprisingly good at this. They weren't a year ago, but now they're very good at it. Um, so that's kind of the way to shave the context thing down, and that's to your question about scaling things out. Um, was there another aspect of scaling that you were interested in or-
- 26:56
Yes, like the getting all the data in. And I guess, like, part of that is, like, cost.
- 27:01
Yeah. It's liveness and stuff too.
- 27:02
How long does it take to do? Like what is-- what data should I even pick to put into this rather than this?
- 27:08
Yeah. It's, it's a good question. So Unblocked, um, at Unblocked, like, we're continuously ingesting content. Uh, we get webhook callbacks for all the events, for example, in, in PRs. And so every time there's a change, we get that webhook event, we ingest the content, we re-embed it, um, and store it in, in the database. Um, that, that's just really just like a distributed systems question about how you scale that out from an ingestion level. Um, you know, storage, same thing. So, yeah.
- 27:37
Yeah. So maybe this will make it more concrete. So what-- if you would say, like, these are the top three data sources, like if you-- someone would go out this-
- 27:44
Yeah
- 27:44
... it seems like GitHub, maybe like Confluence or something.
- 27:48
Yeah.
- 27:48
What would, what comes to mind?
- 27:49
Um, so Slack for sure. A lot of the, like, why questions and, and decision-making happens in Slack. Um, people, uh, in, in the context of this sort of like structural query, um, people usually think about conversations that they've had in the past, um, and they'll be able to query Unblocked and say, "Oh, yeah, you know, roughly three weeks ago, I was having a conversation with so and so." And so that, that's a perfect thing for a structural query. Um, yeah, like I would say
- 28:19
that, um, Slack is the most important for why questions. Um, a lot of the why decision-making is also embedded in PRs. PRs, uh, have all of that, like, organizational history in the comments, and that's just basically black hole, right? So, uh, you can basically run over that data before you serve it and, and kind of mine it for interesting insights. So for example, at Unblocked, we, we create these things called best practice memories, which
- 28:49
are distilled from PR comments in the past. Does that make sense?
- 28:54
Yeah.
- 28:54
Yep. Anyone else? Yep.
- 28:57
Um, question. So instead of letting the model or, like, generate queries and access directly into the database, what-- how is this better than just making an MCP, for example, and like building these layers? Since you have like this disallow all mindset where you-
- 29:14
Yeah
- 29:14
... give it more access, uh, why not just build an MCP that gives it like a very contrived list of what it can do and control that instead of like letting it build its own queries?
- 29:26
Yeah, I mean, you, you can definitely do that. So if you have a structured, if you have a structured query table, you could express that through MCP. You can even do that dynamically by updating the tool every time the schema changes behind the scenes. Um, this, this type of thing is showing like what happens on the other end of that MCP call. So you, you could, you know, have an MCP interface that just takes natural language queries, for example, and then translate that all on the back end. Or you could provide a direct interface to the database. It depends
- 29:56
on like what the customer interface looks like, but yeah. Yeah.
- 30:02
Yep.
- 30:03
Yeah. I have, uh, two questions.
- 30:04
Yeah. Yeah.
- 30:05
You can decide how much time to spend on each. So, um, the first one is gonna be, I actually didn't quite exactly catch how you created a schema from basically like a seamless pool of information, right? So maybe a quick refresher on that.
- 30:21
Yep.
- 30:29
cases, you know, it's, it's better. Do you have any thoughts around kind of like combining RAGs with this kind of technique or, you know, what is the preferred way of doing that?
- 30:41
Uh, how do-- you mean how to combine it or... Let me just make sure I understand your last question.
- 30:46
Uh, both how and like should you-
- 30:50
Yeah.
- 30:51
Or do you feel like this is like the superior like context-
- 30:54
No, not at all. No, no. They're absolutely complementary.
- 30:56
Okay.
- 30:56
Yeah. Yeah. I mean, y-you can have a full text search that does this kind of, um, you know, like that's full text search, right? So it'll identify keywords. You still need some kind of mechanism to, to do like the actual semantic similarity search because they're-- you all have ideas and sentences that are encoded that aren't not ne-necessarily textual matches.
- 31:19
Right.
- 31:20
So you want, you want that like conceptual matching.
- 31:23
Yeah.
- 31:23
Right? So a-as I said, um, at the beginning, like there's... Similarity search is actually really powerful for jump off points. So you can, for example, start with a vector search and then from there give it access to tools like this because then it'll actually have like some interesting starting data-
- 31:43
Mm.
- 31:43
And help it form queries to go and, and fetch the data. So yeah, that, that's, that's something that we do. And so sorry, the first part of your question was?
- 31:52
Um, I didn't quite catch but you created a schema.
- 31:55
Yeah.
- 31:55
As you said, you don't use an LLM because it'll be probabilistic, right?
- 31:58
Yeah.
- 31:58
So how does, I guess, deterministically create the schema by default LLM?
- 32:03
So you take, um, you take, you know, in, in Mongo anyways, like these are all documents and they're, they kinda like form into JSON. So you can merge those things down and look for certain tokens and then, you know, extract the tokens and then that's how you distill the types. So you, you can l-look through the source code and you'll see that at least like this iteration of that algorithm is pretty hacky, right? It'll do things like look to make sure that the character count is between two and eight for, uh, for
- 32:33
type information, and then for enumerations it's, it's looking for particular patterns as well.
- 32:38
Okay.
- 32:42
Yeah.
- 32:43
Thank you.
- 32:43
Yeah.
- 32:43
Hi. Um, so two questions. Uh, so sorry for the recording, but they're really kind of short. Um, so I noticed that you use Longbeam for the agent. Uh, do you also use Atlas sometimes?
- 32:54
Yes.
- 32:54
You do?
- 32:55
Yeah.
- 32:55
Also, and the second question I had was, um, have you guys looked at like hybrid search, uh, as a potential kind of search option?
- 33:05
Now, that, that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about. Do you mean hybrid search in terms of, um, a provider that does semantic similarity-
- 33:16
Correct
- 33:16
... and then, uh, also does a lexical search and then does rank fusion?
- 33:21
Yeah. Yeah. So actually, um, with Atlas, uh, you can actually as part of an aggregation pipeline-
- 33:26
Yeah
- 33:26
... you have a full text, uh, stage and a semantic stage, and then that is a hybrid rank fusion. Yeah, sorry, I forget Longbeam uses RAG.
- 33:34
Yeah. Oh, yeah, yeah. No, so, so we, we do that. Yes.
- 33:37
Oh, okay.
- 33:37
The answer is yes. Yeah.
- 33:38
Great.
- 33:44
Yeah.
- 33:44
And then my first question I had was for schema sampling, the three hundred docs sample.
- 33:45
Yeah.
- 33:45
Um, did you...
- 33:48
How did you get to that number as like this is like a good, good set to sample?
- 33:53
Uh, just empiric-empirical trial and error. And so, I mean, there's, there's no easy solution to this. Um, be-behind the scenes, like at Unblocked, we're, we're using a lot of static schemas 'cause we wanna get the correctness absolutely nailed down. This, this was like a, just a technique that I wanted to show because, um, you know, for exploration of data or experimental, um, uh, for experiments or either to wire up like arbitrary databases, this is a good
- 34:24
technique. So we, we got this through some trial and iteration with customers. And, you know, we'd, we'd ingest a whole bunch of contents from the document store, and we'd get responses back from customers that said, "Oh, you missed the thing that I was searching for," and we'd dig into it, and we'd figure out why, and we kinda got to the sample size that made the most sense.
- 34:44
Yeah. No, it definitely makes sense. Yeah, 'cause I guess something that I've also run into with some of our developers is that like we have to kind of tune like the number of docs you sample to sort of use case.
- 34:56
Yeah.
- 34:58
It just depends on how varied-
- 34:58
It's absolutely true. Yeah. Definitely. You're, you're always actually gonna-- you're gonna miss something. There's always gonna be some document that has that little edgy thing. Yeah.
- 35:09
Anyone else?
- 35:12
I thought I saw a question over there. Oh, yeah.
- 35:15
So, uh, in the talk you mentioned that there is some forcing of the tool calls happening. Um-
- 35:21
So sorry, can you say that first part again? There's-
- 35:23
Uh, you mentioned that you force a tool call, uh, right, to understand the schema, the schema tool call.
- 35:30
Uh, we-- the, yeah, what we do is we force a tool to be called for... not for schema discovery 'cause that happens at the start.
- 35:37
Mm-hmm.
- 35:37
It's for name extraction.
- 35:40
How do you make sure that it is deterministically called every time? Like, do you see cases where it's not called or does it affect-
- 35:48
We, so we have a validator that watches what the agent's doing. So like, if it doesn't call the, the first part, which is name extraction, um... So there's, there's actually two techniques for this. One is you create a tool that does the, the query, and as an argument to the tool, you accept a username. So you can fail right there if the username is not correct or malformed. Um, if you want to do name extraction, uh, via tool call, then you have
- 36:18
to watch what the agent does, and then if it doesn't call it, you just basically insert a user message that says, "You forgot to call the extraction tool. Do so now." Yeah.
- 36:33
Yeah.
- 36:34
I have a question about, uh, how to handle the cha-- like schema involved, like schema change?
- 36:40
How do you handle the changes?
- 36:41
Yeah, like, like schema change. Like the, the query may either be some, like... Like the, the query assumption is based on like the existing schema.
- 36:52
Yeah.
- 36:52
And then also like how to probably ensure like schema is updated and also like the, the general query is always like follow up the, the latest schema.
- 37:04
Yeah. So r-remember, like for this particular example, the schema is, is completely unknown. And so as documents get added, um, that schema could update. So your, your, your choice is either, you know, on a sampling interval, you, you rerun the schema generation. Um, in our case at Unblocked, the schema is, is static and it's tied to our code base, and we control the schema actually. So if the schema of-- If GitHub decides to change the schema, um, that of course rarely happens
- 37:34
'cause that would have serious implications for their customers. But if they did, then we would just adopt whatever changes are made statically and then obviously deploy that through CI. But yeah.
- 37:50
Anyone else?
- 37:55
Awesome. I think we can, uh, reclaim some time in that case. Thanks.