Beyond RAG: A Relational Context Engine That Cuts Token Burn — Unblocked

Are we up? Okay. Can you hear me just like this just fine? Everyone good? Okay. Um, so we're, we're gonna be doing a bit of a speed run here 'cause, uh, uh, we've only got twenty minutes. But, um, I'm c- I'm Peter from Unblocked, and at Unblocked, we build a context engine. So I'm gonna show you a couple of things. The talk's gonna be split into two parts. Um, I'm gonna show you what a context engine is and how it can add value to your organization, and then we're

gonna actually go and build a component of a context engine, and I'm gonna show you a couple of open source projects that you can, uh, go and play with after this.

So there's the agenda. What's a context engine? How does it deliver value? And then we'll build. Um, so let, let's just hop back in time maybe a few years to the times before AI agents. And in those days, um, you were the context layer.

Imagine that you're a new employee to an organization, and you're just trying to get to grips with what's, what's happening. What you usually end up having to do is look at the... look at a whole bunch of code, rip around documentation systems, just trying to find the information that you need to get your work done. Um, and not to mention, like, all of the, the sort of shared history that's built up over time through an organization dealing with incidents and the battle scars that come with it. Um, so what a

context engine does is it takes all of that institutional knowledge, and it pulls it in. And what you get, the scattered context that goes into the context engine, and out the other end, you get grounded context, which means that the context is all interrelated to each other. It's intent-specific. It's personalized. And it's permissions-aware.

And so I'm gonna show you exactly what I mean by that now. Let's imagine

that I'm gonna go into here into Claude. I'm gonna show you a couple of interesting sessions that I had. So, um, I asked Claude here, this is without a context engine attached, to create a plan to optimize a component in our layer called the Source Mark engine. And, uh, I'm sure you're all familiar with what Claude does when it starts a new task. It's like a new employee. It doesn't understand anything about your, your code base or your organization. So the first thing it does

is it greps around your code base and, and tries to figure out what's going on. So you can kind of see that here. And, you know, it, it figures it out eventually, or at least, you know, it, it confidently projects that it's figured it out. Um, and so it provides kind of like a, a recommendation for me there. But now I'm gonna show you what it looks like when you attach a context engine to the back end. So now what it does is it calls the Unblocked context engine, and Unblocked now delivers a very

comprehensive but task-specific context bundle for this task. And what you can see is that it, it's referencing not only the source code, but also, you know, Notion documents that talk about the architecture, um, pull requests, Slack conversations that were had about optimizing this particular component. Um, and the, the, the net out is that you get this really tight context bundle with all these links. So now, uh, if, if, um, you know,

Claude wants to go back and fetch more information, it has all these great jump-off points. But the really interesting thing is, is this. So here I spent, um, you know, $1.29 on this, and it took about a minute and thirty seconds. Um, that was with the context engine. But without, I'm spending up to, you know, $2.60 and three minutes. So what you see essentially is a fifty percent reduction in cost, which is pretty incredible.

Um, now I wanna show you another interesting demo, um, which is, uh, an interesting session that my colleague... Whoops, sorry. Bring this out here. It's an interesting, um, session that my colleague had with Unblocked the other week. So what he... We also have a code review product, and Richie works on that. And he discovered that the number of issues that the code review product was surfacing was, was starting to drop. So he just asked Unblocked, and

Unblocked was able to pull this information, um, and, and, you know, narrow it down very quickly to, uh, a likely cause, which was that we switched from Opus four-eight... or Opus four-six to Opus four-eight, and the behavior characteristics behind the scenes, um, you know, caused all those issues to drop. And, uh, so what he did is he asked Unblocked to just fix it. And now Unblocked, um, was able to just

create a PR behind the scenes, and it has all that context, and it knows what to do. But the really cool part is you can see that it created this description, and it's, it's just remarkable. Like, it, it, um, understood the PR that caused the regression in the first place, and then it found a Slack conversation where the convers-- where, where this was being discussed and pulled on some of the data here to help inform, um, the fix, which, which is just nuts.

So I'm gonna go back to... Whoop, I think I lost my keynote presentation here. Let's see. There we go.

So you have the same problem with agents, right? Cost... Oh, sorry, this is the wrong it's the wrong keynote.

There we go. So now I think what we're gonna do is we're just gonna get right into the building part 'cause we're, we're quite short on time today. Um, and so I- I'm just gonna summarize this. You've got context scattered all over the place. You've got context in PRs, reviews, comments, issues and tickets, docs, conversations. Um, and, and all of that is, uh, semantically queryable, but it's also structured.

And RAG nails the content part. Um, it, it can be, it can handle things like content questions, find the code, the PR that changed it, the Slack thread that's debating it. And if you wrap that stuff in an agent loop, it can go really, really far. You give an agent loop tools to do file search after this, and it, it's very powerful. But it doesn't handle some of the other type of, uh, structural questions, like PRs that I merged last week.

Who reviewed payments the most, open PRs, et cetera. Um, and these are queries, they're not semantic searches. So I wanna just show you again, um, going back to Richie's example here, that he started the, the conversation with Unblocked by asking it a query-based question 'cause there's a temporal element. Can you show me a chart of ratio weekly since January 1st? Okay. So in order to do this,

it's not gonna be able to just pull this information from a vector store 'cause that's not where that information's encoded. So it's, it's gonna have to do that, um, through some sort of query engine.

And so, you know, retrieval has two halves essentially. We've got semantic retrieval, and that can handle those types of semantic-based questions. But today we're gonna work on this, the structured part. We're gonna turn structured queries or natural language queries into structured queries.

But generating the queries is actually the easy part. The hard part is the scaffolding that sits around it. So you need to do things like understand the schema, so you can pass it to the LLM, so it generates the, a tightly bound query. Uh, you need to also be able to map identities. If I ask a question like, "Open PRs by Rasheen," um, it's not gonna know who Rasheen is because his identity in GitHub might be completely different. So we have to be able to map that somehow. And

then when you get the query out of the LLM, it's untrusted. It could have, uh, operations that are dangerous. It could cross tenant boundaries. And so you have to have some way of validating that the query is, is, uh, correct and safe. And then finally, uh, even though you do that validation, there could be errors. Um, and when you actually do the query against the database, there could be errors there as well. And so we can actually get the LLM to try to correct itself if it gets into that

situation. So this is what we're aiming for here. We wanna go to, uh, to this. We wanna be able to say something like, "Open PRs by Rasheen," and then we wanna get this kind of query on the back of that. So just take a sec here. Uh, this is the open source repo that we're gonna be building against, and I'm going to jump right into what that looks like actually.

So this is the repo here. It's called the Document Query Engine, and I wanna show you what it looks like when we just boot it up the first time. So I can say, you know, "PRs by Peter at Ground Zero." Oh, actually, we're actually not running it, so I'll go back.

Let's run it first.

Now I can ask my question.

So when you start, it's gonna have... It's not gonna be wired up at all. It's gonna produce nothing. So we're gonna just gradually build this thing up here, okay?

First thing we're gonna talk about is automatic schema discovery. This is completely dynamic. Mongo is schemaless, not... A- and not all databases are schemaless, but the same kind of principle can apply. You can dynamically discover the schema at runtime. Um, and so what, what the technique involves is sampling a whole bunch of documents in the document store and then deriving a schema from that. And, uh, you know, I just wanna show you, like, one tricky part of that, which is

inferring enums. Um, now the, the, the outcome from this is that you can use an LLM as a Thor hammer, um, and it can do a pretty good job. But actually, you get, uh, really, really far just using traditional procedural methods. So you can... Y- you know, don't always fall back to an LLM to do this kind of what looks like non-deterministic work. This is actually quite deterministic, this kind of thing. So

now that we've discovered schema, I'm just gonna show you what that looks like in our demo. Whoops. First, we have to level up to the next...

And then run. So now we've discovered the schema from our, um, GitHub collection, and this is just pull requests, okay? So it has not only the schema here, but also, um, indexes that will help improve the performance of queries.

The next step is identity resolution. It's that thing I talked about, about mapping, you know, users to, uh, real identities. And, um, again, this is- this can be solved just using simple procedural tricks. So, um, that- that's actually more or less how we do it, uh, in underneath the hood at Unblocked. When we take usernames or identities from different systems and map them across, um, you can do this kind of fuzzy matching, and then in the end, if you have to, you

can pass a list of names to an LLM and have it act as a tiebreaker.

And finally, uh, the last step before we can actually show the cool stuff is this. It's the synthesis part. And roughly speaking, this is actually the easiest part. We just take the schema that we dynamically resolved from earlier, we take the current date and time, uh, maybe some information about the user that's asking the question, so it can easily resolve questions like my PRs and not just Peter's PRs. Um, and then we pass it to the LLM and ask it

to synthesize a query for us. And the outcome, uh, will look something like this.

Let's get to this one.

So now I can go back to my engine and try to run that query again.

And it's going to pull the documents for me now and generate this query.

Now, the next step I- I was talking about was, was validation. This is actually the most important thing. Um, your, your, uh, query could contain really dangerous operators, like the function operator, for example, which would execute raw JavaScript, um, on the DB server, which is definitely not ideal. Um, but there could be other things that you wanna do with this. Like, for example, you might have hidden fields that you don't want, uh, to be surfaced either to the LLM or for the LLM to, to derive and execute.

If you have metadata fields, for example, that contain, um, tenant IDs and you wanna isolate each query to a specific tenant, you're better off making sure that the LLM doesn't interact with those fields at all, and then just constraining it behind the scenes. You can inject those fields underneath.

And that's kind of what that looks like. So again, we're gonna just quickly show the payoff for that and what that looks like.

Let's go up to our validation step. Oops. Run it again. So if I now ask a question like this, PRs with authentication in the title, that is probably gonna derive some kind of like regular expression thing. And there we go. Okay. So, uh, our validation engine said, "Sorry, regular expressions aren't allowed. You're getting bounced." And now we're not allowed to do that.

So validation is cool, um, but what do we do with that? You know, we've got an error, um, but what we can do instead is just take that error and just loop it back in to the LLM and have the LLM kinda self-correct.

And that's kind of what it... That's all it is. That's all it looks like from a technical standpoint. It's just a loop with some sort of maximum, uh, retry.

And so if I try this again...

Bounce up to my retries and run that.

Now, if I try that query again... Oops, that's the wrong one. Let's try this one.

Now what you can see is that it actually took a couple attempts . Um, but it got there in the end. However, um, it constrained it down to use match, and unfortunately, this is an exact title match. We don't have any PRs with that exact title, so we need one more thing,

which is this, full text search. So now if we add full text search to our engine, um, hopefully the outcome that we get is something that looks a little bit like this. So let's try it one more time.

Awesome. So there you go. We got... We went from start to finish there. This is all open source, so you can take a look at how it's implemented and play around with it yourself.

And this is what we built, something that can effectively handle all of the, uh, other half of RAG. So anything that has to do with, uh, structural based data.

You can have filters, aggregations, it's time scoped, and it self-corrects. But what's really cool is that it also powers a chat agent. And so if I go over here and I say something like, um, "What authentication-related things have we been working on last week?"

Now you get this whole thing that will show you exactly what it's doing, all the queries that it creates.

And now it has the data, and it's just kind of reasoning about it behind the scenes, and there we go.

Very cool.

So there you are, structured, queryable context for agents and humans. Now, there's just one last thing. Um, on the description you probably saw that we were promising a context engine simulator. If you don't want to wire up Unblocked, uh, right away because you've got some constraints in your, in your environment, um, we have an open source project that will allow you to do this entirely locally yourself. And so the idea is, uh, it will create

a context bundle for you on a task by task basis, and then take that context bundle and run an AB test against the same task and show you the benefits of having all of that context at your fingertips for your agents. And that's the, uh, PR code for that, if you want that.

All right. And one last thing . If you want some coconuts, come by the booth

. Thank you.