Stop babysitting your agents: building a context engine for mergeable code

Talk transcript

1. 0:00

   [upbeat music] Hello, everybody.

2. 0:16

   It's good to see you. I'm Brandon. I work at Unblocked. It's a great place. Uh, and my goal is to make it so that you don't have to babysit your agents anymore.

3. 0:24

   Um, I'm sure we all have a different take on what that means. What I think of is care and feeding. Basically, agents, whenever you spawn it by typing Claude in your CLI, let's assume, whatever tool you may use, they exist, and it's like a brilliant software engineer has just spawned, and it knows nothing about what it needs

4. 0:39

   to do, and knows nothing about your org. It's completely zero context in its head. So typically what happens is people have to move through building that context, which we'll go through in the beginning.

5. 0:50

   But first, what I'm gonna cover today pretty quickly is three myths about how you can stop babysitting your, uh, your agents, and then three lessons that we learned the hard way building a context engine at Unblocked.

6. 1:01

   So our product basically provides this context for agents, and we'll tell you a bit about how we built that, techniques to care about, and I'll show you a repo we actually constructed at our workshop yesterday that will be open source at the end of the week, uh, which has one component of what's in a context engine, which

7. 1:16

   you can lift for yourself and bring into your org if you'd like.

8. 1:19

   So not long ago, you were the context engine. If you think about that when you're writing code, you thought about everything. You knew everything. You figured it all out.

9. 1:28

   You were dealing with that. And now what's weird is you're in a weird state where you are actually the context engine for your agents. So a useful way to think about this is: how did you build context when you showed up at a company?

10. 1:39

    So day one, you had probably nothing, but you were really smart. You finished school. I don't know, maybe some self-learning. Then over time, you accumulate context by doing stuff at work, meeting market, meeting your team, being like, "Here's a PR," getting it rejected, all these good things built up a lot of your capabilities.

11. 1:56

    And then finally, you became very good at your job because you asked good questions, and you knew how to gather accurate context and shred stuff that wasn't helpful for you.

12. 2:07

    This is where we're at right now. Most people here, if you look at the bottom, is in the you-are-the-context-engine stage because you're either dealing with the early phases of AI, which was just fancy autocomplete, or you're in an agentic IDE where you're triggering every job.

13. 2:22

    What we see with all these businesses that work with us in their AI adoption is it's usually at a varying level of this. This has been adapted from Basim Eldes' work.

14. 2:30

    Um, if you check him out later, great engineer. But basically, this is the type of ladder that we're dreaming for, and far on the side is, like, a dreamy future that maybe Codex figured out, I think.

15. 2:40

    I didn't see Ryan's talk, unfortunately. But everyone else is kind of trying to get there. In order to move through this, you want to get to the curated context layer.

16. 2:49

    That is typically what a lot of teams are doing by creating static repos. So static stores of a bunch of context that says key things about their company. These can include, of course, Claude MD files, AGENTS.md, that-- those types of tooling.

17. 3:01

    But usually, people start to put a bunch of other key corporate context into an area that agents can access to pull data from. The issue with that is those are static content pieces, of course, so someone has to maintain and up them or, sorry, update them.

18. 3:14

    As well as they don't have, uh, the availability of, you know, actual raw runtime data. There's just a bunch of information that engineers obviously need that don't go into these static layers that you're starting to see, which typically look like a file system.

19. 3:26

    Wow. Context engine. What this needs to do is basically have all that static content, of course, but also be able to, at runtime, when a query comes through from the software engineer, typically, "How do I implement this feature?"

20. 3:39

    At runtime, it's able to pull all that static source across your entire corporate knowledge corpus, essentially, whether it's any many SaaS apps, different systems or records, and pull in the runtime singles in order to analyze, reason across all those surfaces, all those different data stores, and then run exhaustively to actually find all of the things that are

21. 3:59

    important and then send a token-optimized, AKA small, response to the agent with all the details it needs to then execute its next steps. So you'll see through this, but typically that means getting the best context up front makes all agent choices and actions after that even better.

22. 4:17

    So if you give it, for example, a key research packet of like, "Hey, I want to do a new integration," and it-- you drop a packet so that it creates a good plan and that information says, "Here's our patterns.

23. 4:27

    We use factory pattern. We do these things," like all the things about your organization, it's then able to trigger its background agent jobs to go grep your code base to do the things it needs to with higher accuracy, which means it's more token efficient and it gets the job done faster.

24. 4:42

    This is the problem from our, our friend Karparthi. The gap is not intelligence at this point [chuckles], it is context.

25. 4:49

    Though mythos sounds pretty cool. Um, so the problem is people think that access is the answer, but it is not understanding. So providing your agent tools with MCPs, with pipes to different sets allows it to access that.

26. 5:04

    But you have to remember, you day one when you showed up at work, you don't know where things are, and you definitely don't know what you don't know. So there's probably some service over there you've never heard of before in your life.

27. 5:16

    You're working on a thing. Your agent's like, "Oh, I did it. I wrote the whole code from scratch." And then your senior engineer is like, "Hey, bro, we have a service," and denies you. [chuckles]

28. 5:26

    So one thing you'll see is we actually triggered this task that's a real prompt. You'll see data later in this, uh, slide about the outcomes. We did it with Unblocked, a context engine, and then without one, but it had an MCP access to each SaaS tool that was required to get the job done.

29. 5:41

    And I'll show you the differences. The short story is you kind of get this. The naive run, which just had the MCP access, basically passed all the code checks it compiled, but the senior engineer was like, "This is totally wrong, and what it tried to do would have broken our entire system if we had shipped it."

30. 5:58

    So three myths about building context. The first, if I do naive RAG over my docs, that is context. Unfortunately, that does not work. [chuckles] Naive RAG picks a bunch of things.

31. 6:09

    It puts a data store there, and the agent can then crawl across that data store. But it typically falls down because there's something known as satisfaction of search. This is a known phenomenon in radiology, but the short story is, if you get an X-ray, because let's say you have a lung problem, the radiologist will scan, and when

32. 6:25

    they find something, they're like, "Oh, there's something on your lung." They stop looking 'cause they think they found the answer to the symptoms you have told them, and this is very bad in medical health, of course, 'cause there may be other things wrong.

33. 6:37

    So what happens with an agent is if you say, "Make a Zendesk integration," it will go, it might call an MCP, and the first piece of data it finds, it goes, "Oh, this, this must be the pattern."

34. 6:47

    It stops looking. So the issue is if it's not exhaustive, it will not find the actual root cause, or it may not find the correct best way to implement, and just a bunch of other problems can happen.

35. 6:57

    And basically, by the time the agent output is done, you read it as an engineer and go, "No." And then you're in a doom loop, where you're like, "Let me correct you.

36. 7:04

    It's actually over here. I'm gonna point to file." And so you're babysitting again. If I just connect enough MCPs, I'm done. I think I've spoken to that. They're there.

37. 7:12

    They're pipes. That's great, but they don't provide understanding or reasoning across it. And then finally, we did think this for a while. We dreamed of the one million context window.

38. 7:22

    It's here. I don't know if anyone's ever whacked it full with something and then tried to get the agent to do anything. [chuckles] It can't. Um, it basically just can't reason over that much data.

39. 7:31

    It's just not super helpful. It just sits there as there's no entities and relationships, and there's all these things that we need for these agents to be most effective.

40. 7:39

    Um, so the bigger context window does not solve it. There's a bunch of compute reasons why even if we got to a hundred million in a context window, it's still not gonna help other than needle and haystack problems, if you're obviously like find, find Wal-Waldo.

41. 7:53

    So basically, this is what we see. The classic waterfall. Code that compiles is what the agent can see. But typically today, they miss all of this 'cause it can't see it.

42. 8:03

    It doesn't know if it's there. It would have to run for so long grepping in a session to actually get your factory patterns or other things across your code base that you burn a bajillion tokens.

43. 8:12

    And then when you close that terminal window, buh-bye. [chuckles] You gotta just do it again, and no one wants to repeat this cost.

44. 8:19

    So this is why you need a context engine. It understands who you are and what information actually matters. So a key component of this is a social graph because you use that as a pivot point.

45. 8:30

    Because if I ask how to do a Zendesk integration, the context engine should know which code bases I work in, where my PR history is, who I work with, and what I mean when I say that.

46. 8:40

    Because at a large org, we deal with companies that have twenty thousand members that are-- that are customers of ours. It's very different. [chuckles] So you need to be able to reason that.

47. 8:50

    And by the way, a context graph is an incredibly useful technique for building these things. We'll talk about it. It should resolve conflicts. I don't know how many times I've looked at source code that's running in main, and we go, "Yeah, that's the source of truth."

48. 9:01

    But there's a Slack conversation where the CTO says that was implemented wrong. Which is right? The context engine must be able to settle that debate. And by the way, a con-- a graph of-- social graph helps with that because if you see the CTO saying in the [chuckles] Slack thread, "That's wrong," the CTO is probably right.

49. 9:18

    So the context engine reasons about that and goes, "Well, the code says this. Slack says this. That's the CTO. We should probably tell the agent what the CTO said, and of course, provide it the source code," et cetera.

50. 9:29

    So it passes what's truth. So it handles truthiness. That's a tough problem. We have a lot of techniques in our product to solve it. It is not fully solved. [chuckles]

51. 9:37

    It should respect permissions of governments. This is pretty basic. It's one of the reasons this is delivered over MCP is you can carry the OAuth model through for data governance and a bunch of other reasons that matter in scaling businesses.

52. 9:48

    I mean, as soon as you're [REDACTED:age], typically this matters 'cause some data should not be accessible to others. So when you build your engine, you do not wanna put everything in there, especially when you think about we ingest Slack conversations and Microsoft Teams convos.

53. 10:01

    So if it's you, we will return responses from private chats. But if someone else asks a question, we will never show them private chats that aren't theirs. That's just one easy way to think about it.

54. 10:12

    Ooh. And it should deliver the right context, the right model at the right time in a token optimized way. This is how ours works. So the short is we ingest a bunch of data sources.

55. 10:21

    It sits in our engine. We have six key differentiators, which I'll go into in a sec. And then on the output side, there should be many surface areas where agents and humans are able to interact with the context engine.

56. 10:31

    One of ours is simple. Human engineers in Slack just chat with it and ask it questions all the time to get data they need. I'll show you an example.

57. 10:39

    Um, but then, of course, agents. As you move to background agents, they need a context engine in order to run headlessly or to run in the background or run in the cloud because they have to be able to ask questions of a machine, not a person.

58. 10:51

    'Cause otherwise you're not gonna wake up to a PR. You're gonna wake up to a, "Am I allowed to use this tool?" And that's not helpful. So these are the six.

59. 10:59

    We're gonna move at pace. But the short is, I've talked to a lot of these, but these are kind of marketing terms. Unified system context, so it should be able to reason across all of your systems of records.

60. 11:08

    It has to be able to do targeted retrieval.

61. 11:12

    Conflict resolution, as described, there are many times [chuckles] where the docs and this and that are conflicting, so how do you settle that? That data governance problem, so secure access model, personalized relevance, building social graphs, knowing who you are, who you work with, and finally, of course, token optimization.

62. 11:26

    This is becoming a pretty big issue. A lot of benefits you get on token optimization is just by having a context engine 'cause you don't rerun those greps for the agent to know.

63. 11:36

    But also with an engine, if you reason across everything, it's then able to compress the response into exactly what the agent needs and only send that back as an answer.

64. 11:47

    That task I talked about, I'm not gonna lie, we asked Claude to do the comparison, so it made these bar charts and numbers, but it did pull out all the key points of this.

65. 11:56

    That same prompt, one was naive. It had all the MCPs it needed to get the data, and the other had our context engine only. This is the difference across key principles of engineering.

66. 12:07

    But these are funny. It, like, didn't catch that we used Bedrock as a, a fallback. It shipped, like, bugs. There was one that broke the custom callers. The short story is, if you're working at any form of scale, again, twenty plus, agents are just gonna try to mock things, and it'll look like prototype.

67. 12:22

    It's not mergeable. If you get a context engine, when I put up this PR, our senior engineer for the one with the engine basically gave me a nitpick and was like, "Yep, you can merge this.

68. 12:31

    Just, just fix that." Great. These are some of our hard lessons. [laughs]

69. 12:38

    We did try to optimize for access. Now understand, we're like, "The model will handle this." Like, the agent's totally gonna figure it out. It'll collapse into mythos or whatever.

70. 12:45

    It, it hasn't. It's been years. So we were like, "We have to solve this problem another way." And I think that's correct based on actually Anthropic's launch last night with cloud agents and Ryan's talk this morning.

71. 12:54

    You have to get context into the harness, and an engine is the way to do that. We hid conflicts instead of servicing them. The agent would actually just pick when we found conflicts at first 'cause we were like, "It can't be that bad."

72. 13:05

    It's that bad. So solving, [laughs] solving conflicts is an important problem. And then finally, this is a really fun one. We thought as good answers happened, we actually got feedback loops on those, so we were caching them for latency.

73. 13:16

    If you cache a good answer, basically it's like when you write docs, right? The moment you write it, it's no longer valid because things are changing. So if you cache a correct answer, and then tomorrow someone asks the same question and you answer it, you, you probably lied to them now because things probably changed in a twenty-four

74. 13:30

    hour clock, so the system is not the same. Obviously, some questions I'm sure are stable, but this led to a lot of problems, so I highly recommend against even if it's optimized.

75. 13:40

    This is where AI forward teams are. They're using context engines in all of these cases. So I know we're all engineers, but I'm sure that we support others in our orgs, like a ask engineering channel.

76. 13:53

    Our context engine, it sits in every customer's ask engineering channel. It detects if a question is asked, it scores confidently, and then it responds automatically. So when support teams, sales teams, whoever's like, "Hey, what's running in prod?

77. 14:06

    What's this?" The context engine just answers them and deals with the issue, just like it would answer an agent asking for data. Um, so there's a lot of ways like that use case I just talked about, but then ticket enrichment, triage, incident management, obviously working with your agents and coding.

78. 14:22

    These are all great ways that you get tons of leverage out of getting one of these into place.

79. 14:29

    Teams then customize them, so most fork. We have like a cookbook. They take that repo. The cookbook's full of skills. Um, but then you dev- devise your skills with your standard operating procedures.

80. 14:38

    You obviously can build your own workflows, whether you do that as a skill or some other technique. And then, of course, custom agents. All of these can leverage that same context engine, so you just get a huge amount of leverage.

81. 14:50

    This is what I'm trying to leave you with. An agent should write code that feels like it was written by someone who's been on your team for years. Like, that's just like we should expect that by now, and this is one of those techniques to get you there.

82. 15:02

    I can do a brief QA and I can give a demo. I have three minutes left, so what's your preference? [laughs] Demo. Cool. So this is the tool I talked about at the workshop.

83. 15:12

    We're gonna open source it. This is one component of a key-- of a social graph. And so

84. 15:19

    when you run it, this will be available, I think it's Monday, we're gonna actually open source. It's closed source right now. There's a whole setup, but we had a bunch of teams hack against it and ship code.

85. 15:27

    But basically it will build you one of these. Uh, I'm gonna zoom in. This is our engineering org. As you can see, there's different nodes and edges, a basic graph, but it's a social graph.

86. 15:37

    Rasheen is a goddamn machine. So he-- that's how much he ships is by the size of the node. If we go look at-- This is algorithmic, procedurally generated, so you'll see the tool.

87. 15:46

    But in short, you can see on the right who he's working with. Sorry, this screen is, like, kind of small.

88. 15:52

    Great. Kind of worked. Um, so you can see who he works with, whose code he reviews, what area, uh, he worked in. We use labeling with an API key from Anthropic.

89. 15:59

    Oh God, now it looks terrible. Let's zoom in.

90. 16:03

    A little better. Um, sorry. It's a fun tool. But basically this expert graph would allow you to, when a query comes across, we go, "Oh, you're Rasheen. You work with these people."

91. 16:13

    Great. It'll pivot on that data. It'll then zoom into the code bases that he's dealing with. So when you as an engineer ask maybe not the best prompt of all time, but you're like, "Hey, I gotta get this done.

92. 16:24

    There's a bug. Gotta fix it," it'll know who you are. It'll pivot on that. It'll probably find the bug that's, like, correct. And again, it'll use this component to do a bunch of things and make decisions as it traverses in order to reason to give the agent the exact answer that it needs.

93. 16:38

    So this, this tool, like, generates this for you. You just point it at your code repo. It'll do a construction. Um, but it do-- it does things like creates experts across, like, various areas in the business, so in our libs, our services.

94. 16:50

    I'm quick, quick scrolling. Apologies. We got time. You can check a heat map grid of who works with each other, like who reviews what, who authors what. You can check peer tables, you know, who Andre works with, [laughs] et cetera.

95. 17:01

    So this data available in a context engine, very useful. Uh, and again, this will be open source. Uh, if-- I think you got a badge scan, I'll, I'll just email you all as soon as it's open source if that's cool.

96. 17:12

    Another quick demo. Ghosty, my boy. So in this one, I actually used our MCP, and I just straight up said, "How do I con-- make a new first class integration to Zendesk?"

97. 17:23

    I just said use the MCP. It would probably have picked it up, but I wanna make sure for this demo that it did. It ran-- it just chose to use our research task tool.

98. 17:31

    You can see that it constructed a query, so the agent did that based on the shape of our MCP, so it wrote the right query, ran that. It did effort high for reasoning.

99. 17:39

    It got the data back, then it triggered its explore agents. This is key 'cause now they're exploring the right place after this research packet came in. Great research results.

100. 17:47

     Did the thing, did the thing, wrote me a plan. So if you look at this plan, you do not know my source. That's fair. But if we just scan it, like, it found all the things that matter, like registering our provider, obviously.

101. 17:59

     We have a factory pattern. Um, I'll just pull through, but, like, the library modules, client. Like, this is, like, one hell of a plan is the short story, and it's, like, pretty correct.

102. 18:08

     I would probably prompt this a couple more times to get it totally right, but at any time while it's executing, it's able to keep calling our MCP. So typically what we see is use the engine for planning, run execution, and then basically as you get to code review, leverage your engine again, 'cause that engine is very good

103. 18:25

     at code review, and it's extremely good at planning.

104. 18:29

     That's all the time I have, so thank you all very much. I appreciate it. Uh, I'm at a booth at G sixteen, so come by if you have questions. [audience applauding] [upbeat music]