Mergeable by default: Building the context engine to save time and tokens

Talk transcript

1. 0:00

   [upbeat music] All right. Thanks everyone.

2. 0:15

   Sorry about the wait. Um, [chuckles] this is gonna be a, a, a bit of a strange session because, um, there is a workshop component to this, so, uh, I guess everyone will be coding on their laps. [chuckles]

3. 0:27

   Sorry. Um, but anyways, sorry, I'm Peter, and, uh, this is my colleague, Brandon.

4. 0:32

   Hey.

5. 0:33

   Um, so we're, we're gonna break this, this session into two different parts. One is a, um, sort of a talk that I'm gonna give about what context engines are, are useful for and, and how you might go about building one, what to think about.

6. 0:46

   Um, and then we'll, we'll launch into the, uh, the second part of it. So, um, just briefly, um, quick, quick agenda. We're gonna talk about three myths that are circulating, uh, right now about, about context engines.

7. 1:01

   And then w- I'll, I'll go over a couple of less- or a few lessons, uh, that we learned along the way building one of these things. Um, and then finally, we'll do this.

8. 1:11

   So we're gonna build a social engineering graph. Uh, this is a component that is super useful in a context engine. And first, just a show of hands, does everyone know what I mean by context engine, or does anyone want clarification on that?

9. 1:27

   Yes, please.

10. 1:27

    Okay. So, uh, in the world of AI agents, uh, you have agents that, uh, when you start off and you start coding, they are basically at ground zero. They have no context about your code, your organization, nothing.

11. 1:43

    Okay? So typically, what happens is, the first thing they do is they start to rip around your code base, uh, based on the task that you give them to try to gain some understanding, uh, sort of background understanding before they start to do their task.

12. 1:57

    So context engineering is, is kind of the art of supplying, uh, all the context that you need, and most importantly, none of the context that you don't need, in a highly optimized way, so that when the agent starts to run, it executes the task, uh, in a streamlined way that's in line with your organization's best practices and

13. 2:19

    expectations and so on. Okay. So we'll get to this after.

14. 2:26

    So not long ago, [chuckles] as in, like, four years ago or less, uh, you were the context engine. Okay? So when, when your agent needed something, um, you would prompt it, you'd grab the, the issue ticket, you'd hand it all of the information that it needed to start its task.

15. 2:45

    And in many cases, even when it was ripping around getting background context, when it got to the end of its task, sometimes it, it got it wrong. In fact, many times it did, and you'd have to kind of, like, reset it, um, re-guide it towards the solution that you were thinking of.

16. 3:01

    Um, or if it completely missed the mark, you'd have to be like, "No, not the, not the JavaScript, dummy. It's the Python source code that I want you to look at."

17. 3:09

    Um, so let's just remember how you built context in an organization. Uh, so we're, we're taking AI out of the picture for it, for, for just a sec. I just want you to pretend that pre-AI, uh, you just joined an organization.

18. 3:27

    Let's remember how we built it up. So over time, you would accumulate this kind of context through experience, right? You would start a job, maybe start code spelunking a little bit to figure out, um,

19. 3:41

    uh, how things work. You'd maybe latch on to a mentor. Um, and eventually you'd, you'd experience real things like incidents and outages and things like that. Those are sort of the, the pain things that stick with you.

20. 3:56

    Those are the battle scars, right? And that is what constitutes organizational context. It's the, um, it's the learnings along the way, the why did we do things the way we did it.

21. 4:13

    And now you're good at your job because, uh, after all of that experience of pain, now you know what questions to ask. You know where to look when an incident happens.

22. 4:24

    And this is the goal. This is what we want to get for our AI agents.

23. 4:31

    So, um, [clears throat] I'm gonna just lift this adoption curve from Bassem Eldef, and, uh, I, I may have butchered his last name, but sorry, sorry, uh, Bassem, if you see this.

24. 4:42

    Um, so let's, let's start at the beginning here. This was, like, four years ago in twenty twenty-two. Everyone, everyone remem- remembers fancy autocomplete, right? Um, so back in those days, context windows in AI were pretty limited.

25. 4:59

    I'm not sure if everyone even remembers this, but it was, like, eight kilobytes or s- or eight K tokens, I should say. And that's not a ton. And so tokens were highly optimized and, um, agentic IDEs like Cursor focused just on the code that surrounded, uh, the code that you wanted to go in and autocomplete.

26. 5:19

    So basically, they took some code before, they took some code after, they put it into a model, and they said, "This user is working on this piece of code.

27. 5:26

    What's the most likely next thing?" And that's what was printed out. Um, it got progressively better than that. ASTs were, uh, were integrated, uh, language servers, and then it...

28. 5:38

    You, you were able to basically pull, like, callers of source code and pull all that into context. And then the LLMs were really good at, at completing, uh, code.

29. 5:47

    Um, so at those levels, you were the context engine, and, uh, in, in many, in many circumstances here, this is kind of where most people are here. They're at the, uh- Uh, parallel agents hooked up with MCP and skills, okay?

30. 6:08

    Just super curious, has anyone gone beyond curated context into the, the last, uh, few degrees of, of agentic freedom, shall we say, where you have background agents running in the cloud doing stuff in YOLO mode?

31. 6:25

    Has anyone, anyone experimenting with that? Okay, cool. That's, that's very cool. That's bleeding edge. Um, but let's just take a moment to recognize [chuckles] that bleeding edge today is, like, yesterday's news in six months, okay?

32. 6:40

    So the, the puck-- I'm [REDACTED:origin], so I'm gonna say this. The puck is going down, down the line towards background agents for sure. Um, and one of the things that we run into right now is this, uh, w- we're becoming the bottleneck as humans, right?

33. 6:56

    I'm not sure if, if people have tried managing parallel agents and, uh, working on several tasks at once, but everyone's starting to feel this, like, cognitive disconnect 'cause you're context switching all the time, and it's just g- it's just really, really painful.

34. 7:09

    Um, it is very difficult to move from that mode, where you're the human managing context, into the background agents mode unless you have some kind of context engine that knows how your code operates, how your organization works, and understands the motivations for historical changes and things like that.

35. 7:31

    So Andre, he nailed it. Um, systems are intelligent. We're reaching the exponential on, on intelligence for code pretty soon. Everyone's seen the, the release about Mythos, um, even though we all haven't had a chance to really try it out yet.

36. 7:48

    Uh, the promise is that from a code intelligence perspective, this thing is, like, pretty much close to, to perfect. Um, but so now the bottleneck is context, of course.

37. 8:00

    Without, um, without context, I'm just gonna reemphasize this point, you'll probably end up in doom loops. Does everyone know what a doom loop is? [chuckles] A doom loop is, like, when you're, uh, you're struggling with the agent.

38. 8:13

    It's, it's not quite doing what you want, and you have to keep iterating on it. The worst-case scenario is you run this thing in YOLO mode and it, it finishes the entire task and it's completely wrong, and you have to go back and correct, you know, various stages.

39. 8:27

    Um, so when you have a context engine, you can get there faster. The problem [clears throat] is that access doesn't equal understanding. So w- we have customers that are on various parts of-- I'm just gonna go back to here.

40. 8:41

    We have customers that are on various parts of this journey.

41. 8:44

    Um, and one of the, one of the interesting things that we've noted is that, uh, people feel that, you know, they're, they're-- they understand their organization best. So when it comes to feeding the right context to these agents, people will try to build some semblance of what a context engine actually is.

42. 9:04

    They'll maybe build a RAG system or they'll build some way to, like, feed organizational data to an agent. Um, so unfortunately though, uh, access doesn't mean understanding. So what that means is you could just wire up a bunch of MCP servers, um, and it's not gonna be able to understand what the relationships are, uh, between all that

43. 9:26

    data, how it was-- how it got there, and how-- why it is the way it is. Um, and then there's another problem, which I'll talk about a, a little bit later called satisfaction of search.

44. 9:36

    So just remember that term. I'll come back to it. Um, okay.

45. 9:43

    So I just wanted to show you this. Um, this was something that, uh, we actually implemented, and we did it in, in two parts. One was just, uh, without any context engine but wired up to a bunch of MCP servers.

46. 9:57

    It did a pretty good job, um, but then when we reached the end, um, it, it, it missed the fact that we had some legacy stuff that depended on this old, um, method of, um, of intelligence size to, to Anthropic.

47. 10:13

    So they have adaptive thinking now, but it used to be you had to supply a, uh, a token budget, and that's how, like, you could increase the size of the thinking window.

48. 10:23

    Um, so we, we had some code that kind of, like, depended on this, and there were reasons for that, um, that the agent didn't understand or see, and so it just basically cloggered all that code.

49. 10:34

    Um, but when we added the context engine, then it saw all those reasons and implemented it the right way. So it made the appropriate changes in the right places, included backwards compatibility for the code that was using the old method.

50. 10:48

    Okay, so now for the myths. Myth one: Naive RAG over my docs is a context engine. Um, so if you implement, um, say, like, vector search, um, or just a couple of search methods, uh, you're gonna run into this-- you're gonna run into a few issues.

51. 11:08

    One is the satisfaction of search problem, where, uh, the agent will search like crazy, consume your tokens, and then in the worst case, you'll reach compaction, okay? So, um, without being able to find the, the end game.

52. 11:23

    Um, there are a few different other techniques. Like, you, you need to have personalization when you build a retrieval system because if you just RAG all your data, especially for very large organizations, there's gonna be things like conflicts that you have to resolve in the data.

53. 11:39

    Um, it won't be focused on the task that you're trying to perform. It might pull in, you know, relevant code from other parts of your organization that, especially if, uh, if you have a really big org and you've got tons of different repos, um, it's, it's just gonna create a huge mess.

54. 11:55

    So you need to have some element of personalization.

55. 11:59

    And then here again, uh, connect a bunch of MCPs. I'm just gonna reiterate this point. I'm done. No, definitely not. Um, so that, that is the thing that h- that really, um, puts an emphasis on the satisfaction of search point, and I'll explain that in a sec.

56. 12:15

    And finally, a bigger context window will solve this. Um, so way back, you know, when the models were starting to get big, people were really excited about a million tokens in your context window.

57. 12:29

    Uh, the first models that tried this, I think it might have been Claude, actually. Was it Claude? I think it was Codex. Oh.

58. 12:36

    Oh. Or Gemini. Or OpenAI? Okay. Okay. Gemini. Gemini. Yes. Sorry. That's right. I'm so sorry. Um, so yeah, Gemini first pr- first model to try this, and it was really good at finding needle in a haystack.

59. 12:46

    So you could feed, like, a huge document to it, and as long as you, you knew what you were looking for ahead of time, it could find it. Um, but it wasn't good at all at reasoning across different data sources, um, understanding the real meaning behind a problem, and then recommending the appropriate solution.

60. 13:05

    So none of that was possible. Um, obviously things have, things have gotten much better. Now, the problem is, most organizations have more than a million tokens worth of context, so trying to fit all that into their context window isn't gonna work anyways.

61. 13:20

    Let's project out to the future and imagine that you could fit, like, ten million tokens, fifty million tokens. Um, at the current, uh, rate of memory consumption, um, just to operate the models, that's not gonna be possible for a really long time.

62. 13:34

    Even if it was, and you fit all that context into your context window, you're still gonna run into problems with understanding what's true, what's false,

63. 13:44

    um, how to select the right information. Okay? So now I'm gonna come back to this second point here, satisfaction of search. This is a, a term that actually comes out of, uh, the medical field in radiology.

64. 13:58

    And the idea is that, um, when techs are looking at X-rays, uh, and they're looking for, uh, the cause of, of, of symptoms, they might find something on the X-ray that explains those symptoms, and then they stop.

65. 14:13

    Um, and i- that's, that's kind of like a dangerous thing medically because, uh, there might be other indicators for things like cancer that get missed.

66. 14:24

    So, uh, satisfaction of search is a, is a real problem in radiology, and there's lots of protocols to prevent just stopping as soon as you find the first thing.

67. 14:32

    Um, this is what happens with agents. When they search around in, say, uh, Notion, in your code, uh, Confluence, they'll stumble across what looks like the, the thing they're looking for, and they'll stop.

68. 14:46

    And then they'll, they'll proceed. But the, the real, like, golden nuggets of information might be in a different place that the agent wouldn't think to look, like in a, in a past Slack conversation or in an incident report, something like that.

69. 15:01

    So here's the, the classic iceberg meme. [chuckles] Um,

70. 15:05

    code that compiles, that's like the baseline. Does the agent produce code that compiles? Um, but everything that, that is actually important is-- happens underneath here. So understanding the user's original intent, um, what was rejected in the past by the team and tried before but failed.

71. 15:23

    Uh, how are you gonna surface that kind of content just by looking at docs and, and code and stuff? Um, so you need to understand that somehow.

72. 15:34

    Um, and e- even worse, like it, it's, it's sometimes hard to know, uh, when things were deleted, like in the absence of information. So you need history as well leading up to decisions.

73. 15:48

    So this is why we think you need a context engine. Uh, a context engine understands who you are, what team you work on, who you work with, who the experts are in your organization, um, and, and what the decisions were that led up to the current iteration of your code base.

74. 16:09

    It's able to resolve conflicts. Uh, so this is like a truth and false type situation, what's true, what's not. Um, sometimes that truthiness is a gray area, right? So the context engine needs to also understand when to instruct the agent, um, that it wasn't able to resolve the conflicts and then, uh, learn from additional user input.

75. 16:34

    Um, this third point is super important, of course, in any large organization or enterprise. Um, there's often, you know, repositories that not everybody can access, secret projects, that sort of thing.

76. 16:46

    So, uh, it's really important that you flow the access controls up. We have-- I'll give you a, a, an example that everyone will appreciate, which is Slack. Um, our context engine integrates with Slack or Microsoft Teams.

77. 16:59

    Um, and when you have, uh, private channels, that, that's really highly sensitive, right? Like, you could be discussing HR information or, uh, maybe something that you just really don't want, um, everyone else to see.

78. 17:12

    And so when, when Unblocked answers questions, uh, it will use private channel information, but it won't-- it, it will only use that information if the person that's asking the question has access to it.

79. 17:23

    And then those answers are not public. Okay? So they're, they're, they're private to you.

80. 17:30

    Um, and then finally, of course, delivering the right context at the right time, and this is about token efficiency. It's about getting to the answer as quickly as possible.

81. 17:41

    So here's a kind of a high-level overview of how, how a context engine might work. Um, on the left we've got data source inputs, so things like planning tools, docs, conversations, code PRs, basically like anything that's relevant to getting work done at the engineering level.

82. 17:58

    Um, and then on the right side we have the outputs. So, you know, th- this all can flow to coding agents, MCP, or CLI tools. Um, you can custom build apps through the API.

83. 18:12

    We've got a-- We have a code review, uh, component that just plugs right into your SCM and provides code reviews and, of course, uh, integrations with social messaging apps.

84. 18:24

    So the-these are the kind of, like, broad six requirements that we think are important. There's actually much more than this, but these are the high-level things. So again, unified system context.

85. 18:37

    Um, this is about building relationships between data. Okay? But it's more than just, um, recognizing when, uh, one piece of data is related to another. Like, for example, in, in Slack, you might have conversations about PRs.

86. 18:54

    That's an easy linkage because you're posting links back and forth, so that's easy. Um, what's less easy is understanding, uh, the reason why decisions were made or your organization's best practices, right?

87. 19:08

    So to understand that, you have to go a little deeper, um, do, do things like distill, um, historical pull request comments on PRs and, uh, try to distill those down to the-their core essence.

88. 19:22

    And then when you see repeated patterns, uh, you can pull those patterns together and store them as, you know, quote-unquote, memories, so that when, uh, someone is working on a similar piece of code, you can load those memories, and then the agent can see that and go, "Oh, yeah.

89. 19:38

    Right. Uh, this is the way this organization does this particular thing."

90. 19:43

    Um, [clears throat] conflict resolution. Super important. Um, we took a, uh, initially a kind of a naive approach to this at first and based it just on recency, right? So we, we would bias towards newer stuff.

91. 19:59

    Uh, unfortunately, when you have in the fullness of all your context, recency is not enough. Um, often you have people, um, writing documents or chatting in, in s- in their messaging platforms, and they might be saying things that are not, like, completely aligned with, uh, how the, the system works.

92. 20:22

    Um, [tsks] uh, so, you know, then we started to bias towards code. So we had recency, and we're like, the main branch is definitely your source of truth, but not always.

93. 20:32

    Because sometimes, um, what's important is what happens next, not the way a system currently works. Like, when you, when you're working on a task, um, what you really want is for the agent to understand where you're going, not necessarily where you've been.

94. 20:49

    Where you've been helps it understand what not to do. Where you're going helps it understand what you should do. Okay? So in, in the Slack case, looking at the conversations that your organization's experts are having is more important than just understanding what, you know, every random engineer is talking about.

95. 21:10

    Um, targeted retrieval and personal relevance are very related, so I'll just talk about them, uh, together briefly. So, um, again, like, when you're pulling context in, it's important that, uh, you're only pulling context in for the relevant task at hand and probably relevant to you.

96. 21:31

    So here's a technique that's kind of interesting. Um, you can understand what repos a person works on most by the number of PRs they submit and contributions. And then if you do a-- if you're doing vector retrieval, you can do a, uh, deep retrieval on those focused repositories and then a wider retrieval on, you know, the rest

97. 21:52

    of the source code and then sort of bias the, the selection towards, uh, the focused repositories 'cause that's more likely where someone's going to be working and spending their time.

98. 22:04

    Um, and then, you know, we've talked about data governance, so I don't think I need to go over that again. Super important, though. [coughs]

99. 22:13

    This was just a, a, a little experiment that we ran, uh, with a larger task. Um, I'm-- I fully admit that some of these numbers are a bit wonky.

100. 22:22

     This is basically like Claude outputting numbers, so don't, don't trust it. Just trust the, the vibe of the thing and not necessarily the numbers. Um, basically, what it's saying is that when we started out, uh, without the MCP server ac- or sorry, without the context engine active, um, it, it really missed the mark on a lot of

101. 22:41

     stuff. Uh, and that's just because it didn't understand how, um, the existing implementation really worked and why it was the way it was, what was tried before and failed.

102. 22:53

     Um, and so it made a lot of those same mistakes. Uh, with the context engine turn-turned on, obviously, it, it, um, it nailed it. The, the key numbers, though, are the, the time and the tokens that it took.

103. 23:05

     So without, um, the context engine, it took two and a half hours to finish this task with twenty-one million tokens, which is a lot of tokens. Um, but with the context engine, it took only twenty-five minutes and ten million tokens.

104. 23:20

     So it's, it's a pretty dramatic difference. Um, okay, so the hard lessons. These are just samples, by the way, but the-these are ones that we thought were kind of interesting.

105. 23:31

     So first of all, uh, initially, we optimized for access, not understanding. So w- our, our first premise was if we just wire up a bunch of tools,

106. 23:41

     um, and provide a, a knowledge graph, it will be able to traverse the knowledge graph and, uh, execute a bunch of retrieval-specific tools for particular integrations and so on and figure everything out.

107. 23:55

     Um, that does not work. So, uh, you'll have to go a little bit deeper than that. [chuckles]

108. 24:04

     Uh, second one is we hid conflicts instead of surfacing them. So, um, by con- by hiding conflicts, I don't mean that we just ignored the conflicts. What we did instead was we tried to resolve those conflicts using those naive strategies.

109. 24:20

     And we didn't surface the conflicts that we weren't able to resolve. So this was a, a really good learning, is that, um, a context engine, I mean, we'll get there eventually probably, but, uh, it can't always tell, uh, what the truth elements are.

110. 24:35

     And when it can't, you should surface that and learn from it. That's the key thing.

111. 24:42

     And then finally, uh, [laughs] I think a, a, a lot of folks try this. It-- this is a really bad idea. So when, when a context engine supplies an answer, um, do not cache the answer and try to serve that same answer up again, uh, to a similar question.

112. 24:57

     The reason is obv-- is, is fairly obvious in retrospect, but, um, s- everything changes constantly, right? Code changes, docs change, the reason for things change. So this just doesn't work.

113. 25:09

     Um, the other thing is, if you try to, uh, use the, the previous answers as context for new answers, you regress towards a mean. So if the model is, like, misbehaving or doing something bad, and you continuously bring that into context, you're obviously gonna pollute, uh, the context.

114. 25:32

     And this is what happens. Okay. So let's now talk about where AI forward teams like, like those that are doing this, like, cloud-based agent thing are, are using and taking advantage of context engines.

115. 25:47

     Um, definitely and especially during the planning phase, okay? This is where you get the biggest bang for buck unquestionably. Um, get the context engine involved, use a skill to bring it in, um, connect it to the MCP server, and, and watch it do its thing.

116. 26:04

     It-- this is where you get the biggest bang for buck. It's also useful to do this during review. So you get planning and review at the end. Um, because, you know, if, if you get an agent to do review, it's basically just gonna pay attention to the code and try to understand where the break points are, um,

117. 26:23

     security concerns, that kind of thing. But without the organizational context, it doesn't understand the motivation for it. So that's the really important thing.

118. 26:32

     Ticket enrichment. Um, this is a, a super cool use case. So you create a ticket for a new feature, and then you just ask the agent that's connected to a context engine to fill in the blanks.

119. 26:44

     Works. Triage. Uh, I use this all the time. When I see an issue in production, I just whack it into, and my agent connects to the context engine, and it just, like, [snaps finger] instantly brings up all the past issues related to this and, um, starts operating right away.

120. 27:04

     Increasingly, we're seeing this one, incident management. Okay? So we, we just, uh, wired up, uh, Datadog. And this-- Sorry?

121. 27:13

     Sentry.

122. 27:13

     Sentry and Datadog, sorry. Um, and this is already proving, like, super cool use case. It, uh, it, it can see the signals, and then it can act on all the signals and relate that to code, uh, relate it to past incidents that you...

123. 27:29

     and, and discussions that you've had in Slack. Having all those things come together at once is, is almost, like, magical. And finally, I think this one's actually my favorite one, and it's the one that customers use the most, is, uh, customer success and sales and engineering support.

124. 27:45

     So what, what a lot of big teams do is they have engineering support channels where other teams can come in and ask questions. If you put a context engine into one of these things, you can have it automatically answer a lot of questions and save engineers a ton of time.

125. 28:03

     All right. So how teams make a context engine their own. Skills. So definitely build, uh, skills that you can use to curate context in a GitHub repo. [clears throat]

126. 28:14

     Um, and it-- you can build other skills around it like typing ticket enrich, give it the issue ID, and then it n- uh, it, it can use the context engine to build the enrichment.

127. 28:28

     Uh, workflows like this one, prepare, prepare an incident timeline, um, and then you can just send it off to your agent again, context engine, blah, blah, blah. It brings everything together.

128. 28:37

     Magical. And this thing here, um, you can wire this up to all kinds of agents. I've got, um, uh... One, one of the things that a lot of customers like to do is wire this up to Claude Code in their CI system.

129. 28:51

     Um, we actually do have a code review component, so you don't have to do this if you're using Unblocked. Um, but people use this for other things, not just code review.

130. 29:00

     As soon as you wire up a context engine in the background, give it an API key, let it, let it run on its own, it, it can do some pretty insane stuff.

131. 29:09

     Um, so I'm just gonna show a quick,

132. 29:12

     um, example of what wiring up a context engine can do. So this is a PR that, uh, my colleague wrote, and, uh, it, it-- Unblocked, like, went through and provided a, a kind of review to this thing.

133. 29:29

     And at the bottom of this review, here's the review part, um, you can see that Richie, who was the author of this PR, was like, "Very cool. This is something I would say."

134. 29:40

     Uh, now the reason for the comment, which was you've basically duplicated a bunch of tests, you can, you can kind of dry that up a little bit, is because, um, this was a best practice that was distilled from a bunch of other PRs.

135. 29:54

     And the, the funny part is that the author of those PRs was Richie. [laughs] So he's the one that actually instilled the best practice in the organization. Uh, so that was, that was just a cool little moment when we discovered that.

136. 30:08

     Um, here's another example. So this was a-- It's a fairly long transcript. I'm not gonna, like, show the whole thing. But we, we sent it on a, on a mission to do a big, large task.

137. 30:19

     Without, uh, Unblocked, it, it, uh, took quite a while. Like, you can see the transcript's quite long. Um, and it, it missed a whole bunch of stuff. With Unblocked, uh, it was a lot more compact.

138. 30:32

     It, it got to the answer, like, very quickly and correctly. And just because we're now AI forward and lazy, um, [chuckles] we took both of those transcripts and ran them into Claude and just said, "Hey, Claude, why don't you just do a- an analysis of both these things and, and give us your, give us your result?"

139. 30:50

     Um, so it, it went through-- I won't, you know, bore you with the details, but just to say that at the end, the verdict is that, uh, the context engine plan is, is what I'd ship with.

140. 31:01

     This other one is good for a prototype, but it's missing a whole bunch of stuff that is important to this organization that was previously discussed. Okay?

141. 31:14

     So this is essentially what, what I've been trying to say. Uh, AI-generated code should just feel like it was written by someone that's been in your team for, like, 20 years, okay?

142. 31:26

     Um, it doesn't-- If it doesn't yet, that's fine. Uh, it will. Um, your-- If you wire up Unblocked, you'll, you'll see, like, a, a huge difference in performance of agents.

143. 31:38

     And if you're building one of these things, the- absolutely. Like, take all these things and, and build and, and let's see where that goes.

144. 31:47

     So just before we get into the workshop component, um, maybe we'll just have, like, five, 10 minutes of Q&A.

145. 31:57

     What up? I'm Brandon.

146. 31:58

     And this is Brandon. So he'll, he'll help with Q-

147. 32:00

     I, I swear I'm on this. [laughs] Well, not maybe. Is that recording?

148. 32:03

     Yes.

149. 32:04

     Oh, thank you. Sorry.

150. 32:05

     Okay. You're fine.

151. 32:05

     Thanks. Um, so it's clear what it does for you and what kind of problems it solves. But to me, a big question mark is: What is the thing? What is the artifact that, uh, uh, that fits the bill?

152. 32:17

     Is it, like, a program you install, an API that's hosted remotely or an MCP server? What is it?

153. 32:25

     It's, it's all of those things. So a, a context engine-- I, I'll explain what Unblocked is. Maybe I can just show a quick demo of it. Um, so

154. 32:34

     broadly speaking, there's a bunch of different surfaces to a context engine. You wanna get it into your agent flow, and you can do that with an MCP server. You can do that with a CLI tool, for example.

155. 32:47

     Um, we also have this dashboard surface where you can ask questions about your code. Um, this is a pretty basic one, but you can see it understands who I am and what I've been working on.

156. 32:58

     Um, and then, uh, we have a Slack, we have Slack connectivity as well. So you can bring Unblocked into Slack, um, drive it in conversations, and have it auto-answer things.

157. 33:10

     Does that make sense? Did that answer your question or-

158. 33:12

     Um, I think my question will get answered in the demo or in the-

159. 33:18

     Okay.

160. 33:18

     Yeah.

161. 33:19

     But yes, API, CLI, MCP-

162. 33:21

     Yeah

163. 33:21

     ... and products.

164. 33:23

     Mostly API.

165. 33:24

     Yeah.

166. 33:25

     Oh, sorry. I got to carry the mic around. I forgot. [laughs]

167. 33:28

     It's okay.

168. 33:29

     Thank you. Um, so my question is-- So as, as far as I understand, it's like a knowledge management and retrieval, um, application.

169. 33:39

     Yeah.

170. 33:39

     And does this relate somehow to things like, um, LLM Wiki, like it was made popular recently by Andrej Karpathy, or the decision traces, uh, and context graphs-

171. 33:53

     Yeah

172. 33:53

     ... which was discussed a lot, uh, a few months ago?

173. 33:56

     Yeah. So you can think of all of those things as kind of, uh, useful components to a context engine. A context engine has to do much more than that because, um...

174. 34:07

     So agents are really good at recursing through a wiki, for example. It depends on how you build this wiki 'cause there's a bunch of things like organizational memories, best practices, you know, experts in your organization and-- that are used as pivot points for context retrieval.

175. 34:24

     So a, a wiki doesn't solve those problems unless it has, like, a, you know-- Y- you could build a structure with it. And I think, uh, Karpathy discovered that if you treat a wiki as, um, kind of like a file system, you can break it down and have the agent, uh, whack through it like a file system.

176. 34:42

     They're, they're-- By the way, agents are, like, highly optimized for file system traversal.

177. 34:47

     The compilation step.

178. 34:48

     Yeah, the compilation step. Exactly. Yeah.

179. 34:51

     Yeah. Sorry. Everyone else is ready, but-

180. 34:54

     Okay.

181. 34:54

     Sorry. Uh, uh, maybe the same question, but is it, is it a, a general purpose, uh, context engine-

182. 34:59

     Yes

183. 34:59

     ... or is it targeted against, uh, code? Because will it be useful as a, say, uh, as a business domain expert, uh, or sort of building up a business domain and then having this context engine use, uh, my-- So I could-- All my other AI agents could use this as context for their business.

184. 35:20

     Uh, or would you say that is more like f- just for the code part of it?

185. 35:24

     Uh, so it, it's definitely engineering-focused. The, the integrations are focused on engineering activities, so, you know, SCM integrations and other, and, and other tools that engineers use. Um, we are increasingly seeing customers using it for other purposes.

186. 35:42

     So business intelligence is a key thing. Uh, and that's usually useful when, uh, people in, in business functions are trying to get an understanding of the product and its function.

187. 35:54

     Um, we don't have, uh, like, say, Salesforce integrations wired up for that. So you couldn't use it to understand, um, you know, any- anything that's sales related. It's, it's really just primarily an engineering-focused context engine.

188. 36:09

     That's not to say that that won't change.

189. 36:11

     To that. [laughs] Uh, more questions?

190. 36:17

     Yeah.

191. 36:19

     On the governance thing-

192. 36:20

     Yeah

193. 36:20

     ... if you're, um, respecting- Access rights, how can it do sort of synthesis across stuff and then develop new knowledge inter- in- internally that it could then surface to people?

194. 36:33

     So the... Y- yes, you're correct to point that out. The, the synthesis, um, is compartmentalized, so there are, you know, places that are compartmentalized, like r- individual repositories. That's kind of the level of access.

195. 36:46

     So if you can synthesize, uh, historical data based off of that, um, and then correlate that with public Slack information, then that's, that's one way to do synthesis without crossing the, the organizational boundaries.

196. 37:00

     Um, so, uh, the o- you know, the other way is to look at and tag when, um,

197. 37:09

     i- synthesized information crosses those organizational boundaries, and you can take something like a group ID approach to that problem, where you attach group ID tags to the synthesized information and then only retrieve it if the person that, uh, has access to that can, can build it out.

198. 37:27

     So first take the compartmentalized approach because that's the-- where you'll get the, the most mileage, and then you kind of build up from there. I mean, this is the core problem with using a technology like GraphRAG, right?

199. 37:39

     Because GraphRAG is like a pyramid where it builds up in layers and then basically summarizes at each layer. But that, uh, like unavoidably crosses, uh, permissions boundaries. So you have to be...

200. 37:52

     you have to create compartmentalized pockets. Yeah.

201. 37:58

     More questions.

202. 37:58

     That's a good question.

203. 38:01

     Time to hack after this, I think.

204. 38:03

     Yeah.

205. 38:04

     Uh, yeah. So you've talked a lot about like all the different sources of information that you consume and putting them all together. When it's like synthesizing those down, is that still sort of like naive RAG vector search, all of that stuff under the hood?

206. 38:18

     Or is it like agents deciding what is appropriate them-- like what, what... Or probably like combinations of all of them, but what is that sort of step?

207. 38:26

     Um, yeah, you're right. It is a combination of all of them. So knowledge graph, like knowledge graph buildup happens in a bunch of different ways.

208. 38:34

     Yeah.

209. 38:34

     Um, the, the PR thing that I showed you, for example, is like, uh, uh, first you build a, a naive knowledge graph procedurally, and then from there you can use an LLM to distill down and summarize and build up, um, those types of techniques.

210. 38:49

     Um, our context engine builds first like a knowledge graph from the base, uh, using-- trying to leverage like all the different entities. It's kind of like a PageRank thing where it builds up the relationships procedurally.

211. 39:04

     And then, of course, it vectorizes data. Um, and then there are procedural tools that fetch data at runtime. Um, a lot of the distillation for, uh, you know, conflict resolution happens in two places.

212. 39:19

     So one is like during data ingestion time, there's, there are tags that relate data to each other so that we can see if we can deconflict at that level and then like rank against each other at that level.

213. 39:31

     And then, of course, at runtime, you have to pass the thing to a judge with the criteria, um, and then it does additional deconfliction in, in real time.

214. 39:41

     Does that make sense?

215. 39:42

     Yeah.

216. 39:42

     Okay.

217. 39:43

     One more question and then we'll-

218. 39:45

     So, uh, I was curious, you said conflicts, but at some point you get conflicts that something means revenue for one company and means revenue for another company is in a totally different meaning, how you can recognize that.

219. 39:56

     So how do you get humans in the loop? How, how do you use their ontologies, and how can you-- uh, do you use it when you run into it?

220. 40:04

     So I'm very curious about that, actually, how, how you use it in the loop.

221. 40:07

     Yeah. So if you-- I can show you just a quick thing here. So, um, you'll notice that at the bottom, the, the references that were used for answers are delivered both like to the human in this interface, but also to the agent.

222. 40:20

     Yeah.

223. 40:20

     So, um, if the agent-- if the context engine isn't able to do the deconfliction, then at this point here, the human can step in and guide the agent when there are, uh, enough si-

224. 40:33

     How does it do that then? So you-

225. 40:35

     So you can, you can literally just reply and say like, "That's not correct," or you can come here-

226. 40:39

     Oh, okay. Yeah

227. 40:40

     ... and-

228. 40:41

     I'm just seeing.

229. 40:41

     Oh, yeah. Sorry, the cursor went down.

230. 40:42

     Yeah. Or you can, y- you can do this, like not helpful and, and give the reason why. Um, like it, it is a bit of a manual process at this stage, but the signals that build up over time-

231. 40:53

     It, it's funny, right? So you use it for production, but some companies you might have-- you catch a lot of human intelligence by this, right?

232. 41:00

     Yeah.

233. 41:01

     That's amazing. Uh, yeah. Okay.

234. 41:02

     It's... Yeah.

235. 41:03

     For, for, for a typical customer, how much-- do you have this volume of data, the metric of the capture?

236. 41:08

     Oh, it's huge. It's, it's, it's amazing. Like I, I was actually really surprised by how willing people are to give feedback. Um, yeah, no, it's-

237. 41:17

     Can you give an example, one project that you catch the thousands or hundreds of, of, of these human feedback things?

238. 41:25

     I mean, at, at small team size, it's, you know, in the hundreds at... So small team size being like twenty, thirty people.

239. 41:32

     Yeah, yeah.

240. 41:32

     At large team size, a, a hundred to two hundred people, it's like hundreds and hundreds of-

241. 41:37

     Oh, wow

242. 41:38

     ... of feedback. Yeah.

243. 41:39

     That's cool.

244. 41:40

     People just really like to interact with agents and tell them in natural language what's wrong. It's, it's just a totally natural thing to do. Yeah.

245. 41:49

     Good for us.

246. 41:51

     Cool. All right. Um, are we, are we good for Q&A?

247. 41:57

     And then we can get on-

248. 41:57

     You can always ask us questions as we hack, but [laughs]

249. 41:59

     Yeah. Let's, let's get onto the, let's get onto the workshop part of this. So, um, we have created a, um... Actually, what I'll do is I'll just do this first.

250. 42:10

     So you can do this now if you'd like. Um, I will come back to this slide in a sec. So the idea here is we're gonna get everyone to join a Slack workspace that we created, and then we're gonna get, uh, everyone into a repo where this, um, where this sample code lives, and then we'll just start

251. 42:27

     hacking away on it together. Okay?

252. 42:34

     Hopefully this is working. Declare endurance.

253. 42:37

     Yeah.

254. 42:38

     Light mode. Thank you. The one time I need it, I'm like...

255. 42:48

     Hopefully that's working.

256. 42:50

     Yeah.

257. 42:50

     Oh, I got some people coming in here.

258. 42:52

     Nice.

259. 42:52

     When you drop in, you'll see an AI Engineering London channel, hopefully. Hopefully.

260. 42:57

     Ah, there we go.

261. 42:58

     Uh, there's a, there's a link to click there.

262. 43:01

     And please, in this thread, drop your, uh, GitHub URL-

263. 43:05

     The-

264. 43:05

     -name, and I'll invite you

265. 43:07

     ... the Unblocked link will not work until you do step two-

266. 43:10

     Yeah

267. 43:11

     ... to get into the GitHub org.

268. 43:13

     Mine opened the... This might be a network problem.

269. 43:17

     Oh, no.

270. 43:18

     I've got, I've got many people coming in, so I'm really hopeful that [laughs]

271. 43:22

     Is it network? Yeah.

272. 43:24

     It might be network.

273. 43:25

     We will find out.

274. 43:26

     Um, okay. While, while folks are doing that, I'm just gonna show you what we're getting into here. So this is the, uh, GitHub organization. Um, what we're, what we're working on is a social graph builder.

275. 43:37

     So what this is gonna do is look at a source code repository. So you can run this on your own repo. It's not gonna upload anything. It's all local.

276. 43:46

     Um, so that you can see this thing building up against your own organization. Um, and it's going to do a bunch of things. We're gonna get basically a social graph out of it, and I'll show you what that looks like.

277. 43:59

     And we're gonna understand who the experts are and which parts of the code they work on. Um, and then there's gonna be a little, like, interactive visualization thing. So what, what the goal of this exercise is, is to get this thing up and running and start-- just start hacking away on it.

278. 44:14

     So, like, start submitting PRs as soon as, uh, we get this going. Um, so this is what it looks like.

279. 44:25

     This graph here is our organization, Unblocked, and, uh, what you're seeing here is a, a relationship graph that shows who's reviewing whose PRs, um, and who's c- who's getting reviewed, essentially.

280. 44:40

     The, uh... This thing is a distillation of all the different teams within Unblocked. So this is roughly accurate, actually. Well, not roughly. It is pretty accurate. Um, we've got...

281. 44:54

     I, I did this all the way back to the start of '25, 2025. When you run the thing, I'd recommend maybe doing it for a shorter timeline 'cause it will be a little bit slow, uh, if you go all the way back to '25.

282. 45:06

     Could take, like, 15 minutes. Um, but it's es- effectively distilled who the teams are, and, um, y- the only AI step in this is to label the teams. You don't have to run the AI step if you don't want to.

283. 45:19

     It'll just use the, the parts of the code that people work on the most.

284. 45:24

     Um, this tab here will show the experts in the organization and what they work on. So this is just broken down by, uh, project area and path, um, and shows, like, what areas of the code have good coverage.

285. 45:39

     Coverage is defined mostly by whether a, a high contributing organizational expert is present and whether, uh, it's, it's an actively contributed to part of the code.

286. 45:51

     And then finally, uh, we'll have this interactive graph that, um, breaks things down by team area and will show, like, you know, who the major contributors are. I'm over here on the AI team.

287. 46:04

     Um, yeah. So that's it. Let's get everybody in, and we'll start hacking away at this.

288. 46:12

     Are you sure the GitHub, uh, URL-

289. 46:13

     Yes, absolutely.

290. 46:14

     Let's put that back.

291. 46:15

     Yeah.

292. 46:18

     Many of you should have an invite who have put your GitHub already in. So please, please give it a check. GitHub is the worst. Makes me-

293. 46:25

     So the private repo, right?

294. 46:26

     Yeah.

295. 46:27

     Yeah.

296. 46:27

     We will, um... It is an MIT license, so we will be making it public later. But for now we needed it locked down.

297. 46:32

     No, but I can't get you to Slack.

298. 46:35

     Oh, did you... Oh, are you still not invited? That's crazy. All right. What's your @? Just what's your-

299. 46:40

     Nothing

300. 46:40

     ...@ and I'll invite you.

301. 46:41

     K-A-S. K-A-S.

302. 46:42

     K-A-S. [laughs]

303. 46:44

     Okay. Um, M. M.

304. 46:47

     Oh, thank you. But you should have an invite.

305. 46:55

     Oh, yeah. Yeah, of course.

306. 46:57

     Julian Haske.

307. 46:58

     Sorry.

308. 47:01

     Oh, just-

309. 47:02

     Julian?

310. 47:02

     Julian?

311. 47:03

     Yeah.

312. 47:03

     Got it.

313. 47:05

     From stage, uh, S-

314. 47:13

     Oh, there. Uh, yeah. [laughs] Probably faster.

315. 47:21

     I'll show you.

316. 47:21

     I'm coming over.

317. 47:22

     I won't put it in. [laughs]

318. 47:24

     What have you done? [laughs] Let me, let me type this out.

319. 47:28

     The, the name looks weird. [laughs]

320. 47:30

     I thought it was... I, I keep trying to type TV and it's not the right-

321. 47:33

     Yeah. No, it's Guus for... This is Dutch. [laughs]

322. 47:36

     Oh, thank you.

323. 47:39

     I also have a site with dogs. It's my daughter, so I just took it that.

324. 47:43

     It's yours. All right, you got it. I'm at Julian in that list by the way, teams.

325. 47:53

     So if I'm-

326. 47:54

     So I think the, the rest of this session is gonna be now just, like, hacking away. So, um, in a sec here, I think I'll, I'll take this, this down if everyone's got it, um, so that Brian and I can concentrate on working with you guys to build features.

327. 48:09

     It's gonna be fun.

328. 48:24

     Oh, when you submit PRs, by the way, you'll notice that Unblocked is sitting there as a code reviewer, so don't, don't feel badly if it, uh, sprays on your PR a little bit. [laughs]

329. 48:36

     Depends on how much slop you want

330. 49:10

     Abdel, you got one heck of a username. Good work.

331. 49:13

     Slack.

332. 49:21

     Is, is everyone good with this? I take it down? Okay, cool.

333. 49:26

     That is it. Brandon, you're, you're on top of the invites?

334. 49:38

     Yeah.

335. 49:38

     Okay, cool.

336. 49:39

     There's a, there's a few more. I'm, I'm onto, uh, Chris.

337. 49:45

     Chris has given two names, so-

338. 49:47

     Oh, yeah.

339. 49:48

     That's okay. I'm gonna send both. Don't worry. [chuckles]

340. 49:50

     Amazing.

341. 49:50

     Yeah. [chuckles] That's just where I am in this list.

342. 50:41

     Oh, forgot to mention a couple of things here, actually. Yeah.

343. 50:51

     Oops. Coming back live? Yeah. Good. Um, just a couple of things. So if, if, uh, you're looking for something to implement and starting with, with any-- with coming up with ideas and stuff, there is a, uh, a set of sort of predefined issues that you can hack away on, so you can just grab one of these, [chuckles]

344. 51:13

     whack it into Clyde and see how it does when it's connected to the context engine.

345. 51:26

     Um, the MCP server for Unblocked is here, so if you want instructions on how to wire this up to, uh, Claude Code or another agent, then you can grab it from the instructions from here.

346. 51:53

     All right. So I'm at Lars. There's two more in here, so I'm, I'm still going by the way, for those just adding their names.

347. 52:03

     What's

348. 53:21

     going on, Brandon?

349. 53:23

     Oh, sorry. Just one of the usernames is invalid.

350. 53:25

     Oh, okay.

351. 53:29

     I'm sending a new. All right. Cat, you should have an invite. Andre, I'm co- you're coming next.

352. 53:43

     I'm just two, two behind it, looks like.

353. 53:47

     Christopher, did you not get the invite yet?

354. 53:49

     No. No, I didn't, so-

355. 53:50

     Oh, that's weird

356. 53:51

     ... I didn't get it yet.

357. 53:51

     Let me double-check. You should have one, but... [chuckles]

358. 53:55

     So you said is it through Claude?

359. 53:57

     Yeah, I should. You should have an email. Um, I'm up to like [REDACTED:age] of you, [chuckles] so-

360. 54:03

     Hard work

361. 54:04

     ... hopefully you'll get it. It's like five clicks to add a member from like GitHub. We need to talk.

362. 54:09

     There are agents doing that for you.

363. 54:11

     I was gonna say it.

364. 54:12

     You know they're-

365. 54:12

     You should be able to use the CLI for this. What's going on?

366. 54:14

     You don't have the agents for it yet.

367. 54:15

     That's right. My bad. No, they keep putting it in my PR, and I don't want it there. [chuckles] Copilot's gonna review for me.

368. 54:23

     Very poorly, but it will review. Uh, Andre, you're good. Simon Lake. Let me double-check. Yes. Okay, wow.

369. 54:36

     Can I ask a question real quick?

370. 54:43

     Yeah, for sure. Do-- Hold on, let me grab you the mic.

371. 54:50

     Sure. Hopefully that's on.

372. 54:56

     Does it work? Yes. Um, so I guess that context engine works very well for asynchronous agents so that you don't need to specify things on your keyboard because they can fetch what they need.

373. 55:09

     Yeah.

374. 55:09

     That's one of the main uses, I, I guess. And, um, so it plays very well, uh, I think with agents like Copilot on GitHub. Do, do you see, uh, uh, if you can share it, uh, which agents are used most, uh, with Unblocked?

375. 55:26

     Whether it's more... Because on the wild, as, uh, developers with our laptops, I think Claude Code is much more used than Copilot, but maybe you see a different picture.

376. 55:36

     Okay. So I'm gonna take this off the screen for a sec [laughs]

377. 55:41

     and try to see if I can pull that up for you.

378. 55:45

     Um, but the answer is yes, we do know roughly what that breakdown looks like. So let me grab that.

379. 55:59

     Uh.

380. 56:13

     Okay.

381. 56:42

     I think this gives you kind of the rough picture.

382. 56:55

     Okay. So this is kind of the rough, the rough picture here. Um, uh, I... unfortunately, because of the way that this is, I, I should probably like-

383. 57:05

     Dark mode

384. 57:05

     ... extend the screen, but I'll just step over here. So, uh, Claude Code is by far the most used, um, followed... This is the, the next one is Cursor, so that, that seems fairly obvious.

385. 57:19

     This last one here is kind of a catch-all, but what's really interesting is that a lot of people use Claude Desktop, which, which was very unexpected, but this is the case.

386. 57:29

     Um, so, and then VS Code and Codex account for a much smaller component. But yeah, it seems like everyone's using either Cursor or Claude Code.

387. 57:37

     I would have expected more of, you know, totally as asynchronous agents, like something that people would-

388. 57:43

     Well-

389. 57:43

     ... just run from a, a PR.

390. 57:45

     Yeah.

391. 57:46

     Okay, you can run Claude from a PR, but it's less common. Maybe sometimes you use Copilot because it's built in.

392. 57:52

     Yeah. Actually, this, this one here, Claude Code, um, may inc- may capture some of that traffic. So that, that's probably what you're seeing, 'cause people will wire up Claude Code in CI-

393. 58:02

     Yeah

394. 58:03

     ... and do things like that.

395. 58:04

     Thanks.

396. 58:05

     No problem.

397. 58:20

     I've, I've got a potentially dumb question.

398. 58:23

     There's no dumb questions.

399. 58:24

     This... Well, we'll see. [laughs]

400. 58:27

     Actually, you know what, you know what?

401. 58:28

     Don't judge too soon.

402. 58:28

     I, I, I had a teacher in grade three that used to tell me there are no dumb questions, only dumb people. Go on.

403. 58:35

     I could, I could be one of them. [laughs] Um, how, like, from, from your point of view, right, you've got-- you've, you can use like sub-agents from like an exploratory standpoint.

404. 58:45

     Yeah.

405. 58:47

     How, how, how does like that plus memory plus just like storing snippets of information that might be able... I, I'm thinking of the like social graph that you just showed, right?

406. 58:58

     Yeah.

407. 58:59

     E-even in an organization that's like several thousand people, you would be able to store that in a very small file.

408. 59:04

     No.

409. 59:04

     Um, you, you would as the graph that you showed.

410. 59:08

     Uh, oh, I see. Uh, d- the social graph component, yes.

411. 59:11

     Yeah.

412. 59:12

     It, it can be compacted.

413. 59:12

     Um, I'm trying to understand how this compares, like what's the kind of like USP compared to the exploratory agents and repeating that.

414. 59:21

     I, I see what you're saying. Okay. Um,

415. 59:24

     so there, there are two, there are two components to that. One is that, uh, an exploratory agent would have to do this every time. So when it starts from ground zero, yes, it might be possible for it to reconstitute, uh, a sort of social graph hierarchy, but it would have to do two things in order to do

416. 59:43

     that. One is it would actually have to write code in order to constitute the, the graph. At least the way that agents are today, or the way that the models are today, you wouldn't be able to just have it like run basic tools around, um, the, the organization and figure out the who's who.

417. 1:00:02

     Um, it would have to write kind of like what that social graph algorithm is, run it, and then get the distillation out the back end. So, um, at that point, you're basically getting close to that component anyways, so the-- you short-circuit it and just run it and use it.

418. 1:00:19

     Um, may-maybe I should explain some of the motivation for that thing, actually. I, I realize now that I may not have done that effectively. Um, social graph is not just about conveying information about who the experts are.

419. 1:00:33

     It's used within the context engine as a pivot point, um, into more like important context. So understanding who the experts are in a particular code area acts as a jump point because, um, another part of, uh, a context engine which happens at the ingestion and processing layer is, um, distilling the, um, we call it bottling the

420. 1:00:58

     expert But it's essentially distilling what that individual has worked on in the past, uh, where they sit in the, in the kind of hierarchy of the organization, um, the dec- the decisions that they've made based on Slack conversations that they've had, based on their PR comments, all this kind of stuff.

421. 1:01:15

     Um, when you distill that down, it's-- and you pass it to the agent, then what happens is, like, let's say that I'm a new employee and I'm coming to work on a particular area of code.

422. 1:01:27

     Um, there are a bunch of different ways of loading context for that code. One is, you know, semantic search via vector search, right? So that's kinda layer one. Another layer is, uh, pre-built memories.

423. 1:01:41

     And then the, the third layer is bottling-- unbottling the expert for that area of code. And getting that expert's learnings into context is, is a really powerful mechanism. It helps drive the rest of the retrieval in an agentic loop, and it helps, um, the agent, uh, directionally, like where to go next.

424. 1:02:03

     Does that make sense?

425. 1:02:03

     Right.

426. 1:02:04

     Yeah.

427. 1:02:04

     I think everybody's in now.

428. 1:02:05

     Awesome.

429. 1:02:06

     So let's, uh-

430. 1:02:07

     Okay. So I think we're-- If we're all in, then,

431. 1:02:12

     uh, the next thing here is, once I get this back up on the screen.

432. 1:02:18

     I'm still s- I'm still sending invites. I saw someone just put-- So please keep coming and, but, uh, we'll, we can keep going as-

433. 1:02:24

     Yep. So, um, feel free to basically just fire this repo at your agent and get it to, like, run it. If you, if you literally just say to Claude Code, "Run this against my repo," um, be sure to give it a time range or a PR limit, otherwise it'll go off the rails and take a really long

434. 1:02:45

     time to finish. So just say, like, "Process the last, like, three hundred PRs," or, "Process up till, you know, September twenty twenty-five," or something like that.

435. 1:02:56

     Um, there's enough information in the README that it should be able to just do it and just run it against your repo.

436. 1:03:06

     This morning I Git cloned, said, "Read the README and make it happen, Claude," and it did. So [laughs]

437. 1:03:11

     Yeah.

438. 1:03:12

     Can I, can I ask another que-

439. 1:03:14

     Yeah.

440. 1:03:14

     What's your roadmap, uh, for-- Do you have a roadmap? What, what's your plans for the coming-

441. 1:03:19

     Yeah

442. 1:03:19

     ... year or so into-

443. 1:03:20

     For, for Unblocked?

444. 1:03:21

     You're an AI type. [laughs]

445. 1:03:23

     Yeah. I, I'm, I'm speaking with long-term, uh-

446. 1:03:26

     Is it, is it about Unblocked or, or about this, this sort of side project? [laughs]

447. 1:03:30

     Uh-

448. 1:03:32

     This-

449. 1:03:33

     No, no, I mean Unblocked.

450. 1:03:34

     Yeah.

451. 1:03:34

     Yeah.

452. 1:03:36

     Um, so I mean, I've, I've sort of alluded to this before, but, like, where the puck is going is with fully autonomous agents. So we're very focused on making sure that autonomous agent flows are highly optimized.

453. 1:03:52

     You-- As I was saying at the beginning of the conversation, you cannot run those things effectively without, like, um, very finely tuned context.

454. 1:04:02

     Yeah. Yeah.

455. 1:04:04

     Yeah.

456. 1:04:05

     So when you think about it, it's like agent swarms. It's gonna be input, output. There will be an IOPS at some point on, like, a context engine. And then shaping that token efficiency helps with retrieval.

457. 1:04:16

     Af- I, I read things like tracing. What, what do a-a-agents-- And you get runbooks out of those. Is that, is that the path you're, you're investing in? Or what, what is it-

458. 1:04:26

     Yeah. What-

459. 1:04:26

     ... retrieval? What, what-

460. 1:04:28

     Are you s- talking specifically about incident management then or?

461. 1:04:31

     Sorry?

462. 1:04:32

     Are you, are you speaking specifically about incident management and that sort of thing?

463. 1:04:35

     No, I'm s- I'm speaking about your-- I, I'm thinking actually pr- more from a business perspective.

464. 1:04:40

     Yeah.

465. 1:04:40

     How can we extract business knowledge that's really deeply embedded into systems nobody knows anymore, and some people know-- think they know, but they don't know.

466. 1:04:48

     Yeah.

467. 1:04:49

     Uh, and, and documents, uh, human knowledge, right, tested knowledge.

468. 1:04:53

     Yep. So I mean, there's, there's two ways of servicing that, either at the product level or, um, through the context engine itself.

469. 1:05:01

     Yeah.

470. 1:05:01

     And increasingly what we see is that people leverage, uh, agents to do their work even at that level. So they'll, they'll go to Claude Code, they'll connect the Unblocked context engine.

471. 1:05:12

     They'll be like, "Do this thing for me." And then the context engine will find all the things that it needs to do that task, and then it'll surface that data.

472. 1:05:20

     Yeah.

473. 1:05:23

     Yeah.

474. 1:05:23

     For us, that means the, the first near-term roadmap is API.

475. 1:05:26

     Yes.

476. 1:05:27

     It's like CLI and-

477. 1:05:28

     CLI API

478. 1:05:29

     ... API endpoints and API layer to interact with it.

479. 1:05:31

     Yeah.

480. 1:05:31

     To your good question earlier about how you expose an-- this thing. It's like API, CLI-

481. 1:05:36

     Mm-hmm

482. 1:05:36

     ... HTTP, et cetera, and some first-class products, as you saw with the, um, sort of QA experience in Slack or-

483. 1:05:41

     Mm-hmm

484. 1:05:42

     ... dashboard. And then the code review is one of our first-class apps as well. But when you use these in your org, it's just good to basically make sure that that access is available to lots of humans and agents.

485. 1:05:53

     Most are-- It's over API at some point.

486. 1:05:58

     Cool. I'm gonna lift this off again, and hopefully people will start s-submitting some PRs, and we can-

487. 1:06:03

     Yeah. Rip, rip some PRs.

488. 1:06:05

     Yeah.

489. 1:06:05

     Ask those questions. Test the agent. Is the, uh-- Once, once you're in that GitHub org, let me actually repost it in the Slack channel

490. 1:06:15

     'cause that link will...

491. 1:06:18

     So this, this org will stay up until the end of the week, um, at which point we'll basically bring it down and, um, release this, uh, as open source.

492. 1:06:28

     And, uh, everyone that contributes obviously is gonna get credited. So, um, your, your name will be on it.

493. 1:06:41

     Should we, like, uh, set up the repo locally and then we then start doing work? Like, what's the, the first thing we should do, basically?

494. 1:06:56

     So I've just finished setting up-

495. 1:06:57

     Yeah. Just, just clone the repo. Um, you, uh-- The easiest thing to do is to take- Uh, an agent like Claude and point it at, um, just launch it from that repo, from that directory, and just say, "Please, uh, bootstrap and launch this, this product," and away it will go.

496. 1:07:16

     If, if you guys run into any kind of technical things, we'll-- we're here, obviously. Yeah.

497. 1:07:20

     I have one question. Back tracking on the,

498. 1:07:24

     on the

499. 1:07:25

     Let's-- Hold on. Let's get you the, the mic. Oh, you got it?

500. 1:07:28

     Yeah, I've got it.

501. 1:07:28

     Okay.

502. 1:07:28

     I've got, I've got a lapel now, so.

503. 1:07:29

     Awesome. Cool. Yeah.

504. 1:07:32

     Can you hear me? Yeah. Perfect.

505. 1:07:34

     Yeah.

506. 1:07:34

     Um, so on the, on the slide where you had, like, the performance and you guys were, like, 80% and without Unblocked, it was 20%.

507. 1:07:42

     Yeah.

508. 1:07:43

     Um, and now I, I see that while you're basically hooking up, like, Unblocked to Claude Code. So, in a way, is it a fair comparison to say

509. 1:07:53

     I will use vanilla Claude Code with access to the MCP and to the skills?

510. 1:07:59

     Yeah.

511. 1:07:59

     And then I will use Claude Code hooked with Unblocked with the same MCPs and the same skills.

512. 1:08:06

     Yeah.

513. 1:08:06

     And here you can do the performance comparison, and here you still have a lot of alpha from, I, I guess whatever you are cooking inside the Unblocked. Is it, is it-- Was it the comparison that was done?

514. 1:08:18

     Or-

515. 1:08:18

     Yeah. It's-

516. 1:08:18

     Was it done without-- Was it done with a vanilla Claude Code but without context?

517. 1:08:23

     No, it was done with MCP servers like GitHub and Slack wired up.

518. 1:08:26

     I see.

519. 1:08:27

     Yeah.

520. 1:08:27

     Cool, cool.

521. 1:08:27

     We, we basically got parity with all the MCP servers of every SaaS vendor in one. It was like vanilla Claude, all MCPs, and the other one was Claude with Unblocked only,

522. 1:08:39

     and then do the task.

523. 1:08:40

     And same context, like the same context file as-

524. 1:08:42

     Yeah.

525. 1:08:43

     Same, same, same prompt.

526. 1:08:44

     And same access. Yeah.

527. 1:08:45

     Yeah.

528. 1:08:47

     It's, it's pretty fun. Yeah. Oh, thank you.

529. 1:08:50

     Um, maybe two questions. So one is, uh, I see that, like, a lot of these, like, social graphs are built with, like, the traditional network, uh, kinda calculation and net statistical aspects of networks.

530. 1:09:03

     Um, is this, like, the approach that you began with and it already worked the best? Or, uh, did you like... 'Cause, 'cause most of memory systems, they work more on, like, filtering out, like, episodic memory, something else, something else, something else, and this is, like, really scoring, really nice scoring system.

531. 1:09:18

     Yeah.

532. 1:09:19

     Uh, that's first question. Is it, like, also with the Unblocked?

533. 1:09:22

     Yeah.

534. 1:09:22

     Second question, um, you mentioned that it works with Teams, uh, Microsoft environment. I wonder what the differences did you observe between building social graphs for different environments? 'Cause on GitHub, I imagine it's very different than on SharePoint, Teams, et cetera, et cetera.

535. 1:09:38

     Yeah.

536. 1:09:39

     Is it also, like, this network stats based, or is it something different, uh-

537. 1:09:44

     Um, so, uh, I mean, our first implementation was, was incredibly naive, right? It was just using, uh, the numbers of PR contributions and comparing that directly with, uh, the number of PRs reviewed by each person.

538. 1:09:59

     So just a simple, like, numbers game. Um, [clears throat] with-- That, that didn't produce accurate team clusters. So then we, we got onto, um, the algorithms that you see here. Um, Unblocked does a little bit more than, than this, so this is kind of like a middle road.

539. 1:10:15

     Um, another strategy that Unblocked uses is, um, like, experts by, by vector clusters. So when we ingest the source code and vectorize it, um, we understand, like, who the, the, the most contributors are for that piece of source code.

540. 1:10:32

     So when we look up individuals, we can see what they've been working on and what the, um, the clusters and proximity are, and then relate people based on their, their cluster proximity.

541. 1:10:42

     So that's more of like an ML type approach. Um, and then there's a, a final layer, which is, um, uh, an sort of AI, LLM-heavy layer that does distillations of, uh, a whole bunch of different context elements.

542. 1:10:57

     Things that people have worked on in the past, conversations that they've been having in Slack. Um, and then when, when you take all that and you weigh it against, uh, the, like, procedurally generated graph, you get a much more accurate distillation.

543. 1:11:12

     There-- This one here, you'll notice, like, some, some people will get pulled into team clusters that you know are, you know, operating across many different teams, for example, and this won't account for that.

544. 1:11:25

     Yeah.

545. 1:11:27

     And then for the differences between, uh, like, different environments like GitHub versus Microsoft versus Slack, like this, uh... Do you need, like, different algorithms, different weights, let's say, that you ascribe there?

546. 1:11:41

     I mean, I don't want to take out all the secrets-

547. 1:11:43

     This-- So, no, no. This algorithm is, like, purely SCM based. So the algorithms for-- You're, you're right, like, um, Slack teams, they're quite a bit different because you don't have these review points.

548. 1:11:56

     Yeah.

549. 1:11:56

     So then it becomes, you know, who's the most active in particular channels, and then you, you need a distillation or a summary of what that channel is about, and then you need to vectorize that, and then you need to score it against the, the most frequent contributors.

550. 1:12:11

     Um, but it's not enough. You have to relate that back to the SCM data in order to figure out who the real experts are. One of, one of the problems that I, I've personally experienced in some organizations I've worked at is that you get, like, the noisy junior engineer, right?

551. 1:12:26

     So they're, they're very noisy, they love to talk, but the signal-to-noise ratio is not great. [chuckles] And, uh, just because someone's not saying a lot of things doesn't mean that their messages are not impactful.

552. 1:12:39

     So part of this game is about assessing the impact of, uh, when people say certain things, you know, how does that relate to the PRs that get spawned off as a consequence?

553. 1:12:50

     How many of those PRs get merged? You know, that sort of thing.

554. 1:12:54

     Yeah.

555. 1:12:54

     Yeah.

556. 1:13:00

     I sent a message to Slack that-

557. 1:13:01

     Oh

558. 1:13:03

     I don't think there's write access to the repo, is there?

559. 1:13:06

     Oh, is there not?

560. 1:13:07

     There should be.

561. 1:13:08

     Okay.

562. 1:13:08

     I'll check that.

563. 1:13:09

     I don't have it.

564. 1:13:09

     Well, you should be able to open a pull request. You can't push to main.

565. 1:13:12

     Okay.

566. 1:13:13

     So if that, if that's the situ- we sh- but I mean, we'll check.

567. 1:13:16

     Yeah, you should, you should be able to create a branch.

568. 1:13:18

     Oh, uh, no, no. Can-- He can't fork the repo either.

569. 1:13:22

     Oh, um, yeah, forks m-might be disabled.

570. 1:13:26

     This will be open source, like, at the end of the week, um, and your, all your contributions will be on it. [clears throat]

571. 1:13:39

     What's really fun is using that social graph tool later against your own repo and, like, showing your team. Yeah. Oh, sorry.

572. 1:13:47

     You can find stuff.

573. 1:13:48

     Oh, I'll do it, yeah. Come out. [chuckles]

574. 1:14:12

     I like that Unblocked tried to answer you for that question. [laughs]

575. 1:14:21

     Oh. You see that? The Slack auto response?

576. 1:14:30

     Yeah, I was like- [coughs]

577. 1:14:33

     Sorry, excuse me. Um. Are you in here?

578. 1:14:38

     That's a camera. Sorry.

579. 1:14:40

     Sorry, did we send it?

580. 1:14:42

     Oh, it's okay. I was just, um... [coughs]

581. 1:15:16

     Oh, okay. Um, let me check to see. That should not be the case.

582. 1:15:29

     Okay.

583. 1:15:31

     Let me know if you still need a GitHub invite.

584. 1:15:35

     Let me just check the members. I think there might... Yeah, there might be an issue here. Just a second.

585. 1:15:42

     Sick. Sick, sick, sick. Oh, these were direct assignments, so I think we have to, like, pull people

586. 1:16:07

     into the-

587. 1:16:08

     Oh, my God

588. 1:16:08

     ... whole project 'cause they're not, they're not org assigned.

589. 1:16:12

     Oh, GitHub. I love you. Zero nines of uptime.

590. 1:16:19

     Yeah. Okay, we'll fix this one here.

591. 1:16:21

     Yeah, and slam everybody in. That's crazy. [laughs] Oh, come on.

592. 1:16:45

     You got it up?

593. 1:16:46

     I, I'm trying to, so... 'Cause now we just need to add people.

594. 1:16:50

     Go to Settings.

595. 1:16:51

     Yeah.

596. 1:16:51

     Go to Collaborators. And then-

597. 1:16:55

     Just start adding people here. It's, it's super painful.

598. 1:17:02

     Unblocked, you all have write access. [laughs]

599. 1:17:25

     It is the name of the company.

600. 1:17:26

     Just, just validate that for us, if you would. [laughs]

601. 1:17:28

     We-- Yeah, please let me know.

602. 1:17:31

     Yeah.

603. 1:17:32

     Oh. Ba, ba, ba, da, da.

604. 1:17:38

     Thank you. I'm okay. I'm sorry.

605. 1:17:42

     Perfect. [laughs] All right.

606. 1:17:48

     I think I was the most, the first one here.

607. 1:17:51

     It's a little box.

608. 1:17:52

     Yeah, but

609. 1:17:58

     All right. We're getting real PRs now. There we go. [laughs]

610. 1:18:02

     Nice. Nice.

611. 1:18:11

     Hell yeah. Now let's do fun things.

612. 1:18:31

     Okay, nice.

613. 1:18:46

     Looks good to me.

614. 1:18:50

     What?

615. 1:18:56

     I think we, we have our, our first approved PR. [laughs]

616. 1:19:10

     I'm send- I'm just sending ridiculous chats to Unblocked, so you can see it try to answer questions in Slack as, as PRs come up [laughs].

617. 1:19:19

     I'm gonna see what it says about this.

618. 1:19:22

     Ask it to-

619. 1:19:23

     It's like, "Oh, let me think about it." [laughs]

620. 1:19:24

     Oh, did you ask it about the PR?

621. 1:19:26

     Yeah, about the PR. I think you accepted, so let's-

622. 1:19:27

     Oh, okay

623. 1:19:27

     ... we'll see what happens. [laughs]

624. 1:19:29

     Yeah. I mean, it did, it did approve it, so you know, Unblocked was-

625. 1:19:32

     Unblocked's like, "This looks good to me, man."

626. 1:19:33

     Unblocked was down. Only visible to you? Oh, no.

627. 1:19:42

     Oh, it's such a good answer, though. Nice PR. Good job, Unblocked. Great answer. [laughs]

628. 1:20:01

     Yep.

629. 1:20:01

     Sorry. Can I, can I... I just joined in later.

630. 1:20:03

     Yeah.

631. 1:20:04

     Can you send me the link or-

632. 1:20:05

     Oh, yeah, yeah. I'll put it back up. I'll put it back up. One sec.

633. 1:20:09

     Uh, where did it go? Actually, I lost the... [sighs]

634. 1:20:30

     Okay. Okay, thanks. It was over here. Boom.

635. 1:20:36

     Thank you.

636. 1:20:37

     Yeah.

637. 1:20:42

     Can we have a sneak peek of how it looks, the, the SaaS inside? I mean, how we can, you know, handle the sources, uh, or maybe see something-

638. 1:20:52

     Do you want-

639. 1:20:52

     ... the app?

640. 1:20:53

     Yeah, for sure. Yeah.

641. 1:20:54

     If that's possible.

642. 1:20:55

     Oh, yeah, yeah.

643. 1:20:56

     Yeah, of course. [laughs] We were focused on you building, but yeah. [laughs]

644. 1:21:00

     Yeah, but like not... Oh, sorry. What am I supposed to do?

645. 1:21:03

     Oh, no, it's okay. I mean, let's go.

646. 1:21:04

     Oh, oh, you-

647. 1:21:05

     Okay, okay. Sorry.

648. 1:21:06

     Oh, I... Sorry. So this, this, uh, this thing that I showed before, it, it is the project that exists in that repo.

649. 1:21:13

     Yeah, I know about it.

650. 1:21:15

     Oh, so the idea is like, um, think, think about features that you wanna add or things that you want to, to fix or like new components, and then just, uh, hack away at it and submit a PR and-

651. 1:21:28

     Sorry, I didn't get that far.

652. 1:21:29

     Yeah. [laughs]

653. 1:21:29

     My bad.

654. 1:21:29

     All good.

655. 1:21:31

     Um, do you wanna open up like a terminal session and show the MCP?

656. 1:21:35

     Oh, sure. Yeah.

657. 1:21:36

     'Cause I'm s- Like, people can obviously use it, but they don't have all our source.

658. 1:21:40

     Let me just to-

659. 1:21:40

     Yeah

660. 1:21:40

     ... first explain where I'm coming from.

661. 1:21:42

     Yeah.

662. 1:21:42

     I'm a consultant. I really like the idea, but if I wanted to try it or propose it to a client, I cannot show the, the, the context graph or maybe get an idea how it works in internal answer.

663. 1:21:53

     Because I really like the idea. This is a-

664. 1:21:55

     Thank you

665. 1:21:56

     ... big problem. [laughs]

666. 1:21:56

     Yeah.

667. 1:21:57

     And, um, but without seeing the product, it's hard, you know, to-

668. 1:22:03

     Well, I mean, w- like one thing that you could do, um, if you're visiting clients is, uh, you can ask them if they run the tool on their, uh, on their, um, repo, and then it will generate this result for them-

669. 1:22:17

     Oh

670. 1:22:17

     ... so they can see on their own project what the value is, right? I think-

671. 1:22:21

     Peter, I think he's just asking about our product specifically, not this.

672. 1:22:25

     Oh, Unblocked.

673. 1:22:26

     Yeah. [laughs]

674. 1:22:26

     You're asking about Unblocked. [laughs]

675. 1:22:27

     My bad, man. [laughs]

676. 1:22:28

     I can go to-

677. 1:22:30

     We're driving this way.

678. 1:22:31

     Yeah. Sorry, sorry. Single track mind. [laughs] Um, okay. So your, your question is how can you demonstrate the value of Unblocked to customers or-

679. 1:22:40

     See the value.

680. 1:22:41

     Exactly. I mean-

681. 1:22:42

     Yeah

682. 1:22:42

     ... I guess the one big question customers have is all of this information enable, for example, if there's a conflict how we can maybe do, uh, data sources that conflict information.

683. 1:22:54

     Yeah.

684. 1:22:55

     And I understood that you can make... Sorry.

685. 1:22:59

     Yeah.

686. 1:23:00

     You can make conflicts emerge in, in your app, but, uh, um... And then there is the compliance layer, which is very interesting for corporate clients.

687. 1:23:09

     Yeah.

688. 1:23:09

     Uh, I was thinking how this, um, is translated to a UX, because you know, m- many people are non... I understand it's mainly for coding.

689. 1:23:20

     Yeah.

690. 1:23:21

     And whether this is for technical people or maybe, you know, people overseeing some, uh, engineers or the engineer itself. I mean, just see how your platform works. But if it is, is out of context, I mean, I...

691. 1:23:35

     It's, it's okay. I-

692. 1:23:37

     No, no, that's, that's totally fine. So this, this dashboard is kind of like the, um, the, the sort of front-end customer interface to the product. So, you know, you come in here and you can ask any question about your code base or your, or your organization and get an answer for it here.

693. 1:23:56

     Um, this is right now, you know, attached to... Sorry, I lost my cursor. This is attached to, um, this test org that we have, but I could use it against Unblocked and I could say like, you know,

694. 1:24:10

     um, well, I have a little hot thing here that I can show.

695. 1:24:13

     Yeah.

696. 1:24:16

     Sorry.

697. 1:24:16

     Whoops. Yeah. So the source mark engine is, uh, an internal component that we use to track source code changes through time, including like where, um, you know, changes move between files and so on.

698. 1:24:30

     Um, so as a demonstration, you know, you can show off... I mean, you can book your, your customers into a demo with us and we can demonstrate this, or you can wire it up to your own organization and demonstrate this flow to customers, um, and try to find, you know, use cases where data sources conflict and demonstrate

699. 1:24:51

     that. The, the challenge with context engines is that it's really hard to demonstrate the value to someone without actually wiring it up. So there, there is a little bit of overhead there where people have to connect it to all their integrations.

700. 1:25:04

     Now, the good thing is, um, Unblocked has, uh, a free enterprise trial period, so people can try out the product in its fullest form before, um- Uh, p-paying for it. [laughs]

701. 1:25:16

     Yeah.

702. 1:25:17

     Sure.

703. 1:25:18

     So if some of that information is incorrect, you can just reply in the chatbot or flag it in the references?

704. 1:25:24

     Yes, exactly. Yeah.

705. 1:25:26

     Several groups of tags and what kind of priorities and measurements-

706. 1:25:29

     Yeah. So you can just-- you can reply here or you can say, "Not helpful," and explain why, and then, uh, it will distill it for the next, the next round.

707. 1:25:38

     So it will adjust some weights or confidence scores internally?

708. 1:25:42

     Uh, well, internally, what it does is it constructs task memory.

709. 1:25:45

     Emotional memory.

710. 1:25:46

     So, um, it looks for those kind of repeated signals and, uh, it-- this is actually where the experts graph comes in. It's used a lot. Um, the experts graph provides, like, weight.

711. 1:25:59

     So when an expert comes in and says, "That's not correct," it's gonna get some, some more weight and distill a memory for it. Um, if, uh, if it's just a new engineer that says, "That's not right," then that's not really a trustworthy source yet.

712. 1:26:13

     So, uh, you have to have a, a trustworthy source to, to base that on. Does that make sense?

713. 1:26:20

     Yeah, it makes a lot of sense. It's like social network-

714. 1:26:23

     Exactly

715. 1:26:23

     ... somehow. [laughs]

716. 1:26:25

     Yeah. Yeah.

717. 1:26:25

     Thanks.

718. 1:26:26

     No problem.

719. 1:26:30

     Sweet.

720. 1:26:33

     Changes.

721. 1:26:33

     Cool.

722. 1:26:38

     What is your memory source look like? Is it files?

723. 1:26:42

     Oh, under the hood?

724. 1:26:43

     Yeah.

725. 1:26:43

     Um, well, when it's presented to the AI, it's presented as, as files. Um, but under the hood, we store it in, you know, database tables and stuff. Um, like, the memories are, are, are constituted from a bunch of different sources, so they're not just, like, flat file-based.

726. 1:27:02

     You know, they'll be-- the whole memory construct will be hydrated at runtime. So yeah.

727. 1:27:07

     And will you just give your agent tools to, like, vary your database based on whatever criteria the user's updating?

728. 1:27:14

     Yeah. Well, for-- Yeah, so yes. Um, there are a bunch of tools for data retrieval. For memory specifically, um, you can't really leave it up to the agent to do memory hydration because that's kind of like part of the seed context.

729. 1:27:29

     In order to get the agent to go in the right direction, you have to seed it with the appropriate data, and experts context is a good jump-off point for the agent.

730. 1:27:39

     So yeah.

731. 1:27:43

     Thank you.

732. 1:27:43

     Yep.

733. 1:27:45

     Thank you. Uh, is there any official benchmark that kind of track the type of value you try to bring? Like, um-

734. 1:27:52

     Yeah.

735. 1:27:53

     Yeah, because it feel like it's not exactly coding.

736. 1:27:55

     Yeah.

737. 1:27:56

     Well, it is, but yeah, I'm curious if there's any, uh, public things that, uh-

738. 1:27:59

     It's-

739. 1:27:59

     ... you're tracking yourself against.

740. 1:28:01

     So we, we, we [laughs] we do have some internal benchmarks. Um, you're right, it's a little bit squishy. Um, so Anthro-- Th-- Have you, have you heard Boris Turney talk, um, at Claude Code?

741. 1:28:11

     He's like-

742. 1:28:12

     Uh-

743. 1:28:12

     ... the creator of Claude Code.

744. 1:28:13

     The creator of Claude Code. Yeah. So he, um, [laughs] did this interview where they were talking about, like, how they measure success, uh, for Claude Code internally. This may have changed 'cause there's a lot of benchmarks now that they have.

745. 1:28:27

     Like, they, they have, like, the, the Shitpost benchmark. You guys have probably seen that one. [laughs] Um, but it-- but what that really distills down to is vibes. [laughs] And so the most important thing in, uh, systems like this is to capture sentiment.

746. 1:28:40

     And so if your sentiment is, uh, is trending upwards, then, um, that's a good thing. Our, our sentiment right now is, uh, on a scale of minus 100 to 100, somewhere around 60, uh, 60 score.

747. 1:28:55

     So on a normalized scale, that's like 0.75 to 0.8.

748. 1:29:00

     So, so the vibe would be captured by something like maybe less back and forth on the PRs or maybe, um, I don't know, you having less back and forth with Claude to get your stuff done, so-

749. 1:29:12

     Yeah

750. 1:29:12

     ... this type of-

751. 1:29:12

     So [laughs] the-

752. 1:29:13

     All right

753. 1:29:13

     ... the vibes are, like, they're, they're people satisfied, right? So satisfaction can come from a lot of different sources, and dissatisfaction can come from a lot of different sources.

754. 1:29:24

     So the way to think about that is that it, it encodes all of those things. Um, but you can capture specific metrics, and we do, how long things take, and we're actually currently working really hard to bring the, uh, response times down because, um,

755. 1:29:41

     uh, you know, even though agents are, um-- Here's the interesting thing. As we move towards a more autonomous universe, response times for MCP servers are actually less and less important.

756. 1:29:53

     The more important thing is that they get the answer absolutely bang on.

757. 1:29:58

     Yeah.

758. 1:29:58

     And the reason is because, um, the, the amount of time that a context engine spends collecting all that information and distilling it is a microcosm of what the full task takes to implement and to, and to traverse.

759. 1:30:14

     So if you can spend a little bit more time and cut the implementation down by, like, 60, 70, 80%, that's a huge win.

760. 1:30:22

     Right.

761. 1:30:23

     Yeah.

762. 1:30:23

     And-

763. 1:30:24

     Oh, go ahead.

764. 1:30:24

     Yeah, sorry. S-very small follow-up.

765. 1:30:26

     Yeah.

766. 1:30:26

     Actually, I'm curious, do you have any, uh, rough, uh, numbers on how much time does it spend retrieving context versus executing the task? Uh, to your point, like, is it 10% right now or 90%, or is it, uh-- I, I have no idea.

767. 1:30:41

     I mean, I have my own experience, but-

768. 1:30:42

     It's like, yeah, um, agent context collection is probably close to that number. It's like 90%. Um, the actual code writing part is really, really fast. If, if-- You can even just watch what an agent's doing, um, when it writes the code that output tokens are, by the way, the, the thing that drags down, um, the, the performance.

769. 1:31:06

     Everyone used to think it was input tokens. We've run tons of experiments with this. You can bring the input token size up and, you know, time to first output token now is, is pretty, pretty good.

770. 1:31:16

     Like, it's pretty highly optimized. The thing that really impacts performance is output tokens. So, um, you have to be, like, judicious with the way that you collect and supply context back to the agent, uh, so that it remains tight on its output loops as well.

771. 1:31:35

     For, um, for one benchmark that Peter mentioned in the talk, we, we gave an ambitious task, 'cause obviously it's prompt-dependent how much time you're adding and, like, with a context engine.

772. 1:31:44

     Um, but the ambitious task we gave was to implement the new adaptive thinking mode in Anthropic's tool chain when they introduced that, which as mentioned, it went from a twenty-five minute wall clock time to it with, with Unblocked, with a context engine.

773. 1:31:59

     The other case without was two and a half hours. It was two hours and twenty-five minutes. But the main reason for that was we gave it all the data, we ran the prompt, and then its first output was, like, totally wrong.

774. 1:32:10

     So you had to-- the human had to loop again and be like, "No, no, this, this, this." And the next output was wrong and the next output. So once you do four loops, you have like a two and a half hour wall clock time versus obviously the twenty-five minute when it did not need that, when there's no

775. 1:32:23

     corrections required.

776. 1:32:25

     Yeah.

777. 1:32:25

     Um, so as mentioned, it's g- think of it as a waterfall. The more high quality, correct, like, s- high signal context you have up front, the better every single thing the agent's gonna do until it says it's done [chuckles] whether it got it right or not.

778. 1:32:39

     Yeah.

779. 1:32:42

     Yeah.

780. 1:32:44

     Man's got it.

781. 1:32:47

     You also mentioned that, uh, the token usage on tool calls and like just information search really decreased. So-

782. 1:32:56

     Yeah

783. 1:32:56

     ... I know that a lot of these tools that provide, uh, or aggregators for tool use, they have insane, like, uh, token usage. So maybe you have like some estimations on how, like, let's say I need a Slack conversation, some summary from one conversation to another, or like how people interact there would be like sixty K tokens

784. 1:33:17

     on Composio. I wonder how many tokens it would be like using, uh, Unblocked.

785. 1:33:23

     Yeah. It, uh, lo-lower. [laughs] We're still very vibes there. Like, it's hard to get real data from other customer or people in the market. Um, but the, again, with that same ta-- I'm gonna keep talking to the same task as ZZ.

786. 1:33:36

     That one went from twenty-one million token total usage to ten million token with the context engine. So a part of that though is 'cause you didn't have to doom loop.

787. 1:33:46

     Mm-hmm.

788. 1:33:47

     So when, when the, of course, like that increased a lot of the tokens expense. Like, so we did drop it by fifty percent on a large task. Again, obviously, if you're like, "Yo, center a div," you're not gonna get a lot again. [laughs]

789. 1:33:58

     It's like probably in the training data. Um, but yeah, like any feature, uh, fix, like so a lot of like, again, a lot of what people are putting through Unblocked are what an engineer is doing every day.

790. 1:34:08

     It's very rare that you're doing a task that's like so, I don't know, minor that like... I mean, then again, I've asked, I've asked Claude to do Git push, so I'm not the only one I bet. [laughs]

791. 1:34:18

     I was like, "You do it." It's like, why did that cost me thirty cents? [laughs]

792. 1:34:22

     I don't know.

793. 1:34:22

     It's running some type of-

794. 1:34:23

     Yeah. I put, I did all the effort to put my GBG keys in the right place, so I'm like, "Claude,

795. 1:34:29

     go." Any more questions while y'all ship? Any confusion? Anything I can unblock for you? It's my purpose in life.

796. 1:34:41

     Sorry, you may have answered this question already, but, um, so you're, are you using knowledge base, knowledge base RAG on, in Unblocked? Or what exactly is the tech that you are surfacing?

797. 1:34:52

     Oh, so many things. [laughs] Uh, I can come talk to you at the side. I'll take my mic off. I'm just gonna answer that question.

798. 1:34:59

     Sure.

799. 1:35:00

     That was just the-

800. 1:35:06

     See, incredibly talented individual. Oh, thank you.

801. 1:35:11

     What's, what's

802. 1:35:30

     the churn rate on the like data? Like the knowledge that you got-

803. 1:35:46

     Oh, it's, it's real time, basically. So, um, there, I guess there's, there's two parts to that question. One is like how much or how frequently Unblocked updates the data on the back end.

804. 1:35:59

     Um, so it's, it's real time for many of the integrations and then on a, a cron job for others because for those, for those particular integrations, they don't have web hooks, basically.

805. 1:36:10

     Yeah, I was gonna say push, pull, right.

806. 1:36:11

     Yeah. But the, the dis- so that means that rebuilding the graph data has to happen on a, on a very frequent basis.

807. 1:36:20

     Zero. Zero.

808. 1:36:23

     Yeah. Yeah.

809. 1:36:29

     If I, if I think of like good parts, it's almost like knowledge as a whole sort of thing. But, um, if I think of like hashing for example, like rehashing something-

810. 1:36:40

     Yeah

811. 1:36:40

     ... changing data-

812. 1:36:41

     Yeah

813. 1:36:41

     ... it's just like recomputing the whole thing.

814. 1:36:43

     Yeah.

815. 1:36:46

     Is that like similar?

816. 1:36:47

     No, it's, it's incremental. So our, our like, you know, social graph builder algorithm has an incremental component to it, so we don't have to rerun the whole thing. Um, but also, uh, social graphs are less sensitive to frequent changes in data because it's unlikely that, you know, a single change is gonna make a huge impact on the

817. 1:37:12

     experts graph unless your organization is brand new. So for-

818. 1:37:16

     There's gonna be attempts to kind of change essentially over time.

819. 1:37:21

     Yeah

820. 1:37:23

     Sometimes collecting data [background chatter]

821. 1:37:25

     Yes. Yeah. So as an example, um, we do best practices distillation on a much lower cadence, like basically w- uh, week by week because, uh, yeah, it just doesn't change that much.

822. 1:37:41

     Yeah. Yeah. [background chatter]

823. 1:37:48

     Do you get, do you get any pushback at all from like customers from like a privacy or, uh, yeah, so, so like how are you, uh, holding that data?

824. 1:37:59

     'Cause that's what I was expecting.

825. 1:38:01

     Um, well, the... Oh, yeah. Re- r- repeat your question. That's a good question, so I wanna make sure we get that one down.

826. 1:38:08

     Um, yeah, in terms of-

827. 1:38:13

     Oh.

828. 1:38:13

     In, in, in terms of customer privacy, data retention, um, kind of-

829. 1:38:17

     Yeah

830. 1:38:17

     ... from, from my point of view, I'm thinking of like enterprise SaaS or even like on-premise type deployments, which I'm, I'm not suggesting that you... I'm, I'm just thinking of that customer kind of-

831. 1:38:27

     Yeah

832. 1:38:27

     ... modality.

833. 1:38:27

     Yeah.

834. 1:38:28

     Um, yeah. Do you get, do you get pushback? Do you... How do they feel about you holding data? It's another processor in the loop.

835. 1:38:36

     Um, well, so the, the, the privacy discussions happen at the organizational level, so it, um, uh, we don't actually run into a lot of friction. Um, there are definitely environments like in government and at banks that have, uh, super sensitive needs.

836. 1:38:54

     And, and so for those needs, we have an on-prem solution. But it's definitely not the path that I would recommend. Like staying cloud-based, like we, we have very large enterprise organizations, uh, that are entirely cloud-based, like fully cloud-based.

837. 1:39:11

     Um, the, you know, the, the secret sauce is kind of like less encoded in source code now and more encoded in, um, uh, the reasoning. So organizations tend to be a little bit more sensitive around things like Slack data, for instance.

838. 1:39:29

     But, uh, the way that we store, uh, data, like we have a whole white paper about how we protect customer data. Um, and it's never been a problem.

839. 1:39:40

     Yeah.

840. 1:39:42

     Can you run on-prem?

841. 1:39:44

     Pardon me?

842. 1:39:45

     Can you run on-prem?

843. 1:39:46

     Yes. We, we do have an on-prem solution, but a- as I say, like it's, it's not the recommended approach, [laughs] but for sensitive environments, for sure. Yeah.

844. 1:39:56

     I missed why it wasn't recommended. Sorry.

845. 1:39:58

     Oh, why it's not recommended? Um, well, the cloud-based integrations, um, g- you know, get updated more frequently, and so there's software patches. It's a little bit harder to maintain within an organization.

846. 1:40:12

     Uh, there's, there's one customer, it's a bank, um, where administering, uh, the platform becomes quite difficult because they have network isolation. And so like now one of us has to, you know, sit within that network and administer the platform, or we have to train, uh, individuals within the company to administer the platform.

847. 1:40:35

     So it's just, it's more of a, a maintenance and, um, hand-holding exercise.

848. 1:40:41

     But yeah.

849. 1:40:46

     Makes sense. Yeah.

850. 1:40:49

     Yeah, exactly. That's exactly right.

851. 1:40:52

     Thank you so much for the talk.

852. 1:41:02

     Yeah. Thank you.

853. 1:41:03

     Really nice.

854. 1:41:04

     Thanks for coming.

855. 1:41:06

     Thanks. [outro music]