Context Is the New Code

Talk transcript

1. 0:00

   [upbeat music] There's, uh, there's a few people who wanna start earlier.

2. 0:17

   Uh, I don't know, I'm gonna take the opportunity to officially open kind of the, uh, architect track. There's no track host, so I do it myself. So thank you for coming here.

3. 0:26

   I hope you already had, like, a good conference. Um, it's amazing that, like, so many people showed up. Um, maybe before I start, um, who's used any AI coding agent in this room?

4. 0:38

   Raise your hand. Like, lower it who hasn't. Raise your hand.

5. 0:44

   Okay, my kind of people. Perfect. All right. [laughs]

6. 0:48

   Um, okay. Context is the new code, um, or context development lifecycle. Um, I feel honored to be here every time I try to do a different talk at the AI engineering.

7. 1:02

   So this is a little bit of, um, you know, thinking ahead. It's an unpolished thought. It's not like everything's there, but is there anything there in AI anyway? But

8. 1:14

   , so let's start. I assume you all are now vibe coding with prompts. I barely touch anymore kind of the code. I just tell the AI to do something different.

9. 1:28

   So I would co-- say, like, okay, you know, context is the new code because it's being generated. A little bit more advanced maybe is, I see myself having a tendency is I had large pieces of code that I was using, maybe some helpers and some other pieces, and I just turned them into a skill.

10. 1:50

    We had that in our-- into our product. It was an onboarding from, you know, uh, AI agents. Uh, people have Python, Node.js, all the various things. Then they have different tools for packaging, and it is impossible to actually code that.

11. 2:06

    Like, it will require a lot of coding. But if I just say, a skill says, "Please first figure out what their package manager is, then figure out what their ecosystem is, and then do these steps together with the user," you know, it solved a lot more problems that we could ever code.

12. 2:24

    So that is another piece that I would say code is also transforming back into context as a skill as well, as a workflow that's reusable. Anyway, leave that with you.

13. 2:37

    I like to think in parallels. In two thousand nine, I don't know if there's any DevOps people in the room, it was kinda me saying, like, "What if ops looked more like dev?"

14. 2:46

    And then we got, like, hey, collaboration, kind of, uh, deployment, all that stuff. So kind of, you know, last year I started thinking, what if context is the code?

15. 2:58

    How do we deal with this in a more consistent way?

16. 3:03

    And it's basically saying, if we have a software development lifecycle, how does a context development lifecycle look like? Because we're basically shifting somewhere else. It's context, it's not code.

17. 3:16

    How does it look like? I came up with this, you know, of course, an infinity loop with some DevOps background. But the whole idea is that we generate a lot of context, then hopefully we test the context, we distribute the context maybe to some colleagues, to some other parts of the organization.

18. 3:34

    We observe whether it works, and if it doesn't work or works, we kinda like, you know, adapt and regenerate the context and then go from there. So that's kind of the loop of the talk that I'll be going for with some examples.

19. 3:47

    So step by step, going through. Generate. It's probably the one that you're all most familiar with because you're all prompting. You're like the human context creation, typing things, right?

20. 4:03

    I was actually amazed that I just asked, "Tell me when my talk is at AI Engineer," that it would fetch the website and would just say, "Here's your talk."

21. 4:10

    Like, blew my mind. But hey, I, I said, like, the context that I've given it, I'm Patrick, all that stuff, right? So very simple context. It's what you do probably a lot, uh, in your setup.

22. 4:23

    If you get a little bit more advanced, you say, "Lab prompting is tedious. I wanna have reusable prompts." So, you know, depending on the flavor of your coding agents, they call it instructions.

23. 4:34

    Luckily, there's a little bit of a standardization now happening where it's like an AGENTS.md and some pieces like that. Boo, Claude, for still calling it CLAUDE.md. But anyway, you get the picture.

24. 4:45

    There's, like, reusable prompts, reusable pieces of context that we're doing.

25. 4:51

    We can also bring other context in. If we have documentation of libraries that we use day to day, we wanna pull that in, because the LLMs might not have the latest documentation.

26. 5:03

    And so it's hallucinating. Is it version two, version three? We don't know. So we give it the context and say, "Please download the documentation," hopefully then agent optimized, and then they will do a better job at generating the code for that version of the library.

27. 5:18

    Another piece of getting better context and creating context from libraries.

28. 5:24

    And of course, it wouldn't be, uh, complete if we would say, "Pull context from wherever." MCP has been instrumental. Get it from your GitLab, GitHub, kind of Slack, all context.

29. 5:36

    We're pulling in, we're creating. Even the ticket is creating context because we're pulling that in while we go there.

30. 5:45

    And then maybe the new kid on the block is, okay, what if we start, like, writing our prompts as specifications, spec-driven development, which then gets broken down by the agent into a planning mode, into step-by-step kind of prompts that it then kind of runs through.

31. 6:02

    So a lot creation happening in that field. You know, simple. This is probably what you're closest to. But when you're typing all that context and creating all that context, you change two lines in your Claude MD,

32. 6:20

    do you know the impact? Is it like YOLO? Looks good to me, let's do it. You have to think about, how do we test things? It's not just about we have a piece of code and we have a piece of context now, we need to write tests to see what is the impact.

33. 6:38

    New coding agent, we don't know whether the lines still work. Now, it's not new in the world of AI engineering, but it's not that common yet in the world of coding with AI that you start writing evals for-- uh, which are tests for your kind of code context.

34. 6:59

    Uh, a little bit hard to read, but, you know, if you think in parallels, we have different levels of testing in code, and the simple one could be linting.

35. 7:09

    Your IDE is-- has the squiggly lines, like, "Hey, this is not..." Like, you know, there's some incorrect syntax or you could do better like that. Here's an example of a validation of a skill where we say, "Well, you need to have the description.

36. 7:24

    It can only be so long." So it's validating according to the spec of the format of the context, in this case. Simple analogy, simple linter that you can run.

37. 7:38

    And then you can do other things, like, and, and I haven't found maybe the good coding equivalent, but think of this as a Grammarly, right? So if you write context, um, is it actually-- c- can the agent understand what you're writing?

38. 7:53

    If you write two words, it's not verbose enough for it to actually understand the context. So what you can do is you can say, uh, ask these, like, "Okay, you know, given this context, what do you think about-- uh, do you understand this?"

39. 8:07

    And then you can get feedback like, oh, it's not explicitly enough written, or it's not complete, like you're missing pieces. So that's kind of feedback that you can get out from tools as well.

40. 8:21

    So whenever you're writing now your context, you get a Grammarly saying, "Hey, do this." Uh, that's why I like to voice code. For some reason, I'm way more elaborate voice coding than typing.

41. 8:32

    I'm a bad typer, two fingers, still after so many years. But when I talk, I was like, you know, I see the, the sentences come on the screen, but it helps to get good context there.

42. 8:43

    All right, another kind of test. So imagine you put in your Claude MD, or Agent MD, I should say. Uh, um, every API point must use the prefix awesome, right?

43. 8:56

    You have some convention in your company, right, which is great. So your prompter will be then, "Add me a new endpoint to save a user." And you expect actually your coding agent to just say the code that's being generated has kind of /a- awesome/user.

44. 9:15

    That's great. But the way we can test this is by asking then

45. 9:22

    an LLM, the code that was generated, does it actually start with /awesome? Now you could do that with regex. I know, this is just for example purposes, but you can ask it to kind of judge your code based on your criteria and whether it did the right thing, right?

46. 9:41

    So imagine you would ask the same question without your context above. No LLM is ever gonna prefix URL with awesome. So that's kind of where your content or your company-specific, your team-specific things come in, and that's why you still write those tests to see if this still works.

47. 10:00

    Now, maybe Gemini kind of, uh, reacts differently than Copilot or something, and in your company, you need to make it more, you know, switchable of context. With this, you run the test, and you can actually tell that's the difference.

48. 10:16

    And then you can make like whole suites, and I would compare that almost to unit tests. I have a bunch of these tests, and they tell me whether that's actually, you know, good code, the code is following the rules, and everything's fine.

49. 10:28

    In this case, it's even kind of infrastructure as code. It doesn't need to be code only. It could be various things, could be config files as well. And I just have-- It's hard to read, but a bunch of kind of criteria that I just run every time to do that.

50. 10:44

    But if you wanna test, you know, whether an endpoint has /awesome/user,

51. 10:53

    there's a real test that we wanna run, which is I wanna test the endpoint. I just don't want only to check the code, I want to have it running.

52. 11:03

    So when you give the judge a tool and the judge becomes an agent and it can do things in a sandbox and execute stuff,

53. 11:14

    it can actually do c- do the curl. So you can bind LLM as a judge with kind of some tooling, and then you can have multitude of tests. Actually, you know, in this case, it kind of ends up being an end-to-end test, right?

54. 11:28

    Because it's not just looking at the file, it's actually running the piece with everything that it's supposed to do.

55. 11:36

    And then I can do this, like given a certain commit in my repo, I wanna run this scenario. Given this piece of context, did it make a difference, yes or no?

56. 11:48

    So you're kind of like building this up while you're committing context also within your repo.

57. 11:55

    And because we now have tests and it gives us feedback whether it's working yes or no or what it's missing, we can optimize context. So that's kind of the, you know, you-- we can put that in a code action or something that says like, "Okay, fix this context.

58. 12:11

    Improve this context." With all the feedback the LLM has given us- To improve that. So, you know, again, coding, uh, improvements, but we start thinking more in testing that piece as well.

59. 12:26

    Uh, one of the first reactions is once you have tests and optimizations, can we run this in a CI/CD system? Because that's perfect, right? That's where we run all of our tests and their test suites and do that.

60. 12:40

    Now, there's a little bit of a weird thing. If you run evals,

61. 12:46

    you run it once, you run it another time, it might not give the same results. Remember, undeterministic things.

62. 12:54

    So you cannot say, "Well, run it once, and then if it passes or not," you're gonna be in for a treat because it's like, "Oh, I can't debug that."

63. 13:03

    So think about this like you run it five times, and out of five, how many times does it succeed? And, you know, maybe in several cases it hits 100% all the time, which is great, but in others not.

64. 13:19

    And depending on how you change your context, it will influence which tests actually work or not. I find it personally helpful to think about this as error budgets. I give a set of tests an error budget that I really care about, so it, it's only allowed like, you know, to fail minimally, and other pieces are okay.

65. 13:41

    So that's how you have to think about testing context. You cannot do like exact testing all the time. It's a different way that this works.

66. 13:52

    All right. So generate, hopefully you understood what the testing could do for you,

67. 13:59

    and distribute. Maybe that's also something you already did. If you maybe have checked context into your repo, right? Which is great, you know. All of a sudden it becomes available, your colleague checks it out, uh, zero friction.

68. 14:13

    I can push, I can share. But we have another mechanism for doing things. Think of this like imagine you have a reusable context that you wanna reuse across multiple projects, across multiple teams.

69. 14:30

    We had the concept of a library. So what if we package kind of pieces of context, and then we are able to install pieces of context that we need for this project?

70. 14:43

    Guidelines, front-end, it doesn't matter for that. And then if we take it up a notch, how to discover what packages exist, that's a registry, right? Now, in that way, it's no surprise that you'll see things like skills and kind of the Tessl registry and the marketplace

71. 15:05

    where you can find a multitude of skills. Now, the reality is

72. 15:09

    99.9, and I mean that in a very sincere way, of the skills is crap.

73. 15:17

    But it's good to learn from others to see what they're doing. But hardly of them, if you run kind of any set of evals on there, is actually up to a quality standard.

74. 15:29

    Now, that will likely improve, but there's also a tendency is that a lot of the skills and pieces, people actually want to put that in their own registry.

75. 15:41

    So I'll come to that later again, but... So you start seeing the gist. A skill not only contains context, it can contain scripts, it can contain documents, contain a bunch of things.

76. 15:56

    So is this kind of the package format? Probably, you know, plugins could now also contain MCP, but you see there's like a standard coming in. Skills all of a sudden, when that came out, all the coding agents said, "We're supporting this as almost like a package format for people to distribute their context on."

77. 16:16

    And then when I have one piece of context, I have dependencies. And I'm sorry, but also with context, we're gonna have dependency hell, right? [chuckles] I, I'm, I'm gonna download this for front-end, and maybe it's conflicting what is in the React context package.

78. 16:33

    And so you start having to deal with that as well. So you start seeing also, uh, packages that, uh, mirror your library versions, your code ver- like your context versions, and kind of pull that in as well.

79. 16:49

    And of course, when we have packages and people are publishing things in registry, we need security, right? OpenCLAUDE. Thank you for that. Like everybody all of a sudden became aware that we need more secure things because we are able to run things on our laptop that are not and coming from strangers, right?

80. 17:06

    So Snyk has a way of scanning context, right? It's doing some credential handling. It's, uh, exposing some third-party pieces. So you start seeing those scanners on the context as well.

81. 17:22

    And then when you think about security, who actually built this skill? How was it built? With what model was this built? So all kind of capturing what we learned in maybe, uh, with packaging, like the SBOM, is kind of the AI SBOM, like the package of context that we're putting in.

82. 17:42

    So you've seen... Still on the path, right? We generate, evaluate, distribute. Let's move into observe.

83. 17:54

    When you are making libraries of skills and context for others, and I don't mean copy and paste this over Slack or something, but when you actually wanna maintain this as something somebody else can use, similar to a library, um, when they start using that, how do you get feedback whether that still works?

84. 18:15

    Now, a great place to get feedback is actually by looking at the agent logs. So-

85. 18:25

    Imagine developer one coding on the project, and the agent is not doing what they want.

86. 18:33

    They could put this into their context, which is great, right? Okay. Let, let me do the TDD almost like, you know, I hit a problem. It's not TDD, but you get my gist.

87. 18:43

    Um, or what if we at a team or an organization scale would look at the logs every time an agent said, "We're missing this piece." And we surface that and say, "If everybody's missing this piece, we should create context for this," and then we distribute the context to everybody, and all of a sudden, the impact of improvement

88. 19:06

    is for everybody. Luckily, like the AGENTS.md, there's now a standards becoming for logs. So we can read from logs, and that's part of our feedback channel to see if the agent is actually using or missing some of the context.

89. 19:24

    Any feedback you get on a PR that's not complete, that's feedback on your context because that PR was created with certain pieces of context. If you say this is not correct, you can kind of keep arguing on the PR, or you can just say, "Let's improve the context," so the next iteration actually improves, uh, and you don't

90. 19:44

    hit that same problem again. What about running code in production that was generated from context and that's not correct? Because yes, we do our PR reviews and we say thumbs up, thumbs down, and we give the feedback, but the actual feedback is also in production when it's running.

91. 20:04

    So this is a tool that actually instruments your code,

92. 20:08

    pushes it out. It's almost like a wrapper. It pushes it out to production. When it fails, it says, "These pieces of code were changed and were failing. Hey, in this case, input, output, it did something wrong.

93. 20:23

    Can we create a test case for this so the next time we don't hit this again in production?" Feedback loop.

94. 20:31

    Now, these are all kind of pretty trivial, like missing pieces of context or improvements. But if you run agents and the equivalent of scanning maybe, you know, in the CI/CD is you need to make sure when it's running in production,

95. 20:49

    is it not doing strange things? So we need kind of a way of looking at that. Now,

96. 20:55

    I've been toying myself with, uh, you know, sandboxing agents, and it is very resourceful at finding things.

97. 21:03

    I like, "Okay, you know, run this thing, try to figure out like anything useful to get break out of the system." And okay, it uses my environment variables. Okay, stupid.

98. 21:15

    Well, let's-- let me remove the secret. Let me look at your memory files. So you have to really make, make sure that like whatever it's doing, you can have a way of tracing this as well.

99. 21:28

    And, uh, apologize again for kind of the slide, but

100. 21:33

     the gist is we can have a sandbox where the agent runs inside.

101. 21:39

     But your Codi agent, by default, without any restrictions, loads your AGENTS.md, you loads your SKILL.md.

102. 21:49

     Like nothing is blocking that. So if you download this, immediately it's loaded. So you can't filter that with sandboxes. You need to have another way. I call that a context filter.

103. 22:04

     Think of this as a web application firewall that just filters out any patterns or prompt injections or stuff that is coming in directly in that piece.

104. 22:13

     And if you take that, there's a lot of talk here as well on harness engineering. Harness engineering itself also has this kind of full observability, looking at logs, looking at traces, looking at feedback.

105. 22:25

     So it's kind of, you know, useful for training pieces, but as much useful for running your own piece as well.

106. 22:33

     Those were the pieces for me today. I would say

107. 22:38

     for a lot of people, there's like create context, test context. Think of this as your library authoring tool loop. And then when you push this into the enterprise, there's an organizational loop.

108. 22:51

     "Hey, I made a library. Somebody else is using it. I'm looking whether that's useful, whether that's still working, whether that's still working for all the other pieces." So that's kinda like the kind of

109. 23:04

     improvement, almost like sonar CI/CD model for context. And then

110. 23:11

     you're currently probably doing a lot at the individual solo model. You're improving, you're honing, crafting your own kind of markdown. What if you start doing this more with your team?

111. 23:21

     Make that a reflex. If it's missing, add some context. What if you put that out to a team of teams and you start having a flywheel? You know, if you fix it here, the other team can reuse it and, and that's kinda like, you know, scaling things out into the organization as well.

112. 23:39

     And so there's a lot of talk about LLMs and coding agents, and I all love them, but the way that I see it is they're just the engine. If you give the engine the wrong fuel, which is context,

113. 23:52

     they're not gonna perform. So... And you can't do anything on the LLMs, at least not me, right? I'm just using the coding agent. I'm using whatever they give me, but I can optimize my context.

114. 24:03

     Um, and that's I think the message, uh, doing this more in an engineered way than just copy and pasting things and hoping for the best in there.

115. 24:13

     If you like this talk, connect on LinkedIn for the slides. Um, give me some feedback, good and bad. If you wanna try Tessl where we implement some of the pieces of this, uh, have a go.

116. 24:26

     And if you're also interested in another conference, I know you can never have enough conferences, uh, visit, uh, AI DevCon, which I curate the content for, uh, here in London, 1st and 2nd of June.

117. 24:38

     And that's it. I can maybe take a few questions. [audience applauding]

118. 24:49

     Any questions?

119. 24:51

     I do have a question.

120. 24:53

     Sure.

121. 24:54

     So I wonder if you have any thoughts about, like, more exotic forms of context, like, you know, as the traditional ones. So for example, one of the things I'm working on is an automated system for, uh, scoping out architectural problems and, like, trying to create hard definitions for them so we can feed that to the agent and,

122. 25:09

     you know, create actual objectives, uh, tests. Oh, cool.

123. 25:14

     Yeah.

124. 25:15

     Microphones. Um, and one of the things I've been testing out is, like, the ability to create consistency as a form of context or as a form of eval. So, um, given this rough, like, very loose definition of what the plan is, if c- you put that, if you try that agent system, turn that into a really crisp

125. 25:32

     definition, and you just have that done in parallel, how often do you get the same crisp definition? And if they're all over the place, then the original definition was so poor and you'd, like, go back to basic principles or to an architect.

126. 25:44

     But if they're all the same, then it's probably a pretty good definition and you can carry on with the downstream process.

127. 25:50

     Yeah.

128. 25:50

     So I guess, like, besides just code and typical evals, um, any other sources of context or generating context that you think is useful?

129. 25:58

     Um, I don't have maybe a, a specific answer to your, like, exotic-

130. 26:02

     Yeah, yeah, yeah

131. 26:03

     ... you know, case. But, uh, I would say the, the, maybe the piece that people underestimate is that once you, you know, you thought you were gonna save time by writing actually your context, uh, instead of all your code.

132. 26:14

     Mm-hmm.

133. 26:15

     But if you take this rigorously, you're gonna spend time on writing the right evals.

134. 26:19

     Right.

135. 26:20

     And that's kinda like, you know, a lot of work to kind of ... Because n- now you don't only have one prompt [laughs] that you're trying to get right. It's, like, all the prompts of the evals.

136. 26:30

     And that, like, if people do almost like a, like, the more advanced people, they almost have their own process, and they, they build their own process on top of, like, for building the right evals-

137. 26:41

     Yep

138. 26:41

     ... on your business case as well. So yeah. Good question. Thank you. Any other questions?

139. 26:49

     If not, I'll be around. Um, say hi. I'm also gonna be at the Tessl booth. So thank you very much, and I'm gonna make space for the next speaker.

140. 26:57

     Thank you. [audience applauding] [upbeat music]