AI Engineer Europe 2026
Lawrence Jones - Fighting AI with AI
Read the talk
Fighting AI with AI: Making Complex Systems Debuggable
Incident.io connects focused evals, downloadable execution traces, and repeatable agent analysis to turn bad AI investigations into testable code changes.
From a talk by Lawrence Jones
Before you start: Familiarity with prompts, evals, execution traces, and coding agents will help; the article explains how these pieces fit into the debugging workflow.
How do you know an investigation is right?
How do you judge an AI investigation that tells you what broke in production and what to fix? At incident.io, that question grows out of an incident response platform that pages engineers, helps coordinate incidents, and supports customer communication. Lawrence Jones, a founding engineer, describes customers including Netflix, Etsy, and Skyscanner. The ambition extends beyond managing major incidents: the team wants to automate production investigations, including questions that begin as routine tickets.
At the time of the talk, Jones says the team had spent roughly 18–24 months building these systems. An investigation runs hundreds of queries across logs, metrics, traces, and historical incidents, then cross-references the evidence with the codebase to propose a cause and a fix. The report on screen makes that result concrete: it describes a PostgreSQL query failure, asks why it happened now, and considers whether a fix is in progress.
Reading the answer is much easier than establishing whether it is correct. A reviewer may need to reconstruct the incident, inspect what responders did, and read a postmortem before deciding whether the investigation was right—or even useful. Jones estimates that understanding an incident well enough to judge its report normally takes about an hour. Behind that report are hundreds or thousands of prompts, operating across customer environments with different failure modes. The debugging problem is understanding the system that produced the answer, not merely reading the answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with a prompt you can test
Jones’s earlier LDX talk, Becoming AI engineers, covered the underlying constructs: prompts, evals, scorecards, traces, datasets, and backtests. Here, those are the starting point. The next layer makes eval datasets easier for agents to maintain, exports debugging interfaces as file systems for Claude Code and Codex, and uses agents to run repeatable analysis pipelines.
An eval is a unit test for a prompt: supply input, execute the prompt, collect its output, and apply grading criteria to decide whether it passes. Incident.io keeps evals in YAML files beside the Go code defining its prompts; its AI work, like the rest of the application, is written in Go. Engineers run those tests before merging prompt changes.
The deliberately simple demonstration translates a message into pirate speak. It has two independent requirements: the output should sound like a pirate, and it should preserve the input’s meaning. A colorful translation that changes the request still fails. A faithful translation with no pirate style fails the other criterion.
The demonstration shows three YAML cases and their execution results. A compact fixture following the same two-criterion design could look like this:
yaml
criteria:
- id: pirate_style
description: The output sounds like pirate speech.
- id: meaning_preserved
description: The output preserves the meaning of the input.
cases:
- name: meeting_time
input: "Meet me at the harbor at noon."
- name: missing_supplies
input: "We have no drinking water left."
- name: keep_the_warning
input: "Do not open the door until I return."
Keeping the criteria separate makes the intended behavior legible without prescribing a single acceptable wording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make large eval fixtures editable
Real incident fixtures are less tidy. The behavior an engineer needs to reproduce may depend on almost an entire incident report. Incident.io added a button that exports a problematic production interaction into the repository as an eval, making it possible to test against the actual input that went wrong. That solves capture, but it does not solve maintenance.
A useful unit test communicates a focused expectation. A production export may instead bury that expectation in a huge YAML document. Jones uses a two-megabyte YAML fixture to illustrate the problem: reading and modifying whole suites can exhaust a coding agent’s context before it has done useful work.
The team’s internal Eval Tool provides a smaller interface to those files. Its CLI can list test cases and edit, replace, or add an individual case. The agent can therefore work on the relevant fixture without loading the entire suite into its context. That access pattern also makes it possible to give the agent a runbook for changing prompts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give prompt repair a repeatable procedure
Packaged as a runbook or a skill, the repair procedure gives the agent a sequence to follow:
- Inspect the prompt and the requested behavior.
- Create an eval that demonstrates the current failure.
- Modify the prompt until that eval passes.
- Run the other evals to check for regressions.
- Consolidate and simplify the prompt so repeated repairs do not leave it swollen with instructions.
The final step matters because a succession of individually reasonable additions can produce a prompt that becomes difficult to understand and maintain.
Jones demonstrates this in Claude Code with a real prompt that converts human requests into Loki log queries. The agent adds an eval, runs it repeatedly, and assesses whether its pass rate is acceptable. The talk does not specify the repeat count or acceptance threshold. This workflow makes a known prompt easier to change with regression checks; it does not yet tell the engineer which prompt caused a bad interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the error inside the hierarchy
A production interaction rarely belongs to one prompt. Jones maps the incident chatbot as a hierarchy of prompts, tools, and agents. Roughly ten agents are visible, and the full graph does not fit on screen. A customer can identify a bad response without knowing which component produced the mistake. The red–green eval loop is useful only after that component has been located.
Investigations add another layer. Each step in the investigation timeline expands into a trace; individual green blocks can themselves contain hundreds of prompts and tool calls. A subtle intermediate error can propagate until the final root-cause analysis presents a completely wrong explanation.
The team built interfaces to help humans inspect these executions. Those interfaces expose the details, but engineers still lack the time to follow every branch, and coding agents cannot readily use the same UI tools. The missing capability is a way to make the existing debugging evidence accessible to an agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Download the debugging interface as files
Claude Code’s ability to navigate file systems with standard tools suggested a direct solution: export the content behind the debugging UI as a file system. Incident.io downloads an interaction into a sandboxed Claude Code environment. The package includes the inputs to the prompts and a self-documenting structure; the agent also has access to the application codebase. It can inspect what happened and identify where a behavioral change belongs.
Once the agent locates the relevant prompt, the earlier eval loop becomes useful again. Nor does exporting the UI require discarding its structural information: Jones shows that a trace can be represented in ASCII text, preserving the relationships that the interface displayed visually. The agent can consume that structure alongside the underlying prompt content.
The resulting debugging session follows a practical sequence:
- Download the interaction associated with a bad experience into the sandbox.
- Ask the agent to inspect it and explain what it thinks went wrong.
- Describe the desired behavior and ask which part of the tool and prompt hierarchy should change.
- Use the available codebase to make the change.
- Verify it with the eval runbook.
Diagnosis and repair now happen in the same session, with the human specifying the intended behavior and the agent tracing its implementation. Incident.io has added these file packages to multiple kinds of AI interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn backtest scores into explanations
Individual debugging sessions still do not cover the whole system. Jones describes running thousands of investigations daily across hundreds of customer accounts. A backtest batches investigations against incident.io’s own account and customer accounts so the team can track whether the system is improving.
Jones’s example backtest reports 86% accurate root-cause analysis on incident.io’s own account. The talk does not supply the dataset size, grading rubric, or evaluation window. More immediately, the aggregate cannot explain why the score rose or fell, or what needs to change for a particular customer.
The team exports the investigation batch as files and feeds it to a Claude Code analysis pipeline. An internal repository called Scrapbook contains Markdown playbooks describing how the agent should inspect the downloaded evidence and move through the analysis. This turns an open-ended debugging task into a repeatable process.
The process first fans out into independent investigation analyses; Jones gives an example of about 25 agents running in parallel. It then clusters those findings into cohorts with similar failures. The resulting report explains patterns behind an account’s performance and proposes improvements, rather than simply repeating the aggregate score. Jones says the team has applied this approach across several internal systems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the analysis through to a production change
A useful analysis pipeline keeps its work, not just its final answer. Subagents analyze individual investigations in parallel and save incremental findings in files inside the downloaded package. Those files let the workflow stop and resume. Giving the analysis access to the codebase also lets it connect a diagnosed failure to an implementation location and propose a concrete change.
The coding agent can then implement that proposal in the same session and use the eval red–green process to check the repair. Jones shows a merged pull request following analysis of a couple of failing backtest investigations: PR #51443 feeds previous hypothesis-review feedback into the next hypothesis build. The example makes the handoff tangible—investigation evidence leads to a discussion of a feature change, then to a code change. Deployment and testing in production complete the loop.
Internal debugging tools deserve the same deliberate use of AI as the product itself. For a complicated AI system, that means making the evidence usable by the coding agents engineers already work with. Jones favors bulk file downloads because an agent can grep through the information and retrieve the details it needs. He judges an MCP layer or computer-use approach less effective for this workflow; he presents that as an engineering judgment, without a measured comparison.
Recurring complex analysis is also a candidate for an AI runbook. Jones argues that capturing the procedure can save days or weeks of work. He closes with an invitation to help build these systems: at the time of the talk, incident.io was expanding its London team after a substantial raise the previous year and hiring for its AI SRE work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Jones's earlier LDX talk on evals, debugging tools, and engineering practices for AI systems. The public page includes an abstract; video access requires registration.
Official documentation for querying Grafana Loki with LogQL, the query language relevant to the talk's log-query example.
Further reading
An account of incident.io's internal Workbench, including interaction scoring, trace inspection, repeated prompt trials, and evaluation across customer accounts.
- You can’t vibe code a promptArticle
Worked examples of prompt tuning that breaks existing cases or memorizes an eval suite, with guidance on using LLMs for evaluation and prompt review.
Updates since the talk
Jones's later walkthrough of a daily AI spend-analysis skill, including separate reference files and iterative tests with fresh subagents.
Read the complete timestamped transcript
- 0:00
[upbeat music] Um, hi everyone.
- 0:17
Uh, so I'm here to talk today about how we use AI to manage the complexity of the AI products that we build, um, at Incident.io. Uh, and to share with you some of kind of the tips and tricks and the internal tools that we use when we're building, um, our AI SRE product.
- 0:32
Uh, but first, I guess, like, who am I? Um, so I'm Lawrence. Uh, I'm a founding engineer at a company called Incident.io. Uh, so we build, if you haven't heard of us, uh, we build an incident response management platform.
- 0:44
So we're used by companies like Netflix, Etsy, Skyscanner, and actually probably a few of you in the room. Um, we page you when things go wrong, and we help you run your incident.
- 0:53
And as you're running your incident, we help you communicate with your customers. Um, but, like, you might be thinking, like, where does AI actually come into this? Um, and actually, the, like, we don't just want to help people respond to these incidents.
- 1:06
Um, our goal is actually to fully automate kind of production investigations. So whether or not it's a big incident or if you just have some ticket and you wanted to look into production, uh, we want to be the place that you turn to ask us questions about actually what's, what's going on.
- 1:20
Um, now it turns out we've been building this for about a year and a half, two years now. Um, and that's actually, like, a really big ask. And the systems that we've had to build to try and support this have been really quite complicated, um, and have been kind of on the edge of what you can do,
- 1:32
uh, with all of the AI technology that's out there. Um, and they often pose a challenge for humans to debug them. Uh, they are now complicated enough that you can't, as a human, really tractably dig into how these things are performing.
- 1:43
Uh, you need assistance to help you. Um, so for example, this is, uh, kind of, uh, one of the investigations that we would actually produce for you, uh, when you have an incident.
- 1:52
Uh, we will end up running, uh, right at the start of the incident, this investigation, which will go through hundreds of telemetry queries. It's gonna look at your logs, your metrics, your traces, any historical incident data that we have.
- 2:03
Um, and it's gonna try and cross-reference this with your code base and go, "Hey, like, I'm pretty sure that the problem is this, and you should probably do this to fix it."
- 2:10
But I wanna pause here and go, like, how would you... if you were building this system, how would you actually figure out how to tell me if this was a good or a bad report?
- 2:17
How do you know if it's right? How do you know if it's wrong? Um, and there's a load of things that you might do. Uh, you might jump into the incident, and you look at everything that happened.
- 2:24
You might look at a postmortem if there was one that was written. Um, but all of this might actually take you a really long time to do. In fact, it normally takes you, like, an hour or so to get a real full understanding of this incident.
- 2:36
Uh, and it's only at that point that you could then look at this investigation and go, "I think it's right," or, "It gave me the information that was really, really useful."
- 2:42
Um, and as I said, behind this investigation is, like, hundreds if not thousands of prompts. So how on earth do we scalably understand how this system is performing, especially across all of our customer accounts when they all have very different things going on?
- 2:55
Um, you end up with a lot of stuff and a lot of AI, and you've got to use AI to try and actually tractably get a handle on this.
- 3:01
Um, so I actually did a talk, uh, a year ago, um, at LDX about becoming AI engineers, uh, where I went through some of the core constructs that hopefully, like a lot of you in the room, given that we're an AI engineering cons-- uh, conference, are familiar with.
- 3:15
So things like prompts, evals, scorecards, traces, datasets, backtests. Um, this talk is going to be about if you assume that you have these constructs put together and you're building these complicated AI systems, um, how can you use AI with the internal tools that you use to understand them to get a better handle of how your system is
- 3:32
performing? So yeah, um, in this talk, I'm going to talk about how you can use AI to help you manage and curate your eval datasets and making it easier for you to work with them, uh, making it easier for coding agents to actually work with your eval tool.
- 3:45
Um, I'm gonna talk about, like, probably what was the biggest unlock for us when we were building these systems, which was starting to translate the UIs that we built to try and debug them, um, into downloadable file systems, which has actually helped us massively using tools like Claude Code and Codex, uh, to dig into how the system
- 4:01
is performing. Um, and then I'm gonna talk about how you can build kind of repeatable analysis pipelines using, uh, AI agents to run through them. Um, but first, evals.
- 4:11
So evals for me are AI unit tests. Uh, so each eval takes a prompt, and it goes, "Here is some input data." Uh, it runs the prompt, it gets the output, and then it has some grading criteria that says, "Does this eval pass?
- 4:23
Does it fail?" Um, and for us, evals live in YAML files right next door to, like, our Go, uh, prompts. Uh, so we do everything in Go at Incident.io, um, including all of the AI work that we do.
- 4:34
Um, and this is how we prove when we make a change to a prompt before we ever go and merge it, that the prompt is actually going to do the thing that we want it to do.
- 4:42
Um, so for us, this is what a prompt looks like. Um, this is kind of a contrived prompt. I would hope no one actually has this in production anywhere.
- 4:50
Um, it takes a message, and it tries translating it into pirate speak. So really simple, bit silly. Um, but what we do for evals is if this is the prompt, uh, we would then define on the left some grading criteria for this prompt where we'll go, there are two things that we care about.
- 5:06
We care that the result actually looks like pirate speak, and we are going to care that the meaning is preserved between the input and what we actually produce as an output.
- 5:15
Um, so this is actually what we're going to use to tell us if the eval passed or failed. Um, and then we have the evals up on the top right, which is just in a YAML file, where we go, here are three different test cases, and we'll run through them, and you can see the results of us
- 5:26
actually running this, uh, on the bottom right. So this works, and it works really quite well, but it does have some problems, and I'm assuming that several people in the room have kind of come across these themselves.
- 5:37
So first, um, like, evals are really, really fiddly. Setting up realistic test data for your evals if you want to actually understand how this stuff is running, um, is quite difficult to do.
- 5:46
Uh, and especially in our environment, our production evals are including almost an entire incident. So you can imagine a full incident report, and that's the only thing that can trigger the bad behavior.
- 5:55
This is kinda hard for you to end up pulling them down and putting them in your eval test suite. Um, they just become extremely unmaintainable very quickly. Uh, now quite early on, we created, like, this little button that we have that allows you to, like, steal an eval from production.
- 6:08
So if anything was going wrong inside of our, like, AI interactions, you could go in and you could just pull that down, put it in the code base, and you could run the eval against it.
- 6:16
Um, but the thing with this is, like, production evals aren't, like, great.
- 6:21
If you think about evals as kind of like a unit test, uh, for your prompts, you want a unit test suite to be reasonably understandable. Like an ideal unit test is very focused and just says, "I expect it to do this thing."
- 6:32
You don't wanna have like two megabytes of all the YAML associated with it. It's just really, really hard to work with. Um, and what we found was, as these YAML files for the evals were growing really, really large, uh, our coding agents weren't able to work with them.
- 6:45
So if you wanna do a quick like read and like modify the eval suite, you'd be booting that into the context, and you quickly hit your context limit. Uh, which is obviously a problem 'cause then you can't work with it effectively.
- 6:56
Um, so what we ended up doing was we ended up creating a small CLI tool that we call Eval Tool, um, that was designed to allow agents to leverage our eval suite files.
- 7:07
Um, so it's just a small CLI that can go, "What test cases do you have in here? I want to edit one, I want to replace one, I want to add one."
- 7:13
Um, and it was by doing this that we allowed agents to work effectively with our eval tooling, and that's why we were able to create this runbook to the right, which is actually a runbook that's designed for, um, a coding agent to use.
- 7:24
So either a runbook or a skill, it depends on how you want to package it. Um, but the cool thing about this is that now that agents can work with the evals, you can end up in a situation where you just ask your coding agent to go, "Hey, I've got a problem here.
- 7:36
Can you look at this prompt? I want it to do these things." And the coding agent's going to turn up, and it will create an eval case where it proves that the thing has failed, and then it will go modify the prompt so that the eval now passes.
- 7:47
Um, and then it will go through this runbook, and one of the most important stages for us is checking at the end that the change that you've made to the prompt hasn't ended up breaking any of the other evals that you had in your test suite.
- 7:57
Um, and we also have like a final pass that tries consolidating the prompt as well. 'Cause if you end up doing this repeatedly, you end up with a prompt that is massive and really, really difficult to maintain.
- 8:07
Uh, so you kinda want every time you make an adjustment to try and simplify as well. Um, so this has actually worked like really well for us. Um, and you can see here, uh, this is like me using it in Claude Code, where you can just point it at the eval and say, "Hey, have a look at
- 8:22
the prompt." This is a real prompt for us, which turns human queries into log queries for a Loki system. Um, and it ends up racing through and goes and adds a new eval.
- 8:30
It checks that it passes with a certain number of repeats. Um, and then it gets to the end, and it's like, "Yep, I think I've added it. Kind of the pass rate is acceptable.
- 8:38
You can go ahead and get going." Um, but the problem with this is, like this solves one problem. Um, and the problem that it solves is that if you know what the prompt is that you want to change, you can now change it fairly reliably, and that's very useful if you're working on these tools.
- 8:51
Um, but one of the biggest problems that you have [chuckles] now is that if you're building these systems, you'll know that they're not just one prompt anymore. Um, in fact, most of the production AI systems you will use on a daily basis are many, many, many prompts.
- 9:04
Um, and to illustrate this problem, I've taken our chatbot. So this is a chatbot that you interact with, um, inside of an incident. And what I've done is I've created a graph of all of the different prompts, tools, agents, and everything in the hierarchy, uh, that powers an interaction with our system.
- 9:20
Um, so you can actually see there's like 10 different agents there. There's 50... I, I don't even know. There's... It, it's actually bigger than this. I couldn't fit it on the screen.
- 9:27
Um, [laughs] it's, it's a lot of stuff. So even if you think that you know-- even if you've got a bad interaction that came in from a customer, um, you don't necessarily know which part of your system is actually the problem and which part to go change.
- 9:39
Um, so even if you have this eval red-green cycle, you're gonna struggle to know where to go to fix it. Um, and this gets even worse for a system like our investigations.
- 9:49
So if you think about trying to run through this process to debug what's going on in Incident, we have a ton of stuff that goes on, uh, inside that system, and you can see all the steps on the left.
- 9:58
Um, and each one of those steps unpacks into the trace that you have on the right. And like really, it's not about the details here. Um, it's more about each one of these green blocks actually expands into possibly hundreds of different prompts and hundreds of different tool calls.
- 10:11
And at any point, if you make a slight subtle error, you can't then easily trace through the system where the error originated, even if it ends up resulting in you having totally the wrong picture on what you think that the incident was and your RCA is totally wrong.
- 10:25
Um, so like we built these UIs so that we could help humans look at them, and they've been really good for humans to look at them. But I guess going back to what I was saying before, um, we just feasibly don't have enough time to do-- go through this stuff. [laughs]
- 10:38
So the problems that we had was we have all these UI tools, but agents can't properly use them. So how do we get to a place where like the agents can use the tools properly?
- 10:47
Um, and I think Anthropic stumbled on this with Claude Code, where, um, they found kind of when they released Claude Code that these agents are fantastic at using file systems and just going through this data using standard tools.
- 10:58
Um, so we kinda thought like, "Can we just download all of the UI that we have as a file system?" Um, and that's kinda what we've done. So now for each of our different AI systems, you're able to download all of the content as a file system, and we drop that into a sandbox Claude Code, uh, at
- 11:13
which point you can just point Claude Code at it and go, "Hey, I've got a problem here. It's behaved in the wrong way." It can see everything that went into all the prompts.
- 11:20
It understands the structure 'cause it's self-documenting. Um, and then it can tell you, because you have access to the code base as well, uh, exactly where you should actually be making the modification to try and change it.
- 11:30
And then you can lean on that red-green cycle from before to try and modify a prompt if you need to. Also, there's more stuff that you can put in this than you might think. [laughs]
- 11:39
There is really not much of a limit as to what you can put into ASCII. Um, so like traces like this can get translated exactly from how you would present them in the UI to a text file, which then the LLM can consume in a really nice way.
- 11:53
So yeah, uh, this is something that has turned the way that we debug our application into we hear that there's a bad experience. We end up downloading that interaction into a sandbox Claude Code.
- 12:02
Um, you sit there in the session, and you go, "Hey, like have a look at this. Tell me what you think has gone wrong. Like, like what is your interpretation of the problem?"
- 12:09
Um, and then you go, "I really want it to do this instead. Like, what part of the system would you change?" And then it will work its way through the hierarchy of all those tools and prompts that you just saw, and it will be able to tell you where you should be making the modification.
- 12:20
And then all from that session, because you have access to the code base, you can just go, "Hey, can I make that change?" And then you can prove it using the Eval runbook that I mentioned before.
- 12:29
So yeah, like we've implemented these file system packages for a load of different AI interactions for us now. Um, so it's really easy for us to just drop this in Claude Code and just get going.
- 12:37
Uh, but we now have another problem, right? 'Cause whilst you can do this on an individual basis, uh, we are running thousands of, of investigations across hundreds of our customer accounts.
- 12:47
Um, and we're doing that daily 'cause we need to know if this system is getting better or worse. So, um, what you can see here is, uh, we have what we call a backtest, which is essentially a batch of investigations that we run on a daily basis against our account and against a load of our customer accounts
- 13:02
as well. Um, and eventually you just get this rolled up number, which is like, oh, cool, eighty six percent accurate RCA on our account, which is, which is great, but like, this doesn't really tell you why the number went up, and it doesn't tell you why it went down.
- 13:14
And if you want to improve the system for someone, you're gonna struggle. Um, so what we've actually done is we've allowed ourselves to download, uh, kind of all of these investigations into a file system that we can then provide into an analysis pipeline that again, is lever- or is run using Claude Code, uh, that can end up
- 13:32
running a structured analysis, like with markdown playbooks that help you run it repeatedly and reliably each time. Um, so what that actually looks like is we created this readbook-- this repo called Scrapbook.
- 13:44
Um, and inside of Scrapbook, we have this like very structured flow that explains exactly how a coding agent should go through all of the information that we've gone and downloaded, how it should understand these investigations, and the process that it should go through to actually run them.
- 13:57
Um, now the key things that, like, I think are very important to these flows are you start and you actually parallelize out all of your agents. So you start maybe twenty five agents in parallel, and they can all individually build their analysis of an investigation.
- 14:11
Um, and then you go into the next stage of the pipeline where you do some cohort clustering, and you look at like meta points around, like, what are the same types of failure or how do we go wrong in different ways.
- 14:20
Um, and by clustering it together, you end up with actually like a really, really useful report that doesn't just tell you how this has gone wrong, but it tells you why is your AI system performing well or badly on this customer account, and actually like what should you do to try and fix it or improve the system.
- 14:34
Um, and like this is, this is like something that we've done several times over for several of our systems now, and I think it generalizes really well for anyone who's building this type of thing.
- 14:43
Um, so the points that make like a really good pipeline for this, um, you should leverage subagents to do that parallel per entity analysis. Um, you should store all of your analysis in files inside these downloads so that you have like incremental analysis built as you run through it, so that you can start and resume the analysis
- 14:59
if you ever need to. Um, and then you want to combine this analysis with the code base that is actually powering the system, so that if it finds a problem, it can look in the code base and go, "Hey, I think that this is the problem and this is the place."
- 15:12
And it can actually do some analysis to go, "I think that you should change it like this." Um, and then at the end, because you have this all loaded in your code session, you can just go fix it and ask the coding agent, Claude Code, Codex, whatever you use, to actually go make the change, and then you
- 15:25
can use that eval red-green process to actually confirm it works. Um, and then like, yeah, this is a PR that is created after you do something like that where because the backtest showed a couple of investigations that were going wrong and I knew exactly what w- the problem was, um, I could have a chat to it about
- 15:41
a feature that we might change in the system. Um, and then we can deploy that, and we can test it out in production and see how the thing goes.
- 15:48
Um, so yeah, that's it. Uh, so, um, the key thing from me is like these patterns do generalize. So for any of you in the room who are building kind of complicated AI systems, um, and you're finding it really hard to understand them or debug them or evolve them, um, you, you really need to be using AI
- 16:03
just as effectively in your internal tools to try and understand these systems and grow them, um, just as you are in the products that you're building yourself. Um, so yeah, make sure that you prioritize any of the debugging tools that you have so that they work really, really well with the coding agents that you're leveraging on a
- 16:18
day-to-day basis. Um, file systems are exceptionally good agent context. Like, we could have put an MCP on top of this or used like human use, uh, agents. Uh, it wouldn't have been half as effective as this ability to just download in bulk all of the information that you need so that the coding agent can grep through it
- 16:34
and find the details. Um, and then, yeah, anytime you are performing complex analysis, look at creating an AI runbook for it instead. Um, [laughs] it will save you literally days or maybe weeks of your life.
- 16:46
Um, and then one final point from me, um, like we are hiring. Uh, [laughs] we're, we're in London. Um, we have just done a fairly big raise last year, and we're looking to expand the team so that we can build some of these systems.
- 16:58
Um, so if any of this work looks interesting to you and you're interested in being on like the edge of building some of this AI, like AI SRE product, uh, then just get in contact and let me know.
- 17:08
I'd love to chat. All right. Thank you. [audience applauding] [upbeat music]