Lawrence Jones - Fighting AI with AI

How do you judge an AI investigation that tells you what broke in production and what to fix? At incident.io, that question grows out of an incident response platform that pages engineers, helps coordinate incidents, and supports customer communication. Lawrence Jones, a founding engineer, describes customers including Netflix, Etsy, and Skyscanner. The ambition extends beyond managing major incidents: the team wants to automate production investigations, including questions that begin as routine tickets.

At the time of the talk, Jones says the team had spent roughly 18–24 months building these systems. An investigation runs hundreds of queries across logs, metrics, traces, and historical incidents, then cross-references the evidence with the codebase to propose a cause and a fix. The report on screen makes that result concrete: it describes a PostgreSQL query failure, asks why it happened now, and considers whether a fix is in progress.

Reading the answer is much easier than establishing whether it is correct. A reviewer may need to reconstruct the incident, inspect what responders did, and read a postmortem before deciding whether the investigation was right—or even useful. Jones estimates that understanding an incident well enough to judge its report normally takes about an hour. Behind that report are hundreds or thousands of prompts, operating across customer environments with different failure modes. The debugging problem is understanding the system that produced the answer, not merely reading the answer.

Jones’s earlier LDX talk, Becoming AI engineers, covered the underlying constructs: prompts, evals, scorecards, traces, datasets, and backtests. Here, those are the starting point. The next layer makes eval datasets easier for agents to maintain, exports debugging interfaces as file systems for Claude Code and Codex, and uses agents to run repeatable analysis pipelines.

An eval is a unit test for a prompt: supply input, execute the prompt, collect its output, and apply grading criteria to decide whether it passes. Incident.io keeps evals in YAML files beside the Go code defining its prompts; its AI work, like the rest of the application, is written in Go. Engineers run those tests before merging prompt changes.

The deliberately simple demonstration translates a message into pirate speak. It has two independent requirements: the output should sound like a pirate, and it should preserve the input’s meaning. A colorful translation that changes the request still fails. A faithful translation with no pirate style fails the other criterion.

The demonstration shows three YAML cases and their execution results. A compact fixture following the same two-criterion design could look like this:

criteria:
  - id: pirate_style
    description: The output sounds like pirate speech.
  - id: meaning_preserved
    description: The output preserves the meaning of the input.

cases:
  - name: meeting_time
    input: "Meet me at the harbor at noon."
  - name: missing_supplies
    input: "We have no drinking water left."
  - name: keep_the_warning
    input: "Do not open the door until I return."

Keeping the criteria separate makes the intended behavior legible without prescribing a single acceptable wording.

Real incident fixtures are less tidy. The behavior an engineer needs to reproduce may depend on almost an entire incident report. Incident.io added a button that exports a problematic production interaction into the repository as an eval, making it possible to test against the actual input that went wrong. That solves capture, but it does not solve maintenance.

A useful unit test communicates a focused expectation. A production export may instead bury that expectation in a huge YAML document. Jones uses a two-megabyte YAML fixture to illustrate the problem: reading and modifying whole suites can exhaust a coding agent’s context before it has done useful work.

The team’s internal Eval Tool provides a smaller interface to those files. Its CLI can list test cases and edit, replace, or add an individual case. The agent can therefore work on the relevant fixture without loading the entire suite into its context. That access pattern also makes it possible to give the agent a runbook for changing prompts.

Packaged as a runbook or a skill, the repair procedure gives the agent a sequence to follow:

1. Inspect the prompt and the requested behavior.
2. Create an eval that demonstrates the current failure.
3. Modify the prompt until that eval passes.
4. Run the other evals to check for regressions.
5. Consolidate and simplify the prompt so repeated repairs do not leave it swollen with instructions.

The final step matters because a succession of individually reasonable additions can produce a prompt that becomes difficult to understand and maintain.

Jones demonstrates this in Claude Code with a real prompt that converts human requests into Loki log queries. The agent adds an eval, runs it repeatedly, and assesses whether its pass rate is acceptable. The talk does not specify the repeat count or acceptance threshold. This workflow makes a known prompt easier to change with regression checks; it does not yet tell the engineer which prompt caused a bad interaction.

A production interaction rarely belongs to one prompt. Jones maps the incident chatbot as a hierarchy of prompts, tools, and agents. Roughly ten agents are visible, and the full graph does not fit on screen. A customer can identify a bad response without knowing which component produced the mistake. The red–green eval loop is useful only after that component has been located.

Investigations add another layer. Each step in the investigation timeline expands into a trace; individual green blocks can themselves contain hundreds of prompts and tool calls. A subtle intermediate error can propagate until the final root-cause analysis presents a completely wrong explanation.

The team built interfaces to help humans inspect these executions. Those interfaces expose the details, but engineers still lack the time to follow every branch, and coding agents cannot readily use the same UI tools. The missing capability is a way to make the existing debugging evidence accessible to an agent.

Claude Code’s ability to navigate file systems with standard tools suggested a direct solution: export the content behind the debugging UI as a file system. Incident.io downloads an interaction into a sandboxed Claude Code environment. The package includes the inputs to the prompts and a self-documenting structure; the agent also has access to the application codebase. It can inspect what happened and identify where a behavioral change belongs.

Once the agent locates the relevant prompt, the earlier eval loop becomes useful again. Nor does exporting the UI require discarding its structural information: Jones shows that a trace can be represented in ASCII text, preserving the relationships that the interface displayed visually. The agent can consume that structure alongside the underlying prompt content.

The resulting debugging session follows a practical sequence:

1. Download the interaction associated with a bad experience into the sandbox.
2. Ask the agent to inspect it and explain what it thinks went wrong.
3. Describe the desired behavior and ask which part of the tool and prompt hierarchy should change.
4. Use the available codebase to make the change.
5. Verify it with the eval runbook.

Diagnosis and repair now happen in the same session, with the human specifying the intended behavior and the agent tracing its implementation. Incident.io has added these file packages to multiple kinds of AI interaction.

Individual debugging sessions still do not cover the whole system. Jones describes running thousands of investigations daily across hundreds of customer accounts. A backtest batches investigations against incident.io’s own account and customer accounts so the team can track whether the system is improving.

Jones’s example backtest reports 86% accurate root-cause analysis on incident.io’s own account. The talk does not supply the dataset size, grading rubric, or evaluation window. More immediately, the aggregate cannot explain why the score rose or fell, or what needs to change for a particular customer.

The team exports the investigation batch as files and feeds it to a Claude Code analysis pipeline. An internal repository called Scrapbook contains Markdown playbooks describing how the agent should inspect the downloaded evidence and move through the analysis. This turns an open-ended debugging task into a repeatable process.

The process first fans out into independent investigation analyses; Jones gives an example of about 25 agents running in parallel. It then clusters those findings into cohorts with similar failures. The resulting report explains patterns behind an account’s performance and proposes improvements, rather than simply repeating the aggregate score. Jones says the team has applied this approach across several internal systems.

A useful analysis pipeline keeps its work, not just its final answer. Subagents analyze individual investigations in parallel and save incremental findings in files inside the downloaded package. Those files let the workflow stop and resume. Giving the analysis access to the codebase also lets it connect a diagnosed failure to an implementation location and propose a concrete change.

The coding agent can then implement that proposal in the same session and use the eval red–green process to check the repair. Jones shows a merged pull request following analysis of a couple of failing backtest investigations: PR #51443 feeds previous hypothesis-review feedback into the next hypothesis build. The example makes the handoff tangible—investigation evidence leads to a discussion of a feature change, then to a code change. Deployment and testing in production complete the loop.

Internal debugging tools deserve the same deliberate use of AI as the product itself. For a complicated AI system, that means making the evidence usable by the coding agents engineers already work with. Jones favors bulk file downloads because an agent can grep through the information and retrieve the details it needs. He judges an MCP layer or computer-use approach less effective for this workflow; he presents that as an engineering judgment, without a measured comparison.

Recurring complex analysis is also a candidate for an AI runbook. Jones argues that capturing the procedure can save days or weeks of work. He closes with an invitation to help build these systems: at the time of the talk, incident.io was expanding its London team after a substantial raise the previous year and hiring for its AI SRE work.