Production software keeps breaking, and it will only get worse. Here's how Traversal is fixing it.

If AI makes writing software easier, what happens to the work of keeping it running? Anish Agarwal, CEO of Traversal, opens with three interspersed engineering activities: designing systems, developing their business logic and DevOps machinery, and troubleshooting them when production breaks. Matthew Schoenbauer, who joins him for the customer example, was the first person to join the company.

Tools such as Cursor, Windsurf, and GitHub Copilot reduce the effort spent on development. The attractive outcome is that AI handles both implementation and production troubleshooting, leaving engineers more time for the creative decisions of system design. But accelerating implementation does not, by itself, automate the investigation that follows a failure.

Agarwal predicts a less pleasant outcome if troubleshooting remains manual. Engineers will have less firsthand context about code that AI wrote, while the ease of writing more software will encourage increasingly complex systems. More complexity and less human context make production failures harder to explain. In that scenario, time saved during development moves into QA and on-call work instead of design.

Observability products such as Grafana, Datadog, Splunk, Elastic, and Sentry process telemetry and make it visible. That gives engineers many views of system health, but it leaves them with a search problem when something breaks. Agarwal calls the first stage “dashboard dumpster diving”: multiple people search dashboards and logs in parallel, hoping to find a promising lead.

A lead is only the beginning. The investigation then moves through a recurring sequence:

1. Find a symptom worth following. A dashboard or log suggests what went wrong.
2. Connect it to a change. Examine a pull request, configuration change, or another modification that could explain the symptom.
3. Inspect the code. Look for the mechanism connecting that change to the observed failure.
4. Expand the investigation. When the explanation remains unclear, bring in more teams and repeat the search.

Agarwal describes incident channels swelling from dozens of people to as many as a hundred. Each additional team can contribute expertise, but the loop still depends on people discovering which evidence matters and assembling the explanation.

Traditional AIOps applies machine learning and statistical anomaly detection to this problem. Agarwal’s objection is that a numerical representation can fail to capture the behavior of a complex, changing production system. The result, in his experience, is excessive false positives: a flood of alerts with very little guidance about which one explains the incident. Detecting unusual behavior does not necessarily identify the change that caused it.

An LLM addresses a different part of the job. Paste a selected log into ChatGPT, and it may help explain the message. But that assumes someone has already found the relevant log. Agarwal describes production environments with petabytes of telemetry and potentially a trillion logs: selecting the evidence is the hard part. A larger context window does not remove the storage, memory, and retention constraints of that data. He also distinguishes the models’ semantic understanding from their weaker handling of numerical structure.

An agent can search by choosing tools, examining their results, and deciding what to do next. Agarwal discusses ReAct-style agents through the example of implementations guided by runbooks or a higher-level workflow. ReAct itself interleaves reasoning and actions; it does not require a fixed runbook. The operational objection here is to depending on instructions that, in the team’s experience, can already be stale when written. A changing system makes an old investigation path a poor guide to a new failure.

Removing that prescribed path creates another problem: a broad search through simple tools can become a long sequential loop. Agarwal warns that such a search could run for days or time out, while he sets a useful incident-response target of two to five minutes. This motivates combining approaches rather than expecting any one to solve the entire investigation.

Traversal’s goal is out-of-sample troubleshooting: investigating a new incident from first principles, without needing a previously solved example of that failure. Its first ingredient is causal machine learning. A production failure often creates other failures around it, so many signals move together. The task is to distinguish the initiating cause from the correlated downstream effects.

The second ingredient is semantic reasoning over log fields, metric-source metadata, and code. Numerical analysis can expose relationships in telemetry; reasoning models can interpret what the fields and operations mean. The third ingredient is the control flow that brings those capabilities together. Agarwal describes swarms of agents making thousands of parallel tool calls to search telemetry efficiently and exhaustively. That is the architecture’s stated approach, rather than a disclosed algorithm or measured coverage guarantee.

The components address different bottlenecks: interpreting numerical behavior, understanding its meaning, and exploring enough evidence within an incident’s time constraints. Agarwal attributes the combination to the team’s research and broader recent work, without naming particular papers in this passage.

The architecture maps back onto the manual loop. Agent swarms and causal analysis look for promising leads in the telemetry. Code agents and vector search then help connect those leads to pull requests or change logs. As the investigation assembles context, it can bring in the appropriate team with an explanation of why that team is needed. The intended improvement is not simply fewer dashboard clicks: it is a more informed handoff between evidence discovery, change investigation, and human action.

Schoenbauer’s motivation came from high-frequency trading. He wanted to spend his time writing good code, but found himself repeatedly debugging production incidents. Eventually he quit. His introductory slide places him inside the familiar fire cartoon: the frustration is having to keep working while the surrounding system burns.

Later, in a causal AI class, the instructor mentioned that causal methods might automate incident-remediation workflows. Schoenbauer approached him about a research project and discovered that he had already started a company to pursue the idea. That conversation became an invitation to join Traversal. Roughly a year later, Schoenbauer presents the customer work that followed.

The customer is DigitalOcean, a cloud provider serving hundreds of thousands of customers each day. Schoenbauer asks the audience to imagine a productive coding session interrupted by a report of a serious customer issue. The example message raises a possible host compromise associated with an application. He presents redacted customer material, explicitly framing it as an account of real work rather than a staged demo.

An engineer enters an incident Slack channel alongside dozens of colleagues, all trying to locate the cause. Schoenbauer describes hundreds of millions of metrics across thousands of dashboards, plus tens of billions of logs from thousands of services. The decisive evidence might be a single log. That asymmetry—an enormous search space and a very small useful result—is what makes the manual investigation so expensive. In his description, even a successful search can consume a few hours before everyone returns to work, until the next incident restarts the cycle.

Schoenbauer reports approximately a 40% reduction in DigitalOcean’s mean time to resolution, measured as the time to find and resolve production incidents. The talk does not state the measurement period, incident count, or comparison methodology. He connects the avoided minutes to both engineer effort and downtime costs, describing the latter as thousands of dollars per minute without providing a cost calculation.

In the workflow Schoenbauer presents, Traversal begins investigating when an incident starts. It receives the same small amount of initial context that the engineers receive, then orchestrates a swarm of AI SRE agents to search observability data in parallel. The search happens in the background while engineers remain in their incident channel.

In the presented case, Traversal returned its findings to the incident Slack channel after about five minutes. It identified a deployment whose changes had caused cascading issues across the system. That finding gave engineers a concrete remediation option: roll back the deployment. Schoenbauer reports that DigitalOcean engineers confirmed the finding as correct and as the explanation that solved the issue. The described action belongs to the engineers; the case does not claim that Traversal autonomously performed the rollback.

Engineers who need more than the Slack summary can inspect the investigation in Traversal’s UI:

• Evidence: relevant observability data is cited alongside the findings.
• Candidate causes: possible root causes have confidence levels and accompanying reasoning.
• Impact: an interactive, AI-generated map lets engineers explore the affected system.
• Follow-up questions: engineers can ask how the incident affects the part of the stack they care about.

This lets an engineer move from a proposed explanation to its supporting evidence, then narrow the investigation to their own operational responsibility.

The broader deployment problem is heterogeneous: different enterprises use different observability tools, and the investigator has to work across those systems. Schoenbauer describes broad integration coverage and an operating scale reaching trillions of logs. At that scale, incident investigation is an AI infrastructure problem as much as an agent problem. Choosing the next investigative action is only useful if the system can also retrieve and process the evidence behind it.

The underlying search pattern extends beyond production debugging: a large dataset contains a small piece of information that explains what matters. Schoenbauer points to network observability and cybersecurity as other domains where parallel agents and exhaustive search could be useful. These are proposed applications of the same pattern, not additional customer results presented in the talk.

The closing invitation reflects the range of work involved. Traversal’s team combines AI researchers, people from developer-tool companies, AI product engineers, and practitioners from high-frequency quantitative finance who understand the cost of downtime. Schoenbauer emphasizes mutual support and enjoyment of working together, then invites interested people to join. The aspiration returns to the opening problem: giving engineers more of their working lives back from production incidents.