# The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal

Eric Schwartz, Traversal

AI Engineer Code 2025

[Talk page](https://ai.engineer/talks/y-OVWZD4j6U-5-levels-self-driving-production) · [Video](https://www.youtube.com/watch?v=y-OVWZD4j6U)

## The 5 Levels of Self-Driving Production

Eric Schwartz explains Traversal’s path from manual incident response to diagnosis across an enterprise and, eventually, verified fixes. The hard part is connecting a visible failure to a distant cause quickly enough to help the on-call team.

### Faster development leaves more production to understand

[0:12](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=12s)

Coding agents shorten the time needed to write software. The hoped-for result is more time to design it. Eric Schwartz, product manager at Traversal, opens with a different observation from the enterprises his team serves: more engineering time is going into troubleshooting. Code reaches production faster, while the people responsible for it may understand less of what has changed.

![Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 127 seconds](https://ai.engineer/api/talk-essay-media/eea4765aa55f6f9de046906fe2bfc13a1005e66066b03aae94583c7b49c7ac6c)

A slide contrasts faster development with growing troubleshooting demands.

The software development life cycle has three broad jobs in this account: decide what to build and how to structure it, write the code, then fix what happens when that code meets the real world. Accelerating the middle job does not automatically accelerate the last one. More code creates more behavior to understand and more interactions to debug. Schwartz includes himself among people who can now contribute production code without formal computer science training; the accessibility is useful, but it also widens the gap between producing a change and understanding its consequences.

Schwartz cites estimates of upwards of $400 billion in annual enterprise spending, 40% of executives identifying troubleshooting as a team problem, and engineers losing seven-plus hours each week to troubleshooting while on call. Those figures come without a named study or measurement method in the recording, so they serve as motivation rather than a precise estimate of the cost attributable to coding agents.

### A failing checkout API can be five hops from its cause

[2:59](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=179s)

Observability tools such as Datadog, Elastic, Splunk and ServiceNow expose failures and correlations. The unresolved job is explaining why the failures happened and what to do next. Schwartz recalls the recurring complaint from his work on ServiceNow observability products: “You're telling me what's broken. You're not telling me why, and you're not telling me what to do about it.” More dashboards can display more symptoms without connecting them into an explanation.

Consider the customer-derived example of a failing checkout API. The visible symptom is at checkout, but the investigation may need five to ten hops through dozens of services and petabytes of data. An expired TLS certificate five hops away is the possible cause Schwartz names. The example does not specify the intervening services or establish the certificate as a confirmed diagnosis; its point is the distance between where a failure appears and where an investigation must look.

![Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 286 seconds](https://ai.engineer/api/talk-essay-media/0f2ccd288dadf8285a2e7a8bb35e1d7e610ed5e349f5c8fb42592e70f5beb60c)

A checkout API failure sits within a dense map of connected services.

That distance becomes an organizational problem. In a small environment, a seasoned site reliability engineer might know the whole stack. In a large enterprise, engineers hold different slices of context. Bringing fifty of them into a war room assembles those slices, but takes time and interrupts many people. Pairing one engineer with a coding assistant still leaves the problem of obtaining context across the environment.

Traversal’s premise is that root cause analysis requires causal reasoning: finding the cause-and-effect relationships that explain the observed failure. Correlated failures tell an investigator where to look; an explanation must connect them to what produced the incident. Schwartz also cites Google and Anthropic as recognizing the difficulty, including limitations of LLMs at finding root causes. The technical direction he develops is to give agents a model of production relationships and a way to search through them.

### From war rooms to a closed diagnosis-and-fix loop

[6:25](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=385s)

Self-driving production is the proposed closed loop: detect an issue, find its root cause, put up a fix and verify that the issue has been fixed. The ambition is to complete that loop without paging the team. Verification matters because a diagnosis or proposed change alone does not establish that production has recovered.

The title’s five levels sit above a manual level zero. Schwartz uses the analogy of autonomous driving to describe increasing capability:

- **Level zero: manual response.** Engineers assemble in Slack or Zoom and spend hours debugging.
- **Level one: rules and automations.** A structured workflow responds to a known alert. Its weakness is novelty: a severe incident may have no existing runbook.
- **Levels two and three: increasing automation.** Level two receives no separate operational definition in the talk. At level three, a homegrown agent may debug a particular service or set of alerts well.
- **Level four: diagnosis across the environment.** The investigation spans hundreds of services and repositories and thousands of log indexes.
- **Level five: fixes and verification.** The system combines enterprise-wide diagnosis with putting up fixes and verifying them.

![Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 476 seconds](https://ai.engineer/api/talk-essay-media/3b170ce790b292b333d849295e2821abaa78bd75fd84b053569bb342bd0a157f)

An autonomy diagram shows increasingly broad production capabilities, from manual troubleshooting to self-driving production.

The largest jump in this framework is from a service-specific agent to one that can investigate across an enterprise. A local agent can work within a familiar slice of the system. An environment-wide investigator must cross the divisions that otherwise require many teams to pool their knowledge. Level five adds a further responsibility: carrying the diagnosis through a change and checking the outcome. Schwartz presents it as the destination, rather than claiming every customer already operates at that level.

### Different production jobs need different outputs

[8:47](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=527s)

Traversal serves companies including American Express, Pepsi, DigitalOcean and Capital One. Schwartz describes environments generating trillions of logs and spans and tens of billions of metrics and events, and reports “eighty percent plus” root cause performance for high-severity incidents at that scale. The recording does not define the scoring criteria or evaluation population for that result, so it should be read as a reported product outcome rather than a benchmark for comparing systems.

The product work extends beyond answering a single incident question:

- **Alert intelligence** triages and prioritizes the hundreds or thousands of alerts an on-call team may receive in a day.
- **Incident root cause analysis** aims to pinpoint a cause in minutes instead of requiring hours and dozens of engineers.
- **Self-healing** adds a fix to address or mitigate the diagnosed issue.
- **Production support** answers everyday questions about production and retrieves specific data points, including work that never becomes a major incident.
- **Code resilience** examines changes before production with the goal of making the system more reliable over time.

![Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 603 seconds](https://ai.engineer/api/talk-essay-media/e0446466b47eb59764742323c1112572cc0a18865a6dab930cfc13fe5679559b)

A slide lists alert intelligence, incident RCA, self-healing, production support, and code resilience.

These jobs ask for different kinds of help. Alert intelligence changes what deserves attention; incident analysis changes what teams know about a failure; self-healing changes what the system does about it. Schwartz develops the first two through customer examples, leaving the other capabilities at the level of use-case descriptions.

### Pepsi: turn an alert backlog into decisions

[11:06](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=666s)

Pepsi’s supply-chain software helps move raw materials and finished products, including the critical passage from warehouse to trucks to retailers. Before Traversal, the team received thousands or tens of thousands of alerts each week. A single engineer could have a backlog of 700 alerts. The operational difficulty was deciding where to look: important failures could slip through the noise, while trying to inspect everything risked burnout.

The change is a filtered, prioritized set of alerts that Traversal has already investigated. Investigation happens before the engineer must choose what to open. That creates several possible decisions instead of treating every alert as equally urgent:

- **Investigate now.** Flag alerts worth an engineer’s attention.
- **Change the alert rule.** Identify opportunities to reduce noise by updating the underlying rules in code.
- **Dismiss for now.** Set aside alerts that do not currently need attention.
- **Schedule later work.** Create tickets for technical debt that matters but does not require immediate response.

The causal sequence is practical: a large undifferentiated queue makes attention scarce; pre-investigation supplies context; prioritization directs immediate attention; rule changes and deferred tickets address the remaining work differently. The reported outcome is a more useful queue, with no replacement backlog count given. This is progress toward autonomous production through better triage, even before the system fixes an incident itself.

### American Express: investigate before paging broadly

[13:06](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=786s)

The American Express example starts with an incident response that pages five to ten teams and twenty to fifty engineers. The particular incident comparison uses sixty minutes to resolution, although Schwartz says incidents can take hours or days. He names inability to pay a credit card bill or log into the mobile app as examples of the stakes.

In the workflow Schwartz describes, Traversal becomes the first responder whenever American Express declares an incident, internally called a bridge. Declaration dispatches the system. Within three minutes, it posts a detailed root cause analysis in the incident’s Slack channel; it can also update ServiceNow tickets. The initial explanation then informs who needs to be called.

That order changes the response. Rather than assembling many teams to discover what broke, responders start with an analysis and may page one or two teams to verify it—or no teams. In the specific comparison, Schwartz describes avoiding pages to fifty-plus engineers. The three-minute figure is time to an analysis, not time to a verified repair; the demonstrated benefit here is a narrower, less chaotic response.

![Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 858 seconds](https://ai.engineer/api/talk-essay-media/bec4d6b6043352a3007569bc1a6efbf38ef33156f98a329e4f9a388d0e965a8d)

A slide shows an incident timeline with an analysis followed by a smaller response flow.

What changes when investigation precedes broad escalation? The flow below shows how the initial analysis becomes an input to the paging decision. Human verification remains part of the described workflow, while fewer people need to enter the incident just to locate its cause. For incidents in the middle of the night, that is also a concrete reduction in interrupted sleep.

Analysis informs escalation at American Express
The reported three-minute output is a root cause analysis. It helps decide which teams, if any, should verify the findings.
- Incident declared — American Express declares a bridge and dispatches Traversal.
- Root cause analysis in Slack — Traversal posts a detailed analysis within three minutes; ServiceNow updates are also possible.
- Focused paging decision — No teams, or one or two teams to verify findings, instead of broad escalation.
- Incident declared → Root cause analysis in Slack (Dispatch and investigate)
- Root cause analysis in Slack → Focused paging decision (Use the initial findings to narrow response)

### Five questions to ask an AI SRE

[14:42](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=882s)

The closing evaluation questions return to the checkout example’s central difficulty: reaching a distant cause through a large, changing environment. Schwartz proposes testing the operational requirements that make such an investigation useful:

- **Data coverage:** Can the AI SRE see all the production data it needs? Missing slices of context can block a detailed diagnosis.
- **Search cost and load:** Can it search petabytes of data and hundreds of billions of logs without excessive cost or overwhelming observability infrastructure?
- **Relationships:** Can it map how the entities in that data relate? Reading data is insufficient if the system cannot connect it.
- **Knowledge maintenance:** Does its understanding improve autonomously, or does the team have to maintain Markdown files and repeatedly supply engineering help?
- **Multihop speed:** Can it reach a non-obvious root cause far from the symptom in minutes?

Speed is a condition for adoption during an incident. Schwartz’s observation is that after more than five minutes, on-call engineers lose patience and return to their established habits. An investigation therefore has to be useful before the team has already committed to another way of working. The American Express three-minute analysis illustrates the kind of response window he has in mind.

### A production world model gives causal search somewhere to go

[16:42](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1002s)

Traversal’s final architectural sketch separates three jobs. The bottom of the stack integrates with production data and makes it available for analysis while aiming to control cost. The middle is the production world model, which maps relationships in that data. The top packages the resulting capability into useful workflows and experiences, including alert intelligence and incident analysis.

How does production data become an incident answer? The stack diagram makes the intermediate relationship model visible. Data access supplies material for an investigation; the world model organizes the connections; the causal search engine gives agents a harness for searching through the mapped data. The output reaches engineers through the use case they are working on.

![Selected presentation frame from The 5 Levels of Self-Driving Production — Eric Schwartz, Traversal at 1017 seconds](https://ai.engineer/api/talk-essay-media/3a0669b63ee6ad34d7b0dbfc79a02486a160a51e6c53e0cf4296a300f73634d5)

A layered diagram labels data integration, the production world model, and Traversal’s SRE platform.

The recording describes these responsibilities without explaining how relationships are learned, how causal hypotheses are tested, or how search controls infrastructure load. It also does not develop the execution and verification procedure for autonomous fixes. The production world model and causal search engine are the named foundations for Traversal’s proposed route to level five: understanding an enterprise-wide failure well enough to fix it and verify recovery.

From production data to useful investigations
The world model represents relationships; causal search lets agents investigate the mapped data; product workflows deliver the result.
- Production data integrations — Connect to data and support analysis with attention to cost.
- Production world model — Map and make sense of relationships in integrated data.
- Causal search engine — Give agents a harness to search through the mapped data.
- Team workflows — Package findings into useful experiences such as alert intelligence and incident analysis.
- Production data integrations → Production world model (Organize relationships)
- Production world model → Causal search engine (Search mapped data)
- Causal search engine → Team workflows (Package investigation results)

## Key takeaways

- Faster code generation can increase troubleshooting work when production grows faster than engineers’ understanding of it.
- Enterprise root cause analysis requires connecting symptoms across services and teams; a failing checkout API may be several hops from its possible cause.
- Level four extends diagnosis across an environment. Level five adds fixes and verification, completing the proposed production loop.
- The Pepsi and American Express examples change the order of work: investigate alerts before asking engineers to triage them, and analyze incidents before paging broadly.
- Evaluate an AI SRE on data coverage, search cost and load, relationship mapping, autonomous knowledge maintenance, and fast multihop investigation.

## Slides

### Self-Driving Production

Eric Schwartz introduces how AI agents change the software development life cycle, the problems they create for engineers, and where AI for site reliability engineering fits.

[0:18](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=18s) [Traversal title slide introducing self-driving production, presented by Eric Schwartz.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5e4402e9b49c2ef54ab1bda37a4aee605604771632416df47fc2dfbd494f358.webp)

Traversal

Self-Driving Production:
AI Wrote Your Code. AI Should Fix It Too.

AI Engineer World’s Fair
June 30, 2026

Eric Schwartz
Founding Product Manager, Traversal

### Three Core Engineering Workflows

Schwartz divides engineering into designing what to build, writing the code, and troubleshooting problems as software interacts with the real world.

[0:57](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=57s) [Three boxes identify system design, development, and troubleshooting as software engineering workflows.](https://assets.ai.engineer/sites/aiecode2025/sha256/01adfcba95ce55eaf70236e31adddba9c1944ac431b35066467ed73fc38349fe.webp)

Software engineering breaks into three core workflows.

System Design
Development
Troubleshooting

### AI Compresses Development Time

Coding agents shorten development time, raising the question of what happens to system design and troubleshooting.

[1:29](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=89s) [Development is highlighted between dimmed system design and troubleshooting boxes, with coding-agent logos and a question about the other workflows.](https://assets.ai.engineer/sites/aiecode2025/sha256/b60589c9fc746d6718789a9c3547455dec823cde8a707c3fe4a6dba6fa0431fc.webp)

AI dramatically compresses time spent on
software development...

System Design
Development
Troubleshooting

?
CLAUDE CODE
Codex
CURSOR
?

But what about the other two workflows—system design and troubleshooting?

### Production Troubleshooting and On-Call Toil

Schwartz says the enterprises he works with spend more time troubleshooting as more code reaches production with less human understanding and environments grow more complex.

[1:59](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=119s) [Troubleshooting is highlighted; the slide argues that less human context and greater system complexity will make it more time-consuming.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff2c5c71ae7637cb9644842fc3f64d8cb935be459d31126f6838524d5caede26.webp)

...but production still breaks down the old way.
AI is pushing engineers deeper into on-call toil.

System Design
Development
Troubleshooting

The gap will keep widening, making troubleshooting even more time-consuming:
• Code is written by AI so humans will have less context
• Systems will become even more complex

### The Cost of Production Failures

The slide pairs downtime’s financial cost with its operational burden: Global 2000 companies lose $400B annually, 41% of executives say customers first detect downtime, and engineers lose 7 hours a week troubleshooting.

[2:27](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=147s) [Three statistics report $400 billion in annual downtime losses, 41% of executives saying customers first detect downtime, and seven hours per engineer per week lost to troubleshooting.](https://assets.ai.engineer/sites/aiecode2025/sha256/a33ee32e593aca4160ae484a8b9a024171d1b0b953a8d8a40ef3cce7ec4e3ce6.webp)

Production failures are already costing you billions.

$400B
lost by Global 2000 companies annually to downtime

41%
of executives say downtime is first detected by customers

7 hours
lost per engineer each week to troubleshooting

Sources: Splunk & Oxford Economics, Hidden Costs of Downtime (2024); New Relic, 2025 Observability Forecast; GitLab, 2025 Global DevSecOps Report.

### AI and Production Complexity

The slide contrasts rising complexity—with more code, telemetry, alerts, and incidents—against human troubleshooting capacity. Schwartz argues that AI coding tools add to that complexity.

[2:49](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=169s) [A rising complexity curve contrasts human troubleshooting capacity with more code, telemetry, alerts, and incidents.](https://assets.ai.engineer/sites/aiecode2025/sha256/6487d5af717a266ea370049d6a82d515c8e2b2d28f5c6f2f17e393b749e86c99.webp)

AI is exponentially increasing production complexity.

← Pre AI    AI →

CURSOR

More code
More telemetry
More alerts
More incidents

CLAUDE CODE
Codex

Complexity
Human troubleshooting capacity
Humans cannot keep up

### Observability and causation

A rising curve fills with dashboards and tool logos as Schwartz argues that observability identifies broken or correlated signals but does not explain their root cause.

[3:10](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=190s) [A slide says observability shows correlation but cannot explain causation, alongside observability and AI tool logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/9bda4272646b63ff2180b14a9276986c8b05962c346d610f5ffb37dd7b1dec54.webp)

Observability can’t keep up.

It shows correlation.
It can’t explain causation.

CURSOR
DATADOG
dynatrace
servicenow
splunk>
CLAUDE CODE
bigpanda
Codex

### Correlation is not causation

Google’s SRE book describes the difficulty of finding a failure’s cause. An Anthropic SRE lead says LLMs struggle to distinguish causation from correlation, a challenge for humans too.

[3:58](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=238s) [A slide presents quotations attributed to the Google SRE book and an Anthropic SRE lead about the difficulty of identifying causes.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad375ef44706de0a2f3884fe7eb0a7ff50d5f547dda1fc9931b543dd1d88be91.webp)

Don’t take our word for it. Correlation ≠ Causation.

Humans alone can’t solve it with observability tools

Monitoring and alerting enables a system to tell us when it’s broken... “In a distributed system, it is often difficult to determine the cause of a failure.”

– Google SRE Book
Google

LLMs alone can’t solve it

“...We can't trust LLMs for incident response...The problem is its inability to "step back and start discerning between causation and correlation... For us humans, it is hard as well."

– Alex Palcuie, Anthropic SRE Lead
ANTHROPIC

### Tracing an incident to its root cause

A human and LLM trace a checkout API failure through service dependencies to a possible expired TLS certificate five hops away.

[5:18](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=318s) [A service dependency diagram shows a human and LLM investigation tracing a checkout failure to an expired TLS certificate.](https://assets.ai.engineer/sites/aiecode2025/sha256/00787b15d76ba7e021c949d2520f5e9bf82bd58d6e41c78ffabd5bee1e04a114.webp)

Without production-wide causal reasoning,
investigations take hours, if not days.

Session-replica
Payment service
Incident alerts
Auth-service
Token cache
Order-service
Check-out API

Human & LLM investigation

One root cause =
expired TLS certificate

### The vision for self-driving production

Traversal’s six-level scale moves from manual troubleshooting to autonomous production. At L4, AI investigates across the stack while humans gate critical decisions; L5 adds autonomous remediation and prevention.

[6:24](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=384s) [A six-level scale progresses from manual troubleshooting through autonomous investigation to self-driving production.](https://assets.ai.engineer/sites/aiecode2025/sha256/27b436839b001574070a722428d090faa5e77802074a1a56812c25193890b31d.webp)

Our vision is self-driving production.

L0
Manual troubleshooting
Debugging via war rooms, dashboards, and whoever’s awake.

L1
Rules-based automation
Static alerts and runbooks handle known problems. Breaks on novel ones.

L2
AI summarization & context surfacing
AI summarizes what happened. Humans diagnose and fix.

L3
Single-domain investigations agents
Agents investigate within one slice. Cross-service, cross-layer root causes missed.

L4
Autonomous cross-environment investigation
AI reasons causally across the full stack. Humans still gate critical decisions.

L5
Self-driving production
The system knows your environment deeply enough to investigate, remediate, and prevent autonomously

### Enterprise scale and root-cause accuracy

Traversal reports over 80% root-cause accuracy in under five minutes for high-severity incidents. The slide lists 2T logs, 1T spans, and 20B metrics.

[8:46](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=526s) [Customer logos appear above data-volume figures and a claim of over 80% root-cause accuracy in under five minutes.](https://assets.ai.engineer/sites/aiecode2025/sha256/cef46eda7b6376ad89d449fc980e0ef57f9f533ca9665f679977ec8ae2c4bb92.webp)

Traversal has already enabled the world’s most complex enterprises to be at L4–L5.

AMERICAN EXPRESS
PEPSICO
Capital One
DigitalOcean
kraken

Accuracy that leads directly to remediation—in real time, at petabyte scale.

2T Logs
1T Spans
20B Metrics
20M Events
70K Changes
5K Applications

Identifying root cause with over 80% accuracy in under five minutes.

### Agentic SRE platform capabilities

Traversal groups alert intelligence, incident analysis, self-healing and support after production, with code resilience before production. The slide claims 85%+ MTTR reduction and 550+ engineering hours saved monthly.

[9:29](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=569s) [Five capabilities span alert intelligence through code resilience, with claimed reliability and savings metrics below.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c4b5be55684889a8bbc235c746c4b84941d3ed4f2e767541492e6f89739f43f.webp)

Traversal’s agentic SRE platform drives reliability across the entire software development lifecycle.

POST PROD
Alert intelligence
Incident RCA
Self-healing
Production support

PRE PROD
Code resilience

85%+
MTTR reduction

82%+
Accurate remediation

$10M+
first-year savings

550+
engineering hours saved monthly

AMERICAN EXPRESS
PEPSICO
Capital One
DigitalOcean
kraken

### Alert intelligence at PepsiCo

Traversal filters PepsiCo’s alert backlog into prioritized, pre-investigated signals. The slide claims 500+ engineering hours saved per month and 36% fewer incidents from automated triage.

[11:04](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=664s) [A dense backlog of alerts is compared with a filtered set of prioritized alerts, alongside claimed time savings and fewer incidents.](https://assets.ai.engineer/sites/aiecode2025/sha256/99731c00ff089358f32a7ef27a9ebadf4aa335fbf2fd9eb21f7ed138e78f780f.webp)

Your engineers are the alert filter.
That doesn't scale.

PEPSICO

Before & After: Alert Intelligence

Before Traversal

After Traversal
CPU saturation — middleware node
Anomaly surfaced 8 mins before incident threshold

Redis degrading
Preventable if actioned now

Order-service error rate rising
Downstream effect — monitor only

700+ backlogged alerts.
Critical signals get lost and incidents become inevitable.

Isolated to the alerts that actually matter.
500+ engineering hours saved per month and 36% fewer incidents from automated triage.

### Incident response before and after Traversal

American Express timelines compare 63 minutes and 58 engineers before Traversal with 25 minutes and 5 engineers after. Traversal shares root cause analysis at the start of the bridge for Mission Control to validate and execute.

[13:38](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=818s) [Before and after incident timelines compare response times and staffing, with claims of reduced resolution time and first-year savings.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e766451a6885efd13514a5eca2bbe40387d14c5414da7cef839722d248f562c.webp)

Every minute of downtime costs you.
We give those hours back.

AMERICAN EXPRESS

Before & After: Incident RCA

Before Traversal
Alert Fired — 10:00
INCI Created — 10:02
Bridge Started — 10:06
Failover traffic to IPC1 — 10:15
Db flush for Redis 12340 — 10:33
Recycle 60 pods — 10:48
Bridge Closed; Remediated — 11:09
2 min
4 min
63 min; 5 teams paged, 58 engineers involved in Slack

After Traversal
Alert Fired — 3:22
INCI Created — 3:24
Bridge Started — 3:28
Bridge Closed; Remediated — 3:53
2 min
4 min
25 min; 2 teams paged, 5 engineers
~3 min
Mission Control validates/executes; no RCA needed

Traversal RCA shared at start of the bridge:
Isolated impact to IPC2
Flagged Redis connectivity failures
Detected pod errors

Traversal’s value
Time to root cause decreased by 30+ minutes

MTTR cut from 63 minutes to 25
$10M+ in first-year savings across 1,000+ applications

### Five criteria for AI SRE

Gaps in production data make detailed root-cause analysis harder. Traversal says AI SRE must also search economically, map entity relationships, learn autonomously, and trace distant causes in minutes.

[14:40](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=880s) [Five cards ask about data coverage, reasoning at scale, production mapping, autonomous learning, and finding distant root causes.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e10d12301cdee46360a1f2f57a068172bec49f64c923cee8665cfdc4ad67d36.webp)

Solving causality at enterprise scale requires five new criteria for AI SRE.

Does it see all your production data without gaps?

Can it reason at scale without blowing up costs?

Can it map all of production—millions of entities—in real time?

Does it get smarter itself, or require a team of FDEs (Forward Deployed Engineers)?

Can it find root cause far from the initial symptom in minutes?

### Traversal’s Enterprise AI SRE Platform

Traversal captures and compresses production data, maps its relationships in a production world model, and surfaces it through capabilities such as incident RCA and self-healing.

[16:37](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=997s) [Platform diagram showing data capture, compression, a production world model, knowledge bank, causal search, and enterprise capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ef680c43143397f2187c4a83a8105b58963fac29b09682d423e67326e21baab.webp)

Traversal’s Enterprise AI SRE Platform.

AGENTIC ENTERPRISE CAPABILITIES
Alert intelligence
Incident RCA
Self-healing
Production support
Code resilience

Can it find root cause far from the initial symptom in minutes?
Does it get smarter itself, or require a team of FDEs?
Can it map all of production—millions of entities—in real time?
Can it reason at scale without blowing up costs?
Does it see all your production data without gaps?

Causal Search Engine™
Knowledge Bank™
Production World Model™
AI-Native Compressor™
Agentless Data Capture™

MELT telemetry · Source code · Deploys · Runbooks · Historical incidents · Jira

### Causal Reasoning at Scale

Traversal’s Production World Model maps integrated data, while its Causal Search Engine gives agents a way to search it. A highlighted path traces a route through connected nodes.

[17:30](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1050s) [A highlighted path through network nodes illustrates the Production World Model and Causal Search Engine.](https://assets.ai.engineer/sites/aiecode2025/sha256/f82d9bc209ec770ea33e3feaab0e662898db25c49fc7595ee80efa3cb9153319.webp)

We deliver causal reasoning at scale with our
Production World Model™ + Causal Search Engine™.

### Self-Driving Production

Traversal’s autonomy ladder culminates in L5, where a system knows its environment well enough to investigate, remediate, and prevent autonomously. Schwartz aims to help Fortune 500 enterprises reach that level.

[17:57](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1077s) [A progression from L0 to L5 highlights self-driving production at level five.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e706840920c6663e6d32ec7149078cd6abf72452c1b491a4a2d4a720eb0140b.webp)

Your future is self-driving production.
Traversal gets you there.

L0
L1
L2
L3
L4
L5
Self-driving production
The system knows your environment deeply enough to investigate, remediate, and prevent autonomously

### Thank You

The closing slide invites attendees to see Traversal in action at Booth #U-G11.

[18:11](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1091s) [Closing slide invites attendees to see Traversal in action at its booth.](https://assets.ai.engineer/sites/aiecode2025/sha256/6720976fafb585af60c09b05ffc82be313d539a8367d6972064ee4096b3f63cd.webp)

Thank You
See Traversal in action at Booth #U-G11

Traversal

## Full transcript

[0:12](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=12s) Hey everybody. Um, my name's Eric. I'm a product manager at Traversal, and today we're gonna talk about self-driving production. So just a quick primer on what I'll cover. Uh, I'm gonna talk a little bit about how AI agents are changing the software development life cycle, uh, the problems that that introduces for engineers, and how AI for site reliability engineering fits in, uh, which is where Traversal comes into play. And then I'll talk about a few

[0:42](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=42s) examples of, um, what this looks like in practice for enterprises that we work with at Traversal, uh, and a bit about how we built our AI SRE, uh, to deliver the results that we're seeing with Fortune five hundred companies. So, uh, I guess to start some quick context on, uh, kind of the different phases of software engineering. Uh, so sure you're all familiar, but at a high level, you could think about software engineering as kinda three

[1:11](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=71s) buckets. There's system design, like what is it that I wanna build? What's the architecture that I wanna build? There's development, which is actually writing the code, building the thing, and then there's troubleshooting. So once you build something and it's interacting with the real world, it starts to break, uh, and you need to fix it. And so with the advent of coding agents, the development portion has shrunk. Development is a lot faster. Folks like me without any formal computer science training can now contribute, uh,

[1:41](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=101s) production code. And so this is great. Development's a lot faster. Teams can do ten X more. But then what happens to either end of the spectrum? What happens to system design and what happens to troubleshooting? The hope is that you spend more time on the design part thinking creatively about what it is that you wanna build. But what we actually see with the enterprises that we work with is that, um, more and more time is being spent on troubleshooting. Uh, much more code is being written. People have

[2:11](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=131s) less understanding of the code that's being pushed into production. And so what you end up with is more issues, more comp-- uh, more complex environments. Uh, and instead of spending all your time on the creative architecture and design, your teams are bogged down on troubleshooting.

[2:29](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=149s) And the, the data kinda bears this out. This is a big problem. Um, by some accounts, enterprises are spending upwards of four hundred billion dollars a year. Uh, forty percent of executives say that this is a problem that their teams face, and on average, engineers are losing seven-plus hours each week, uh, just troubleshooting when they're on call. And this is gonna become an increasing issue that we'll all hear more about, that we'll all experience more over time as tools like Claude Code and

[2:59](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=179s) Codex and Cursor, which are fantastic, um, uh, they're fantastic, but they result in more code and more complexity. And so what are we supposed to do about it? Um, there are all kinds of great observability tools. Tools like Datadog, Elastic, Splunk, ServiceNow, where I worked for several years before joining Traversal. Um, these are great, but, uh, there's only so much that they can do. Like, these tools will tell you that something is broken. They'll point out all the things that have

[3:29](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=209s) broken. Um, they'll point out things that are maybe correlated, but they won't tell you the root cause of the issue. And when I worked on observability products at ServiceNow, this was something I heard time and again from the customers that I served, which was like, "You're telling me what's broken. You're not telling me why, and you're not telling me what to do about it." And so no matter how many dashboards you create in these tools, it's not really helping teams keep up with the pace of innovation, uh, and complexity that's being added to their environments.

[3:59](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=239s) And this is not just something we believe at Traversal, this is kind of a widely held view. Um, Google, who kind of wrote the SRE handbook, like the Bible of site reliability engineering, uh, has commented on this directly, and we also heard recently from Anthropic, um, quotes where they described in their own words that LLMs are not great at this problem. They're not great at pointing out what is the root cause of an issue.

[4:27](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=267s) Now, like, why is this so hard? What makes this so challenging? Um, in this particular example that we have on the screen, like, which was based on something we saw with one of our customers, um, if a checkout API is failing, uh, you might need to make five to ten hops across, like, dozens of services and petabytes of data to actually get to the root cause. And so in a very small contained environment, perhaps there's a single seasoned SRE at your company that knows the full

[4:57](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=297s) stack and can debug everything and has all the tribal knowledge to sift through all your data. But if you're a Fortune fifty enterprise or a Fortune one hundred enterprise, there's no single engineer that has all of that context. And so what you end up with is a war room with fifty engineers, like, pulled in, uh, and many, many hours trying to debug this issue 'cause everyone has their own particular slice of context. No one has kind of the full scope and breadth of this. And so no-- it's really

[5:27](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=327s) challenging for a human or a human paired with Claude Code to get kinda the full spectrum of context needed to jump from a checkout API is failing over here, five hops away to the expired TLS certificate that maybe caused the issue.

[5:46](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=346s) And so that's kind of what motivated, um, our founders to, to launch Traversal. Fundamentally, the belief that we have at Traversal is that, uh, root cause analysis, um, is not an observability problem, it's a, it's a causal problem. And so that's why our four founders, three of them come from academia, one comes from quant finance, uh, and they've dedicated, like, decades of their lives towards causal machine learning, which is how do you

[6:16](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=376s) find cause and effect, uh, and have applied it to this problem because it's a massive one that enterprises are really struggling with.

[6:25](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=385s) And so that kind of brings us to the topic of this talk, which is self-driving production. Um, what we're building at Traversal is the ability for an enterprise to basically have a self-driving production environment. What that means is a closed loop system where, uh, as issues break, the AI system finds the issue, the-- finds the issue or the incident or the alert, finds the root cause of what caused that issue in the first place, puts up a fix,

[6:55](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=415s) verifies that it's been fixed, and the loop is closed with-- hopefully without ever having to page or disrupt any of your team members so that they can keep focusing on building. And this is a spectrum. Um, a lot of companies are not fully there yet. We like to think of it in terms of levels, almost like self-driving cars, uh, or autonomous driving. We think about it in the same frame. So level zero is where most companies are today, where everything is fully manual. You're pulling a bunch of people into a Slack

[7:25](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=445s) war room or onto a Zoom call. You're spending hours debugging things. Level one might be where you have some rules and automations. Uh, we're hearing a lot about, like, loops that teams are building now with tools like Claude Code or Cursor or Codex. Uh, and so these might be, like, rule-based automations where you can build out a very structured, specific workflow in response to an alert. Those rule-based automations kinda break down when it's a novel situation, and usually a sev one is something you haven't

[7:55](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=475s) seen before. There's not really a runbook for it. And so the rule-based automation breaks down there. And level two and level three is kind of where you start to get a little more automation built in. In level three, perhaps you have, like, a, a homegrown agent that is really good at debugging a s-- issues for a specific service or a specific set of alerts. But really where we see the leap is going from level three to level four, which is cutting across an entire environment. So

[8:25](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=505s) hundreds of services, like hundreds of repos, thousands of lo-log indexes. It's quite challenging to home grow a solution that can handle that. And level five is kind of the holy grail, which is not only can we diagnose issues across a full production environment for a large enterprise, but also put up the fixes and verify the fixes as well.

[8:47](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=527s) And so, uh, we're really proud that at Traversal we get to work with and serve some of the biggest companies in the world, companies like American Express, who I spend a lot of time with personally, uh, Pepsi, DigitalOcean, Capital One. Um, and it's really cool to see the volume and scale of data that Traversal is able to handle. So, you know, trillions of logs that are generated, trillions of spans, tens of billions of metrics and events, um,

[9:18](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=558s) and hitting eighty percent plus root cause, uh, for these high severity incidents at this scale is a technical feat that we're pretty proud of. And so beyond just these kind of results at a high level, um, as a product manager, I like to think about the use cases that we build for our customers. And so there's, there's a few different types of scenarios and use cases that we found that can deliver a lot of value. I'll dig into a few in more detail, but at a high level, there's alert intelligence, a product that I spend a

[9:48](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=588s) lot of time on, which is helping on-call teams triage and prioritize hundreds or thousands of alerts that they may be receiving in a single day, helping you deal with alert fatigue. There is incident root cause analysis, which is where we got started. How do we help you find the root cause-- pinpoint the root cause of an incident in minutes when it otherwise would have taken hours and dozens of engineers? There is self-healing, which is not only telling you what the root cause was, but putting up the fix to address and mitigate that

[10:18](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=618s) issue. There's production support, which is the, the variety of questions and issues that your team might be dealing with in any given day, wanting to chat with their production environment, get specific data points that maybe aren't part of a, a broader incident, but are still taking up a lot of time and toil for your teams. And then pre-production, there is code resilience. Like when you're putting up changes, how do you make sure that this code is making your system more resilient and reliable over time? And

[10:47](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=647s) so at Traversal, we cover kind of the whole gamut of use cases, um, but I'll dig into alert intelligence and incident root cause just to give you a bit of a more detailed sense of what that might looks like-- what that looks like, and I'm happy to answer any questions after about the others.

[11:06](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=666s) So for alert intelligence, the example that I have here is, uh, based on our engagement with Pepsi. And so Pepsi uses Traversal across a bunch of their different applications, but a very core one is for their supply chain. They have a lot of internal tooling and software to make sure that, um, the raw materials and the finished product can move through their supply chain. And a very critical piece of that is making sure that finished product gets from their warehouse to

[11:36](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=696s) their trucks to their retailers at the end. And, um, before Traversal, their team would get thousands or tens of thousands of alerts every single week. Uh, and at any given time, a single engineer might have a backlog of seven hundred alerts. This is kind of a crippling state to be in. Like, you have so much- Noise that you don't know where to look, you don't know what's broken, and things slip through the cracks. Uh, or you just try to keep up and burn out. And so this was a big problem

[12:06](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=726s) that Pepsi was facing, and they used Traversal, uh, to help parse the signal from the noise of these tens of thousands of alerts. And so now, instead of each engineer having a backlog of seven hundred alerts, they get a very pri- a prioritized and filtered set of alerts that have already been pre-investigated. We flag to them the alerts that are actually worth digging into. We flag to them opportunities to reduce alert noise, whether that's updating the underlying alert

[12:36](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=756s) rules in their code, uh, dismissing alerts that don't need to be addressed right now, uh, or creating tickets for alerts that are maybe technical debt that can be handled later on, but aren't super important to deal with at this moment. And so we've seen a lot of great results with Pepsi, uh, on this alert use case and, and many of our other customers as well. Um, the next example is around incident root cause analysis. And, and this is a case study from our engagement with Amex, um, where I've personally

[13:06](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=786s) spent a lot of time myself. Um, so before working with Traversal, what an incident looked like at American Express is anywhere from five to ten teams get paged, anywhere from twenty to fifty engineers. And it could take, in this particular example, sixty minutes, but it could take hours or days to get resolution to, to an incident. You can imagine if customers aren't able to pay their credit card bill, like that's a big problem. Or if they're not able to log into their mobile app,

[13:36](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=816s) that's a big problem. And so now, uh, Traversal is the first responder to every single incident that's created at American Express. And so what that looks like is the second the bridge is declared, that's the term they use internally, the second the incident is declared, Traversal is dispatched. Within three minutes, we'll post a very detailed root cause analysis in the Slack channel where that incident is being managed. We can post updates to ServiceNow tickets if they want to as well.

[14:06](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=846s) And rather than paging five teams and fifty-eight engineers, because you have that initial analysis, it's a lot less chaotic. You have a sense of what actually broke, and either no teams get paged, or maybe you page the one or two teams just to verify the findings that Traversal posted. Uh, and so we save, in this case, like fifty-plus engineers, uh, from the time and headache of being paged, uh, into this incident. Um, and that's especially appreciated when these incidents are happening in the middle of the

[14:36](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=876s) night. Like this is a, a good night of sleep that fifty-three engineers can have now.

[14:42](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=882s) And so those are, those are two of the use cases that we're especially proud of at Traversal, but like I mentioned, there's a bunch of others. Maybe in the last couple minutes, I'll just highlight like AI site reliability engineering is a very hot space. You've probably-- If you're familiar, you've probably heard of a lot of companies in the space or claiming that they're solving these problems. Um, we think that there's a few really key questions you should be asking, uh, that are critical to getting this right. So the first is,

[15:12](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=912s) can your AI SRE see all of your production data? If you're only giving it a slice, or if it has gaps in its understanding, it's gonna be very hard to get to a very detailed root cause. The second is, can it search through this data? Can be-- It, it's often petabytes of data, hundreds of billions of logs, um, without blowing up costs or taking down your observability infrastructure. Um, this is really not trivial to do at scale for a Fortune one hundred

[15:42](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=942s) enterprise. The third is, can it map out the relationshi- uh, relationships between all of these entities in your data? Even if you're able to ingest and read through all this data, if you don't know what to do with it and if you're lost, it's kind of worthless. And then four, does it get better and smarter over time autonomously without having to dedicate engineering resources to maintaining a bunch of markdown files or having a bunch of forward deployed engineers taking up your, your team's time to

[16:12](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=972s) maintain the system knowledge? And finally, um, can it make multiple hops? Can it find the non-trivial, non-obvious root cause that's far away from the initial symptom in a matter of minutes? What we've seen is like anything more than five minutes, and you've kind of lost the plot, you've lost the patience of the on-call team, and people will fall back onto their own habits. So it needs to be really good, it needs to be really fast. And I won't go into this in too much

[16:42](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1002s) detail, but this is kind of how we think about these five questions at Traversal. So at the bottom of the stack is how we think about kind of a-analyzing all your data, uh, without increasing costs. So basically, how do we ingest and map all of the data, uh, that we're connected to and integrate with all of that data? In the middle of the stack is how we make sense of that data. And so we call that our production world model. How do we map out all the relationships between all the

[17:12](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1032s) denti-- the data that we're ingesting? And then finally, how do we surface that and package that up in either use cases or user experiences that are valuable to your team? And so there's a lot of components to getting this right, uh, especially at Fortune five hundred scale.

[17:32](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1052s) And so, um, that's a bit about what we've been building at Traversal, what we're delivering at Traversal. The core pieces, like I mentioned, are the production world model, so how we map and make sense of all the data that we integrate with. And then what we call, uh, what we call-- refer to as our causal search engine. Basically, the, the harness and mechanism that we give our agents to search through all of that data, uh, that we've mapped out. And so the production world model and the causal search engine are how we

[18:02](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1082s) believe we are gonna help Fortune five hundred enterprises get to the level five of autonomy, fully self-driving production.

[18:12](https://www.youtube.com/watch?v=y-OVWZD4j6U&t=1092s) Cool. Thank you for your time.
