# Your Agents Need a Save Button

Hamza Tahir

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/bZISsg7H7DA-your-agents-need-save-button) · [Video](https://www.youtube.com/watch?v=bZISsg7H7DA)

## Your Agents Need a Save Button

Checkpointed execution lets you change a model or tool response, replay from that point, and compare outcomes before deciding whether a cheaper agent is worth shipping.

Familiarity with tool-calling agents, execution traces, and basic Python will help you follow the replay examples.

### Why did the agent do that?

[0:00](https://www.youtube.com/watch?v=bZISsg7H7DA&t=0s)

Why did an agent make that decision? Would another model have reached the same result more cheaply? Would a different tool response have changed what happened next? These questions require more than an explanation of the completed run: they require a way to resume it under changed conditions. Hamza Tahir’s starting analogy is the familiar save button. Documents retain state as we work; an agent needs an equivalent checkpoint if we want to revisit a decision.

A trace records emitted telemetry: tool calls, inputs, outputs, and spans. But the trace alone does not necessarily retain the variables, files, executable code, and environment that produced those events. In Tahir’s framing, the missing connection is between **observability spans and executable state**. A read-only record in an observability backend can show what happened without providing what is needed to run the computation again.

![Diagram showing execution in a process sending spans and events through a one-way arrow to an observability backend marked read-only.](https://ai.engineer/api/talk-essay-media/d6040230884b6e006c5b35474e8653aded0ad66897f36cc0f8ca70a65d60f10f)

Execution state and read-only observability traces live in separate systems.

Once that state is available, replay supports several distinct experiments:

- **Model substitution:** try a cheaper model and inspect its subsequent decisions.
- **Tool substitution:** override a tool’s return value to test a different policy or external response.
- **Failure injection:** deliberately degrade a tool response and observe how the agent handles bad information.

Each experiment needs a retained starting point so the changed execution can be compared with what originally happened.

### Turn production checkpoints into evaluations

[2:03](https://www.youtube.com/watch?v=bZISsg7H7DA&t=123s)

The architectural move is to put a **durable runtime beneath the agent harness** and connect its saved execution state to the traces. The harness still coordinates the agent; the runtime retains the material needed to resume its work. With automatic checkpoints during production execution, the system accumulates both the original behavior and starting points for later experiments.

Consider a support agent handling a chargeback dispute and a possible refund. A replay can test whether an order-status response in another language changes the outcome, whether the request should have escalated, or whether a smaller model could have handled it. Successive checkpoints act like autosave: the experiment can begin near the relevant decision instead of reconstructing the whole interaction.

That leads to an evaluation loop:

1. Select a cohort of production runs—for example, runs that cost too much or took too long.
2. Replay a specific change from saved state.
3. Diff the new executions against their original baselines.
4. Use the differences to decide what to route or ship.

The distinction matters: these are evaluations using **production checkpoints**, not merely a collection of production traces. The baseline tells you what happened; the checkpoint lets you test what could happen under an intervention.

### Ground simulations in production behavior

[4:08](https://www.youtube.com/watch?v=bZISsg7H7DA&t=248s)

Tahir introduces DoorDash as an example of using simulated customer conversations to test changes against production behavior. Its order-status validation compared 302 simulated conversations in five minutes with 175 production conversations collected over seven hours. Escalation rates were 46% in simulation and 44% in production—a two-percentage-point difference in escalation, not a measure of overall accuracy.

DoorDash separately reported 90% fewer hallucinations in simulations. These results concern its particular simulation system and validation exercise. Production-derived scenarios, simulated customers, and mocked dependencies can make experiments useful without establishing that the system restores runtime checkpoints or uses Kitaru. The relevant connection is grounding what-if experiments in interactions that have already occurred.

![DoorDash slide with three columns: 302 sims in 5 minutes, 2 pts within target, and −90% hallucinations.](https://ai.engineer/api/talk-essay-media/dedc4f0bb1db06efecedab5bbdeb42b8f72bd9afeabc830e84884c911ca0bd8a)

Tahir’s DoorDash slide reports 302 simulations in five minutes, two points within target, and 90% fewer hallucinations.

### Inspect what a checkpoint retains

[5:03](https://www.youtube.com/watch?v=bZISsg7H7DA&t=303s)

The live demonstration uses [Kitaru](https://github.com/zenml-io/kitaru), developed by ZenML, where Tahir identifies himself as a co-founder. In the demonstrated architecture, Kitaru supplies the runtime beneath the harness, connects execution to traces, and supports checkpoint-based replay. This is the earlier runtime workflow; the later product reorganized execution around sessions and Workers, so current commands should not be substituted for the recording’s interface.

The example application is a support agent that processes customer requests and escalates to a human when necessary. Its execution view shows tool calls and a timeline. Selecting a checkpoint exposes the configuration, code, and input/output artifacts associated with that step. Tahir also describes the execution environment, such as a Docker image or sandbox, as part of the saved context. That description should not be read as a guarantee of automatic capture of arbitrary process memory or every filesystem dependency; the concrete replay mechanism here depends on retained checkpoint results and executable context.

The timeline also exposes how long steps took, including the tool calls surrounding model requests. That gives the first experiment a precise target: pick a point in the support execution and ask whether a cheaper model would make the same subsequent decisions.

### Change one part of the execution

[6:42](https://www.youtube.com/watch?v=bZISsg7H7DA&t=402s)

For the first replay, Tahir selects an execution point, copies the replay invocation into his terminal, and changes the model to **GPT-5 nano** after a particular tool call. A new execution launches. The demonstrated fork skips its first three checkpoints because Kitaru already has their saved state, then resumes execution at the changed checkpoint. The unchanged prefix is reused; the suffix runs under the new model choice.

The result initially looks similar, so the next experiment changes a different variable. Tahir replaces the lookup-policy tool with another function in the codebase that returns a different policy, while holding the model constant. This separates the policy intervention from the model intervention: if downstream behavior changes, the comparison is no longer confounded by changing both at once.

A Python tool replacement can be a small function with the same calling contract as the original. For example, this policy override returns a modified copy while preserving fields outside the intervention:

```python
from typing import Any


def lookup_policy_override(
    original_policy: dict[str, Any],
) -> dict[str, Any]:
    return {
        **original_policy,
        "requires_human_review": False,
    }
```

The field here illustrates the substitution pattern rather than Kitaru’s CLI syntax or the demo’s exact policy schema. The replacement supplies a different tool result; the replay must still execute the agent’s downstream decision. Access to the code makes that intervention practical. In the demonstration, the second fork produces different logs and an altered artifact, leaving three executions to compare: the baseline, the model fork, and the policy fork.

### Compare decisions, not just trace shapes

[8:57](https://www.youtube.com/watch?v=bZISsg7H7DA&t=537s)

Kitaru’s diff command takes the original execution ID and the replay IDs, then produces a URL for a side-by-side comparison. The command emits some artifact warnings before the comparison page opens. In the execution view, the saved prefix remains identical across branches. The policy fork diverges at the changed tool call, with differences in subsequent execution and timing.

Opening the final `support_decision` artifacts makes the behavioral difference explicit:

| Execution | Account charge | Review status | Next action |
| --- | --- | --- | --- |
| Baseline | Restricted | `needs_review` | `escalate_to_human` |
| Model fork | Restricted | `needs_review` | `escalate_to_human` |
| Policy fork | Restricted | `safe_to_answer` | `answer_directly` |

All three retain the restriction, but the policy fork changes whether the agent should answer directly. That is a substantive decision difference even when the executions look broadly similar. Whether it is acceptable depends on the intended policy.

![Kitaru comparison page displaying three support_decision JSON artifacts; the first two show needs_review and escalate_to_human, while the third shows safe_to_answer and answer_directly.](https://ai.engineer/api/talk-essay-media/8c6c8777b578cdab0e9d9a8c2d3eb3dbf13afc0b67a50ef896f872c77256fe45)

Side-by-side support decisions show two needs_review results and one safe_to_answer result.

Tahir describes both replay forks as cheaper in input/output token cost, without giving exact savings. The comparison view supplies more detail for analysis, but a lower cost does not establish that a changed review decision is correct. Nor does this single support case establish that the model substitution is safe across other requests.

### Apply the intervention across a cohort

[10:54](https://www.youtube.com/watch?v=bZISsg7H7DA&t=654s)

The next step is to select a collection of runs, such as the most expensive support requests, and apply the same intervention across matching configurations. The demonstrated replay-many operation resumes each execution from a selected point and changes either the relevant tool behavior or the model. The output is a distribution of outcomes rather than one promising example.

Tahir does not execute the cohort replay live because it would take time. Instead, he opens an already emitted JSON report. At this scale, manually comparing every branch in the UI becomes cumbersome, so he asks an agent to analyze the report using the Kitaru MCP server and recommend what to do across the cohort.

The useful capability is not just summarizing a large JSON file. A queryable runtime lets the analysis agent inspect executions and retrieve artifacts when it needs evidence about a decision. Skills and MCP access become more valuable as the workload grows from a handful of comparisons to thousands. The analysis begins looking for problematic decisions and continues in the background while Tahir returns to the presentation.

### A cheaper request can still be a worse service

[13:21](https://www.youtube.com/watch?v=bZISsg7H7DA&t=801s)

A model swap can make an individual execution much cheaper, but Tahir says naive substitutions often fail in his experience with customers. A support bot that spends less while leaving requests unresolved may miss the product’s actual objective. Cost is one dimension; the value of the completed work is another.

The [Braintrust cost-efficiency study](https://www.braintrust.dev/evals/test-agent-cost-efficiency) provides a useful framing for this trade-off: assess support policies under explicit tool, safety, and response-quality requirements. The false economy is accepting inadequate resolution quality because a policy looks fast and cheap. It does not require claiming that cheaper models always have a higher dollar cost per resolved request; the matching study’s cheap-only policies still report a lower cost on that measure. The release decision needs a quality threshold as well as a cost comparison.

Repeated reliability adds another dimension. In the original [τ-bench paper](https://arxiv.org/html/2406.12045v1), **GPT-4o** with function calling achieved 61.2% single-trial success on retail tasks, while retail success across all eight trials was below 25%. This is **`pass^8` reliability**: success on every trial for a task, judged by the final database state. It is not textual self-consistency or a universal rule converting a roughly 60% pass rate into a quarter.

**One replay is an anecdote.** Cohort evaluation exposes variation across requests, while repeated trials expose instability on the same task. Both matter when deciding whether a change is dependable. Replaying large workloads can itself become expensive, so selection needs to be deliberate: spend evaluation effort on the populations and decisions that matter to the service.

### The cohort changes the release decision

[15:10](https://www.youtube.com/watch?v=bZISsg7H7DA&t=910s)

An operational replay process starts with real production runs and checkpointed state. Build cohorts around expense, latency, or risk, rather than shipping after one or two favorable examples. The evidence should lead to a concrete decision: ship the change, route suitable requests to it, or hold it back. Automate as much of the loop as is useful, including agent-assisted analysis, while retaining a human at the final decision.

When Tahir returns to the completed cohort analysis, its recommendation is **do not ship**. The single model replay had appeared cheaper and reached a similar result, but the analysis across the demonstrated support cases recommends against using the cheaper model. That verdict applies to this cohort; another dataset could support a different decision. The demonstration does not provide a numerical failure breakdown, so the result is a cohort-specific recommendation rather than a general judgment about GPT-5 nano.

The architectural requirement is what makes that reversal discoverable: an agent harness backed by a runtime that can checkpoint state and replay it from code under changed scenarios. It lets the team test both alternate design choices and interventions in the agent’s behavior before deciding what belongs in production. Tahir closes by pointing to Kitaru’s repository and describing the tool as open source and free to use.

![Slide headed “Wrap your agent. Start replaying.” with open-source and self-hosted text, github.com/zenml-io/kitaru, uv add kitaru, and a repository QR code.](https://ai.engineer/api/talk-essay-media/0ffb694927285d7491a57303d78e225b13b5a688596aea19e04d56df972ced49)

Kitaru’s closing slide presents its repository and installation command.

## Resources

- [Kitaru](https://github.com/zenml-io/kitaru) — Open-source agent replay and evaluation software, with setup instructions, examples and MCP integration.
- [Kitaru’s original launch](https://www.zenml.io/blog/kitaru-launch) — The original introduction to flows, checkpoints, persisted artifacts and replay with cached earlier results.
- [Testing agent cost-efficiency](https://www.braintrust.dev/evals/test-agent-cost-efficiency) — A support-agent experiment comparing model selection, retries and escalation through resolution quality and cost per resolved request.
- [τ-bench: Tool-Agent-User Interaction](https://arxiv.org/html/2406.12045v1) — The original retail and airline benchmark, including repeated-trial reliability measured with pass^k.
- [The new Kitaru: production traces to repeatable evals](https://www.zenml.io/blog/introducing-the-new-kitaru) — Hamza Tahir’s August 2026 account of imported sessions, reviewed cohorts, evaluators and the new worker execution layer.

## Slides

### What if you could replay a run?

Change one thing in a past agent run and replay it to explore what might have happened, without re-running it in production. The speaker asks whether a different action could have been cheaper or faster.

[0:00](https://www.youtube.com/watch?v=bZISsg7H7DA&t=0s) [Opening slide asks what would happen if one thing changed in a past run. A speaker appears in a small camera overlay.](https://assets.ai.engineer/sites/aiecode2025/sha256/00ffe28177612c2613d31171121818cec7aa2a2acc8eb23d2f05fd38cb5c2262.webp)

KITARU

What if you could ask “what if?” about a run that already happened?

Change one thing. Replay it. See what would have happened — no re-running in production.

### Your agents need a save button

Hamza Tahir introduces a save button for agent runs as a way to explore whether different actions could have been cheaper or faster.

[0:13](https://www.youtube.com/watch?v=bZISsg7H7DA&t=13s) [A save icon above the statement that agents need a save button, with the presenter’s name below.](https://assets.ai.engineer/sites/aiecode2025/sha256/70293d6faffe2929b5069a440f3a0f2691fb0fafe134005350dd2c1a5a78d5cd.webp)

KITARU

Your agents need a save button.

Hamza Tahir · Co-founder of ZenML · Building Kitaru

### The execution is gone

A saved file can be closed and reopened with its state intact. The slide contrasts that with agent state, which lives in the process and disappears when it exits.

[0:18](https://www.youtube.com/watch?v=bZISsg7H7DA&t=18s) [A comparison shows file state persisting after closing while agent state lives in the process and disappears when it exits.](https://assets.ai.engineer/sites/aiecode2025/sha256/8cc45e4bf280b982a4db5fccb726588838d708c02d4768ccf826b8e4748c5109.webp)

CHAPTER 1 · THE EXECUTION IS GONE

We’ve had the save button since the early 1980s.

So why don’t our agents have them?

FILE · 1984
⌘S
State persists. Close it, reopen it, keep working.

AGENT · 2026
+ no save
State lives in the process. When it exits, gone.

### A trace is a description

The process sends spans and events one way to a read-only observability backend, leaving the execution’s live state behind.

[0:33](https://www.youtube.com/watch?v=bZISsg7H7DA&t=33s) [A diagram shows execution in a process sending spans and events one way to a read-only observability backend.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7c06ce720eaeb946eb77f1fc64bdcbe06e2e4debf4bd503b1d5c824f0d109a0.webp)

CHAPTER 1

A trace is a description, held by a system that never ran your code.

Your execution ran in your process. The telemetry went somewhere else — Braintrust, Langfuse, LangSmith. Nothing connects the two.

YOUR PROCESS
The execution
• tool calls run here
• real state lives here
• then it's gone

spans · events
ONE-WAY

OBSERVABILITY BACKEND — off to the side
run · draft-flow · completed  ENDED 14:09:52
✓ research  12 calls · 4.2s
✓ write_draft  [illegible] · 11.6s
✓ publish  3 calls · 2.1s
✓ done
READ-ONLY

### Choose a new configuration and replay

Replay saved state to test what-if changes: swap the model, mock or drop a tool, fail a call, or degrade the retriever.

[1:26](https://www.youtube.com/watch?v=bZISsg7H7DA&t=86s) [Five replay options: swap the model, mock a tool, fail a call, degrade a retriever, and drop a tool.](https://assets.ai.engineer/sites/aiecode2025/sha256/59775462aced5a594a421b9684ee74ffc78db0bdc036ef0c83948baa5636b53d.webp)

CHAPTER 2

Agents have many dials – simply choose your new configuration and replay.

one replay call

Swap the model
overrides={model}

Mock a tool
overrides={tool}

Fail a call
raise on step

Degrade retriever
swap the index

Drop a tool
remove from set

### Add a durable runtime layer

A durable runtime sits beneath agent frameworks, checkpointing calls and augmenting traces with execution state for replay.

[2:01](https://www.youtube.com/watch?v=bZISsg7H7DA&t=121s) [Several agent frameworks and SDKs are shown above a durable runtime layer labeled as checkpointing each call.](https://assets.ai.engineer/sites/aiecode2025/sha256/bab0517b50f18cde2f3827d3cd42946715fa79b97386e5329f1add3a79a936d8.webp)

CHAPTER 2

Add a durable runtime layer.
Augment your traces with true checkpoints.

Claude Agent SDK
LangGraph
python
Pydantic
Google Antigravity
OpenAI Agents SDK

Durable runtime
checkpoints every call underneath

### Production runs as a test set

Production runs capture real inputs, tool responses and edge cases. With state checkpoints, teams can revisit a refund after a chargeback and ask whether escalation or a smaller model would have worked.

[2:28](https://www.youtube.com/watch?v=bZISsg7H7DA&t=148s) [A slide contrasts imagined synthetic cases with examples from real production runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/d270c0b456f896859c715040da66cb01601883e2a31802c45ad74b768f7a56f5.webp)

CHAPTER 3 · YOUR TEST SET WROTE ITSELF

Your production runs are the ultimate test set.

Synthetic cases imagine what users might do. Real runs are what they did — real inputs, real tool responses, real edge cases, including the ones that already went wrong.

SYNTHETIC
imagined — what users might do

Customer asks refund
Check my order status
Reset my password - request

PRODUCTION
real runs — what they actually did

Refund after a chargeback dispute
Order status — in Spanish, mid-chat
Refund request, tool timed out

### The evaluation loop

Select production runs that matter, replay one change, and compare the result with the original. Use cost, decisions, and divergence to decide whether to ship, route, or hold; then repeat.

[3:22](https://www.youtube.com/watch?v=bZISsg7H7DA&t=202s) [A repeating four-step loop: select a cohort, replay one change, diff, and decide.](https://assets.ai.engineer/sites/aiecode2025/sha256/55bbe82ce2537489e2d8ae78f2e93a9d6769dc6db0601c334cecb0d4f09cedca.webp)

CHAPTER 3 · THE LOOP

Close the loop and turn past runs into full-blown evals.

1 Select a cohort
the runs that matter

2 Replay one change
same call, one keyword

3 Diff
cost · decisions · divergence

4 Decide
ship · route · hold

repeat

### Four steps: checkpoint, replay, diff, decide

Checkpoint costly, escalated, or failed runs. Replay a change using cached work from before it, then compare cost, decisions, and divergence to decide what to ship, route, or hold.

[3:57](https://www.youtube.com/watch?v=bZISsg7H7DA&t=237s) [A four-row summary describes checkpointing, replaying, comparing results, and deciding what to ship, route, or hold.](https://assets.ai.engineer/sites/aiecode2025/sha256/d42b47e5fc2f111259b3beed57f17d82996e35353c38fd76fc74c013f53142ba.webp)

CHAPTER 3

Four steps. The methodology is the skill.

01 Checkpoint — the expensive, the escalated, the failed.
02 Replay — everything before your change, handed back from cache.
03 Diff — cost, changed decisions, where it first diverged.
04 Decide — ship what matches, route what's marginal, hold what drifts.

### DoorDash simulation and evaluation example

DoorDash describes a simulated environment for replaying customer bots and exploring what-if scenarios for support chatbot evaluation.

[4:17](https://www.youtube.com/watch?v=bZISsg7H7DA&t=257s) [A slide points to a DoorDash article about simulation and evaluation for support chatbots.](https://assets.ai.engineer/sites/aiecode2025/sha256/496696af616268ae9c5b0648b564ec419eee699bab527a2fdf85863dc8e9f4ad.webp)

CHAPTER 4 · THIS WORKS AT SCALE

Don’t believe me? Doordash and others are doing it too.

Blog
Inside DoorDash’s one-click simulation and evaluation platform for support chatbots
June 1, 2026 | [illegible] | [illegible]

https://careersatdoordash.com/blog/doordashs-one-click-simulation-and-evaluation-platform-for-support-chatbots/

Interface text:
DOORDASH
Mission & Values · Working at DoorDash · Belonging · Blogs · Career Areas · Engineering · SevenRooms · Search Jobs

### DoorDash simulation results

DoorDash ran 302 simulations in five minutes, while 1% of live traffic took seven hours. Simulated escalation was within two points of production, and hallucinations fell 90% once the eval flywheel was in place.

[4:29](https://www.youtube.com/watch?v=bZISsg7H7DA&t=269s) [The slide reports 302 simulations in five minutes, an escalation rate within two points of production, and 90 percent fewer hallucinations.](https://assets.ai.engineer/sites/aiecode2025/sha256/6746773eefe13c6035ce4f8706002f3bbc35acc9861d3d88707556784ab87042.webp)

CHAPTER 4 · DOORDASH

302 sims
in 5 minutes
where 1% of live traffic took 7 hours

2 pts
within target
simulated escalation rate vs production

−90 %
hallucinations
once the eval flywheel was in place

### One run to ten: ask the agent

The demo introduces a production support agent with every model and tool call checkpointed, then sets up a walkthrough from one run to ten and asking the agent.

[4:52](https://www.youtube.com/watch?v=bZISsg7H7DA&t=292s) [A demo introduction describes a production support agent with every model and tool call checkpointed.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8f117dce5b664fdf542dbf5ab121e493d93586974cc8aa734028eedb3c7ec8f.webp)

DEMO

One run → ten → ask the agent.

A support agent in production. Every model and tool call checkpointed.

### Kitaru repository

The demo opens Kitaru’s GitHub repository. The presenter describes Kitaru as a runtime layer below the agent harness that connects to traces for checkpointing and replay.

[5:02](https://www.youtube.com/watch?v=bZISsg7H7DA&t=302s) [GitHub repository overview for Kitaru, described as an open-source platform layer for AI agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f1724ed1b19f15a8ee1eb348df1357014dd53f098a05f220bdf9d1b1af44ab9.webp)

kitaru Public

About
Open-source platform layer for AI agents in production
kitaru.ai
python · mcp · replay · observability · ai-agents · checkpoints · mlops · pydantic · workflow-orchestration · agent-framework · llm · durable-execution · pydantic-ai
Readme
Apache-2.0 license

strickvl  Note agent-harness key scrubbing in kitaru-release smoke step
85118e6 · 37 minutes ago
576 Commits

.claude  Note agent-harness key scrubbing in kitaru-release smoke...  37 minutes ago
.github  Bump the all-actions group with 3 updates (#462)  2 days ago
assets  Helm chart for kitaru (#51)  3 months ago
docker  Add sandbox stack and adapter command support (#423)  last week
docs  Add adapter cost calculators (#465)  yesterday
examples  Point Agent Harness Platform docs at the ZenML Agents g...  2 days ago
helm  Add sandbox stack and adapter command support (#423)  last week
scripts  Fix release smoke sandbox checks and refresh kitaru-rel...  last week
src  Add adapter cost calculators (#465)  yesterday

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Thu 25. Jun 11:09
Executions - Kitaru
Your Agents Need a Save But...
zenml-io/kitaru: Open-source
github.com/zenml-io/kitaru
Work  New Chrome available
zenml-io / kitaru
Type / to search
Code  Issues 28  Pull requests 12  Discussions  Actions  Security and quality 14  Insights  Settings
Edit Pins  Watch 2  Fork 13  Starred 190
develop  76 Branches  29 Tags  Go to file  Add file  Code

### Kitaru runtime

Kitaru sits beneath the agent harness, adding checkpoints, replay and resume while teams keep their existing SDK, prompts, tools and model.

[5:12](https://www.youtube.com/watch?v=bZISsg7H7DA&t=312s) [Kitaru README describes a self-hosted agent runtime with checkpoints, replay and resume.](https://assets.ai.engineer/sites/aiecode2025/sha256/24fcf5e9628c04fdfa7f2573cc831129ec70be788f6513f334088427f32d6f01.webp)

Kitaru

The runtime layer underneath your agent stack.

Kitaru (来る, "to arrive") is a self-hosted, framework-agnostic runtime for autonomous agents — underneath the harness your team already picked. You keep your agent SDK, your prompts, your tools, your model. Kitaru adds durable execution: checkpoints, replay, resume, wait(), versioned deployments, and isolated runtimes, running on your own infrastructure.

pypi v0.17.1
python 3.11 | 3.12 | 3.13
license Apache-2.0

Docs · Quick Start · Examples · Getting Started Guide · Roadmap · Community

Embedded dashboard:
Kitaru · Flows › content_pipeline · ACTIVE
Overview · Transitions · Execution · Logs · Cost · Diff
PIPELINE content_pipeline · RUNS 126 · FAILURES 23 · AVG COST $0.048
content_pipeline
EXECUTIONS 126 · SUCCESS RATE 73% · AVG COST $0.040 · AVG DURATION 10.2s

Interface text:
Chrome · File · Edit · View · History · Bookmarks · Profiles · Tab · Window · Help
Thu 25. Jun 11:10
Executions - Kitaru
Your Agents Need a Save But
zenml-io/kitaru: Open-source
github.com/zenml-io/kitaru
Work · New Chrome available
pyproject.toml · Release 0.18.0 · 1 hour ago
uv.lock · Release 0.18.0 · 1 hour ago
wrangler.redirect.toml · Migrate Kitaru docs to GitBook + standalone SDK referenc... · 3 weeks ago
wrangler.toml · Migrate Kitaru docs to GitBook + standalone SDK referenc... · 3 weeks ago
README · Contributing · Apache-2.0 license · Security

### Support-agent executions

Kitaru lists completed and failed runs with durations and costs. The speaker’s support agent handles customer requests and escalates them to humans when needed.

[5:37](https://www.youtube.com/watch?v=bZISsg7H7DA&t=337s) [Kitaru lists support-agent executions with statuses, durations and costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5bc707fec42c23acc369f4ee1d390c7362458ca32dafb051e35c7fc3c925d24.webp)

support_copilot_flow · not deployed

Executions  Invoke

local  All versions  Status  Stack  Range  Search #num or flow...  Refresh

Execution | Status | Duration | Cost | ID | Author | Created
#0071 | Completed | 1m 9s | $0.00[illegible]1325 | 9e135198-e12e-4f32-9f76-bd4cdb0ed037 | AD admin | 25/06/2026, 10:49:35
#0070 | Completed | 47s | $0.00138725 | 7f3e3411-f73c-4c92-bd0d-daa9c7044346 | AD admin | 25/06/2026, 10:39:54
#0069 | Completed | 37s | $0.00098615 | 8e07850d-d954-4c98-8541-359cbfdf1e3b | AD admin | 25/06/2026, 10:25:59
#0068 | Failed | 6s | | ea434beb-2755-474c-a6f6-185a47932b51 | AD admin | 25/06/2026, 10:25:48
#0067 | Completed | 35s | $0.00071735 | 1a48977e-0d46-4095-b9cb-b50a33abe41d | AD admin | 25/06/2026, 10:25:04
#0066 | Completed | 36s | $0.00090655 | 37689f60-1b57-4a03-a163-2d41bc9b26a7 | AD admin | 25[illegible]

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Thu 25. Jun 11:10
Executions - Kitaru
Your Agents Need a Save But[illegible]
zenml-io/kitaru: Open-source
preview.demo.kitaru.zenml.io/flows/cc12f224-3790-47c8-af9b-7418cf99bda3/v/local/ex...
Work  New Chrome available
Flows / support_copilot_flow  local
Flows  Executions  AD

### Execution checkpoints

The support-agent execution lists six checkpoints: three model requests, two tool calls and a final publishing step. Their displayed durations range from 2 to 10 seconds.

[5:45](https://www.youtube.com/watch?v=bZISsg7H7DA&t=345s) [An execution detail lists six model, tool and publishing checkpoints.](https://assets.ai.engineer/sites/aiecode2025/sha256/30806962d7c2e6a7b27a192994e799a1de45c8325bfdc03840fef723c3e03314.webp)

6 checkpoints
llm  support_copilot_model_request  10s
tool  gather_context_tool  2s
llm  support_copilot_model_request_2  5s
tool  lookup_policy_tool  2s
llm  support_copilot_model_request_3  7s
publish_support_decision  2s

Interface text:
preview.demo.kitaru.zenml.io/flows/cc12f224-3790-47c8-af9b-7418cf99bda3/v/local/ex...
Flows / support_copilot_flow / local / #0071
Execution  Logs
EXECUTIONS
List  Timeline  Filter steps...
#0071
#0070
#0069
#0068
#0067
#0066
#0065
#0064
#0063
#0062
#0061
#0060
#0059
#0058
#0057
#0056
#0055
#0054
Sele[illegible]

### Kitaru execution checkpoints

The timeline places six checkpoints across a 38-second execution. Selecting gather_context_tool opens its code, alongside tabs for configuration and other checkpoint details.

[5:48](https://www.youtube.com/watch?v=bZISsg7H7DA&t=348s) [A timeline displays the durations and sequence of six execution checkpoints.](https://assets.ai.engineer/sites/aiecode2025/sha256/20c53809a8524a8259f62300a91608f8741be42158956b37ce5c6a2cd62452ed.webp)

6 checkpoints · 38s

Timeline: 0.0s  10.0s  20.0s  30.0s
llm  support_copilot_model_request  10.0s
tool  gather_context_tool  2.0s
llm  support_copilot_model_request_2  5.0s
tool  lookup_policy_tool  2.0s
llm  support_copilot_model_request_3  7.0s
publish_support_decision  2.0s

Interface text:
preview.demo.kitaru.zenml.io/flows/cc12f224-3790-47c8-af9b-7418cf99bda3/v/local/ex...
Flows / support_copilot_flow / local / #0071
Execution  Logs
EXECUTIONS
#0071
#0070
#0069
#0068
#0067
#0066
#0065
#0064
#0063
#0062
#0061
#0060
#0059
#0058
#0057
#0056
#0055
#0054
List  Timeline  Filter steps...  1.0x
Sele[illegible]

[5:59](https://www.youtube.com/watch?v=bZISsg7H7DA&t=359s) [A selected checkpoint displays a short Python function in its code tab.](https://assets.ai.engineer/sites/aiecode2025/sha256/913af423f4ba84bc66c389ebca031d7c4150457a24b3a0282cd6d93562d8c094.webp)

6 checkpoints

[illegible]copilot_model_request  10s
[illegible]context_tool  2s
[illegible]copilot_model_request_2  5s
[illegible]policy_tool  2s
[illegible]copilot_model_request_3  7s
[illegible]t_decision  2s

tool  gather_context_tool  2s
Logs  Checkpoint  Configuration  Code  Artifacts
_utils
kitaru/adapters/pydantic_ai/_utils.py

def _turn_with_tool_args(
    tool_args: Any,
    _cache_key: str | None = None,
) -> Any:
    return body()

Interface text:
[illegible]taru.zenml.io/flows/cc12f224-3790-47c8-af9b-7418cf99bda3/v/local/ex...
[illegible]cal / #0071
Flows  Executions  AD
Refresh
Filter steps...
Work  New Chrome available

### Checkpoint artifact

The selected tool checkpoint shows its JSON output alongside the execution steps. The speaker says the code, artifacts, and execution environment are snapshotted between checkpoints.

[6:06](https://www.youtube.com/watch?v=bZISsg7H7DA&t=366s) [An execution list appears beside a checkpoint’s JSON output.](https://assets.ai.engineer/sites/aiecode2025/sha256/2aee5adf1305657ec2dc9e21f45b580fd2bc2be70c834a389571c2a9f697cedd.webp)

Flows / support_copilot_flow / local / #0071

Execution  Logs

EXECUTIONS
#0071
#0070
#0069
#0068
#0067
#0066
#0065
#0064
#0063
#0062
#0061
#0060
#0059
#0058
#0057
#0056
#0055
#0054
#0053

List  Timeline  Filter steps...  6 checkpoints
llm  support_copilot_model_request  10s
tool  gather_context_tool  2s
llm  support_copilot_model_request_2  5s
tool  lookup_policy_tool  2s
llm  support_copilot_model_request_3  7s
publish_support_decision  2s

tool  gather_context_tool  2s
Logs  Checkpoint  Configuration  Code  Artifacts
OUT (1)  output
output
JSON  182 chars decoded  Rendered  Raw
{
  "intent": "change_sso_permissions",
  "category": "permissions",
  "triage": "high",
  "requested_change": "grant_admin_access_to_sso_set[illegible]
  "customer_tier": "enterprise"
}

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Thu 25. Jun 11:10
Execution #0071 - Kitaru
Your Agents Need a Save But
zenml-io/kitaru: Open-source
preview.demo.kitaru.zenml.io/flows/cc12f224-3790-47c8-af9b-7418cf99bda3/v/local/ex...
Work  New Chrome available
Flows  Executions  AD  Refresh

### Replay checkpoints

The replay skips the first three checkpoints using saved state, then runs the policy lookup and continues with GPT-5 nano to test a cheaper model.

[7:26](https://www.youtube.com/watch?v=bZISsg7H7DA&t=446s) [Execution view shows five checkpoints, including a policy lookup and a running model request.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb774b8a19e7a793c3b92d3df85a8cc8666c84a1a9b15117720b387df31b7cd2.webp)

Timeline
Filter steps...
5 checkpoints

support_copilot_model_request
gather_context_tool
support_copilot_model_request_2
lookup_policy_tool  3s
support_copilot_model_request_3  1s

tool  gather_context_tool
Logs  Checkpoint  Configuration  Code
OUT (1)  output
output
JSON  182 chars decoded  Rendered  Ra[illegible]
{
  "intent": "change_sso_permissions",
  "category": "permissions",
  "triage": "high",
  "requested_change": "grant_admin_access_to_s[illegible]
  "customer_tier": "enterprise"
}

Interface text:
[illegible]v.demo.kitaru.zenml.io/flows/cc12f224-3790-47c8-af9b-7418cf99bda3/v/local/ex...
Work  New Chrome avail[illegible]
flow  local  #0072
Flows  Executions
Refresh

## Full transcript

[0:00](https://www.youtube.com/watch?v=bZISsg7H7DA&t=0s) Have you ever looked at your agent execution and asked yourself the question: Why did it do that? What if it had done a different thing? Would it have been cheaper?

[0:10](https://www.youtube.com/watch?v=bZISsg7H7DA&t=10s) Would it have been faster? Well, you can do all these things if your agents have a save button.

[0:20](https://www.youtube.com/watch?v=bZISsg7H7DA&t=20s) We've had the save button for documents for decades now. Since the nineteen eighties, people have been used to pressing Control S, Command S, uh, or auto-saving while you're working to have a persistent state.

[0:34](https://www.youtube.com/watch?v=bZISsg7H7DA&t=34s) But agents, they don't have that today. The only thing we have which is closest is a trace. A trace gives you the emitted telemetry data of how an agent calls tools and the input and output of that state.

[0:48](https://www.youtube.com/watch?v=bZISsg7H7DA&t=48s) Now, while this is a good start, it is actually very disconnected from the runtime in which these agents actually execute. So all the variables that are in state, all the file system that is in, in flight, um, the decisions that it makes in the code, uh, the actual code itself, all of that is lost, and it is

[1:08](https://www.youtube.com/watch?v=bZISsg7H7DA&t=68s) only stamped as a read-only trace by the end, which is sitting in another tool far away from where the actual code is. And I think this is what's missing today in the industry, is that we don't have a clear connection between the observability spans that are emitted with OTel and the execution.

[1:28](https://www.youtube.com/watch?v=bZISsg7H7DA&t=88s) And maybe at this point you might be wondering, "But why even bother? Why do I need to have a save button?" Well, save allows you to replay. You can go back in history and ask the what if question.

[1:41](https://www.youtube.com/watch?v=bZISsg7H7DA&t=101s) What are the types of questions you might wanna ask? Well, you might wanna swap the model. Maybe you use a, an open source model that is cheaper. Maybe you mock a tool and you override the, what it returns.

[1:53](https://www.youtube.com/watch?v=bZISsg7H7DA&t=113s) Maybe you degrade it intentionally to see what would happen if things are wrong.

[1:58](https://www.youtube.com/watch?v=bZISsg7H7DA&t=118s) And these sorts of questions are only possible if you have that state.

[2:03](https://www.youtube.com/watch?v=bZISsg7H7DA&t=123s) And there is a category of the stack emerging which actually allows that, and this sit on top of the harness, sit on top of the frameworks that allow you to create agents, and put a durable runtime below that, um, and augments the traces that are emitted with actually the code execution and the things that are around it

[2:26](https://www.youtube.com/watch?v=bZISsg7H7DA&t=146s) to actually complete the state of this-- of the, of the system.

[2:31](https://www.youtube.com/watch?v=bZISsg7H7DA&t=151s) And the good news is that once you have such a system, in production, you already have the information you need to ask those questions that are relevant to making your system better, cheaper and faster.

[2:46](https://www.youtube.com/watch?v=bZISsg7H7DA&t=166s) Production already has the traces. It already has these state checkpoints,

[2:53](https://www.youtube.com/watch?v=bZISsg7H7DA&t=173s) ideally from the runtime, that can allow you to go back in time and ask those questions. For example, let's take an agent example which does a customer resolution and refunds after a chargeback dispute.

[3:06](https://www.youtube.com/watch?v=bZISsg7H7DA&t=186s) Well, you can then see if the order status is changing languages or whether the request should have been escalated or maybe a smaller model would have handled it if the runtime is checkpointing each of the state as it goes along, almost like an auto-save, a Command S, a Control S in your agent.

[3:24](https://www.youtube.com/watch?v=bZISsg7H7DA&t=204s) And once you have that, you can even close the loop. Uh, this conference is all about loops, so this is nothing different. You have a cohort of runs that you think maybe matter because maybe they're too expensive, they took too long.

[3:39](https://www.youtube.com/watch?v=bZISsg7H7DA&t=219s) You replay a change, you diff it, you see what would have happened when you have the baseline, which you know what happened in the first place, and then you decide and you route and you ship back.

[3:51](https://www.youtube.com/watch?v=bZISsg7H7DA&t=231s) That's closing the loop on your evals. It's, it's basically evaluating using your production traces.

[4:01](https://www.youtube.com/watch?v=bZISsg7H7DA&t=241s) So it's basically evaluating using your production checkpoints.

[4:08](https://www.youtube.com/watch?v=bZISsg7H7DA&t=248s) So checkpoint, replay, diff, decide. And this is really the methodology that, that I've seen and I've seen others do, uh, which has really scaled. For example, DoorDash. DoorDash has, uh, a blog post on the first of June where they talk about having a simulated environment where they replayed customer bots and they've done what if scenarios and seen

[4:32](https://www.youtube.com/watch?v=bZISsg7H7DA&t=272s) how they could have made it better. And where, and where it used to take them hours and hours to, to do this, now they've reduced it to five minutes, uh, with hundreds of simulations, have ninety percent less hallucinations, and they're still two points within what they've seen in production.

[4:49](https://www.youtube.com/watch?v=bZISsg7H7DA&t=289s) So the simulations are pretty good because they're grounded in what's already happened.

[4:54](https://www.youtube.com/watch?v=bZISsg7H7DA&t=294s) So we're gonna just walk through this, um, in an example, and we're gonna see how this works.

[5:03](https://www.youtube.com/watch?v=bZISsg7H7DA&t=303s) For this demo, we're going to be using a tool called Kitaru. Kitaru is a very new tool that is launched by the team at, uh, ZenML. ZenML is-- has been around for many years and is a player in the orchestration space.

[5:17](https://www.youtube.com/watch?v=bZISsg7H7DA&t=317s) I'm one of the co-founders. And Kitaru is something we've launched recently, which allows you to have a runtime layer below your harness layer and also connect to your traces, um, and do all the checkpointing that we were just talking about, and then running replay scenarios.

[5:33](https://www.youtube.com/watch?v=bZISsg7H7DA&t=333s) So you can see here I have all of my... I have a support, uh, agent which looks at my customer requests and escalates it when it needs to to humans.

[5:44](https://www.youtube.com/watch?v=bZISsg7H7DA&t=344s) You can see various things as you might expect. Um, every tool call, uh, I can see a timeline view if I wanted. And here the difference is that if I, if I click on a particular, um, um, checkpoint, like a tool call, I can see the configuration where it ran, uh, the code it, it took.

[6:04](https://www.youtube.com/watch?v=bZISsg7H7DA&t=364s) Um, and the artifacts, uh, which came in and out of it. Um, and this combination of code and the artifacts that it created and the environment in which it ran in, whether it was a Docker image or a sandbox, those are all snapshotted in state here between the checkpoints.

[6:20](https://www.youtube.com/watch?v=bZISsg7H7DA&t=380s) And you can see that here in this particular example, there was a few tool calls every time it went to the LLM, and I can actually see, um, you know, how long it took.

[6:30](https://www.youtube.com/watch?v=bZISsg7H7DA&t=390s) But imagine I wanted to do something different. Imagine I wanted to change the model. If I... maybe at this point I wanted to use a cheaper model. Would it have done the same stuff afterwards?

[6:42](https://www.youtube.com/watch?v=bZISsg7H7DA&t=402s) So to do this in Kitaru, it's quite easy. All you have to do...

[6:48](https://www.youtube.com/watch?v=bZISsg7H7DA&t=408s) Now, to do this in Kitaru is very easy. All you have to do is you have to take your execution and replay it at a particular point. So I'm just gonna copy this over, and I'm gonna put it in my terminal and see what happens.

[7:04](https://www.youtube.com/watch?v=bZISsg7H7DA&t=424s) So here what I'm doing is essentially I am saying, okay, after this particular tool call, change the model to GPT-5 nano, which is obviously a bit cheaper. Um, and then what's gonna happen here is

[7:21](https://www.youtube.com/watch?v=bZISsg7H7DA&t=441s) there's gonna be new execution that's launched from seventy-one, and you can see the first three checkpoints are skipped. So they're all skipped because Kitaru already has the state of all the checkpoints before that.

[7:35](https://www.youtube.com/watch?v=bZISsg7H7DA&t=455s) All it needs to do is just change this particular checkpoint and then execute, start executing from here. And this is really cool because now I can see what would have happened in this scenario if there was a cheaper model.

[7:47](https://www.youtube.com/watch?v=bZISsg7H7DA&t=467s) So this looks pretty similar to me. But, um, what if I had wanted to do a even different change? What if I wanted to mock a tool or, or if I wanted to change a tool call?

[7:57](https://www.youtube.com/watch?v=bZISsg7H7DA&t=477s) Well, in order to do that, um, you can mock up, for example, the lookup policy tool. And here it's also very easy. I can just go back here,

[8:08](https://www.youtube.com/watch?v=bZISsg7H7DA&t=488s) replay, and this time rather than changing the model, I am changing the lookup policy. And I'm mocking it with another function in my code base, uh, which returns a different lookup policy.

[8:20](https://www.youtube.com/watch?v=bZISsg7H7DA&t=500s) And I'm just trying to understand if the policy had changed, what would have happened. This time I'm holding the model constant. Now, the interesting thing here is that because I have the code, it's very easy for me to do tool calls and to change these particular things and do more experiments than I would have had if I

[8:37](https://www.youtube.com/watch?v=bZISsg7H7DA&t=517s) was completely disconnected from the code base. And here you can see that the code base is looking a bit different. So I see my logs here. Um, they look a bit different.

[8:47](https://www.youtube.com/watch?v=bZISsg7H7DA&t=527s) And here you can see that I got a slightly different artifact maybe from the tool call, and it published the thing. So now again, I have three runs. I have the original run, and I have the two replays.

[8:57](https://www.youtube.com/watch?v=bZISsg7H7DA&t=537s) Now, what if I wanted to see them side by side, right? So Kitaru, um, actually has this very handy diff command that lets me give an original ID of an execution and then allows me, um, on the other side to actually see them side by side so I can see what happened.

[9:18](https://www.youtube.com/watch?v=bZISsg7H7DA&t=558s) So there's a few warnings here, uh, about some artifacts, but in a second it will go and give me a URL. I can copy this URL, and I can actually put it directly here.

[9:31](https://www.youtube.com/watch?v=bZISsg7H7DA&t=571s) And now I have a very nice, um,

[9:36](https://www.youtube.com/watch?v=bZISsg7H7DA&t=576s) comparison of the original and the two forks. Um, I can see a bunch of things here, but I think what's really interesting is this view. So in this view, you can see that the first part of the, the baseline is the same, right?

[9:47](https://www.youtube.com/watch?v=bZISsg7H7DA&t=587s) So these things were skipped. They were exactly as it is. The state is exactly where it was. Um, but now right after that, in the third replay, it's a little bit different, right?

[9:56](https://www.youtube.com/watch?v=bZISsg7H7DA&t=596s) The tool call happened a bit differently because we used a different policy, and then something changed after here. Um, here it took a little bit longer. And you can really start looking at the final result.

[10:07](https://www.youtube.com/watch?v=bZISsg7H7DA&t=607s) And if I click on the final result, I can actually see the artifact side by side. I can see what decision, um, it actually ended up making. So you can see that it, uh, here it restricted the account charge.

[10:20](https://www.youtube.com/watch?v=bZISsg7H7DA&t=620s) Here it also restricted it. Here it also restricted it. The first two it actually needs review, and the third, because we changed the policy, is safe to answer. So this might or might not be good in your scenario, but if...

[10:32](https://www.youtube.com/watch?v=bZISsg7H7DA&t=632s) is this what you expected to see? Well, maybe. Because th- these two were cheaper at the end of the day. For, for, for the number of tokens consumed, uh, they were cheaper, input and output.

[10:43](https://www.youtube.com/watch?v=bZISsg7H7DA&t=643s) And you can see a bunch of detail here, uh, in the, in the UI, which might be useful for these ana-- for these analysis. Okay. But this is just one point, right?

[10:54](https://www.youtube.com/watch?v=bZISsg7H7DA&t=654s) What if I wanted to do this across a cohort? What if I wanted to have a bunch of runs, right? Which actually, um, maybe sorted by cost, so maybe I took all of my expensive ones, and I wanted to do one change across the entire cohort.

[11:11](https://www.youtube.com/watch?v=bZISsg7H7DA&t=671s) So change the-- change all the tool calls that happened, uh, that happened for this particular set of configurations across the cohort. Or change the model itself, um, and use a cheaper model across the cohort.

[11:25](https://www.youtube.com/watch?v=bZISsg7H7DA&t=685s) So this gives me a bigger distribution of information and replays that I can use. Um, and now I can just replay this. Um, and the way you would replay it is you can use the replay many command where you actually, um, start from a particular point and you do the same as we did before, just across the

[11:45](https://www.youtube.com/watch?v=bZISsg7H7DA&t=705s) cohort. Um, this is gonna take a while, so I'm not gonna do it now. But once you execute it, in this particular case, I've just emitted it to JSON.

[11:55](https://www.youtube.com/watch?v=bZISsg7H7DA&t=715s) So I have a very nice, uh, JSON

[12:00](https://www.youtube.com/watch?v=bZISsg7H7DA&t=720s) here, and this JSON gives me a bunch of things. So this is a bit hard to do the UI, right? So there's a lot of things going on. You can't just do the many, many, many comparisons here.

[12:09](https://www.youtube.com/watch?v=bZISsg7H7DA&t=729s) Um, but what you could do, and what I love to do personally, is use the Kitaru MCP server. And here what you do is you can just say, "Hey, read this, read this JSON report and do an analysis on what you think I should be doing across the cohort."

[12:29](https://www.youtube.com/watch?v=bZISsg7H7DA&t=749s) And I think this is what, what is really important, is to be using agents and LLMs to analyze these cohorts across a plethora of data. Because at some point, uh, I mean, ten is probably easy to do, but what if you have thousands?

[12:43](https://www.youtube.com/watch?v=bZISsg7H7DA&t=763s) Um, and doing thousands and thousands is hard, and this is where skills and MCP servers get really relevant, and having the runtime be queryable and go into your execution and fetch the artifacts is very important.

[12:56](https://www.youtube.com/watch?v=bZISsg7H7DA&t=776s) So it's going to be, it's going to be doing a lot of things. It's, it's, it's trying to, you know, um, run an analysis around these decisions, and it's going to flag any, um, red flags that, that happen.

[13:10](https://www.youtube.com/watch?v=bZISsg7H7DA&t=790s) So while this is going on, um, maybe we can go back and continue our presentation. So, um,

[13:21](https://www.youtube.com/watch?v=bZISsg7H7DA&t=801s) here you can see, yeah. Yes, if you change the model, it can get very cheap. Um, and of course, you don't want to do it across just one sample, but across many, um, because then you can see a bigger variation.

[13:37](https://www.youtube.com/watch?v=bZISsg7H7DA&t=817s) I think one thing which is important here is, and to be very honest, is that while we've been doing it-- this with our users and customers, what I've personally seen is that, um, having a naive model swap usually or oftentimes doesn't work.

[13:52](https://www.youtube.com/watch?v=bZISsg7H7DA&t=832s) So just changing, for example, to a cheaper model and just looking at the cost one-- single dimensionally, obviously it could be that you're, you're spending a lot less money, but what happens if your support bot is not resolving the requests, right?

[14:06](https://www.youtube.com/watch?v=bZISsg7H7DA&t=846s) So this is a study from Braintrust, uh, excellent study, where they actually looked at that, and they saw that there could be a false economy if you s- do a naive model swap, because it might look on paper that you're faster and you're cheaper, but at the end of the day, you have to look at the value

[14:20](https://www.youtube.com/watch?v=bZISsg7H7DA&t=860s) created, right? So it's a trade-off between how much money you want to spend and, and, um, yeah, the result. Um, and also if you look at the τ-bench, uh, what we must understand is that a model that passes sixty percent of the time is only self-consistent about a quarter of the time.

[14:36](https://www.youtube.com/watch?v=bZISsg7H7DA&t=876s) So, which basically means that one replay is just an anecdote, and having a cohort, uh, analysis is way, way, way better. Um, and because then you can really see across a population and across scale what would have happened, um, not just looking at one estimate.

[14:56](https://www.youtube.com/watch?v=bZISsg7H7DA&t=896s) Um, this can get very expensive, of course, and this is where you have to be really smart about what you replay and have tooling that really helps you. Um, and this you can really bake into your production process as well, right?

[15:10](https://www.youtube.com/watch?v=bZISsg7H7DA&t=910s) So again, if you, if you take a step back and look at the playbook, you can start from real runs, um, not synthetic, but real runs, real production, uh, checkpointed state.

[15:23](https://www.youtube.com/watch?v=bZISsg7H7DA&t=923s) Build cohorts that matter. Um, maybe take the expensive ones, maybe take the long ones, maybe take the risky ones. Um, never ship anything by just replaying one or two things, um, and just do this at scale.

[15:35](https://www.youtube.com/watch?v=bZISsg7H7DA&t=935s) And, uh, ship, route, and hold, and try to automate that loop as much as possible. Maybe there's an agent that's doing that for you, um, that's even better. But you just have a human in the loop at the end.

[15:47](https://www.youtube.com/watch?v=bZISsg7H7DA&t=947s) But you just have a human in the loop at the end.

[15:50](https://www.youtube.com/watch?v=bZISsg7H7DA&t=950s) Now, let's see where our cohort analysis has gotten. Okay, so we're done. So the verdict is, don't ship. So even though it looked like from a single replay that it was cheaper to do and we'd reach the same result, across a bunch of those support cases, you actually saw that our agent concludes that you shouldn't be using

[16:09](https://www.youtube.com/watch?v=bZISsg7H7DA&t=969s) a cheaper model in this particular case for your data. But this might be different for your data.

[16:16](https://www.youtube.com/watch?v=bZISsg7H7DA&t=976s) In conclusion, um, if you want to be replaying your agent executions and answering the questions, what if I had done something different while designing this agent? Or what if the agent could be driven to do something different?

[16:34](https://www.youtube.com/watch?v=bZISsg7H7DA&t=994s) Well, you can do this if you model your agent

[16:39](https://www.youtube.com/watch?v=bZISsg7H7DA&t=999s) with your harness in a runtime that can checkpoint state and is able to replay that state from code with different scenarios. If you want to use Kitaru, the, the, the tool that I showed that allows you to do it, um, you can scan the repo.

[16:55](https://www.youtube.com/watch?v=bZISsg7H7DA&t=1015s) It's open source, free to use, and, uh, we'd appreciate the feedback and love. Thank you so much and see you guys on the next one.
