# Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

Sofia Jakovcevic, Arize AI

[Talk page](https://ai.engineer/talks/v5tm5EAzsAs-your-agent-fix-just-broke-something-else) · [Video](https://www.youtube.com/watch?v=v5tm5EAzsAs)

## Your Agent Fix Just Broke Something Else

A confirmation rule makes a support agent safer while reducing task completion. The example shows how to build regression tests from production traces, compare candidate changes, and connect coding agents to runtime evidence.

![Recording frame at 4:20: Your Agent Fix Just Broke Something Else](https://ai.engineer/api/talk-essay-media/ffd0b5a906ed9231c8c65b9f073c9af762723e1faca773dfb9f2bae76ba7c7c7)

Your Agent Fix Just Broke Something Else — recording at 4:20

### One confirmation rule, two opposing results

[0:12](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=12s)

A small agent fix can disturb behavior that was already working. The speaker, a forward deployed engineer at Arize AI, begins with that regression problem: tracing a failure helps explain it, but the next change still needs to be checked against the other workflows users depend on. Observability supplies the record of what the agent does, step by step.

The support-agent example starts with a security engineer proposing one line of policy: require confirmation before taking any action. Natural language makes the change feel easy to reason about. A reviewer understands the instruction and assumes that understanding extends to its effect on the agent. Pushing the change on that intuition is what the speaker calls “coding on vibes.”

In the scenario, the change goes into production, and evaluations run a week later reveal two opposite movements:

- **Action safety: +17 points.** Requiring confirmation improves the safety score on these runs.
- **Task completion: −17 points.** Every action now waits for confirmation. If a user leaves the session, the remaining work never finishes.

The average of those two changes is zero. That number hides the release decision: the agent became more cautious and stopped finishing as many tasks. Keeping the scores separate exposes a tradeoff that the one-line edit did not make obvious. Improving one dimension is insufficient evidence that the agent improved overall.

### Why a prompt diff misses the regression

[4:41](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=281s)

The code review contains a one-line edit to a system-prompt Markdown file. To understand why that is an incomplete view, compare the edited instruction with the execution paths it affects: the new policy adds steps and changes how the agent calls tools or works with subagents. A local text change can alter many possible trajectories through the system.

![Recording frame at 5:26: Why a prompt diff misses the regression](https://ai.engineer/api/talk-essay-media/d8ab72e7011f41a3175b5217bdefeb2f93bd516497fa355c4d0d2fdfdc4e9213)

Why a prompt diff misses the regression — recording at 5:26

A **trace** is a structured record of what happened during a run. It captures the path the agent actually took, including its calls and interactions. Those recorded events expose consequences that a prompt diff alone cannot establish. The distinction is captured in the line, “the diff tells you what changed, but the trace tells you what actually happened.”

For this kind of regression, a readable prompt file is insufficient evidence of a successful workflow. The support agent can follow its new confirmation policy and still leave the task incomplete. These are runtime outcomes, so regression testing has to start from observed executions and the failures they reveal.

### Step 1: Preserve failures and successes in the dataset

[6:40](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=400s)

The first step is to select a dataset from production traces. Real user queries, inputs, and tool calls provide material for testing the workflows the system actually encounters. When that material already exists, there is no need to begin by synthesizing a replacement workload.

![Recording frame at 7:35: Step 1: Preserve failures and successes in the dataset](https://ai.engineer/api/talk-essay-media/f36f2495b51dfe7b82fde54ca5c326c36092cad4770ec88ea08e4e9129911997)

Step 1: Preserve failures and successes in the dataset — recording at 7:35

Selection needs to cover four different kinds of cases:

- **Representative workflows:** Include the ordinary tasks users bring to the system.
- **Known failures:** Keep runs that error or fail an evaluation, so a proposed repair is tested against the problem it is meant to solve.
- **Edge cases:** Include uncommon but consequential workflows that average traffic can underrepresent.
- **Successful workflows:** Keep cases that already work, so the suite can detect when a fix breaks them.

Successful cases are easy to forget because they do not look like debugging material. Yet they test the other half of a repair: whether the candidate preserves useful behavior. For the support agent, examples of successful task completion belong in the suite alongside the cases used to assess action safety. Testing only the problem that prompted the change leaves previously working behavior unprotected.

### Step 2: Give each quality dimension an evaluator

[7:40](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=460s)

Step two defines how each run will be scored. The dimensions can include hallucination, accuracy, tool trajectory—the sequence of tool actions—and step efficiency. These capture different questions about quality, so the team builds an evaluator for each dimension it cares about.

For criteria that require judgment, an **LLM-as-a-judge** reads a span, a full trace, or other logged data under a rubric. That guide defines the distinction to assess, such as factual versus hallucinated content or efficient versus inefficient execution. The model applies those criteria to the recorded behavior, automating the scoring step across the dataset.

The task-completion judge reads the support-agent trace and asks whether the agent fulfilled the user’s request with no step left unfinished. It labels the run complete or incomplete, yielding a binary 0/1 score.

![Recording frame at 9:12: Step 2: Give each quality dimension an evaluator](https://ai.engineer/api/talk-essay-media/f895b66008744bcb8e8a38f78ce23e74f3d091607b508f15c37a68979657f048)

Step 2: Give each quality dimension an evaluator — recording at 9:12

A score makes runs easier to compare, while its meaning remains tied to the rubric and the execution data being judged. Other evaluators can use a 0–5 scale or multiple labels. Binary classification is one useful scoring format, rather than a requirement for every quality dimension.

Different metrics call for different scoring mechanisms:

- **Code-based evaluators:** Apply deterministic checks that can be expressed as rules.
- **LLM judges:** Assess criteria that require interpretation of the output or execution.

ML-based evaluators are another option. Together, the selected evaluators form the testing suite.

### Steps 3–5: Establish a baseline, compare, and gate the merge

[9:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=571s)

Step three runs the current agent on the selected production-derived dataset and scores the resulting outputs. Those results establish the baseline. The SDK experiment interface takes the dataset, the task that executes the agent, and the evaluator suite. The recorded production cases supply the inputs; the experiment executes the agent again to measure its behavior.

![Recording frame at 11:35: Steps 3–5: Establish a baseline, compare, and gate the merge](https://ai.engineer/api/talk-essay-media/7736bae022d8181d0db232b009e7ece450cba43dd641e523ea1da623ae23c8bd)

Steps 3–5: Establish a baseline, compare, and gate the merge — recording at 11:35

Step four introduces one change, then reruns the experiment with the same cases and evaluators. A prompt policy can be tested this way, as can a handoff, a tool, or a model. The comparison measures the candidate’s behavior before deploying it to production, allowing several alternatives to be tested against an established baseline.

The model-selection example makes the spending decision explicit. If a proposed model costs five times as much, the team needs to determine whether its measured improvements justify that cost. The evaluation scores provide evidence about what the extra spending buys, rather than leaving the decision to enthusiasm for a new model.

Step five puts the experiment into CI. For example, a GitHub Actions workflow can run the experiment for each pull request and block merging until the candidate meets every regression criterion the team has declared. In the example below, task completion can drop by at most 0.02 from baseline, action safety must reach at least 0.95, and the valid-tool-arguments score must equal 1.00. A failing candidate returns for revision and another experiment.

Two experiments feed one regression gate
Run the current agent first, then rerun the changed agent with the same dataset and evaluators. Both score sets inform the comparison; its pass and fail branches control whether the change may merge or needs another revision.
- Production dataset — Selected real inputs include ordinary workflows, failures, edge cases, and successes to preserve.
- Evaluator suite — Scoring rules for each dimension the team wants to measure.
- Baseline experiment — Run the current agent first and score its outputs.
- Candidate experiment — Apply a change, rerun the cases, and score the new outputs.
- Compare against merge criteria — Assess candidate results relative to the baseline under the team’s chosen regression criteria.
- Merge permitted — All required regression criteria are met.
- Revise and rerun — A required criterion fails; change and test again.
- Production dataset → Baseline experiment (Supplies case inputs)
- Production dataset → Candidate experiment (Supplies the same case inputs)
- Evaluator suite → Baseline experiment (Scores the outputs)
- Evaluator suite → Candidate experiment (Scores with the same rules)
- Baseline experiment → Compare against merge criteria (Baseline metric scores)
- Candidate experiment → Compare against merge criteria (Candidate metric scores)
- Compare against merge criteria → Merge permitted (All chosen criteria met)
- Compare against merge criteria → Revise and rerun (A chosen criterion fails)

### The refund-agent experiment in Arize

[11:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=691s)

The demo opens existing refund-agent experiments in Arize so the tradeoff can be compared across changes. One run has the familiar +17 points in action safety and −17 in task completion. Another candidate is reported at +18 points in action safety and +6 in task completion; the particular edit that produced that result is not specified.

![Recording frame at 13:35: The refund-agent experiment in Arize](https://ai.engineer/api/talk-essay-media/36f15429b291d5744bdf03671f3c7f665e5b841e8d2973b56856f7e695377fba)

The refund-agent experiment in Arize — recording at 13:35

Each listed run represents a tested configuration, involving a prompt edit and possibly a model change. Keeping safety and completion separate shows why the first fix need not be the stopping point: the improved candidate gains on both dimensions. The experiment comparison gives the team a way to choose among measured tradeoffs.

To configure an experiment, **Prompt Playground** represents a single agent with its prompt, attached tools, model, and model settings. The walkthrough loads version one of a prompt from Prompt Hub and selects a 100-row dataset containing edge cases, known failures, and successful workflows to protect.

Duplicating the setup creates a side-by-side comparison. One copy can be edited directly, or another saved prompt version can be loaded. Task completion and action safety are added as evaluators. The available options also include hallucination and user-frustration checks, code templates, and security checks, with custom evaluators supported.

The experiment workflow runs the agent on every dataset record, scores the output, and records the run in the dataset’s experiment history. That history tracks evaluation results alongside latency, total cost, and error rates. A team can therefore compare the operational effects of playground runs and pull-request changes over time, as well as their quality scores.

### Evaluate the full system through Agent Playground

[14:21](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=861s)

For changes to communication between subagents, the executable application itself becomes the test target. **Agent Playground** connects to a remote agent, for example through an HTTP call. The team changes the orchestration in its codebase while the playground invokes that running system.

![Recording frame at 15:08: Evaluate the full system through Agent Playground](https://ai.engineer/api/talk-essay-media/01c48e36f1f6617197ba3a7c8356b9198f213439eb9974e2b07d5204c5712d11)

Evaluate the full system through Agent Playground — recording at 15:08

Each dataset record becomes input to a remote call. The application executes the workflow and returns an output; the evaluator suite then scores that output. The scoring mechanism remains the same as in Prompt Playground, while Agent Playground includes the application’s multi-agent orchestration in the execution being tested.

The workflow is also available through the SDK, REST, CLI, and CI integrations. These access paths let experiments run programmatically, so testing a candidate does not require someone to operate the playground by hand.

### From production failures to fixes proposed by agents

[15:47](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=947s)

Coding agents are another way to drive the workflow. Arize’s public skills repository is presented as a way for them to access the platform’s UI capabilities from their coding environment. That connection puts production behavior within reach of the same agent that is proposing a code change.

![Recording frame at 17:08: From production failures to fixes proposed by agents](https://ai.engineer/api/talk-essay-media/51aeaed6f024a47fc16616704440022163fb60c0929aefc0f07d10743a4f934b)

From production failures to fixes proposed by agents — recording at 17:08

In the human-run workflow, engineers collect traces for production runs, evaluate them, make a prompt, model, tool, skill, or policy change, and test it through experiments. A passing change can ship. A failure supplies another case to investigate and retain.

The persistent result of that investigation should be a new regression case. A failed production trace enters the testing suite so future changes are checked against it. The rule is “every failure should become a test.” Alongside the successful workflows already being protected, these additions let the suite accumulate experience: the next candidate is tested against more of the system’s known failure modes.

The next step is a proposal: let an agent perform the investigation and propose the fix. That proposal targets the time people spend reviewing dashboards, triaging failures, shipping fixes, and reviewing pull requests. It extends observability from a tool for human diagnosis to context that an agent can use when deciding what to change.

A coding agent restricted to the repository shares the earlier reviewer’s blind spot: it can read the new confirmation rule without seeing the extra waits and unfinished sessions. Connecting it to production traces gives it evidence of those consequences. The proposed autonomous loop would use that evidence to investigate a failure, propose a repair, and evaluate the candidate against the regression suite. That is the work behind the closing ambition: “The best agents fix themselves.”

## Key takeaways

- Check quality dimensions separately. In the support-agent example, a confirmation rule adds 17 points to action safety and removes 17 from task completion; their average hides the completion regression.
- Build regression datasets from representative production workflows, known failures, edge cases, and successful workflows. Successful cases test whether a repair preserves behavior that already worked.
- Give each important quality dimension a scoring rule. LLM judges apply rubrics to execution evidence, while code evaluators suit deterministic checks; the resulting scores make experiments comparable.
- Run a baseline, change the agent, and rerun the same cases and evaluators before deployment. CI can require every pull request to meet the team’s declared regression criteria before merging.
- Retain failed production traces as new regression cases. Giving coding agents access to that runtime evidence supports the proposed next step: agents that investigate failures, propose repairs, and evaluate their own changes.

## Related talks

- [Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI](https://ai.engineer/talks/fnuNVFgLjHA-building-self-learning-loops-your-agent-fuad) — A related talk on building self-learning loops for agents.
- [How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase](https://ai.engineer/talks/_4Ph2dhmUFI-know-when-ai-agent-work-unsupervised-evan) — A related talk on deciding when an AI agent can work unsupervised.

## Slides

### Your agent fix just broke something else

[0:14](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=14s) [Sofia Jakovcevic introduces a regression loop for testing an agent before shipping each change.](https://assets.ai.engineer/sites/aiecode2025/sha256/31cbf26a16b600702c80e1e80572ffacfcf4cd4e22d9a92f8eb1fed06b648bbb.webp)

Your agent fix just broke something else

### Agents fail at runtime

[4:46](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=286s) [A graph illustration introduces the section on runtime failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/db5eb8a4cdd3ba8f206407d1f6b8471785dcd1dbd00c5419a7aa22ced7d1ec53.webp)

Agents fail at runtime

### Why agent regressions are different

[5:02](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=302s) [The slide compares a one-line change in a system prompt with the execution path it produces.](https://assets.ai.engineer/sites/aiecode2025/sha256/11455e4ac212d9f0a100f878a76b1889d485f57709cd0189f2398696ec521921.webp)

What code review sees · What runs in production

### Traces as the new source of truth

[5:49](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=349s) [Two panels contrast logic written down in code with behavior that emerges at runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a1e8d99b98ca9b3765b982d9fb0849fb2e4f731e8645216263ce77837cfcecb.webp)

The logic is written down. The logic only exists at runtime.

### Select a dataset

[6:47](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=407s) [Production traces supply examples of a representative workflow, known failure, edge case and success to protect.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad928a251f61257cf8d441edca0a9d8623906c10802a08aa0a5d2a133dfc694b.webp)

Select a dataset

### Add evaluators

[7:49](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=469s) [A support-agent rubric pairs an LLM completion judge with a code check for valid tool arguments.](https://assets.ai.engineer/sites/aiecode2025/sha256/c9f9e6e2b2a53169d55a5642c88a93f84f59e407c94d97906a4c3bb97f6a7e7b.webp)

Add evaluators · LLM-as-a-judge · Code evaluator

### Run a baseline experiment

[9:49](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=589s) [An experiment measures task completion, action safety and valid tool arguments for the unchanged agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/595f91133368357a226c6b56e84c2040c922d30b2e46d361ca2b2b7aa1da1bc3.webp)

Run the baseline experiment

### Make one change and rerun

[10:25](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=625s) [A results table compares scores for the baseline and changed agent across three evaluators.](https://assets.ai.engineer/sites/aiecode2025/sha256/6be506add9ae9c93e015fe7bf81452349c095d9d4dc1067f6e930b25cd25b8f9.webp)

Make one change and rerun

### Run the experiment in CI

[11:13](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=673s) [A continuous integration example runs regression tests against the baseline when a pull request changes the agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ed3aeebd02e6a8b2496648c8d5358abc838e62f67e9f83704bb4540b38660b4.webp)

Put the experiment in CI

### Comparing experiment scores

[11:54](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=714s) [Several colored lines compare results across experiment runs in the live dashboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/7865065ce19d419dc8f9d45bffc092f1dc01b0a2d1d0098a2e1f95c63c0125c4.webp)

Experiment comparison

### Tracing individual runs

[12:35](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=755s) [The demonstration opens a list of traces with per-run status and timing.](https://assets.ai.engineer/sites/aiecode2025/sha256/97608998cecbd13f565472b454c14e8c2a6014c696d5628bb8610bcd4e2b9900.webp)

Traces

### Comparing two runs in detail

[13:03](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=783s) [A side-by-side comparison exposes differences in the records for two experiment runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/49ea658ca35869e5a93fb49f092ab3b769f2a45f25c34d3bf3a4d4a740dfa286.webp)

Experiment comparison

### How engineers close the loop

[16:27](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=987s) [A flowchart runs from observing failures to evaluating, changing, testing and improving the system.](https://assets.ai.engineer/sites/aiecode2025/sha256/48afe93cc1dec3b115fa01fee190dde011bf538e79a79089f32b717eccee8d96.webp)

Observe · Evaluate · Change · Test · Improve

### Turning a failure into a test case

[17:13](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1033s) [A single slide states the rule for what to do with a production failure.](https://assets.ai.engineer/sites/aiecode2025/sha256/08a79b187df9eebc6d422d5edf97b9729949f6ec58e8bf040a2f7eb29b478668.webp)

Every failure should become a test

### From observability to self-improvement

[17:52](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1072s) [Two columns contrast a manual bottleneck with closing the feedback loop at agent speed.](https://assets.ai.engineer/sites/aiecode2025/sha256/a88e519c52a9f174e0a5f7c700a2f3484c41e55fa1ac3574a37d31c9690c3d54.webp)

The Manual Bottleneck · Closing the Loop at Agent Speed

### Learning from failures

[19:00](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1140s) [The closing slide describes agents that act on failure data to improve themselves.](https://assets.ai.engineer/sites/aiecode2025/sha256/d11b7f0d2a265fc9f0f42714c6456ca1200b4c95d97af324adbcfb77769ac853.webp)

The best agents don’t just learn from failures. They fix themselves.

## Full transcript

[0:12](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=12s) Hello, everyone. My name is Sofija Kovcevic. I'm a forward deployed engineer at Arize, and for those who aren't familiar, Arize is an observability backend that is designed to make AI work. Now, what does that mean? Building AI systems can be inherently intuitive in some sense, especially when a lot of that low-level abstraction is... a low-level programming is abstracted into the mind of an agent. Unfortunately, though, that mind can be a bit of a black box. When that black

[0:42](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=42s) box is difficult to understand, it's difficult to predict and therefore difficult to debug, and that's where observability becomes essential. Observability helps you understand what your systems are actually doing. It's essentially shining a light in the black box. And once you understand what the system is doing step by step, you can then evaluate it, run evaluation metrics to track its performance, and then use that data to iterate and improve your agents over time. But I want to take this one step further because the biggest challenge

[1:12](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=72s) isn't determining what broke. The biggest challenge is that when you do submit a fix, you want to make sure that you don't break ten other things along the way. Now raise your hand if you've had that exact issue before. You push... Yep, seeing a lot of hands. You push a simple change, you know, should be, it should be easy, no issues, and next thing you know, everything's on fire. So today we're going to discuss how you can use your observability as a way to build a robust regression process. Not only something that your

[1:42](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=102s) engineers can do, but an automatic process that feeds into your CI/CD. We're gonna break this down to three parts. First, we'll discuss why agent regression is so different and difficult to detect. And the first thing we're gonna touch on is that agents don't fail at the code level, they fail at runtime. We're then gonna use this premise to build out our regression loop. It's gonna be a five-step process. We'll touch on it at a high-level overview, and then we'll move on to the Arize platform and show you exactly how you can do it today. Then we'll close

[2:12](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=132s) off the session by closing that regression loop at agentic speed and touching on where we're seeing the next frontier of agent observability move to. And if you stick with me till the end, I'll introduce a paradigm shift that I hope you'll all find interesting. So let's get started, first with a concrete example. So imagine with me that you have a support agent and you have a security engineer that has approached you, and that security engineer has reviewed the policy of your support agent and determined that he wants to add one line, a simple one line to

[2:42](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=162s) your agent that will ask your agent to confirm before it takes any action. Sounds simple, sounds harmless. Your security engineer says he has the intuition, and oftentimes we're quick to make this change. I'm not even gonna ask if you've done this before, 'cause I know everyone has pushed a one-line prompt change and just assumed everything will go well. And the reason we make that assumption is because we think that because it's written in natural language, that we have this intuition on how these agents are performing.

[3:12](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=192s) We're really coding on vibes, and we don't want to be. We want to be able to code with a numeric evaluation suite backed data. We want to have scores that prove that these changes are actually making a positive impact and not just crossing our fingers. But let's say in this scenario, you did push that one line change, and now you are running your evaluations on your production data a week after it's been submitted. And the first thing you notice is that you actually have a seventeen-point improvement on a action safety. So your security engineer

[3:42](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=222s) was correct. Asking the agent to confirm before taking any action does improve its overall safety. But your observability engineers are continuing to evaluate, and they also check for task completion, and they notice a seventeen-point degradation. Now, this is occurring because if you have an agent that's confirming before every single action it's taking, now tasks are being uncompleted. If a user leaves the session, that session is never completed to the end. And now you have

[4:11](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=251s) incomplete task, and you see this unlikely result. You may not have caught this initially when you were thinking about the project, but your evaluation suite has caught this. And ironically enough, the overall average of both is net zero. And what I want you to take away from here is just because you improve the dimension of one agent does not mean that you're improving the agent as a whole. The question I have for you is, would you have shipped this change? Many of us would have before the evaluation test. And this is why it's so important that we

[4:41](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=281s) get these numeric results so that we're not shipping on vibes, we're shipping with backed data. So to understand where the mistake that we made is, is we made an unrealistic assumption, and that assumption is that the error is going to occur at the code, but the error rarely occurs at the code level. Instead, these agents are failing at runtime. What the code reviewer saw is here on the left-hand side. It was a one-line change in a system prompt Markdown file.

[5:11](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=311s) Easy change, simple to understand, simple to approve. But in reality, this code change was local, but the behavioral changes of this code change was in production. It was in the runtime. This one simple line has shifted how the agent operates. It's adding additional steps. It's changing how the agent works with other sub-agents. It's changing how the agent calls tools. All of these different trajectories, which could be thousands of possible trajectories, are not in your static code. They're only in your observability backend. They're only in your

[5:41](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=341s) traces. And so what we are seeing here is that the diff tells you what changed, but the trace tells you what actually happened. And there's a shift here that traces are becoming the new source of truth. These structured logs of what your agents are actually doing actua-- is what truly represents what these systems are doing. In traditional software, logic was written down. It was very clear to determine what that runtime was before you ever pushed to production. But now the logic only exists at

[6:10](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=370s) runtime. The observability becomes essential when we're looking at regression testing. This gap between what was written and what actually happens is filled with silent errors. So to capture that, we need to start our regression testing by looking at our traces. So let's build out this regression loop together. It's a five-step process. We're gonna start with our production traces. We're gonna build evals, run experiments, and then close it all up to be an automatic gate at CI. First, I'm gonna do a

[6:40](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=400s) five-step high-level overview, and then we're gonna go into the actual product and show you how you can do it yourself. So step one is we're gonna select a dataset. Seems simple enough. We want to pull real production traces. We don't need to synthesize data if we already have real user queries, real inputs, real tool calls. We're pulling from that directly. It's important that our dataset is a representative workf-- is representative of the average workflow that our users are going to be pushing into our systems. But it's also

[7:10](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=430s) important that we capture known failures. We filter out co-- m-moments where it often errors, often fails at eval metrics. Capture those into your data-- datasets. You also wanna capture edge cases, maybe not as common, but just as important. And one thing that we often see users forget is capturing successful workflows that we want to protect. Because, again, we're not just trying to make sure that the fix is gonna work, but we also want to test that we're not gonna break something that was

[7:40](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=460s) already working. So it's important that we capture all four of these in our dataset. Once we have this production dataset, we can then build out the rest of our regression testing suite. The second step is we wanna have a scoring mechanism, and this is where we bring in evaluators. Now, show of hands, how many here are familiar with the concept of evaluating agents?

[8:01](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=481s) I'm so happy to see so many hands. This is a core foundational part of agentic engineering. Evaluators allow us to take metrics and dimensions that are important to your engineers, your product team, and actually quantify them. What do I mean by that? Let's take some metrics that are important for your agents. Things like hallucination, accuracy, tool trajectory, step efficiency. All of these metrics are subjective in nature, and you may have a human that reviews them and can potentially determine

[8:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=511s) whether or not something was hallucinated, whether or not the most efficient pathway was taken. But we wanna be able to automate this process. We wanna turn a subjective metric into a numeric score, and we do this by leveraging LLMs. We use a concept of LLM-as-a-judge, where we judge a span, a trace, some log data against an LLM. The LLM has a particular guide of how it determines whether or not something was hallucinated or factual, something was efficient or not efficient.

[9:01](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=541s) And now we've done essentially a binary classification. In this example at the top right corner, you can see a task completion eval where the LLM-as-a-judge is... has the task of determining whether or not something was complete or incomplete. Now we've taken a subjective metric, and we've essentially turned it into a binary score, zero one. You can translate the score across zero to five. You can have multiple different labels. But you've now turned a quantitative measurement into a qualitative score that you can measure against. You're not limited to LLM-as-a-judge. You can also use

[9:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=571s) code-based evals or ML-based evals as well. Determine-- Based on what that metric is, a code evaluator is great for your deterministic checks, but an LLM-as-a-judge is good for your non-deterministic checks. So once we have a list of all of our metrics that are important to us, we build evals for each of them. Now we have a testing suite that we can go against. Step three is we're gonna run our baseline experiment, and what this experiment is, is we're going to take our agent and run it against all of our production traces, um, all of our production

[10:01](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=601s) datasets. We're-- After we get the output of those production data, production datasets, we're then gonna run our eval testing suite on top of that, and that's what's gonna be our baseline. Now, you can do this in the SDK. It's just run experiment. You'll pass in that dataset, pass in that task, pass in your evaluation suite, and you'll get those results. We're gonna t-touch on how you can do this in the UI, as well as how you can do this with your coding agents. But once you have that baseline experiment, you're then gonna add your one change. Change that policy, rerun the evaluation.

[10:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=631s) What are the eval results? A simple process now that you can quickly iterate on, and not just with one policy change, but test multiple changes within your prompts. Test multiple models. If your team comes with y-- comes to you and says, "We wanna switch to the new Fable model, it's five times more expensive," you're not gonna switch without proving that you're gonna get a good ROI. This is how you can prove that with a numeric set of measurements to det-determine whether or not it actually is improving your investment to switch to a more expensive model. And you can

[11:01](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=661s) adjust this policy, rerun and compare. Rerun and compare how changing the prompt, changing handoffs, changing tools, changing models is all going to affect your overall output, all without having to deploy to production. And then we can automate this. We push this experiment into CI to complete this automation. Now, each time that you have a pull request, let's say you set this up in GitHub Actions, for example, each pull request, each PR is run through an experiment workflow against the evaluations that your team

[11:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=691s) has deemed important, and you don't merge until it meets all of the regression criteria that you have listed. Now you're no longer pushing on vibes. You're pushing with confidence that you actually do have a positive impact on your performance. And if it is negative, you go back, you test, and you reiterate. So let's actually show how to do this in the platform. Let me switch over. Um, here we have, here we have the Arize platform, and I'm showcasing this exact refund agent. You can see in our baseline experiment

[12:01](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=721s) what our scores were, and then you can see that plus seventeen points on action safety and minus seventeen points on task completion. Each one of these points, each one of these line items here is a different test that I ran with different prompt changes, potentially different model changes, and I'm evaluating each one of them to see which one has the best trade-off to get to where my end goal is. And I can see that overall, I'm gonna have a plus eighteen per-- eighteen points on action safety and plus six on task completion. Now, how can we build these individual points? Well, there's a couple

[12:31](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=751s) ways. We'll do this first in the UI. I'm gonna jump into the playground here. I'm gonna create a new playground. We're gonna start with the Prompt Playground to test a single agent. Let's call this AIE NYC. Perfect. Once I'm in the Prompt Playground, here is where I can test a single agent. An agent, a single agent, all is essentially just a single prompt with a set of tools attached to it, a model, LLM configurations, and so we're gonna replicate that here. Now, I've already stored my

[13:01](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=781s) prompt in my Prompt Hub, so I can pull it right here. I've got version one, and then I can pull up my dataset. I've already selected a dataset that I can use here and that I've gone through and pulled out those edge cases, those known failures, and those successful workflows that I want to protect, all listed in this one hundred-row dataset. I can test this right now, or I can test multiple prompts at the same time. So I'm actually gonna duplicate this, and I can change a line item directly in here, or I can load

[13:30](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=810s) up a separate version right here and test side by side, so I can see just how quickl-- just how both of these prompts are affected. Now, of course, we have to add our scoring mechanism, so that's gonna be our evals. I'm gonna click here. I'm gonna add in my task completion, and then I'm also going to add in my action safety. And you can see here that there's a ton of other pre-built hallucin-- uh, pre-built evaluators you can select from, hallucination, user frustration, um, we've got some code templates,

[14:00](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=840s) security checks, et cetera, and you always have the ability to customize your own. Once I run this, it's going to run on every single record, and then it's gonna pull out those eval scores. And then when I go into my datasets here, go into my

[14:21](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=861s) experiment run flow, it'll pop up under here as well. And I can pull up all those metrics that are important for me. I can look at latency, I can look at my evals, I can look at my total costs and error rates and track that over time. So I can look this for PR changes, for playground runs. And there's one more other option that I wanna show here too, and that's if I go into Playground, I click here, and I do Agent Playground.

[14:46](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=886s) Call this AIE... Oh, there we go. Agent Playground allows you to not only evaluate at a single agent but at multiple agent levels. If you give us access to connect to your agent remotely, for example, through a HTTP call. If I go into remote agents here, you can connect your remote agent directly here so that you can change how different sub, sub-agents communicate with each other, all directly in your code base, and then we can call that. So each record in your dataset, we'll call against your agent,

[15:16](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=916s) we'll test, we'll get the output, we'll run those evals, and then score it in the same mechanism. So you have two ways of running pl-- running these experiments within the UI. You can run them in the Prompt Playground, testing a single agent, or run them in the Agent Playground and test a very complicated multi-agentic system. Now, this is in the UI, but of course, you can do this programmatically with the SDK, REST, or CI, or CLI. But my favorite method to do this is actually with coding agents. How familiar is this group with the concept of agent skills?

[15:47](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=947s) Oh, I love to see when people are saying advanced. Agent skills allows you to connect your coding agents with external databases in advanced contexts. Arize has a public repository of skills that allows you to connect to every feature available in the Arize UI. This means that you can close the loop between what your coding agents are producing in your IDE, the code that they're producing, with what is actually happening in your observability backend. So let's actually touch more on that closing the loop.

[16:15](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=975s) So how are we gonna close this loop at agent speed? We've got the same loop, but instead of running it by people, we wanna run it by agents. Now, I'm gonna highlight two key takeaways. We only have two minutes left here. And the first key takeaway I wanna touch on is what the engineers are doing. Today, engineers are the ones that are closing the loop. And to summarize what that loop is, is we have observability first. That is, as your systems are running, we are taking that log data, that tracing data, and we are tracking it for every run in production. We then take that, evaluate it, get

[16:45](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1005s) those scores. Then we can implement a change, and that could be a prompt change, model change, tool, skill, policy change. And then we test that change. We iterate on it with the experiments workflow. Finally, we improve it. If it passes, great, ship it. But if it fails, we close the loop. We take those failed cases, and we bring it back to the observability layer. These new production traces that failed then become new cases in our regression testing suite, and that's the first takeaway here,

[17:15](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1035s) is that every failure should become a test. Every failure should make your agent stronger and harder to break. Once you have these regression tests that are-- regression test suites that are constantly building on themselves, you have a more robust method of continuously iterating and improving. That's takeaway number one. Now, takeaway number two, going back to that loop, I have a question for the group, and that is, what if an agent could do the investigation and propose the fix? Because right now the manual

[17:44](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1064s) investigation is that-- that's the bottleneck. Humans are the bottleneck here. So what we're seeing is a shift from observability to self-improvement. The manual bottleneck here is that humans are still reviewing dashboards. They're still triaging and shipping those fixes and reviewing those PRs, and that is the ultimate bottleneck. So we wanna not only close this loop, but we wanna close it at agentic speeds. We don't wanna just give humans the ability to debug agents. We wanna give agents the ability to debug and understand themselves, and that's the

[18:14](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1094s) shift that we're seeing in observability, is observability is no longer about humans understanding agents. It's about agents understanding themselves, agents having the tools to improve themselves, agents having the tools to be accountable. Remember what we talked about in the very beginning. That is that traces is where the real runtime is. That's where the real logic is stored. If your coding agents are pushing PRs, and I know they are, they're pushing PRs without actually knowing what's happening in production. That means they're pushing PRs blind. They're reliant on the static code.

[18:44](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1124s) They don't know what's actually happening in production. By closing that loop, by connecting your coding harnesses, your coding agents with what is actually happing-- happening in your production database, you are giving your agents the full understanding of what these changes are actually making. And that is the second and final takeaway. That is that the best agents don't just learn from failures. The best agents fix themselves. And that's the future of observability that we're seeing, is that agents can not only see, but they can evaluate and improve themselves on their

[19:14](https://www.youtube.com/watch?v=v5tm5EAzsAs&t=1154s) own. Thank you.
