Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

A small agent fix can disturb behavior that was already working. Sofia Jakovcevic, a forward deployed engineer at Arize AI, begins with that regression problem: tracing a failure helps explain it, but the next change still needs to be checked against the other workflows users depend on. Observability supplies the record of what the agent does, step by step.

The support-agent example starts with a security engineer proposing one line of policy: require confirmation before taking any action. Natural language makes the change feel easy to reason about. A reviewer understands the instruction and assumes that understanding extends to its effect on the agent. Pushing the change on that intuition is what Jakovcevic calls “coding on vibes.”

In the scenario, the change goes into production, and evaluations run a week later reveal two opposite movements:

• Action safety: +17 points. Requiring confirmation improves the safety score on these runs.
• Task completion: −17 points. Every action now waits for confirmation. If a user leaves the session, the remaining work never finishes.

The average of those two changes is zero. That number hides the release decision: the agent became more cautious and stopped finishing as many tasks. Keeping the scores separate exposes a tradeoff that the one-line edit did not make obvious. Improving one dimension is insufficient evidence that the agent improved overall.

The code review contains a one-line edit to a system-prompt Markdown file. To understand why that is an incomplete view, compare the edited instruction with the execution paths it affects: the new policy adds steps and changes how the agent calls tools or works with subagents. A local text change can alter many possible trajectories through the system.

A trace is a structured record of what happened during a run. It captures the path the agent actually took, including its calls and interactions. Those recorded events expose consequences that a prompt diff alone cannot establish. The distinction is captured in the line, “the diff tells you what changed, but the trace tells you what actually happened.”

For this kind of regression, a readable prompt file is insufficient evidence of a successful workflow. The support agent can follow its new confirmation policy and still leave the task incomplete. These are runtime outcomes, so regression testing has to start from observed executions and the failures they reveal.

The first step is to select a dataset from production traces. Real user queries, inputs, and tool calls provide material for testing the workflows the system actually encounters. When that material already exists, there is no need to begin by synthesizing a replacement workload.

Selection needs to cover four different kinds of cases:

• Representative workflows: Include the ordinary tasks users bring to the system.
• Known failures: Keep runs that error or fail an evaluation, so a proposed repair is tested against the problem it is meant to solve.
• Edge cases: Include uncommon but consequential workflows that average traffic can underrepresent.
• Successful workflows: Keep cases that already work, so the suite can detect when a fix breaks them.

Successful cases are easy to forget because they do not look like debugging material. Yet they test the other half of a repair: whether the candidate preserves useful behavior. For the support agent, examples of successful task completion belong in the suite alongside the cases used to assess action safety. Testing only the problem that prompted the change leaves previously working behavior unprotected.

Step two defines how each run will be scored. The dimensions can include hallucination, accuracy, tool trajectory—the sequence of tool actions—and step efficiency. These capture different questions about quality, so the team builds an evaluator for each dimension it cares about.

For criteria that require judgment, an LLM-as-a-judge reads a span, a full trace, or other logged data under a rubric. That guide defines the distinction to assess, such as factual versus hallucinated content or efficient versus inefficient execution. The model applies those criteria to the recorded behavior, automating the scoring step across the dataset.

The task-completion example shows how a judgment becomes a score: the evaluator decides whether the task was complete or incomplete, then represents that classification as a binary 0/1 result. The concrete relationship to inspect is between the completion criterion, the assigned label, and its numeric representation.

A score makes runs easier to compare, while its meaning remains tied to the rubric and the execution data being judged. Other evaluators can use a 0–5 scale or multiple labels. Binary classification is one useful scoring format, rather than a requirement for every quality dimension.

Different metrics call for different scoring mechanisms:

• Code-based evaluators: Apply deterministic checks that can be expressed as rules.
• LLM judges: Assess criteria that require interpretation of the output or execution.

ML-based evaluators are another option. Together, the selected evaluators form the testing suite.

Step three runs the current agent on the selected production-derived dataset and scores the resulting outputs. Those results establish the baseline. The SDK experiment interface takes the dataset, the task that executes the agent, and the evaluator suite. The recorded production cases supply the inputs; the experiment executes the agent again to measure its behavior.

Step four introduces one change, then reruns the experiment with the same cases and evaluators. A prompt policy can be tested this way, as can a handoff, a tool, or a model. The comparison measures the candidate’s behavior before deploying it to production, allowing several alternatives to be tested against an established baseline.

The model-selection example makes the spending decision explicit. If a proposed model costs five times as much, the team needs to determine whether its measured improvements justify that cost. The evaluation scores provide evidence about what the extra spending buys, rather than leaving the decision to enthusiasm for a new model.

Step five puts the experiment into CI. For example, a GitHub Actions workflow can run the experiment for each pull request and block merging until the candidate meets every regression criterion the team has declared. A failing candidate returns for revision and another experiment. The comparison below makes the gate’s inputs explicit: baseline and candidate scores produced from the same cases and scoring rules.

The demo opens existing refund-agent experiments in Arize so the tradeoff can be compared across changes. One run has the familiar +17 points in action safety and −17 in task completion. Another candidate is reported at +18 points in action safety and +6 in task completion; the particular edit that produced that result is not specified.

Each listed run represents a tested configuration, involving a prompt edit and possibly a model change. Keeping safety and completion separate shows why the first fix need not be the stopping point: the improved candidate gains on both dimensions. The experiment comparison gives the team a way to choose among measured tradeoffs.

To configure an experiment, Prompt Playground represents a single agent with its prompt, attached tools, model, and model settings. The walkthrough loads version one of a prompt from Prompt Hub and selects a 100-row dataset containing edge cases, known failures, and successful workflows to protect.

Duplicating the setup creates a side-by-side comparison. One copy can be edited directly, or another saved prompt version can be loaded. Task completion and action safety are added as evaluators. The available options also include hallucination and user-frustration checks, code templates, and security checks, with custom evaluators supported.

The experiment workflow runs the agent on every dataset record, scores the output, and records the run in the dataset’s experiment history. That history tracks evaluation results alongside latency, total cost, and error rates. A team can therefore compare the operational effects of playground runs and pull-request changes over time, as well as their quality scores.

For changes to communication between subagents, the executable application itself becomes the test target. Agent Playground connects to a remote agent, for example through an HTTP call. The team changes the orchestration in its codebase while the playground invokes that running system.

Each dataset record becomes input to a remote call. The application executes the workflow and returns an output; the evaluator suite then scores that output. The scoring mechanism remains the same as in Prompt Playground, while Agent Playground includes the application’s multi-agent orchestration in the execution being tested.

The workflow is also available through the SDK, REST, CLI, and CI integrations. These access paths let experiments run programmatically, so testing a candidate does not require someone to operate the playground by hand.

Coding agents are another way to drive the workflow. Arize’s public skills repository is presented as a way for them to access the platform’s UI capabilities from their coding environment. That connection puts production behavior within reach of the same agent that is proposing a code change.

In the human-run workflow, engineers collect traces for production runs, evaluate them, make a prompt, model, tool, skill, or policy change, and test it through experiments. A passing change can ship. A failure supplies another case to investigate and retain.

The persistent result of that investigation should be a new regression case. A failed production trace enters the testing suite so future changes are checked against it. The rule is “every failure should become a test.” Alongside the successful workflows already being protected, these additions let the suite accumulate experience: the next candidate is tested against more of the system’s known failure modes.

The next step is a proposal: let an agent perform the investigation and propose the fix. That proposal targets the time people spend reviewing dashboards, triaging failures, shipping fixes, and reviewing pull requests. It extends observability from a tool for human diagnosis to context that an agent can use when deciding what to change.

A coding agent restricted to the repository shares the earlier reviewer’s blind spot: it can read the new confirmation rule without seeing the extra waits and unfinished sessions. Connecting it to production traces gives it evidence of those consequences. The proposed autonomous loop would use that evidence to investigate a failure, propose a repair, and evaluate the candidate against the regression suite. That is the work behind the closing ambition: “The best agents fix themselves.”