Agents That Self Improve While You Sleep - Fuad Ali, Arize AI

In Fuad Ali’s opening failure scenario, a shopping assistant tells customers it sells no board games, and an engineer gets paged at 4 a.m. Ali, a product manager at Arize, starts with a familiar reliability problem made harder by agents: different execution paths create different failure modes, and the volume of activity makes manual investigation difficult.

The workshop’s running example is Wonder Toys, a local chat application for finding and purchasing toys. Following along requires the application, Node and Python, an Arize AX space ID and API key, and an LLM provider key. Coding-agent skills help set up the application and connect it to Arize’s tools. The exercise follows one agent from ordinary customer requests into the evidence needed to improve it.

Tracing records what happened; evaluation asks whether the behavior was right. Completing the workflow requires someone—or another agent—to inspect those traces, preserve failures as regression tests, and verify a proposed repair. Signal, Arize’s investigation agent, connects these activities by reading traces, identifying failures and helping produce fixes, datasets, evaluators and experiments. A person can then review and merge the change. Each run leaves more context for the next investigation instead of making the team rediscover the same failure.

Wonder Toys uses FastAPI, Next.js, ChromaDB and agent SDKs, and sends traces to Arize AX. Its demonstration bug is planted in a tool’s filter. Instrumentation starts through skills that wrap the CLI and MCP integrations, with auto-instrumenters for supported providers and agent frameworks. This makes the application’s internal activity available for inspection without requiring every attribute to be recorded by hand.

The trace view answers two practical questions: where did the request spend time, and what information was available at each step? For “Show me toys for nine-year-olds,” the request passes through the agent, model calls and tools such as product search. The trace records start and end times, latency, inputs, metadata, the stage of execution and database results. Those details let an investigator connect a tool’s output to the agent’s next decision.

A tool-call span records the arguments sent to the tool and the results it returned. Model-call records add prompts and token counts. The AX CLI’s guided setup turns these records into the trace tree, while session grouping connects multiple exchanges with one customer. That grouping matters when the behavior worth evaluating spans several turns. For a related application to coding workflows, Arize’s coding-agent tracing walkthrough explains how similar records expose file operations, commands, retries and resource use.

Signal watches a project receiving live traces. Its default investigation schedule is every six hours, with other cadences configurable. Connecting a GitHub repository gives it two complementary inputs: execution history showing what the agent did, and source code defining how its tools work. Together, they let an investigation move from a recurring symptom to a proposed code change.

For Wonder Toys, Signal groups redundant product-search retries into an issue. The traces show three sequential LLM calls and high latency associated with product search. The investigation points toward a casing problem in category filtering. The visible symptom is extra work, but the evidence directs attention to the values crossing the model-to-tool interface.

The issue becomes a starting point for several concrete actions. Its failure cases can be saved to a dataset, an evaluator can detect a recurrence, and GitHub integration can open an issue or pull request. Ali has already asked Signal to attempt the repair, so the next step is to inspect that investigation session and the reason it gives for the failure.

Comparing the tool argument with the stored category format explains the empty search result. After six minutes of work, Signal identifies that product categories are stored as lowercase, hyphenated slugs, while the LLM supplies human-readable names with title case and spaces. Without normalization between those representations, the filtered search returns no result. The model’s category name can look sensible to a person while failing to match the database’s representation.

That empty result changes the agent’s behavior: it broadens the search to every product. The fallback and additional model calls drive the reported latency up to roughly 14–17 seconds. The proposed repair therefore needs to make the category value used by the filter agree with the stored values. Counting retries reveals the operational cost; joining the trace to the repository explains what has to change.

Agent Studio generalizes the investigation setup: choose an agent configuration, point it at a project, give it a task and connect skills. The task determines which additional evidence is useful.

• Cost investigations: A cost agent can flag expensive models doing simple work, suggest cheaper routing and estimate savings.
• Permission checks: A skill can connect to a system holding role-based access information so an investigator can compare tool activity with the agent’s permissions.
• Operational context: A Datadog connection can supply logs beyond the agent’s own trace.

These agents run in Arize-managed sandboxes, with options to bring a different coding harness or sandbox provider. The workshop names Codex, Cursor, Claude Code and OpenCode as harness options, and Modal, Daytona and E2B as sandbox examples. Skills extend what the investigator can call, provided the needed API access is available to its harness.

An investigation can start on a schedule or in response to an alert. Ali’s example triggers one when latency rises above ten seconds. Once findings arrive, Signal keeps one ticket per problem and deduplicates later occurrences. A team can resolve or ignore an issue without receiving hundreds of separate tickets for the same underlying evidence. That keeps repeated failures available for analysis while limiting repetitive triage.

Saving a failure to a dataset turns an incident into a reusable test case. Ali calls these growing collections “living fixtures”: they retain cases that actually failed, then expand as the product gains new use cases and production reveals new failure modes. A later repair can be checked against this accumulated history. The definition of acceptable behavior develops with the application instead of remaining fixed at its first release.

The dataset preserves the cases; an evaluator defines how to judge behavior on them. Different checks call for different mechanisms.

• Code checks: Deterministic rules can test whether a field exists, JSON is valid, a filter was applied or a tool received the required arguments. These are inexpensive checks when the rule can be stated precisely.
• LLM as a judge: Another model scores a trace when the judgment is qualitative or less easily expressed in code. Each evaluated record adds a model call, so filtering and sampling control the workload.
• Agent as a judge: For a more open-ended goal, an agent can choose how to assess completion and which scores to produce. Multiple model calls make this the most expensive approach in Ali’s comparison.
• Decision models: Ali presents these as a cheaper option for bounded decisions, with outputs such as yes/no, a choice or a score.

Combining evaluation results with traces gives a coding agent a more specific repair task. It can inspect the execution that occurred and the criteria that execution failed. Signal supplies that context to a coding harness in its managed sandbox, so a proposed change can respond to both the implementation and the intended behavior.

Three human decisions shape the usefulness of this automation.

• Review the repair: Read the proposed code change and decide whether it addresses a real problem.
• Curate the dataset: Choose which findings deserve to become regression fixtures. The demonstrated add-to-dataset action is deliberately manual so a person can make that judgment.
• Design the evaluation: Specify what good behavior means. An agent can help write the check, but its criterion still expresses a product decision.

Signal has found two issues in Wonder Toys: the planted redundant-search bug and an orphan-span issue caused by Ali’s instrumentation mistake. Creating an evaluator from the retry issue needs a model change during the demo, after which the workflow opens Alyx, Arize’s in-house agent. Alyx uses the issue’s supporting traces to draft the evaluator prompt.

The generated template makes the proposed failure criterion inspectable. It examines the complete trajectory, including tool calls, looks for unnecessary calls, and classifies a run as a redundant retry when product search is called more than once. It then places a score and classification on the trace itself. The evidence has become a recurring test with an explicit input, condition and recorded result.

That criterion encodes an assumption about the task: repeated product searches in this trajectory are unnecessary. Reviewing the assumption is part of evaluator design. A clearly written prompt can still enforce the wrong expectation if the product’s intended behavior has not been defined carefully.

Once deployed on incoming traces, the evaluation can feed charts, monitoring and alerts. Its own spend can also be tracked. If the development workflow later catches the same failure through a regression check, the team may choose to retire the production evaluator and stop paying for another model call on each trace. Evaluation has an operating cost and a lifecycle of its own.

A plausible pull request still leaves two questions: does the change fix the observed failure, and does it damage behavior that previously worked? Agents make the second question especially important. A small system-prompt change can alter how other tools are used, so validating only the symptom that motivated the patch can miss a regression elsewhere.

The category-fix experiment combines the saved failure cases, the proposed version of the agent and evaluators that score its new behavior. It reruns the agent on those cases instead of merely judging the patch’s explanation. Through the AX CLI, a coding agent with access to the source can replay the cases against the second version and collect the resulting scores.

Ali accepts an 88.9% result in this demonstration with a quick “Ship it.” The metric and denominator behind that percentage are unspecified, so it serves here as his demo acceptance judgment. The acceptance threshold remains a decision for the application rather than a number supplied by the improvement loop itself.

The intended validation package pairs the PR with behavioral comparisons: rerun the previously failed cases, count tool retries and measure latency again. Fewer retries and lower latency would show progress on the original symptom. Ali’s example also includes three existing evaluators that should not regress. Keeping those checks in the experiment makes the proposed repair answer to more than the one issue it was written to solve.

Experiment outputs remain available for inspection. If a reviewer disagrees with an output or its evaluation, an annotation becomes additional context for the next run. These experiments can also become CI/CD checks before deployment, although replaying an agent and invoking model-based judges on every push can be expensive. Model choice, execution frequency and sampling determine how much validation the team can afford to run routinely.

Separating when work starts from where its evidence goes clarifies what continues unattended. In the described configuration, the evaluator runs on every new trace and supplies scores for the next Signal investigation. Task automation can append failed evaluations to a dataset, growing the regression fixture. Signal separately runs every six hours or responds to an alert or monitor; weekly digests summarize its findings.

The remaining release decisions concern whether an issue is real, whether the output and trajectory meet expectations, and whether the code is ready to promote. Annotations record what the desired result should have been. Auto-merging is an option Ali leaves to the team’s discretion, while deploying the validated version is a distinct step after evaluating the evidence and reviewing the change.

Confidence scores can make human review more selective. A decision model can return an explicit score, and configured thresholds can route uncertain outputs into an annotation queue. The useful threshold depends on the task and industry. It gives the reviewer a prepared set of cases where judgment is needed, rather than requiring a person to find those cases by reading every trace.

In Ali’s spam-detection example, 4.6% of cases fell between scores of 0.3 and 0.7 and were routed to people; the remaining cases had 99.5% accuracy. Those figures describe that spam test. Choosing a review interval for another application needs its own evidence. The routing map below separates this human-review destination from failure collection and the events that start an investigation.

The managed workflow allows an investigation session to be inspected, interrupted, continued or handed to a local agent. Ali’s longer-term vision is an end-to-end loop with very little human involvement until something needs direct testing or UI validation. Making that vision practical depends heavily on the environment needed to run the agent under test.

Wonder Toys makes replay comparatively simple: start the agent, let it create its own database, and send the requests through again. An application with consequential external dependencies raises harder choices. Should the experiment connect to a mock database, stub particular tools, or run the entire application in a Docker image? Which parts can be reproduced safely? Those choices determine what the experiment can exercise and whether the relevant failure conditions are present.

The stated goal is one click from issue to experiment, including starting the infrastructure necessary to test the repair. Local agents already have access through APIs, the CLI and skills, while connecting Signal to places such as Slack is part of the integration direction. A useful handoff preserves the evidence alongside the proposed change: Ali’s example includes five failed traces in the PR description so a reviewer can inspect the cases behind the patch.

The evaluators themselves also need maintenance. Signal can propose evaluations and inspect their health. A check that almost always passes may be a candidate for retirement to save tokens; a poor evaluator may need refinement. Judge alignment asks whether the evaluator’s scores agree with the human standard of good behavior. Generating more scores has little value when the criterion producing them is wrong.

These integrations work best when human judgments, evaluation scores and trace evidence remain connected in a shared record. The next investigation can then use both what happened and how people assessed it. Automation does the large-scale reading and prepares the reviewer’s work; the reviewer’s decisions become context the system can use again.

The Q&A moves from a relatively measurable bug to a harder problem: deciding whether an agent produced something useful. The category mismatch has a clear chain from inconsistent casing to retries and latency. Qualitative outcomes need a richer definition of success. A single labeled example can encourage overfitting or fail to generalize, so the growing fixtures and annotations become especially important.

Alyx’s filter-generation feature illustrates the gap. A user searching agents or traces can write a filter manually or ask Alyx to generate one. The generated filter may execute successfully and return results while still missing what the user wanted. Successful execution supplies one check, but judging the usefulness of the chosen filter requires more context.

Two forms of evidence help establish that context.

• Session behavior: Inspect the whole interaction. Did the user ask again? How many additional filters did they create? Subsequent actions provide clues about whether the first result helped.
• Expert annotations: Read the request, inspect the proposed filter and its results, and record whether they match the intended task. A product manager or domain expert can supply the expectation that successful execution alone does not express.

Combining these observations with new use cases produces material for the evaluator template. A coding agent can help turn the context into a prompt, after which the practical optimization problem is to preserve useful judgments on a sufficiently inexpensive model to run at scale. The evolving examples explain what the judge should distinguish; model and prompt choices determine how economically it can make those distinctions.

The severity of a rare failure changes the value of evaluation. Even 99.9% correct behavior can leave a catastrophic 0.1%. That qualifies the earlier suggestion to retire checks that almost always pass: the decision depends on the consequences of the failures still being caught, as well as the cost of running the evaluator.

Ali separates two places to spend the evaluation budget.

• Development and CI/CD: Keep known failure cases in a fixture and rerun them on pushes or before merging a PR. Some checks should be required so an already understood failure cannot quietly return.
• Live production traces: Evaluate a selected share of new activity to look for behavior the existing dataset does not cover. Sampling and model choice control the cost of this continuing observation.

These two uses cover different weaknesses. Testing a historical dataset gives confidence about those cases, while production monitoring encounters new ones. Evaluating every production trace can be costly, and relying entirely on development fixtures assumes coverage the team is unlikely to have. The useful balance depends on evaluation cost, failure cost and the model chosen to do the judging.

The investigator is observable too. Arize traces its evaluations, and Signal produces traces of its own work. Those records answer how an issue was produced: they expose the instructions, shell calls and internal MCP calls used during the investigation, followed by issue creation. This makes the reviewer’s behavior available for the same kind of inspection applied to Wonder Toys.

Arize uses Signal to evaluate Signal internally, but the available success criteria eventually become indirect. PR merge rate, session transcripts, handoffs, dataset creation and evaluator creation can provide feedback. They remain proxies for whether the proposed work actually helped a customer. At some point, the customer knows more about the desired outcome than the platform can infer from its own activity.

That is a practical stopping condition for recursive evaluation. Once another reviewer is only rereading an agent transcript without additional ground truth, the extra cost can produce little additional confidence. Ali’s sharper description is “vibing with the agent’s output.” More layers of judgment need more meaningful evidence, or they risk making the process look productive without establishing that it improved anything.

The demonstrated evaluators inspect an agent after it has run. Ali sees low-latency decision models as an opportunity to place checks inside execution, where a judgment could help route a request while it is still being handled. This is a direction he is excited about, distinct from the post-execution workflow demonstrated earlier. Cheaper judgments could also make evaluation practical for more teams and more traces.

Small models also suggest possible on-device uses. For multimodal evaluation, latency and subjective judgment combine: an evaluator may need to assess a generated image or an agent’s interpretation of a clickable UI element. Confidence scores offer a way to route uncertain cases to a person sooner. The benefit here is a faster path to the right reviewer when the model’s own judgment is insufficient.

An investigation can include downstream application evidence as well as agent traces. The Datadog skill can pull metrics, logs and APM traces around an alert, and custom skills can connect other sources. The selection still matters: connect the systems relevant to the investigation, give the agent suitable access, and avoid filling its context with information that does not help explain the problem.

The improvement agent also has a bill. Harness choice, underlying providers and execution frequency all affect its cost, which can be inspected through Signal’s own traces. Ali’s hypothetical makes the comparison concrete: if the shopping assistant costs $8 to run and its improvement loop costs $20, what does that additional spend buy? There is no fixed budget share offered as the right answer for every application.

The available controls operate at different points in that bill.

• Context and connected skills: Supply the information that changes the investigation’s usefulness; additional context can add cost without helping.
• Evaluation design and cadence: Choose which checks run, how often they run and what share of traces they inspect.
• Execution environment: The capabilities are also available through APIs, so a team can export work to its own coding agents when that arrangement has more favorable costs.

Apply the same cost accounting to the investigator that you apply to the customer-facing agent. Then adjust its context, cadence or execution setup in response to the value of the results. The overnight work becomes useful when the failures it finds and the repairs it helps validate justify the resources spent finding them.