# Agents That Self Improve While You Sleep - Fuad Ali, Arize AI

Fuad Ali, Arize AI

[Talk page](https://ai.engineer/talks/UGuNt2_Oz8s-agents-that-self-improve-while-you-sleep) · [Video](https://www.youtube.com/watch?v=UGuNt2_Oz8s)

## Agents That Self Improve While You Sleep

A broken category filter in Wonder Toys turns a product search into repeated model calls. Fuad Ali follows that failure through Signal investigations, growing regression datasets and replay experiments, while showing where human judgment, test environments and cost still determine how much to automate.

![Six-step improvement loop: Trace, Signal, Dataset, Eval, Fix and Experiment, with human confirmation and review points.](https://ai.engineer/api/talk-essay-media/b9b5e9065002741684305b3c7cfb01a5fd4cfdfcfe9a3e0de0dabe3e1f73852b)

The workshop’s six-step loop moves from traces to an experiment, then watches new traces for further failures. Confirmation and review remain explicit steps (8:11).

### From a 4 a.m. shopping failure to a reviewable fix

[0:12](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=12s)

In Fuad Ali’s opening failure scenario, a shopping assistant tells customers it sells no board games, and an engineer gets paged at 4 a.m. [Ali](https://ai.engineer/speakers/fuad-ali), a product manager at Arize, starts with a familiar reliability problem made harder by agents: different execution paths create different failure modes, and the volume of activity makes manual investigation difficult.

![Wonder Toys storefront with product cards, category filters and a shopping-assistant input.](https://ai.engineer/api/talk-essay-media/29a598a0a11a75e26ae6690c6d1bd75ccb0794cf4f20357b104ea3d122d6ee7f)

The Wonder Toys storefront supplies the demonstration’s catalog, category filters and shopping-assistant interface (4:39). This frame shows the app, not a completed failure investigation.

The workshop’s running example is Wonder Toys, a local chat application for finding and purchasing toys. Following along requires the application, Node and Python, an Arize AX space ID and API key, and an LLM provider key. Coding-agent skills help set up the application and connect it to Arize’s tools. The exercise follows one agent from ordinary customer requests into the evidence needed to improve it.

Tracing records what happened; evaluation asks whether the behavior was right. Completing the workflow requires someone—or another agent—to inspect those traces, preserve failures as regression tests, and verify a proposed repair. Signal, Arize’s investigation agent, connects these activities by reading traces, identifying failures and helping produce fixes, datasets, evaluators and experiments. A person can then review and merge the change. Each run leaves more context for the next investigation instead of making the team rediscover the same failure.

### Trace the request, including what each tool returned

[8:43](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=523s)

Wonder Toys uses FastAPI, Next.js, ChromaDB and agent SDKs, and sends traces to Arize AX. Its demonstration bug is planted in a tool’s filter. Instrumentation starts through skills that wrap the CLI and MCP integrations, with auto-instrumenters for supported providers and agent frameworks. This makes the application’s internal activity available for inspection without requiring every attribute to be recorded by hand.

The trace view answers two practical questions: where did the request spend time, and what information was available at each step? For “Show me toys for nine-year-olds,” the request passes through the agent, model calls and tools such as product search. The trace records start and end times, latency, inputs, metadata, the stage of execution and database results. Those details let an investigator connect a tool’s output to the agent’s next decision.

![Arize trace viewer with nested model and tool spans, a selected product-search call and returned product records.](https://ai.engineer/api/talk-essay-media/799cd487a9f40a2c2bd1851ef07ef6aa14b53534d0c22e4e7da4a1db9c09b6f4)

A product-search span exposes the returned records beside the request’s trace hierarchy, helping connect an agent action to its actual tool output (10:59).

A tool-call span records the arguments sent to the tool and the results it returned. Model-call records add prompts and token counts. The AX CLI’s guided setup turns these records into the trace tree, while session grouping connects multiple exchanges with one customer. That grouping matters when the behavior worth evaluating spans several turns. For a related application to coding workflows, Arize’s [coding-agent tracing walkthrough](https://arize.com/blog/open-source-coding-agent-tracing/) explains how similar records expose file operations, commands, retries and resource use.

### Signal connects repeated searches to a category mismatch

[12:10](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=730s)

Signal watches a project receiving live traces. Its default investigation schedule is every six hours, with other cadences configurable. Connecting a GitHub repository gives it two complementary inputs: execution history showing what the agent did, and source code defining how its tools work. Together, they let an investigation move from a recurring symptom to a proposed code change.

For Wonder Toys, Signal groups redundant product-search retries into an issue. The traces show three sequential LLM calls and high latency associated with product search. The investigation points toward a casing problem in category filtering. The visible symptom is extra work, but the evidence directs attention to the values crossing the model-to-tool interface.

The issue becomes a starting point for several concrete actions. Its failure cases can be saved to a dataset, an evaluator can detect a recurrence, and GitHub integration can open an issue or pull request. Ali has already asked Signal to attempt the repair, so the next step is to inspect that investigation session and the reason it gives for the failure.

Comparing the tool argument with the stored category format explains the empty search result. After six minutes of work, Signal identifies that product categories are stored as lowercase, hyphenated slugs, while the LLM supplies human-readable names with title case and spaces. Without normalization between those representations, the filtered search returns no result. The model’s category name can look sensible to a person while failing to match the database’s representation.

![Signal issue describing redundant search retries, with supporting traces and actions to create an evaluator, add a dataset item or open GitHub.](https://ai.engineer/api/talk-essay-media/dfeb157ea7bf676421f7324fd27f68c3d30c812849ca22fb6a70c83c8823aced)

Signal groups redundant searches into an issue with supporting traces and follow-up actions for an evaluator, dataset or GitHub work (14:57).

That empty result changes the agent’s behavior: it broadens the search to every product. The fallback and additional model calls drive the reported latency up to roughly 14–17 seconds. The proposed repair therefore needs to make the category value used by the filter agree with the stored values. Counting retries reveals the operational cost; joining the trace to the repository explains what has to change.

### Give investigation agents the context their task needs

[16:22](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=982s)

Agent Studio generalizes the investigation setup: choose an agent configuration, point it at a project, give it a task and connect skills. The task determines which additional evidence is useful.

- **Cost investigations:** A cost agent can flag expensive models doing simple work, suggest cheaper routing and estimate savings.
- **Permission checks:** A skill can connect to a system holding role-based access information so an investigator can compare tool activity with the agent’s permissions.
- **Operational context:** A Datadog connection can supply logs beyond the agent’s own trace.

![Agent Studio configuration form with a project selector, task instructions and a Run control.](https://ai.engineer/api/talk-essay-media/c69080955d820a509a8bfc1f7fc9c47d15a23abb7645900b7fb51dee608702c6)

Agent Studio exposes the project and task configuration before an investigation runs, so the investigator can receive the context for its assigned job (17:05).

These agents run in Arize-managed sandboxes, with options to bring a different coding harness or sandbox provider. The workshop names Codex, Cursor, Claude Code and OpenCode as harness options, and Modal, Daytona and E2B as sandbox examples. Skills extend what the investigator can call, provided the needed API access is available to its harness.

An investigation can start on a schedule or in response to an alert. Ali’s example triggers one when latency rises above ten seconds. Once findings arrive, Signal keeps one ticket per problem and deduplicates later occurrences. A team can resolve or ignore an issue without receiving hundreds of separate tickets for the same underlying evidence. That keeps repeated failures available for analysis while limiting repetitive triage.

### Save the failure, then define what should pass

[18:51](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1131s)

Saving a failure to a dataset turns an incident into a reusable test case. Ali calls these growing collections “living fixtures”: they retain cases that actually failed, then expand as the product gains new use cases and production reveals new failure modes. A later repair can be checked against this accumulated history. The definition of acceptable behavior develops with the application instead of remaining fixed at its first release.

The dataset preserves the cases; an evaluator defines how to judge behavior on them. Different checks call for different mechanisms.

- **Code checks:** Deterministic rules can test whether a field exists, JSON is valid, a filter was applied or a tool received the required arguments. These are inexpensive checks when the rule can be stated precisely.
- **LLM as a judge:** Another model scores a trace when the judgment is qualitative or less easily expressed in code. Each evaluated record adds a model call, so filtering and sampling control the workload.
- **Agent as a judge:** For a more open-ended goal, an agent can choose how to assess completion and which scores to produce. Multiple model calls make this the most expensive approach in Ali’s comparison.
- **Decision models:** Ali presents these as a cheaper option for bounded decisions, with outputs such as yes/no, a choice or a score.

Combining evaluation results with traces gives a coding agent a more specific repair task. It can inspect the execution that occurred and the criteria that execution failed. Signal supplies that context to a coding harness in its managed sandbox, so a proposed change can respond to both the implementation and the intended behavior.

### Turn the retry issue into an evaluator

[23:26](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1406s)

Three human decisions shape the usefulness of this automation.

- **Review the repair:** Read the proposed code change and decide whether it addresses a real problem.
- **Curate the dataset:** Choose which findings deserve to become regression fixtures. The demonstrated add-to-dataset action is deliberately manual so a person can make that judgment.
- **Design the evaluation:** Specify what good behavior means. An agent can help write the check, but its criterion still expresses a product decision.

Signal has found two issues in Wonder Toys: the planted redundant-search bug and an orphan-span issue caused by Ali’s instrumentation mistake. Creating an evaluator from the retry issue needs a model change during the demo, after which the workflow opens Alyx, Arize’s in-house agent. Alyx uses the issue’s supporting traces to draft the evaluator prompt.

The generated template makes the proposed failure criterion inspectable. It examines the complete trajectory, including tool calls, looks for unnecessary calls, and classifies a run as a redundant retry when product search is called more than once. It then places a score and classification on the trace itself. The evidence has become a recurring test with an explicit input, condition and recorded result.

That criterion encodes an assumption about the task: repeated product searches in this trajectory are unnecessary. Reviewing the assumption is part of evaluator design. A clearly written prompt can still enforce the wrong expectation if the product’s intended behavior has not been defined carefully.

Once deployed on incoming traces, the evaluation can feed charts, monitoring and alerts. Its own spend can also be tracked. If the development workflow later catches the same failure through a regression check, the team may choose to retire the production evaluator and stop paying for another model call on each trace. Evaluation has an operating cost and a lifecycle of its own.

### Replay the failed requests against the proposed fix

[26:50](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1610s)

A plausible pull request still leaves two questions: does the change fix the observed failure, and does it damage behavior that previously worked? Agents make the second question especially important. A small system-prompt change can alter how other tools are used, so validating only the symptom that motivated the patch can miss a regression elsewhere.

The category-fix experiment combines the saved failure cases, the proposed version of the agent and evaluators that score its new behavior. It reruns the agent on those cases instead of merely judging the patch’s explanation. Through the AX CLI, a coding agent with access to the source can replay the cases against the second version and collect the resulting scores.

On the check for whether the agent called the expected tool, eight of nine scored runs received a high score (88.9%) and one received a low score (11.1%). Ali accepts that result in this demonstration with a quick “Ship it.” The acceptance threshold remains a decision for the application.

![Regression experiment for the category-code fix, showing a score distribution and individual input/output rows.](https://ai.engineer/api/talk-essay-media/284dfc450ea7148009f7ff5dec865036a2804588c9ccd12b71d25d7f03fb1a12)

The category-code regression experiment shows a score distribution and individual runs after the proposed fix (28:57). Its displayed percentage describes this demo, not general agent reliability.

The intended validation package pairs the PR with behavioral comparisons: rerun the previously failed cases, count tool retries and measure latency again. Fewer retries and lower latency would show progress on the original symptom. Ali’s example also includes three existing evaluators that should not regress. Keeping those checks in the experiment makes the proposed repair answer to more than the one issue it was written to solve.

Experiment outputs remain available for inspection. If a reviewer disagrees with an output or its evaluation, an annotation becomes additional context for the next run. These experiments can also become CI/CD checks before deployment, although replaying an agent and invoking model-based judges on every push can be expensive. Model choice, execution frequency and sampling determine how much validation the team can afford to run routinely.

### Keep evidence flowing and route uncertain cases to people

[31:03](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1863s)

Separating when work starts from where its evidence goes clarifies what continues unattended. In the described configuration, the evaluator runs on every new trace and supplies scores for the next Signal investigation. Task automation can append failed evaluations to a dataset, growing the regression fixture. Signal separately runs every six hours or responds to an alert or monitor; weekly digests summarize its findings.

The remaining release decisions concern whether an issue is real, whether the output and trajectory meet expectations, and whether the code is ready to promote. Annotations record what the desired result should have been. Auto-merging is an option Ali leaves to the team’s discretion, while deploying the validated version is a distinct step after evaluating the evidence and reviewing the change.

Confidence scores can make human review more selective. A decision model can return an explicit score, and configured thresholds can route uncertain outputs into an annotation queue. The useful threshold depends on the task and industry. It gives the reviewer a prepared set of cases where judgment is needed, rather than requiring a person to find those cases by reading every trace.

In Ali’s spam-detection example, 4.6% of cases fell between scores of 0.3 and 0.7 and were routed to people; the remaining cases had 99.5% accuracy. Those figures describe that spam test. Choosing a score range for human review in another application needs its own evidence.

Which events start Signal, and where do evaluation results go?
Data arrows transfer traces, scores and cases. Control arrows start scheduled or alert-driven work. Evaluation results can feed regression tests and human review while also becoming evidence for a later investigation.
- New agent traces — Execution records provide the trajectory to evaluate and evidence to investigate.
- Production evaluator — Attaches scores and classifications to incoming traces; decision models can also provide confidence scores.
- Regression dataset — Configured task automation appends failed evaluation cases to the growing fixture.
- Human annotation queue — Configured confidence thresholds select uncertain outputs for review.
- Investigation schedule — The default cadence is every six hours and can be changed.
- Alert or monitor — A configured rule can use evaluation results or a condition such as latency above ten seconds.
- Signal investigation — Reads trace evidence and evaluation context to find and investigate issues.
- New agent traces → Production evaluator (Data: trajectory to score)
- New agent traces → Signal investigation (Data: execution evidence)
- Production evaluator → Regression dataset (Data: failed cases saved by a task)
- Production evaluator → Human annotation queue (Data: cases selected by confidence)
- Production evaluator → Signal investigation (Data: scores for the next run)
- Production evaluator → Alert or monitor (Control: result meets a configured alert rule)
- Investigation schedule → Signal investigation (Control: scheduled start)
- Alert or monitor → Signal investigation (Control: alert-driven start)

### The test environment sets the limit on automation

[34:24](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2064s)

The managed workflow allows an investigation session to be inspected, interrupted, continued or handed to a local agent. Ali’s longer-term vision is an end-to-end loop with very little human involvement until something needs direct testing or UI validation. Making that vision practical depends heavily on the environment needed to run the agent under test.

Wonder Toys makes replay comparatively simple: start the agent, let it create its own database, and send the requests through again. An application with consequential external dependencies raises harder choices. Should the experiment connect to a mock database, stub particular tools, or run the entire application in a Docker image? Which parts can be reproduced safely? Those choices determine what the experiment can exercise and whether the relevant failure conditions are present.

The stated goal is one click from issue to experiment, including starting the infrastructure necessary to test the repair. Local agents already have access through APIs, the CLI and skills, while connecting Signal to places such as Slack is part of the integration direction. A useful handoff preserves the evidence alongside the proposed change: Ali’s example includes five failed traces in the PR description so a reviewer can inspect the cases behind the patch.

The evaluators themselves also need maintenance. Signal can propose evaluations and inspect their health. A check that almost always passes may be a candidate for retirement to save tokens; a poor evaluator may need refinement. Judge alignment asks whether the evaluator’s scores agree with the human standard of good behavior. Generating more scores has little value when the criterion producing them is wrong.

These integrations work best when human judgments, evaluation scores and trace evidence remain connected in a shared record. The next investigation can then use both what happened and how people assessed it. Automation does the large-scale reading and prepares the reviewer’s work; the reviewer’s decisions become context the system can use again.

### For qualitative output, evaluate the whole session

[40:08](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2408s)

The Q&A moves from a relatively measurable bug to a harder problem: deciding whether an agent produced something useful. The category mismatch has a clear chain from inconsistent casing to retries and latency. Qualitative outcomes need a richer definition of success. A single labeled example can encourage overfitting or fail to generalize, so the growing fixtures and annotations become especially important.

Alyx’s filter-generation feature illustrates the gap. A user searching agents or traces can write a filter manually or ask Alyx to generate one. The generated filter may execute successfully and return results while still missing what the user wanted. Successful execution supplies one check, but judging the usefulness of the chosen filter requires more context.

Two forms of evidence help establish that context.

- **Session behavior:** Inspect the whole interaction. Did the user ask again? How many additional filters did they create? Subsequent actions provide clues about whether the first result helped.
- **Expert annotations:** Read the request, inspect the proposed filter and its results, and record whether they match the intended task. A product manager or domain expert can supply the expectation that successful execution alone does not express.

Combining these observations with new use cases produces material for the evaluator template. A coding agent can help turn the context into a prompt, after which the practical optimization problem is to preserve useful judgments on a sufficiently inexpensive model to run at scale. The evolving examples explain what the judge should distinguish; model and prompt choices determine how economically it can make those distinctions.

### Balance release checks with production sampling

[43:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2616s)

The severity of a rare failure changes the value of evaluation. Even 99.9% correct behavior can leave a catastrophic 0.1%. That qualifies the earlier suggestion to retire checks that almost always pass: the decision depends on the consequences of the failures still being caught, as well as the cost of running the evaluator.

Ali separates two places to spend the evaluation budget.

- **Development and CI/CD:** Keep known failure cases in a fixture and rerun them on pushes or before merging a PR. Some checks should be required so an already understood failure cannot quietly return.
- **Live production traces:** Evaluate a selected share of new activity to look for behavior the existing dataset does not cover. Sampling and model choice control the cost of this continuing observation.

These two uses cover different weaknesses. Testing a historical dataset gives confidence about those cases, while production monitoring encounters new ones. Evaluating every production trace can be costly, and relying entirely on development fixtures assumes coverage the team is unlikely to have. The useful balance depends on evaluation cost, failure cost and the model chosen to do the judging.

### Evaluate the investigator until you run out of ground truth

[46:07](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2767s)

The investigator is observable too. Arize traces its evaluations, and Signal produces traces of its own work. Those records answer how an issue was produced: they expose the instructions, shell calls and internal MCP calls used during the investigation, followed by issue creation. This makes the reviewer’s behavior available for the same kind of inspection applied to Wonder Toys.

![Trace of an investigation with Bash spans and an MCP artifact span, with one shell command’s input and output selected.](https://ai.engineer/api/talk-essay-media/74bd9bbdef28575fe361b75d6ce55170607a829d9b48f2699a80f9f4d70dec7b)

The investigator’s own trace exposes shell work and an MCP artifact call. The selected span shows a command and its output rather than the entire investigation payload (46:36).

Arize uses Signal to evaluate Signal internally, but the available success criteria eventually become indirect. PR merge rate, session transcripts, handoffs, dataset creation and evaluator creation can provide feedback. They remain proxies for whether the proposed work actually helped a customer. At some point, the customer knows more about the desired outcome than the platform can infer from its own activity.

That is a practical stopping condition for recursive evaluation. Once another reviewer is only rereading an agent transcript without additional ground truth, the extra cost can produce little additional confidence. Ali’s sharper description is “vibing with the agent’s output.” More layers of judgment need more meaningful evidence, or they risk making the process look productive without establishing that it improved anything.

### Faster judges could move checks into execution

[48:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2886s)

The demonstrated evaluators inspect an agent after it has run. Ali sees low-latency decision models as an opportunity to place checks inside execution, where a judgment could help route a request while it is still being handled. This is a direction he is excited about, distinct from the post-execution workflow demonstrated earlier. Cheaper judgments could also make evaluation practical for more teams and more traces.

Small models also suggest possible on-device uses. For multimodal evaluation, latency and subjective judgment combine: an evaluator may need to assess a generated image or an agent’s interpretation of a clickable UI element. Confidence scores offer a way to route uncertain cases to a person sooner. The benefit here is a faster path to the right reviewer when the model’s own judgment is insufficient.

### Connect useful signals, then price the whole loop

[50:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3006s)

An investigation can include downstream application evidence as well as agent traces. The Datadog skill can pull metrics, logs and APM traces around an alert, and custom skills can connect other sources. The selection still matters: connect the systems relevant to the investigation, give the agent suitable access, and avoid filling its context with information that does not help explain the problem.

![Datadog integration card marked required and a GitHub skill card marked optional.](https://ai.engineer/api/talk-essay-media/1b58eab276c7644992016c0e75fde28e5f188535f876c8598476396f0a08f21e)

This investigation configuration requires Datadog context around the alert and offers GitHub work as an optional skill (50:32).

The improvement agent also has a bill. Harness choice, underlying providers and execution frequency all affect its cost, which can be inspected through Signal’s own traces. Ali’s hypothetical makes the comparison concrete: if the shopping assistant costs $8 to run and its improvement loop costs $20, what does that additional spend buy? There is no fixed budget share offered as the right answer for every application.

The available controls operate at different points in that bill.

- **Context and connected skills:** Supply the information that changes the investigation’s usefulness; additional context can add cost without helping.
- **Evaluation design and cadence:** Choose which checks run, how often they run and what share of traces they inspect.
- **Execution environment:** The capabilities are also available through APIs, so a team can export work to its own coding agents when that arrangement has more favorable costs.

Apply the same cost accounting to the investigator that you apply to the customer-facing agent. Then adjust its context, cadence or execution setup in response to the value of the results. The overnight work becomes useful when the failures it finds and the repairs it helps validate justify the resources spent finding them.

## Key takeaways

- Trace evidence and repository code explain different parts of a failure. In Wonder Toys, the traces expose repeated searches and latency, while the code reveals the mismatch between human-readable category names and lowercase, hyphenated storage values.
- Preserve production failures as growing regression fixtures, then replay them against proposed changes. Evaluate the original symptom and existing requirements so a repair is supported by behavioral results.
- Human judgment shapes dataset selection, evaluation criteria and code review. Confidence-based routing can concentrate that attention, but useful thresholds depend on the task.
- Automated experiments depend on a runnable test environment. Wonder Toys starts its own database; more complex applications require deliberate choices about external dependencies, mocks, tool stubs and containers.
- Qualitative evaluation needs evidence of user intent and usefulness, such as whole-session behavior and expert annotations. Evaluating the evaluator eventually has diminishing returns when additional ground truth is unavailable.
- Combine required checks on known failures with production sampling, and account for the improvement loop’s own cost. Rare catastrophic failures can justify continued evaluation even when pass rates are high.

## Resources

- [Arize coding-harness tracing](https://github.com/Arize-ai/arize-harness-tracing) — An open-source way to apply the workshop’s observability approach to coding-agent workflows, capturing prompts, tool calls, retries, latency and token usage for inspection and comparison.
- [JSON webpage-generation prompt-learning notebook](https://github.com/Arize-ai/prompt-learning/blob/main/notebooks/JSON_webpage_generation.ipynb) — A separate exercise that generates webpage JSON, evaluates the outputs and retains revised prompts with their results. It develops the instruction-revision side of learning from evaluation feedback.

## Related talks

- [Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI](https://ai.engineer/talks/fnuNVFgLjHA-building-self-learning-loops-your-agent-fuad) — Follows a different Wonder Toys failure—a broken price filter—and develops a budget-specific judge, investigation with repository context and replay against a development endpoint.
- [How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase](https://ai.engineer/talks/_4Ph2dhmUFI-know-when-ai-agent-work-unsupervised-evan) — Extends the human-review question into policy-driven workflows, with bounded agent tasks, representative evaluations and a progression from full verification to sampled review and exception handling.

## Slides

### Agents that self-improve

[0:14](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=14s) [The opening slide introduces the workshop on traces, signals, data sets and experiments.](https://assets.ai.engineer/sites/aiecode2025/sha256/adde270ed9386d54bd60b2c9ec6190e0ebde66c21d6528c8d6aba24e849b7ff2.webp)

AI ENGINEER NYC • WORKSHOP • CT 12
Agents that self
improve while you
sleep
Traces, Signal, datasets, evals and experiments, wired
into one loop
Fuad Ali • Product, Arize • Turing Room, 1:15-2:15

### The four-in-the-morning bug

[2:21](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=141s) [Two cartoon panels contrast a bug waking an engineer with an agent fixing it while the engineer sleeps.](https://assets.ai.engineer/sites/aiecode2025/sha256/e77cbba11d6bd98cb6d73b057fd9d0c455ac52e3f1803b9b1daaabda34c7d729.webp)

THE HOUR, IN ONE PICTURE
Same bug, same 4am, very different night
BEFORE
AFTER
"we sell 0 board games"
customers are noticing
4:07
• signal filed it
/ eval caught it
• fix: 31% - 100%
/ PR ready for you
WHEN THE AGENT BREAKS AT 4AM
WHEN THE AGENT FIXES ITSELF AT 4AM
02

### What most teams have today

[5:40](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=340s) [Boxes distinguish traces and evals from the later steps that are often missing.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a306ae80c034fa191140e311c2ae09929f7e29bcfc805227c3eab49cfb5852f.webp)

THE STARTING POINT
Most teams have the traces, maybe some evals, and
none of the rest
Usually there
Traces
Some
Evals
Missing
Someone
reading the
traces
Missing
Failures saved
as tests
Missing
Proof a fix
broke nothing
An agent that improves itself is a loop. Today we build the missing three and connect them.
04

### Six steps in the loop

[7:49](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=469s) [A feedback diagram links traces, signals, a data set, an evaluation, a fix and an experiment.](https://assets.ai.engineer/sites/aiecode2025/sha256/7cf52e3a21e515aaaac5f7c011e86550c8f3f27ebf913676a75d7ddcc4d2087e.webp)

Six steps, each feeding the next
01 Trace → 02 Signal → 03 Dataset → 04 Eval → 05 Fix → 06 Experiment
Online evals and Signal keep watching new traces, and reopen the issue if it comes back.

### The Wonder Toys shopping agent

[9:05](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=545s) [A concrete shopping-agent failure sets up the workshop exercise.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcccd67c776388ab8983625dc2127c13ae8cedcab632291413c74cd56b537487.webp)

DEMO • THE AGENT
Meet Wonder Toys, a shopping agent with a bug in it
STACK
THE BUG WE PLANTED
OpenAI Agents SDK, gpt-5.4-mini, FastAPI and Next.js, ChromaDB for the
catalog, tracing to Arize AX
TOOLS
[One sentence: what the user asks,
what the agent returns instead, and
why no error fires]
search_products
get_product_detail
purchase_product
Every tool call succeeds. The answer is
still wrong.
check_order_status
cancel_order
07

### A trace a reviewer can read

[11:30](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=690s) [Four points describe the information a reviewer needs to understand a run.](https://assets.ai.engineer/sites/aiecode2025/sha256/21a5200f97b473da5c215606fbb18a86e056d906e9cd6733eb6338b933cbcbb6.webp)

STEP 01 • TRACE
A trace that a reviewer agent can read
/ A root agent span with the user input and the final answer
• A session id, so turns group into one conversation
A span per tool call, with arguments and results
NOT INSTRUMENTED YET?
$ ax instrument
Guided setup for the CLI, the skills and your coding
agent. Or ask your coding agent to use the arize-
instrumentation skill.
Model calls with prompts and token counts
08

### Turning traces into signals

[12:08](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=728s) [Examples on the right show the kinds of findings a reviewer can file from production traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/9df873362a5b89de961d8094f38ce6a878aa410f4f1f19920c89fbdcd8534bbe.webp)

STEP 02 • SIGNAL
Signal reads your traces and files what it finds
AN ISSUE
severity
Overview what goes wrong, in plain words
Evidence links to the traces that show it
Proposed fix where in the agent to look
Matches how many traces hit it
Next steps create an evaluator, add to a dataset,
open a PR
Turn it on
Project, then Signal, then Enable. The first run starts right away.
On a schedule
Every 6 hours by default, from every 3 hours to monthly. It reviews recent
traces each run.
On every plan
Free: 10 issues a month. Pro: 25. Enterprise: unlimited.
09

### One ticket per problem

[18:35](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1115s) [A three-column workflow shows how to open, update and track a recurring problem.](https://assets.ai.engineer/sites/aiecode2025/sha256/315c53759ec09d841a85d1762377daaeb840c45b553fda6486ce5bdc3a25cec8.webp)

STEP 02 • SIGNAL
It keeps one ticket per problem, the way a reviewer
would
Open
A new finding
Later runs that see the same
bug update this issue instead of
filing another.
Resolved
You fixed it
If the bug shows up in new traces, the
issue reopens.
Ignored
Not a problem
Muted, and never filed again.
Your first checkpoint. Open the evidence traces and confirm the issue is real before it becomes a test.
10

### The failure becomes a dataset

[18:53](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1133s) [A sequence of steps shows how to turn observed failures into cases for later tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/9aefdcfa92e0a87cfca9310d99bd464166f527b43553b9262e77b24e67d2b327.webp)

STEP 03 • DATASET
The failure becomes a dataset
01
02
03
04
From the issue, choose Add to dataset. Each evidence trace
becomes one example.
Add columns that say what should have happened.
Add passing examples too, so the experiment can catch what the
fix breaks.
Optional: an auto-add task keeps adding matching spans as
they arrive.
COLUMNS WORTH ADDING
input
expected_tool
forbidden_behavior
failure_mode
source_trace_id
FROM A TERMINAL
$ ax datasets append DATASET_ID \
--json '[{...}]'
11

### Writing the eval

[20:21](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1221s) [A comparison pairs a vague criterion with a question specific to the shopping agent's problem.](https://assets.ai.engineer/sites/aiecode2025/sha256/746fc83016010db7c5028beb439b92c26170f77ce4d15cd5e9a02fa6021550ae.webp)

STEP 04 • EVAL
Write the eval from the problem Signal found
GUESSED UP FRONT
"Was the answer helpful?"
Scores everything a little. The planted bug passes,
because the answer sounds helpful.
FROM THE FINDING
"[Did every product returned respect the
limit the user gave?!"
One question, tied to a failure you have seen. Easy to
check, easy to trust.
01
From the issue, Create evaluator. Alyx drafts it and starts
it on new traces.
02
Run it on the dataset. It should fail the bad rows and pass
the controls.
12

### Choosing the cheapest judge

[20:51](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1251s) [A table compares judge choices against what each can observe and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/ec8cde7bc695ce28a0b27f7b9f907725b450126f9e815b27dea2cdec07477c86.webp)

STEP 04 • EVAL
Pick the cheapest judge that can see the failure
Judge
Code check
What it sees
Fields and values
LLM judge
Agent-as-a-Judge
Jev-as-a-Judge
One span's input and output
The whole span tree
Typed yes, no, choice or score
Good for
Exact rules: a filter applied, a tool
called
The quality of one answer
Multi-step behavior and wrong paths
Always-on checks on every trace
Cost
Next to nothing
One model call per record
Highest; reads the trace
$0.045 per 1,000
Same benchmark: a frontier LLM judge cost about $14.30 per 1,000. Jev is experimental.
arize.com/blog/decision-model-benchmark
13

### Handing an issue to a coding agent

[22:54](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1374s) [The slide describes the evidence handed to a coding agent and the human review that follows.](https://assets.ai.engineer/sites/aiecode2025/sha256/16178e5cba11052347b644493243a7c47ec32e896d3c4a3176010aef369ef38c.webp)

STEP 05 • FIX
Hand the issue to a coding agent
ANY PLAN
Copy issue details into
your coding agent
Claude Code or Cursor with the
arize-trace skill pulls the evidence
traces itself.
ENTERPRISE, REPO ATTACHED
Open PR from the issue
A managed agent writes the
patch from the issue and its
example traces.
PULL THE EVIDENCE
$ ax traces export PROJECT I
--filter "..."
--output-dir. arize-tmp-traces
20
Second checkpoint. A person reviews and merges. Signal never ships a change on its own.
14

### Checking the fix for regressions

[26:57](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1617s) [A before-and-after comparison shows the result on the target behavior and a separate control.](https://assets.ai.engineer/sites/aiecode2025/sha256/f08c480c6449bb7cb387ea669d312b500a446fdf68e4ddce9e5470024c11eae6.webp)

STEP 06
• EXPERIMENT
Prove the fix, and prove nothing else broke
Same dataset,
same evals
Before
After
Rows Signal found
0.89 → 0.72
Task completion after a fix that raised action
safety. The average looked fine.
Control rows
L_I
LL
$ ax experiments run \
--name fix-v2 \
-- dataset DATASET \
--task task.py
0.89 → 0.72: Sofia's talk, Tue 3:20
15

### Where to keep a person involved

[32:52](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1972s) [Six boxes mark decisions a person can make in the feedback loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/8eb13ecbca9abc752309e15e72bce32ccca63297bb9f2e496ad4cebfd0cf3005.webp)

PEOPLE IN THE LOOP
Six places to keep a person
01
Confirm or ignore what
Signal files
02
Write the expected behavior
into the dataset
03
Review and merge the fix
04
Annotate runs where the judge
looks wrong
05
Accept or dismiss evals an
agent proposes
06
Send low-confidence judge
scores to a person
4.6%
In TypeSafe's spam test, sending the 4.6% of cases scored between 0.3 and 0.7 confidence to people left the
rest at 99.5% accuracy.
arize.com/blog/typesafe-jev-11m-judge
17

### A future feedback loop

[34:13](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2053s) [The slide outlines a progression from data and traces to experiments and reviewed fixes.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e85c5374f3db85de5de602e770c63fad93f9ecc8dc006e1430aec9fdb44e6d7.webp)

WHERE WE'RE HEADED
Our vision for the future
01
02
03
04
05
One click from issue to experiment
Signal wherever you work
Issue to reviewed PR, for every team
Evals that write themselves
Experiments on any agent
An issue sets up the dataset, the eval and the before-and-after run
Your terminal and coding agent, through an API, the CLI and skills
A coding agent drafts the fix from the issue and its traces
Signal proposes the online evals that watch every new trace
Run against your agent wherever it lives, your laptop included
More of the loop runs itself. The decisions stay with you.
18

## Full transcript

[0:12](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=12s) Well, thank you guys for coming to the workshop. Really appreciate you. My name's Fuad Ali. Uh, I am a product manager at Arize. Arize is-- helps you guys build the self-improvement loop for your agents. Um, today, today, I'm gonna be talking through what that loop is, how it gets built, um, and I'm gonna be walking through a little bit of an example agent as well to kinda show how that agent works. Um, now we have an hour for this workshop. With that in mind, I know that workshops can be hectic, Wi-Fi can go out,

[0:42](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=42s) so I have everything ready on my screen. I'll be walking through it. It's optional for you guys to follow along. Highly recommend you do. I personally learn a lot better when I have answers to the keyboard. Uh, but if you are having issues, don't worry. And we also have Laurie in the house who is more than happy to kinda help troubleshoot along as well. Sound good? Cool. Uh, so I'm Fuad. I work in product at Arize, and we work with a lot of agents. And, uh, I'm assuming you guys are all very familiar with what agents are and how they fail and how they are non-deterministic. Um,

[1:12](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=72s) and you know, this is sort of like a classic engineering problem that has reared its ugly head, you know, ten times worse with agents because of the indeterminism, right? So when you have agents that can do a lot of things and have a lot of different ways of doing a lot of things, understanding their failure modes becomes trickier. Um, and then add on to that the fact that there are thousands of these agents. Maybe you have internal agents, maybe you have coding agents, uh, you have marketing agents, and you have, obviously, customer-facing

[1:42](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=102s) agents as well. Uh, and that all adds a ton of noise to the problem. And so how do you actually start to build to a loop where these agents use the feedback that you provide as a human being, customer feedback, and obviously actual evidence, so things like traces, um, to inform how they can improve? Uh, and that's what I'll be talking about today. Uh, Signals are out-of-the-box version of this, which I'll be showing heavily today. We have a managed agents platform that

[2:12](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=132s) lets you kinda define on your own terms what these do. Um, obviously, datasets, evals, experiments, uh, all that will be part of this as well. Uh, but this is what we're trying to solve, and, uh, apologies for the meme. Claude's, uh, sense of humor is not that good. But, um, I think it illustrates the use case pretty well. Um, in the previous world of software engineering, um, your SRE would probably be in the first picture, right? So classic situation, PagerDuty s--

[2:43](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=163s) uh, notification goes at four AM, your agent broke something in production. Um, in this case we're, we're doing like a little, you know, uh, shopping assistant. And shopping assistant is saying, "We sell zero board games to all your customers on production." Not a situation you wanna be in. Uh, there have been worse use cases of agents as well, where, uh, I don't know if you guys are familiar with what happened in Canada with the car dealership where someone was able to get, um, a, uh, I think it was like a, a Prius or something for like five

[3:13](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=193s) dollars, uh, through a chatbot. They obviously didn't honor the sale, but that's legal's problem now. Um, and so these agents can fail in a number of different ways for a variety of reasons, and it becomes really important to try to move to a world where we're in the after stage, where the agents can learn how to fix themselves and build that feedback loop into it. And I'll talk a little bit about what-- where humans still add value in, in that sort of world as well. So this is what you'll need to follow along. Again, like I said, this will be

[3:43](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=223s) optional. I'll be walking through the flow for you guys. Um, we do have an hour. Um, we can spend the rest of the time with questions. You can spend some of the time trying out the loop. Um, I would recommend, if you have some time, I'll, I'll put it up here for, you know, thirty seconds-ish to a minute. Um, if you can sign up for a completely free Arize AX account, uh, you'll need a space ID and an API key, which I can show you how to get. Um, you'll need some sort of coding, um, API... Or sorry, um, LLM API

[4:12](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=252s) provider key, OpenAI or Anthropic both work. Uh, you'll need Node and Python on your machine. If you don't have that, don't worry, we'll, we'll still talk through it. Um, Claude Code or Cursor help kinda set this up for you as well. Um, I assume none of you guys are strangers to those tools either. Uh, and it's very simple. So this is kind of the app we have. So it's a chat-based toy buying experience. It's called Wonder Toys, right? Very simple. You can ask in the chat about different products that are

[4:42](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=282s) available. You can chat to purchase. Um, this is all running locally on my machine. So this is the agent that we're gonna be dealing with today. Uh, and we're gonna be walking through how we instrument it, how we set it up so that we can build that feedback loop. Um, so I'll pause here real quick. I'll give you guys thirty seconds to try and do some of this. Again, optional. I don't wanna, you know, block the entire workshop on API keys, for sure. Cool. So really three

[5:12](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=312s) things, getting the repo,

[5:15](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=315s) running the actual app, making sure you have an API key that actually powers the agent, um, so not, uh, a provider API key like OpenAI or Anthropic. Um, then this, this part here is around installing the skills that helps your coding agent work with our tools, uh, in a more kind of opinionated manner. Um, Node and Python just to run the repo, and that's about it. Cool. I'm gonna continue in the interest of time. So open question for you guys.

[5:45](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=345s) Of these, you know, six-- did I count that right? Five boxes. Five boxes. Which of these boxes would you classify your team is in? And, you know, the... Usually it starts from, okay, we are tracing what our agents are doing. So we know- When they fail, we know what tools they call. We can see the entire trace tree for the interaction points of the agent, uh, and that's box one. Show of hands, does anyone have tracing for their agents right now?

[6:16](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=376s) Cool. Okay, we got a bunch, actually. That's good. Then the next stage of that is, okay, well, I know what, what the agent is doing, but is it doing the right thing? Um, and how do I actually score and evaluate that? So we call those evals in the industry, and so I would say, you know, some people probably are in that box. Are, are any of you guys evaluating your agents? You r-- have evals running on traces? Okay, we got a couple. Okay, a lot. Yep. Then there's someone reading the traces, presumably, uh, and that's sort of

[6:46](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=406s) what we're trying to unlock here, right? With the agents. Um, but realistically, if you have coding agents internally, there's no way someone's actually reading all these traces, right? Or, you know, a chatbot serving millions of customers, no one single human being is gonna be able to read those traces, right? Then you actually wanna be able to save those failures as tests, right? So every time something fails, you capture that, and you're able to use that as a regression test moving forward so you don't,

[7:16](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=436s) you know, mess the same thing up twice. Um, and then finally, you wanna prove that-- You wanna make the fix, and you wanna prove that you actually fixed the thing, right? So in those last three boxes, we got anyone kind of in, in there, in that stage where they're actually... Awesome. Cool. We got a couple. Cool. So how do you actually thread all these things together, and how do you make this workflow as easy as possible and, you know, have humans in that workflow in their highest value leverage points,

[7:46](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=466s) but not necessarily reading tons of traces, right? Um, the flow generally goes trace, Signal, which is our out-of-the-box kind of SRE agent, which I'll show you guys in the platform. Signal then runs-- reads the traces, creates a dataset off the failures, proposes an eval for you, automatically fixes what's going wrong, and then runs an experiment that actually proves that that fix is, um, something that's

[8:16](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=496s) valid. And then you review that fix, you choose to merge that fix, and you go back in the loop. And every single one of these steps builds additional context as your agent continues to improve, continues to, you know, move into new use cases. Let's say I wanna add, you know, one-click checkout on this, on this agent or something like that, um, that then feed not just your product roadmap, but your engineering reliability organization as well.

[8:43](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=523s) So, uh, this is kind of what we will talk through. I'll walk through the loop. Uh, I'll show you a bit of the agent and its traces, Signal reviewing the agents, the failure dataset, the eval that we build off of that failure dataset, fixing it, proving it with an experiment, what the agents did themselves, and what still needs a person, um, and where you guys can add the most value, and then we'll have some time for questions as well. So Wonder Toys is a shopping as-assistant agent that we're doing today. Uh, it's got a bug in it, um, that we

[9:13](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=553s) planted for the purposes of this demo, although I hope you guys are not planting bugs in your own agents. Um, it's built with, uh, like I said, the repo has a, a bunch of different agent frameworks out there, um, mostly FastAPI, Next.js, ChromaDB, uh, and then agent SDKs as well. Uh, and it sends traces to Arize AX. It has a few tools available to it. Um, and one of those tools, specifically the one with the bug, um, misfires a filter. And

[9:43](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=583s) so I'll kind of show you at a base level what the tracing experience looks like. Um, we made it very, very easy for you to set up traces, so you're not instrumenting things manually to get, you know, the information out of your agents. Uh, we have skills set up that wrap our CLI and our MCPs, uh, that help you get started, uh, with tracing. We have provider, um, all the major providers, uh, instrumented already with auto-instrumenters, uh, and all the major agent frameworks as well. It's all open source, so very

[10:13](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=613s) easy to get started with tracing. Sounds like a lot of you guys are familiar with tracing, and so you'll be very familiar with what this trace tree looks like. Um, a user asks, "Show me toys for nine-year-olds," um, and that goes through an entire request path, right? So the agent's called, the agent calls a model, the agent has an end, uh, LLM call. It has some tools available to it, whether it's searching products, et cetera. Um, and you can kind of see all the

[10:43](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=643s) information available, um, to as to what the agent's doing under the hood. Uh, so all these attributes kind of show, you know, start time, end time, latency, what the input was, what metadata was passed, what stage in the, in the graph it is, um, results that it fetches from the DB, uh, and what-- at every single one of these stages, what information is available to the agent. Uh, so this is valuable, but again, you probably don't wanna read this at four AM. Um, I think it would be kind of tough to read through thousands of these at four AM just to find where an error lies.

[11:14](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=674s) Um, so what we wanna do is we wanna enable agents to make use of this information and then see how we can actually close that self-improvement loop.

[11:28](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=688s) So I kind of walked through the trace. If you want to instrument, the AX CLI is what we use. Um, it has a guided setup. It's very easy to get started with it. All you need is an API key from Arize, and it will take care of translating all those nasty little attributes into the beautiful trace tree that you saw there. Uh, you get a span per tool call with arguments and results, uh, to the tool call. You get model calls, prompt, token counts, all, all the basics that you need to actually monitor what's going on, um, as well as

[11:58](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=718s) grouping. So you can see sessions, uh, where there's multiple back-and-forths with a customer instead of just individual traces that are a single back-and-forth with a customer.

[12:10](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=730s) So enter Signal, which is sort of the second step in this platform, right? So once you have these traces, what Signal does is Signal is an always-on agent that is able to read your traces on a schedule, uh, and you can configure this in the platform to be kind of like whatever schedule you want. Um, by default, it's, uh, set to every six hours. Uh, you point it to a project that's, you know, has live traces coming into it. Um, you enable it,

[12:40](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=760s) and it reviews those traces and can then do a number of different actions via its skills, right? So the base action that we're gonna show today is Signal connected to a repo on GitHub. So this agent Signal is reading another agent's traces. It's reading the source code of that agent based on sort of what's actually happening in production traces from that agent and the coding definition. It's able to actually

[13:10](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=790s) propose fixes to what you're seeing. Uh, so I'll show you guys what that looks like here real quick. So we have this Wonder Toys agent. It's publishing traces in. We can see all the user requests. You can see I did a lot right before, uh, this presentation at one p.m. We can see, you know, kind of base facts like latency, token count, cost, uh, of running this project, and you'll see this Signal tab. So what Signal is doing is it's clustering issues it finds with the traces and

[13:40](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=820s) actually proposing and identifying, um, a fix for, for these issues. Um, so for the purposes of this agent, the bug we planted, uh, is causing redundant search retries. And so we're see-- what Signal is seeing is that based on the traces that are coming into this project, we are, uh, causing three sequential LLM calls and high latency because of something that's happening with the search products call, right? And specifically, what that's

[14:09](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=849s) happening-- what's happening is that, um, the casing on some of these queries for the filtering is not working, right? So what Signal does is, instead of you getting paged at four a.m., um, if you remember the meme in the beginning of this presentation, um, Signal is finding this issue, analyzing, you know, clusters of these issues, writing an overview and analysis, writing a fix for you, and then what you can do is you can create evals to catch this issue moving

[14:39](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=879s) forward automatically and trigger off those, uh, evals. So, you know, if we run into this issue at any point in the next, you know, thirty days, we automatically fire an alert, and that gets triaged by our team immediately. You can add failure cases to a dataset here as well so that you have a snapshot, uh, for future regression tests on exactly this failure mode. You'll see a GitHub icon here as well to connect and open a PR or open an issue as well, so that you can

[15:09](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=909s) actually get Signal to write a PR. I've actually already done that, and so you can see the PR attempts, uh, that Signal has done to fix this, right? So I'm gonna go click into the session.

[15:22](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=922s) And the agent worked for six minutes, found the root cause within the coding repo as well based on the traces, and found that product ca-- ca-- product categories are stored as lowercase hyphenated slugs, but the LLM is passing human-readable names with title case and spaces. Uh, so when someone's searching for car games, for example, without that normalization, we actually never get back a tool result. The agent is then going and searching

[15:52](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=952s) every single product, right? So you can imagine why that causes the latency spike up to fourteen, seventeen seconds. So everyone following along so far? We have an agent. We have an issue. Signal is reading the traces for you, proposing a fix in connection with GitHub, uh, so actually reading the, the definition of this agent, uh, and then doing some analysis for you and kind of making sure that that's available to you. So this is one of

[16:22](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=982s) many skills you can connect to Signal. Um, you'll see here on the Managed Agents platform, you have the ability to use one of our out-of-the-box templates, so Signal is reviewing your project traces on a schedule, clustering failures, suggesting fixes. We have a cost agent that can flag expensive models doing cheap work, recommend cheaper routing, and actually quantify your savings. Uh, but the interesting part is you can actually start from scratch. Um, and so what the Agent Studio allows you to

[16:52](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1012s) do is you choose an agent config, you point it to a project where traces are coming in, you give it a task, and you can connect skills to it as well. So like I mentioned, GitHub is one of the sources. Maybe you have, um, you know, a third-party system that holds your RBAC information that-- and you wanna check permissions on this agent, or you wanna make sure what the agent is calling is actually within its permission boundaries. You can connect a skill to do that. You can connect a skill to, uh, another APM system like Datadog if there's more logs you

[17:22](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1042s) wanna pull in. Um, there's really no limit to what skills you can define as long as you have an API key and it's callable from the agent harness itself. Um, and we're actually under the hood running these as Arize managed sandboxes, uh, within the platform. Uh, you can bring your own harness, bring your own sandbox provider as well, uh, to kind of, you know, whether you're using Modal, Daytona, E2B, whatever it is or whatever harness you're using as well. We support Codex, Cursor, Claude Code, and OpenCode with more harness support rolling out.

[17:51](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1071s) So does everyone kind of understand, you know, we have a managed agent doing this in the platform. You can set up your own automations to, you know, either be triggered on a time schedule or off of an alert. So anytime we see latency spike above ten seconds, we automatically kick off this agent to do an investigation. Um, and investigation agents are part of it, kind of solves the SRE use case. There's a cost lens to this. There's a product classification lens to this. Um, there are many ways in which you

[18:21](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1101s) can kind of use this agent platform to operate on your traces. Cool. Uh, I will continue through these. So Signal is reading your traces, filing them what it finds. It keeps one ticket per problem, uh, the way a reviewer would. So you're not getting hundreds and hundreds of issues on the same, uh, piece of evidence. Uh, it opens those, you fix it. You can either resolve it if you think it's resolved, you can ignore it, um, and it will dedupe those issues moving forward so

[18:51](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1131s) you don't, you know, kind of get inundated with noise. Um, the add to dataset field or, uh, flow, I think is one of the most important parts here because how do you build the actual continuous improvement? You have to have a record of what's failed in the past and guard against those as regression checks moving forward, right? So adding these to your dataset give you that sort of living fixture quality. So, you know, you define test cases in code before you ship code. Ev- everyone should be doing

[19:21](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1161s) that, and hopefully we are all doing that with the help of some agents now. Um, but this idea of living fixtures that change over time as you add new use cases, as you see new failure modes, is a pretty novel one, and that kind of helps with the self-improvement, uh, loop, where, you know, you're not beholden to one definition at one time of what has failed and what are failure modes, but you're organically growing that definition of failure mode as your agent is failing in production, right? So,

[19:52](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1192s) um, think of it like, you know, every time you go to try and ride a bike, you notice something about how you're riding, and then o- over time, those become so ingrained in your memory that riding a bike feels like nothing. You don't even think about the failure modes of... Um, I actually fell off a bike and broke my foot two months ago, which is why I'm mentioning this, this little anecdote. But typically, you don't think about, you know, putting your left foot on the left pedal anymore because it's been so ingrained into your learning process, and that's what we're trying to kind of accomplish with these agents.

[20:23](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1223s) Um, so on the dataset side, once you get these living fixtures, you actually wanna codify the behavior on these fixtures, right? And that's where the eval comes in. Um, so I saw a few show of hands that people are running evals. Uh, can I get a quick gauge of the room? Who here is running LLM-as-a-Judge evals or you're running code evaluators? Cool. So if you were in any of the earlier workshops with Arize, we went into evaluators in great depth. Um, there are many, many different forms of

[20:53](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1253s) evaluators. Um, so just quick, quick overview. There are code checks that are, you know, really, really cheap to run that are for deterministic checks, right? Is this field present? Is this valid JSON? Exact rules. Was this filter applied? Was this tool called with the right arguments? Very, very easy for that, right? LLM-as-a-Judge evals also help with this use case where you need to have more of a qualitative or nebulous kind of analysis

[21:22](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1282s) of the trace, right? And so this is using another LLM model to score the results of one model's trace. Um, obviously, costs will be higher here because you're making another model call. Um, per record, y- obviously you can, you know, kind of set your own sampling, filtering. You don't have to run these on a hundred percent of your traces that you're running at. Um, there's some trade-offs in terms of eval design that we can discuss, but that's sort of what an LLM-as-a-Judge is for. Then you have,

[21:53](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1313s) for the even more open-ended use cases, you have this concept of Agent-as-a-Judge. So where you're not actually sure what you're evaluating, you have a goal in mind, um, and you leave kind of the, the determination of whether that goal has been achieved to the agent. So the agent can actually pick its own rails, it can actually pick its own scores, it can go through that process itself. And that's obviously the highest cost method because there's multiple LLM calls happening under the hood. Um,

[22:23](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1343s) decision m- models have been around for, you know, what? A month-ish now. And w- Lori actually, and, uh, at Arize has published a ton of material on sort of benchmarking them and how they operate and what trade-offs are present with decision models. I will say they are a very, very cheap way to run some of these checks on your traces versus, uh, using a classic LLM. Um, and they give you, you know, type, type choices, yes, no, choice, score.

[22:56](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1376s) So once you have sort of this eval signal, right? So you, you understand, you know, qualitatively, is this in the right direction? Is it not in the right direction? Is it passing my eval checks? And you have the trace data. That's all really rich context for a coding agent to go, um, implement a fix, right? So Signal, in this case, is, is running on, on an Arize managed sandbox with a coding harness, implement those fixes for you. But

[23:26](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1406s) if you're in your development phase, you can actually pull these traces live into Arize through the CLI as well, right? So it's instant context to feed other agents to actually work on those PRs for you. Um, and this is where we have kind of the human check and balance in, in the room. Um, so I would say the, the places a human adds the most value here are, one, obviously validating the fix, right? Reading the PR. There are, you know, maybe some of us in the camp of,

[23:56](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1436s) we don't want to validate every human PR. I assume if you're in a high-stakes use case like finance, you're very much in the camp that humans read the PR before they merge, right? So that's one very high leverage point for a human to be here. In the dataset curation phase as well, so adding that to a dataset is part of why we left it manual, is, are these good enough findings to add to my fixture, right? So that's a manual step in the process, uh, and that relies on kind of human judgment as well. Um, the eval design, so we have agents that kind of help

[24:26](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1466s) you write these evals. Um- But the eval design portion is something where humans can definitely be in the mix because you're imparting your taste of what is good onto the agent, whether that's in the form of an LLM-as-a-Judge or a more deterministic check. Um, and then folding all this information into the coding agent gives it really rich context for it to work from. Um, so just as an example of kind of like how you would create an eval off this, I'm gonna go to my Traces tab. I'm working off of this Wonder

[24:56](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1496s) Toys project. I have Signal running, and Signal's found two issues so far. One is the orphan spans issue, um, which was actually just my mistake instrumenting this, uh, but it did catch it. Um, and two is the redundant search retries. That's the bug we planted. Um, so I'm able to create an evaluator with one click off of this, and that actually, uh... Looks like I need to use a different model.

[25:28](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1528s) That actually opens up Alyx, uh, which is our in-house agent that I'm doing a talk about tomorrow, um, that is able to go and create that eval off of that evidence for me, right? So based on the traces that are supporting the evidence here, we can kind of codify in a prompt template what we should be expecting moving forward and how we, how we make sure that the eval fires and fails on any mistakes that are made. So this is the eval it came up with. "You're a helpful AI bot that checks for redundant search

[25:58](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1558s) product retries. Inspect the complete trajectory, including tool calls. See if there's any unnecessary tool call. Return a redundant retry when search product is called more than once, um, and put this score and classification choice on the trace itself."

[26:14](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1574s) And then you can run this eval on incoming traces that are coming into your project. So you make sure you catch that. You can monitor that. You can graph that. You can create an alert off of that, uh, and you can track spend associated with that eval as well. So over time, you know, maybe your development flow has codified this as a regression check, and you no longer need to deploy this eval on production data. You don't wanna spend the model tokens on running this on every trace. You can actually view the cost over time and retire that eval as well.

[26:47](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1607s) Cool?

[26:50](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1610s) So the next part of this is, okay, we have a fix. Signal found something. It analyzed the repo. We create an eval to catch it for the next time. We added the evidence to a dataset. So now we have a living fixture that's growing with time. How do we know that the fix is right? How do we actually know that, you know, it wasn't just slop code that the agent produced and tried to find an issue that didn't actually exist? Um, and this is where the world of a-agent experimentation comes into

[27:20](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1640s) play. And this is a very, very rich and deep world, um, so I won't spend, you know, time talking about all the different frameworks and providers that exist out there. Phoenix team did a good job earlier today talking a little bit about the Harbor integration they have in Phoenix. Uh, but really what we're concerned about is proof that the fix works and proof we didn't break anything else, right? Um, and there are many cases where, you know, in this non-deterministic world, if you make a small change to a system prompt, it can then have ramifications on other

[27:49](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1669s) tools that you didn't think were gonna break, uh, and, you know, kind of, um, redundancy issues there. Um, and so making sure that we can prove that this fix actually worked and that nothing else broke is really, really important, and that's kind of the experimentation stage. Um, so I will talk a little bit about this through the flow as well. So in Arize, we support a bunch of different types of experiments. Um,

[28:20](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1700s) this is the category code enum fix experiment that I just ran. And so you'll see here that we have actual evals running on this experiment. So what we're essentially doing is we have this living fixture of failure cases. We have this proposed fix. We actually wanna rerun the agent on these failure cases and score the results. And if they meet a certain, um, threshold, that's kind of up to you to define, uh, in, in my case,

[28:50](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1730s) eighty-eight point nine percent I thought was pretty good, so I was like, "Ship it." Um, if they meet that threshold, you actually have results to prove that fix from the agent, right? Um, and the agent can actually run these tests for you through the AX CLI as well. So as Signal is going and finding these failure cases, and it has access to the agent code, it can actually pipe those failure cases through, um, and replay them through the agent with the version two fix and then score those results themselves, right? So you come back in the morning.

[29:21](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1761s) You weren't working at before four AM. You not only have a PR that has a proposed fix, you have some validation results that say, "Hey, I proved this fix because I actually took the test cases that we failed on, and I re-measured latency or tool call retries, and I saw that we were no longer retrying tools and latency was lower. And hey, by the way, you had three other evals that you don't wanna fail on either, so I made sure to check that those evals are not regressing," right? And

[29:51](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1791s) so as that living fixture grows over time and you have new cases, these experiments become more and more robust, right? Um, and in the platform, obviously, you can go view those experiment results, see what the output is, see what the evals are. If you disagree with the agent, you can annotate that output, and that becomes another piece of information to the context for the next run, right? So everyone kind of clear on this experimentation flow at a high level, like how you can run some of these experiments, um, to take a test case, take an

[30:21](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1821s) eval that you've created as a benchmark, and actually validate your fix before it goes out, before it goes to prod? Um, and you can shunt this into the CI/CD process as well. Um, up to you on how you wanna configure it. Obviously, running the, the agent may be expensive. Um, so your discretion on how you wanna run that. Evals as well, right? If you're using an LLM-as-a-Judge eval, you wanna be careful about which model you're picking, when you're running those evals, what sampling rate you're running those evals at, uh, to make sure you're not

[30:51](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1851s) just ballooning costs and, you know, just sending a ton of spend, um, on validating these fixes on every push.

[31:00](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1860s) Cool. So,

[31:03](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1863s) kind of continuing on, we approve the fix. We look at the task completion rates. Um, we validate those results. And while you kind of step away from this process, what keeps running is that the eval that you created runs on every new trace and becomes additional context to the next Signal run. The task system that we have in Arize can also automatically add failed eval results to new databases, uh, or datasets so that that

[31:33](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1893s) fixture grows over time. And Signal keeps rerunning every six hours or off of an alert, um, or off a triggered monitor, and makes sure that, um, new failures are surfaced, new f- evals are created, and that dataset grows with those new examples over time. Um, we also have, you know, weekly digests where you can kind of consume these in a little bit more of an easy-to-read result where you just have a summary of what's going on with the agent. Um, what is waiting for you is confirming that

[32:03](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1923s) these issues are real, right? Um, you can't just trust every single issue that comes out of this agent. I would say use it as a starting point, validate those fixes, and use your judgment on when those issues are actually real. Writing down what the right answer looks like, so this kinda goes in the annotation flow, right? You run the experiment. You get a new output. Maybe you don't think this is the perfect answer or the perfect trajectory for the agent, and you can actually use the annotations product to kind of codify that taste. Uh, merging the fix, this is up to

[32:33](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1953s) you. Um, you can auto-merge these PRs. I will leave that to your discretion. Uh, and then actually deploying and promoting the new version, right? So I have V2. I validated it. I think the issue's real. I created the eval. I looked at the score. I validated the fix. I like the code. I reviewed the code. Now it's time for me to actually upgrade the agent.

[32:54](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=1974s) Um, I think the, the other thing that's really interesting about this is, like, you can codify the confidence scores, especially with these decision model models that actually return an explicit confidence score and have those auto-route to human beings, uh, in annotation queues, right? So, um, not only are you, you know, responsible for reviewing the outputs, but you actually get a ready-made queue for you that is based on all the low-confidence results. And, um, you

[33:24](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2004s) know, in some of our benchmarking, uh, we've seen these confidence scores to be pretty good. It, it depends on your use case and, uh, specifics in the industry, for sure. Uh, so I wouldn't say that an eighty percent confidence score is the right threshold for everyone. Um, but setting those thresholds is important. And then once you have those thresholds defined, you can kind of auto-route those low-confidence cases to a person to, to go annotate. Um, so I mean, there's, you know... In the spam tests that we saw,

[33:54](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2034s) four point six percent of cases scored between point three and point seven confidence to people, left the rest at ninety-nine point five percent accuracy, right? So that is one example of an industry where, you know, specifically for spam detection, where those confidence thresholds might be more defined. It, it depends on the use case, for sure. Um, so I kinda talked about some of the stuff we've built so far with Signal Managed Agents and traces, right? So

[34:24](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2064s) the ability to actually read those traces at scale, the ability to add things to datasets, uh, the ability to run those experiments with the help of a coding agent, um, and actually review those results in line. You can also interrupt sessions. You can look through the model's reasoning. You can kind of continue a session, kick it off to a local agent, uh, so that you can continue development on your own side if you don't want this all managed by Arize. But I think our vision for the future is this, running this full end-to-end loop with almost zero human

[34:54](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2094s) involvement until you need to go test something, right? Until you need to go click around in a UI and validate something. Um, and so there are a lot of providers out there for that experimentation setup, right? There's the Harbors of the world. There's many, many different ways that you can test the agent. Um, I think most of it, and, and this is where a lot of the discussion is happening, is how many dependencies do you have to run to run this agent, right? Um, running an agent in the experimental phase, does that mean you connect it to a mock DB? Do you stub

[35:24](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2124s) some tools? Um, do you actually run the entire thing in a Docker image? Can you even do that safely? Um, those are all sort of questions that I would encourage you guys to think about as you start thinking about this experimentation flow. With this agent, you know, it's very simple. We just rerun the agent in its entirety. There are no dependencies. It spins up its own DB, and then we run requests through that. So that might be a very simple use case. But when you have external dependencies that really matter, your

[35:54](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2154s) use case become, can become a bit more convoluted. Um, so our goal for this is really enabling the one click from issue to experiment. "Hey, I have this proposed fix. Go test it." Spin up all the infrastructure necessary to go test it. Um, in addition to this, and we've, we've worked on the CLI around this, is connecting Signal where you work as well. So whether that's in Slack, so tagging, uh, from Slack to communicate with Signal. Whether that's your local agents, which we have through the API, CLI, and

[36:24](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2184s) skills, where you can actually ask your local agent to go do a digest of what Signal is finding and, and check those. Um, making sure that we have the fix drafted Straight from the issue and its traces, and that's part of what we've done already, and actually having the evid-evidence in line. So here are the actual five traces that failed in the PR description, so you can actually go validate those. Um, we have Signal now proposing evals as well for you. So it'll actually check your evals, see which ones

[36:54](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2214s) are healthy. As in, you know, if this eval's passing ninety-nine point nine nine nine percent of the time, maybe we don't wanna spend the tokens on this eval, and so we can retire this eval going forward, uh, we feel pretty confident on it. Or, you know, vice versa, if an eval is in poor health, a- making that eval more robust. And we have a whole process on eval alignment, which is, how do I make sure the judge is actually aligned to what I think and isn't just outputting scores that I don't believe in? Um, so there's a whole

[37:23](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2243s) process for doing that. And then, again, the experimentation flow. Running your agent against-- or running these datasets against your agent wherever it lives, uh, whether it's your laptop, in my case, for this very simple agent, or whether it's on more complicated infrastructure. Cool. So I think it would be really great to get started with some of these. Uh, so I have the workshop repo URL. Um, we have docs pages as well for agent experimentation.

[37:53](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2273s) And, um, we have a few other talks throughout this week as well. Um, Sophia has one tomorrow at three twenty about, um, agent fixes and their runoff, um, i- implications. So, you know, you fix something somewhere, and that eventually has an impact downstream. Um, and I have a talk tomorrow at three forty-five talking about Alyx, which is our in-house agent. Um, and we use Alyx to build Alyx, so it's all kind of the perfect dogfooding tool for us in

[38:24](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2304s) the agent space. And so I'll talk a little bit about how we designed that agent, you know, how we designed context, memory, uh, and some of the decisions and lessons we learned along the way. Um, but really, I would encourage you guys to try to get some traces from the agent. We have this built out for you, but if you have an, your own example and you haven't traced your agent yet, it's very, very simple to install our skills in CLI, have it instrument your agent for you, and send those traces to Arize. Um, once you get those traces in, it's one click to

[38:53](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2333s) enable Signal. Um, and then you can start to see the clustering of issues and patterns across your traces as well. Um, so yeah, I would say the things I wanna leave you guys with are

[39:08](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2348s) the hard work of reading a ton of data should definitely just be done by an agent. Understanding where to add the highest leverage human points is kind of the theme of this presentation. Just because you can automate something doesn't mean it should be automated, but you should use that automation to really inform the reviewer. So things like auto-routing low confidence threshold scores to that reviewer queue are really important. Um, and making sure that all the

[39:38](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2378s) context is fed back to the same system. And obviously, we're here with Arize, and, you know, we're telling you how great Arize is, but really, I-- you wanna make sure that this context is recorded somewhere, right? And that's the only way you build this self-improvement loop, is when the human judgment, the eval scores, and the traces and the, the trace evidence are all held together in the same place, feeding that context for the next run, so you're catching new issues instead of just fighting to stay afloat. Um, and that's what I have for you guys.

[40:08](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2408s) Thank you so much. Um, I'm happy to walk around and kinda help you guys get started on some of these. Um, you can check out the repo, clone down the agent. If not, uh, we-- you can install the skills, instrument your own agent. And yeah, I can take questions as well, if you would like. I would say there's, like, two lenses to take how we're actually evaluating. There's how do we evaluate Signal as a product line, and that-- one of the key factors there would be, okay, how often is this right about our PRs, right? That's sort of like an internal evaluation. Do you mean more of that, how we

[40:38](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2438s) evaluate our own product, or how you can evaluate an, an agent use case, generally speaking? Uh, we will do a whole talk about evals for Alyx tomorrow. But to give you the long and short of it, um, eval design is, you know, as an, as an AI PM, I think that's probably my primary job, is like, how do I actually design good evals that actually measure the right things? The shorter answer to that question is, I think codifying

[41:08](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2468s) qualitative decisions is very hard. Codifying things like latency, retry count, like, these are more classical issues that... For example, the issue Signal found, I think, is relatively simple. Like, you found an inconsistency in casing, you saw evidence in retries, make the fix, measure the latency. That's really easy to do, right? Now, qualitative outcomes, I think, are a little bit harder. And I would say there's, there's a couple things to index on there. One is

[41:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2496s) the, the actual, like, definition of what is good is only emergent from as much signal as you can give it. And I use signal as, like, the word signal. Um, so what I mean by that is, okay, I can look at one example and say it's right or wrong. I can't get an agent to learn from one example because then it's just gonna overfit to that one example, or that example might not generalize properly. Um, and so part of what I was talking about with the living fixtures and the annotations is a lot of what we do. So I'll give you an example. Alyx

[42:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2526s) generating, um, filter queries, right? So we have Alyx in our platform. Uh, we have a flow where, okay, if you wanna search agents or search traces, you can obviously type in a filter, or you can ask Alyx to filter with AI. How do we actually measure the performance of that? There's no right or wrong answer, 'cause it could return a filter that succeeds, and it doesn't, you know, parse this JSON, and it will give you some results. But how do you know that's actually a good result? So there's a few things.

[42:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2556s) One We look at the entire session of users, right? So we're not just saying, "Okay, for this turn, was this a good response?" But did that user ask again? Did it-- Was it happy with the results? How many additional filters did they create, right? So that can be one thing to, as, as Signal for you to evaluate on. The second is, I, I mean, annotations, like reading through, okay, what was the query? Does this look like a right filter? Did it return the right eval results? And actually annotating that by hand as a PM who knows what to expect. Um, and

[43:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2586s) then you combine both of those with new use cases, and that rich context becomes the eval template, right? So you can actually use another coding agent to feed that context in and write a really good prompt template, and then it becomes like a, a, a prompt optimization essentially, right? So like, how good can I get this eval template on the lowest cost model so I can run this at scale? Um, so that's kind of how we think about it. We have some more specifics in the talk tomorrow where I'll talk through a little bit about like which models we choose

[43:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2616s) and, um, how we think about which use cases are worth running an evaluator or not. Um, but yeah, that's the short answer. So it is kind of like a, you know, post-hoc, ergo propter hoc. Like, where can we make sure, like, what we define as healthy is actually healthy, and where should we define where it breaks? So what I mean by that is, okay, it's right ninety-nine point nine percent of the time. If that point one percent is catastrophic and we need to catch all of those, it's a very different discussion, right?

[44:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2646s) And then evals, when I, when I say like evals, there's kind of two lenses. There's the evals that you can run in the development flow as CI/CD checks, as regression checks before you deploy code. There's evals that you're running on live traces, which that becomes a cost-benefit ratio question, right? If I'm running this on a hundred percent of live traces, is the cost of running this model on all my traces worth it compared to the cost of this point one percent failure, right? So

[44:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2676s) I think a lot of it boils down to ROI. I think general guidelines are whatever you can catch in CI/CD, you should, and I understand it's a challenge to run some of these, like, and, and convince teams to run some of these in CI/CD. I would probably make some of these non-optional checks in the CI/CD process, right? So you have a dataset of fixtures that shows where this eval has actually failed on the point one percent of cases, and you run, ru- you rerun those with an eval score every single time you push or before a

[45:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2706s) PR is merged. So you're at least-- you at least have some confidence that that set of cases is not gonna fail. And then you run an eval that has a sampling rate on your prod traces, right? Um, so instead of saying, "Okay, we're gonna ignore the development process, evaluate everything that comes in production," that can be a very high cost thing to do versus, "Okay, we ignore prod completely and only test in development and try to capture every case," which you probably can't do, um, in a dataset with coverage, and then

[45:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2736s) just let it happen in prod. I would say both of those are extremes, and the answer is somewhere in the middle, and it depends a bit on the ROI of the eval, right? Um, and what model you decide to use has an implication on cost, et cetera, et cetera. It's a very good question, and it gets very meta, and, uh, I ho- I hope you haven't noticed how meta this conversation has gotten so far, where I'm using Signal in m- multiple different ways. Um, the short answer is that you can do that entire loop again, right? So

[46:07](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2767s) we have a project for eval traces. This is Arize created. These are all the evals, and those evals are traced, right? And you, you can have an agent reading the eval traces and making sure the evals are good. And then each of those agents actually produces their own traces, right? So when you're running Signal, these are Signal's traces. So you'll see here, you are Signal, an automated reviewer for Arize. These are the steps you have to do. These are the instructions. This is what Signal actually called under the hood. So made a few bash calls. It called our

[46:37](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2797s) MC-- internal MCP to go do this investigation. It emits it, writes it as an issue, right? And then you can run Signal on top of Signal as well to make sure that... And that's kind of what-- how we evaluate it internally, right? Uh, it's kind of what I was alluding to as the, the ultimate dogfooding tool. Um, now I, I assume your question is more of like a when does this break down? When is this useful to do? Um, I think the harness changes are useful up to a certain point, and I'll talk a little bit about

[47:07](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2827s) this in, in my talk tomorrow on like how we evaluate Alyx. Like, there are certain use cases where you just a- after a certain point, you don't need to continue evaluating, um, and there's just diminishing returns on cost from there. Um, where we've seen it is, you know, we monitor Signal, and we have, you know, traces coming in from Signal. We have some evaluation score criteria, but there is a certain point at which we actually don't know what is good for a customer beyond what the customer knows

[47:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2856s) themselves. Um, so I mean, you can turn on Signal, and it can propose a PR, and we can use some proxy metrics like merge rate on the PR or the session transcript or whether the session was handed off or whether a dataset was created or eval was created as feedback mechanisms. But beyond that, and beyond that and the merge rate of the PR, we can't really tell that much more. So it becomes, you know, diminishing returns to continue to evaluate from there. Um, 'cause then you're just kind of relying on an agent

[48:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2886s) transcript, and you have no ground truth, right? You're just vibing with the agent's output and pretending you're increasing shareholder value, right? So, um, I don't think that's useful. Uh, but I think we have-- To answer your question, we have used Signal to evaluate Signal as well. I think in execution time checks because such low latency is like probably the most exciting, like where you can actually route a request during execution time. So these, these evals that you're seeing are, are, you know, post-hoc where the,

[48:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2916s) the agent has already run, and we're evaluating it from that. So due to the low latency of decision model models, I think that's something I'm actually genuinely excited for, uh, in the execution loop. Um- Like, selfishly, or maybe not even selfishly, but, like, for my personal enjoyment, I like that people are much more likely to use evals now because they're cheaper. Um, so that's, like, just, you know, purely good for the ecosystem health. Uh, like, more people thinking about evals, more people thinking about actually cost-finding their

[49:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2946s) traces. I think it'll unlock that to a large degree just because it's cheaper. Um, and then maybe not related to, to this talk, but, like, just my, my personal opinion is I think, like, um, because these models are so small that there's some really interesting stuff you can do with on-device stuff, um, that I'm really, really excited for. Um, we're also, like, kind of pioneering it for some multimodal evals as well, um, because I think multimodal evals can be really complicated because of latency, um, and they can also

[49:36](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=2976s) be super, super qualitative. So in the multimodal eval space, like, trusting an agent's taste on an image that was generated, for example, or how it interpreted a click box can be very hard, but getting confidence scores allows you to at least route this to human a lot faster. So I think that's also a space where decision models will help a lot. Yes, a hundred percent. So I kind of briefly alluded to this, uh, when I, uh, talked about skills, uh, but you can actually create your own agents. Um, and,

[50:06](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3006s) um, we have the out-of-the-box Signal agent, obviously. Um, let me do this one.

[50:18](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3018s) So you'll see here we have a Datadog skill as well, where you can pull metrics, logs, and APM traces around the alert. Um, we have a bunch of providers available for you to do that as well. You can also write your own skills. So the, the agent is-- doesn't just have access to the agent traces, but downstream system application metrics as well. Um, so you can connect it to basically any data source. Um, obviously, there's, like, an element of taste to the data source. Like, connect the ones that you need to connect, make sure it

[50:48](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3048s) has the right access, make sure, you know, it's scope key, make sure that you're not overflooding context because more is not always better to actually triage the alert. Uh, but yeah, we have the ability to do all that as well. The, the shitty answer that everyone always gives is it depends, but it, it does kind of depend. So for example, like, what harness are you using? Um, what underlying providers are you using? How often are you running this? And we expose those all as configurations, and we allow you to kind of monitor the cost too. So you'll see here that this is the Signal traces.

[51:18](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3078s) This is how much it's-- Well, I guess this is still flowing in right now. But the same way that you can view kind of your cost for, like, Wonder Toys, you can view the cost of Signal. Uh, so we make it easy for you to understand the ROI. Like, okay, if this shopping assistant agent cost me eight dollars and it cost me twenty bucks to improve with the self-improvement loop, is that really worth doing or not? Um, now, like, I don't have, like, a hard number as, like, five percent of your agent budget should be allocated

[51:48](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3108s) to the self-improvement loop or something like that. I think it really just depends. Um, but I will say, like, that is exposed to you and available to use, and there are many ways in which you can kind of pull the levers to make that a little bit cheaper or more expensive, depending on how you're seeing results. Um, so one is the amount of context you're providing, right? So which skills are you including up to a certain point without, like, just spending more money to give it more information that doesn't actually help. Um, and then the other

[52:18](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3138s) is, like, the eval kind of design question, how often you're running these evals, sampling rate, et cetera. Um, and then the other thing is, like, you-- I mean, you can use this part of the platform, but all these are consumable via API as well. So if you want to shunt this to coding agents, which you get a subsidized cost for or something like that, you can just export that to a coding agent and have it do many, many of the similar things. Thank you for the wonderful questions. I'll be around over the next three days. We have the Arize booth as well. Um, feel free to sign up. Try

[52:48](https://www.youtube.com/watch?v=UGuNt2_Oz8s&t=3168s) this loop out for yourself. Completely free to sign up. Completely free to use. Um, and yeah, happy debugging.
