Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

Observability usually ends with a person reading a dashboard and deciding what to change. Fuad Ali, who works on agent experimentation and tooling at Arize AI, opens with a different destination for telemetry: the coding agent that can investigate and repair the application. His reference point comes from backend distributed systems work at SpaceX, where a rocket consumes its own telemetry to correct its course. The proposed agent workflow applies that feedback principle to software behavior.

The progression has three forms. A developer reads observability data and writes code; a developer reads it and prompts a coding agent; or the coding agent consumes the telemetry directly and begins the investigation itself. The last form removes the repeated human instruction to go inspect a backend. Ali expects that distinction to matter as teams operate thousands of agents, beyond what people can inspect individually.

That requires observability to offer programmatic interfaces an agent can query and act on. A dashboard helps a person notice a problem. An agent also needs access to the underlying events, enough context to explain them, and a way to propose a change. The workshop develops those connections through a deliberately faulty shopping assistant.

The motivating failure is ordinary enough to escape a launch checklist: a shopper asks for a birthday gift under $30, and the assistant returns products above that budget, including $50 items in the opening example. Its confident response supplies no warning that the constraint failed. Existing evaluations can pass while an unanticipated production case remains wrong; the first useful signal may arrive as a complaint or support ticket.

Wonder Toys Assistant makes that failure concrete. It uses the OpenAI Agents SDK with a FastAPI backend, vector search and a text-search fallback. Its tools can search products, retrieve product details, purchase an item, check order status and cancel an order. The web interface lets a user ask about a product or request a purchase; purchases in this demonstration do not involve a real credit card.

The application is already instrumented to send traces to Arize. Getting the example running means cloning the repository, entering it, adding API credentials and starting the development server. The two services explicitly identified are Arize for tracing and OpenAI for the agent. Interacting with the assistant then produces traces automatically, giving the investigation a record of actual application behavior.

Ali deliberately inserted a price-gating bug: most queries work, but a request for a gift under $30 can return items over $33. The useful diagnostic connection is between the requested constraint in the trace and the code path meant to enforce it. The recording identifies a filtering failure but does not establish the exact faulty expression or patch, so the mechanism to retain is the missing enforcement of the budget, rather than a guessed code correction.

Signal is Arize's managed investigation agent. It runs in a sandbox, is enabled through project settings, and continuously examines incoming production traces. It groups failures by root cause and deduplicates recurring failures across runs. That turns repeated incidents into issues an engineer can investigate together, rather than treating every trace as a separate problem.

Its harness can call external tools. Connecting GitHub gives it application code; connecting a downstream system such as Datadog gives it additional operational context. Signal can combine those inputs with traces to write an investigation, assign severity and entity type, gather evidence and open a pull request. The code helps explain how the observed behavior arose, while the traces show which behavior actually occurred.

Arize skills offer another place to run that investigation: inside the coding agent already working in the repository. The agent can query Arize traces while inspecting local application code. The available skills cover trace analysis, creating datasets from failures, running evaluators and running platform tasks. Repository access also lets the coding agent edit the application directly.

The choice is mainly how the work starts and where it runs:

• On demand: an engineer asks a coding agent such as Claude Code to inspect traces alongside the current repository and work on a repair.
• Continuously: Signal watches production traces in the background, develops investigations and can propose pull requests through its repository connection.

These are alternative routes to the same combination of runtime evidence and implementation context; neither depends on running the other first.

What does the assistant actually do between receiving a request and returning products? In the Wonder Toys trace, the workflow starts, the agent responds, calls product search, takes another turn and returns the products it found. That sequence makes the tool-mediated behavior inspectable: an investigation can examine the search activity and resulting answer instead of relying only on the final chat message.

The hands-on setup runs into practical obstacles before that point. Participants report stalled cloning over the venue Wi-Fi, and Ali returns to installing dependencies, locating the .env file, entering credentials and running npm run dev. In response to a question about account access, he clarifies that evaluations are available on the free account used for the workshop, while some features, including code evaluations, are restricted. Those restricted features are not required for the trace-analysis route being shown.

Signal needs traces before it can begin producing analysis, and Ali says startup takes a few minutes after traces start flowing. With Wi-Fi still slowing the room, he switches to an already configured example. This separates the live application's trace inspection from the prepared Signal investigation that follows.

Signal's issue view groups problems into categories and shows how they trend over time, with examples available for inspection. This adds a second scale of analysis above the individual trace: engineers can see a recurring class of failure, then open a concrete occurrence to understand it.

What should an investigation contain before someone reviews its proposed fix? The prepared examples include Python serialization problems corrupting agent input, an already opened pull request, and a stock-agent investigation involving fabricated results. The investigation brings together an overview, a hypothesis, evidence within an individual trace and a proposed correction. These are separate examples from the shopping-assistant budget bug; their value is showing the form of the investigation. Human code review remains part of the demonstrated workflow, and Ali recommends retaining it.

Returning to Wonder Toys, the causal bridge is specific. The trace contains the user's request for a price under $30. The investigation checks the code path responsible for filtering prices. That connection supports a proposed repair to the faulty filter, rather than a vague instruction for the assistant to be more careful. A plausible repair still needs a separate verification step.

Continuous investigation can also start from configured alerts. Ali gives examples of an evaluation trigger rate that must remain below 1%, a latency spike or a cost spike. Signal can use the application's taxonomy and traces, while custom judges can run in CI on proposed pull requests. Detection therefore includes both semantic failures, such as the wrong price, and operational changes that deserve investigation.

A fixed LLM judge works well when the team already knows what to score: supply the relevant input and response, apply a rubric, and return an assessment. Ali uses “Harness as a judge” for an evaluator that can inspect an execution trajectory and use tools while deciding what happened. The distinction matters when the failure involves intermediate actions or information outside the final response.

The trajectory supplies concrete questions for adaptive analysis:

• Tool selection: did the agent choose an appropriate tool for the task?
• Arguments: were the values passed to that tool valid?
• Handoff context: did an action use the right information from another agent?

The harness reads the trace tree directly and runs in a sandbox. A plain-language evaluation instruction can guide it without requiring the user to map trace fields into an evaluation template first.

Access checks show why tools change what a judge can establish. A model reading only an input and output may have no way to know a user's permissions. A connected harness can query an external system for the access that user should have, then compare that answer with the actions recorded in the trace. The permission lookup supplies missing facts; the trace supplies the behavior being assessed.

For Wonder Toys, the resulting price-correctness rubric is direct: pass if every returned product's price falls within the requested minimum and maximum; fail if any result exceeds the budget. The harness can also inspect the filter code. Once the missed constraint becomes an explicit evaluation rule, subsequent runs can be checked for that class of error instead of relying on one remembered birthday-gift example. The rule evaluates the returned set; one affordable recommendation does not excuse an over-budget item beside it.

A proposed code change answers what might fix the problem. Agent experiments answer whether the changed application handles the cases that exposed it. The setup places a development version behind an endpoint and sends it a dataset drawn from production failures, with automated execution available through CI.

Ali's illustrative dataset contains 25 failed budget-constraint searches, such as birthday-gift requests under $30. Arize skills can curate those examples from traces. The repair may come from a coding-agent session or from Signal; either route produces a candidate that can be deployed to development. The experiment sends the dataset inputs to that endpoint and collects fresh responses from the changed agent.

What changes when the same failure cases reach the repaired version? Score its responses with the same judge rubric used in production, then compare them with the recorded production behavior. For Wonder Toys, inspect the queries, returned products and price-correctness scores. Include latency and cost in the comparison: a quality improvement may carry an operating penalty that affects whether the change is worth shipping.

The data flow below separates two uses of the failure dataset. It preserves the production behavior for comparison and supplies inputs for a fresh development run. The candidate code must be deployed before that fresh run can test it; the historical production result is not regenerated by the development endpoint.

Experiments are exposed through skills and APIs, so the investigating agent can continue beyond opening a pull request. It can collect failures, create a dataset, deploy the proposed version to development and test it before notifying a reviewer. In the budget example, the complete chain links the under-$30 request, the over-budget response, the filtering diagnosis, the candidate patch and replay results.

The proposed handoff is a pull request containing both the code and the experiment's explanation of what improved. Ali describes a 36% performance increase on a specific evaluator. The accompanying example slide makes the comparison explicit: price-correctness rises from 64% to 100%, a 36-percentage-point increase. These are the presented example results, rather than an independently reproduced result for the workshop. The reviewer can inspect the evidence, validate the code and decide whether to ship.

The substantive separation is between producing a repair and testing its behavior. An agent's statement that its patch is correct is weaker than running the changed application on real failure cases and evaluating the results. The validation described here tests the collected failures with the chosen judge; it does not by itself establish correctness on every future request.

The closing setup advice starts with the prerequisite: “If you're not collecting tracing, nothing else works.” Instrumentation supplies the execution record. Signal or a coding agent with Arize skills investigates it; a harness judge evaluates behavior; agent experiments measure the candidate against replayed cases. For the workshop application, locating the environment file and configuring API credentials remain the practical first steps.

To add the coding-agent integration shown at the end, the installation command is:

npx skills add Arize-ai/arize-skills

The installer offers an interactive selection or installation of all skills, including trace analysis, dataset curation and evaluators. A useful first request is to inspect traces for errors and cross-reference them with the agent code in the current repository. That gives the coding agent both the observed failure and the implementation it needs to explain it.