AI Engineer World's Fair 2026
Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Slide chapters
20 slide and demo groups · Timestamps are approximate. 19 additional groups were omitted from this pass.
Read the talk
Building Self-Learning Loops for Your Agent
Fuad Ali follows a broken shopping-assistant price filter from production traces to a proposed repair, then explains how tool-using judges and replayed failure cases can test the change before human review.
From a talk by Fuad Ali
At a glance
Ideas worth remembering
Combine traces with repository code: the trace records the violated user constraint, while the code reveals the path meant to enforce it.
Signal provides continuous investigation; Arize skills let an engineer initiate similar work inside a coding agent.
A tool-using judge can inspect trajectories and retrieve missing facts, such as user permissions, that an input-and-output-only evaluation cannot establish.
Replay failure-derived requests against the development endpoint, use the production judge rubric, and compare quality alongside latency and cost.
An autonomous repair workflow can prepare a pull request with validation results before human review; testing the changed application remains distinct from generating the patch.
Give the coding agent access to what happened
Observability usually ends with a person reading a dashboard and deciding what to change. Fuad Ali, who works on agent experimentation and tooling at Arize AI, opens with a different destination for telemetry: the coding agent that can investigate and repair the application. His reference point comes from backend distributed systems work at SpaceX, where a rocket consumes its own telemetry to correct its course. The proposed agent workflow applies that feedback principle to software behavior.
The progression has three forms. A developer reads observability data and writes code; a developer reads it and prompts a coding agent; or the coding agent consumes the telemetry directly and begins the investigation itself. The last form removes the repeated human instruction to go inspect a backend. Ali expects that distinction to matter as teams operate thousands of agents, beyond what people can inspect individually.
That requires observability to offer programmatic interfaces an agent can query and act on. A dashboard helps a person notice a problem. An agent also needs access to the underlying events, enough context to explain them, and a way to propose a change. The workshop develops those connections through a deliberately faulty shopping assistant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A confident answer can still break the budget
The motivating failure is ordinary enough to escape a launch checklist: a shopper asks for a birthday gift under $30, and the assistant returns products above that budget, including $50 items in the opening example. Its confident response supplies no warning that the constraint failed. Existing evaluations can pass while an unanticipated production case remains wrong; the first useful signal may arrive as a complaint or support ticket.
Wonder Toys Assistant makes that failure concrete. It uses the OpenAI Agents SDK with a FastAPI backend, vector search and a text-search fallback. Its tools can search products, retrieve product details, purchase an item, check order status and cancel an order. The web interface lets a user ask about a product or request a purchase; purchases in this demonstration do not involve a real credit card.
The application is already instrumented to send traces to Arize. Getting the example running means cloning the repository, entering it, adding API credentials and starting the development server. The two services explicitly identified are Arize for tracing and OpenAI for the agent. Interacting with the assistant then produces traces automatically, giving the investigation a record of actual application behavior.
Ali deliberately inserted a price-gating bug: most queries work, but a request for a gift under $30 can return items over $33. The useful diagnostic connection is between the requested constraint in the trace and the code path meant to enforce it. The recording identifies a filtering failure but does not establish the exact faulty expression or patch, so the mechanism to retain is the missing enforcement of the budget, rather than a guessed code correction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Signal and skills connect traces to the repository
Signal is Arize's managed investigation agent. It runs in a sandbox, is enabled through project settings, and continuously examines incoming production traces. It groups failures by root cause and deduplicates recurring failures across runs. That turns repeated incidents into issues an engineer can investigate together, rather than treating every trace as a separate problem.
Its harness can call external tools. Connecting GitHub gives it application code; connecting a downstream system such as Datadog gives it additional operational context. Signal can combine those inputs with traces to write an investigation, assign severity and entity type, gather evidence and open a pull request. The code helps explain how the observed behavior arose, while the traces show which behavior actually occurred.
Arize skills offer another place to run that investigation: inside the coding agent already working in the repository. The agent can query Arize traces while inspecting local application code. The available skills cover trace analysis, creating datasets from failures, running evaluators and running platform tasks. Repository access also lets the coding agent edit the application directly.
The choice is mainly how the work starts and where it runs:
- On demand: an engineer asks a coding agent such as Claude Code to inspect traces alongside the current repository and work on a repair.
- Continuously: Signal watches production traces in the background, develops investigations and can propose pull requests through its repository connection.
These are alternative routes to the same combination of runtime evidence and implementation context; neither depends on running the other first.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with an inspectable application run
What does the assistant actually do between receiving a request and returning products? In the Wonder Toys trace, the workflow starts, the agent responds, calls product search, takes another turn and returns the products it found. That sequence makes the tool-mediated behavior inspectable: an investigation can examine the search activity and resulting answer instead of relying only on the final chat message.
The hands-on setup runs into practical obstacles before that point. Participants report stalled cloning over the venue Wi-Fi, and Ali returns to installing dependencies, locating the .env file, entering credentials and running npm run dev. In response to a question about account access, he clarifies that evaluations are available on the free account used for the workshop, while some features, including code evaluations, are restricted. Those restricted features are not required for the trace-analysis route being shown.
Signal needs traces before it can begin producing analysis, and Ali says startup takes a few minutes after traces start flowing. With Wi-Fi still slowing the room, he switches to an already configured example. This separates the live application's trace inspection from the prepared Signal investigation that follows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn recurring failures into a reviewable explanation
Signal's issue view groups problems into categories and shows how they trend over time, with examples available for inspection. This adds a second scale of analysis above the individual trace: engineers can see a recurring class of failure, then open a concrete occurrence to understand it.
What should an investigation contain before someone reviews its proposed fix? The prepared examples include Python serialization problems corrupting agent input, an already opened pull request, and a stock-agent investigation involving fabricated results. The investigation brings together an overview, a hypothesis, evidence within an individual trace and a proposed correction. These are separate examples from the shopping-assistant budget bug; their value is showing the form of the investigation. Human code review remains part of the demonstrated workflow, and Ali recommends retaining it.
Returning to Wonder Toys, the causal bridge is specific. The trace contains the user's request for a price under $30. The investigation checks the code path responsible for filtering prices. That connection supports a proposed repair to the faulty filter, rather than a vague instruction for the assistant to be more careful. A plausible repair still needs a separate verification step.
Continuous investigation can also start from configured alerts. Ali gives examples of an evaluation trigger rate that must remain below 1%, a latency spike or a cost spike. Signal can use the application's taxonomy and traces, while custom judges can run in CI on proposed pull requests. Detection therefore includes both semantic failures, such as the wrong price, and operational changes that deserve investigation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A judge can inspect tools and ask for missing context
A fixed LLM judge works well when the team already knows what to score: supply the relevant input and response, apply a rubric, and return an assessment. Ali uses “Harness as a judge” for an evaluator that can inspect an execution trajectory and use tools while deciding what happened. The distinction matters when the failure involves intermediate actions or information outside the final response.
The trajectory supplies concrete questions for adaptive analysis:
- Tool selection: did the agent choose an appropriate tool for the task?
- Arguments: were the values passed to that tool valid?
- Handoff context: did an action use the right information from another agent?
The harness reads the trace tree directly and runs in a sandbox. A plain-language evaluation instruction can guide it without requiring the user to map trace fields into an evaluation template first.
Access checks show why tools change what a judge can establish. A model reading only an input and output may have no way to know a user's permissions. A connected harness can query an external system for the access that user should have, then compare that answer with the actions recorded in the trace. The permission lookup supplies missing facts; the trace supplies the behavior being assessed.
For Wonder Toys, the resulting price-correctness rubric is direct: pass if every returned product's price falls within the requested minimum and maximum; fail if any result exceeds the budget. The harness can also inspect the filter code. Once the missed constraint becomes an explicit evaluation rule, subsequent runs can be checked for that class of error instead of relying on one remembered birthday-gift example. The rule evaluates the returned set; one affordable recommendation does not excuse an over-budget item beside it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replay the failures against the proposed fix
A proposed code change answers what might fix the problem. Agent experiments answer whether the changed application handles the cases that exposed it. The setup places a development version behind an endpoint and sends it a dataset drawn from production failures, with automated execution available through CI.
Ali's illustrative dataset contains 25 failed budget-constraint searches, such as birthday-gift requests under $30. Arize skills can curate those examples from traces. The repair may come from a coding-agent session or from Signal; either route produces a candidate that can be deployed to development. The experiment sends the dataset inputs to that endpoint and collects fresh responses from the changed agent.
What changes when the same failure cases reach the repaired version? Score its responses with the same judge rubric used in production, then compare them with the recorded production behavior. For Wonder Toys, inspect the queries, returned products and price-correctness scores. Include latency and cost in the comparison: a quality improvement may carry an operating penalty that affects whether the change is worth shipping.
The data flow below separates two uses of the failure dataset. It preserves the production behavior for comparison and supplies inputs for a fresh development run. The candidate code must be deployed before that fresh run can test it; the historical production result is not regenerated by the development endpoint.
Recorded failures become both replay inputs and a comparison baseline
Read the diagram as text
- Production failure dataset. Curated budget-search traces retain failed requests and recorded production behavior.
- Candidate repair. A proposed filter fix from the coding agent or Signal.
- Development agent endpoint. The deployed candidate processes replayed requests and produces new responses.
- Price-correctness judge. Score the new responses using the same rubric used in production.
- Before-and-after comparison. Inspect production versus development behavior, evaluator results, latency and cost.
- Candidate repair → Development agent endpoint: Deploy changed version.
- Production failure dataset → Development agent endpoint: Replay dataset inputs.
- Development agent endpoint → Price-correctness judge: Evaluate new responses.
- Price-correctness judge → Before-and-after comparison: Development evaluation results.
- Production failure dataset → Before-and-after comparison: Recorded production baseline.
- Development agent endpoint → Before-and-after comparison: Development latency and cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring the repair and its test results to human review
Experiments are exposed through skills and APIs, so the investigating agent can continue beyond opening a pull request. It can collect failures, create a dataset, deploy the proposed version to development and test it before notifying a reviewer. In the budget example, the complete chain links the under-$30 request, the over-budget response, the filtering diagnosis, the candidate patch and replay results.
The proposed handoff is a pull request containing both the code and the experiment's explanation of what improved. Ali describes a 36% performance increase on a specific evaluator. The accompanying example slide makes the comparison explicit: price-correctness rises from 64% to 100%, a 36-percentage-point increase. These are the presented example results, rather than an independently reproduced result for the workshop. The reviewer can inspect the evidence, validate the code and decide whether to ship.
The substantive separation is between producing a repair and testing its behavior. An agent's statement that its patch is correct is weaker than running the changed application on real failure cases and evaluating the results. The validation described here tests the collected failures with the chosen judge; it does not by itself establish correctness on every future request.
The closing setup advice starts with the prerequisite: “If you're not collecting tracing, nothing else works.” Instrumentation supplies the execution record. Signal or a coding agent with Arize skills investigates it; a harness judge evaluates behavior; agent experiments measure the candidate against replayed cases. For the workshop application, locating the environment file and configuring API credentials remain the practical first steps.
To add the coding-agent integration shown at the end, the installation command is:
sh
npx skills add Arize-ai/arize-skills
The installer offers an interactive selection or installation of all skills, including trace analysis, dataset curation and evaluators. A useful first request is to inspect traces for errors and cross-reference them with the agent code in the current repository. That gives the coding agent both the observed failure and the implementation it needs to explain it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Extends the use of execution traces into reflective optimization of prompts, agent programs and repository skills, including how evaluators guide candidate selection.
Read the complete timestamped transcript
15 slide groups · 5 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 19 additional groups were omitted from this pass.
- 0:12
My name's Fuad Ali. I work in product at Arize AI, and I've been working on agent experimentation and generally agent tooling across the stack at Arize. Uh, you just heard from Laurie, uh, who is also part of our team here at Arize. He's gonna be flitting around the room. Aparna, our co-founder and CPO, will also be flitting around the room to help you guys out if you require during this workshop. Uh, what we're gonna be talking about today is building self-learning loops for your agent. Um, and just to give you a bit of background
Building self-learning loops for your agent
Fuad Ali introduces self-learning loops in Arize, comparing them to a rocket consuming telemetry to correct its course.
Text in image
arize AI ENGINEER WORLD'S FAIR 2026 WORKSHOP #2 OF 5 Building self-learning loops for your agent “Shipping is easy. Keeping it working is hard.” New in Arize AX: Signal • Harness-as-a-Judge • Agent Experiments Fuad Ali · Arize AI Interface text: [illegible] to navigate F fullscreen New in Arize AX 01 / 25 1 / 25
- 0:42
on myself and, and how I got into this space, I think this concept of self-learning loops is really new to this idea of AI engineering. Uh, and there's a lot of features in Arize that kind of help you support it. But to kind of frame this, um, I actually worked in backend distributed sy- systems at SpaceX, and it is actually quite interesting how we're kind of coming full circle in this world of telemetry, where, you know, if you think about it, uh, a rocket system that is continuously course-correcting every millisecond is
- 1:12
actually consuming its own telemetry and then actioning that telemetry in order to self-correct. And basically, what I'm gonna explain is how we've sort of built this idea of self-learning loops for your agent in Arize and how you can use that to self-correct. Cool. So to give you a bit of history on Arize, we made a few big bets. Uh, we were first to the market with LLM as a judge back in twenty twenty-three. Uh, we created the first set of semantic conventions for LLM applications
Our big bets
Arize traces its bets from LLM-as-a-judge in 2023 through agent evals in 2025, then names self-correcting agents with feedback loops as its next bet for 2026.
Text in image
arize HISTORY OF ARIZE HISTORY OF ARIZE Our big bets Defining the next era of GenAI evals. Timeline: 2023 — LLM as a Judge 2024 — OTel / OpenInference for GenAI 2024 — Real-time AI datastore (ADB) 2025 — Agent evals + Alyx 2026 — OUR NEXT BIG BET Self-correcting agents with a feedback loop Traces become evidence. Agents close their own loop. Interface text: [illegible] to navigate F fullscreen New in Arize AX 02 / 25 2 / 25
Our big bets
Arize traces its bets from LLM-as-a-judge in 2023 through agent evals in 2025, then names self-correcting agents with feedback loops as its next bet for 2026.
Text in image
arize HISTORY OF ARIZE HISTORY OF ARIZE Our big bets Defining the next era of GenAI evals. Timeline: 2023 — LLM as a Judge 2024 — OTel / OpenInference for GenAI 2024 — Real-time AI datastore (ADB) 2025 — Agent evals + Alyx 2026 — OUR NEXT BIG BET Self-correcting agents with a feedback loop Traces become evidence. Agents close their own loop. Interface text: [illegible] to navigate F fullscreen New in Arize AX 02 / 25 2 / 25
- 1:42
with OpenInference, open sourced. Uh, we created the first real-time AI data store, ADB. Uh, in twenty twenty-five, we released Agent Evals and Alyx, which is our in-house agent, uh, to kind of help you with AI engineering. Think of Cursor for engin- AI engineering. And in twenty twenty-six, the next big bet we really wanna make is how you can create self-correcting agents with a feedback loop. Um, and I think everyone understands sort of, like, how that is going to be extremely powerful
- 2:12
to this idea of, of agents generally. Um, and so I really think the, the industry is evolving quite a bit, um, when it comes to software engineering. Uh, you sort of have the classic, you know, software one point oh where there's an app, and it's traced to Datadog, and there's some sort of observability platform that human beings read. A human dev then writes code to actually correct it. I think two point oh was really humans interacting with this in prompts. Um, so same
The evolution of software engineering
The slide moves from developers reading telemetry and writing code, through prompting an agent, to coding agents reading telemetry directly. This shift calls for programmatic observability interfaces.
Text in image
The evolution of software engineering Who consumes the telemetry? Phase 1 · Software 1.0 Application → telemetry → Observability Platform → reads → Human Dev → writes code → IDE IDE → deploy → Application Human is the sole consumer of telemetry Phase 2 · Software 2.0 Application → telemetry → Observability Platform → reads → Human Dev → prompts → Agentic IDE Coding Agent · Claude Code Agentic IDE → deploy → Application Human prompts, agent writes code inside the IDE Phase 3 · Autonomous Application → telemetry → Observability Platform → reads → Coding Agent Claude Code / Cursor Coding Agent → notifies → Human Coding Agent → deploy → Application [illegible] Agent autonomously consumes telemetry Observability has to evolve from human dashboards to programmatic interfaces that agents can consume. Interface text: arize WHERE THIS IS GOING New in Arize AX 03 / 25 3 / 25
- 2:42
thing, but instead of you writing the code, you write a prompt that then corrects the behavior that you're seeing in your app. Uh, what we're moving to really is phase three, which is the fully autonomous loop. In this loop, your application produces telemetry events, observability, and then what is actually reading that is your coding agent. It's not just a human reading it and then putting a prompt in. Um, if any of you guys are using things like Cursor, Claude Code, the Datadog CLI, you're already seeing some of this.
- 3:12
Uh, the gap is you're still probably prompting that to go read that information from a backend like Datadog or whatever it is. Um, and I think when you evolve to the, to the needs of, like, you know, thousands of agents across a stack, it starts to become pretty much impossible for a human being to actually read those things. And so, um, gonna be talking a little bit about some of the fe- new features in Arize that we've released that kind of support this, uh, flow. But what I want you guys to center on is that observability has to
- 3:42
evolve from humans reading dashboards to really programmatic interfaces that agents consume, agents can operate on, and agents can self-reflect on as well.
Our proof in the market
Arize reports 1 trillion spans and inferences processed, 60+ million evals per month, and 3+ million downloads per month, alongside a list of customers.
Text in image
OUR PROOF IN THE MARKET 1 Trillion spans & inferences processed 60+ Million evals per month 3+ Million downloads per month TRUSTED BY TEAMS THAT SHIP Walmart Vanguard Uber Tripadvisor Priceline PagerDuty Spotify Skyscanner SIEMENS ROBLOX reddit Flipkart Handshake Ford DoorDash CVS Health Wayfair Booking.com AT&T Siemens Healthineers Air Canada Wells Fargo Lowe’s Klaviyo + 1,000s of other customers across healthcare, finance, travel & airline, and insurance. Interface text: arize OUR PROOF IN THE MARKET New in Arize AX 05 / 25 5 / 25
- 3:58
So just some proof in the market, uh, Arize has processed over one trillion spans. Uh, we've run over sixty million evaluations per month. Uh, and we have over three million downloads per month. Uh, trusted by a ton of different cool teams, uh, whether it's Spotify, Reddit, Uber, TripAdvisor. So just to, you know, build some confidence that I'm not just here vibe coding something and telling you guys how it should work. Um,
The failures that cost you are invisible at launch
A shopping agent can pass evals and staging yet return gifts over the requested budget or purchase the wrong item. The failure may only surface when a user complains.
Text in image
arize THE PROBLEM The failures that cost you are invisible at launch A shopper asks for "a birthday gift under $30" — gets a list of items over $50. The agent confidently purchases the wrong item because it never verified the product. These don't show up in staging. They show up when users complain. THE GAP It works in eval. It works in staging. You ship it — and the first time you learn it's broken is a Slack message saying "something seems off." New in Arize AX 6 / 25 Interface text: 06 / 25
- 4:25
and I think if you look at the industry, um, the industry is really kind of moving towards this as well. It's not just, you know, these teams, but it's also teams that aren't even Arize customers, like Ramp Labs, and how they've kind of moved to this idea of making software self-improving and self-healing. Um, and I just point this out to say the industry is really moving in this direction. It's not just us. You want agents that can detect a problem, investigate it, ship a fix in the same
Teams are making their own systems self-maintaining
Ramp Labs describes an agent that monitors production, triages alerts and proposes fixes for Ramp Sheets, with engineer review before merges. The slide presents this as a model for a detect–investigate–fix loop.
Text in image
arize INDUSTRY SIGNAL THE INDUSTRY IS ALREADY MOVING Teams are making their own systems self-maintaining Ramp Labs @RampLabs How we made Ramp Sheets self-maintaining An agentic system maintains Ramp Sheets — it continuously monitors production, triages alerts, and proposes fixes without human intervention. No code merges without engineer review. 1,000 AI-generated monitors running in production 1 / 75 one monitor for every 75 lines of code 0 human triage in the loop — until review WHY IT MATTERS The pattern is the same one we're building today: detect → investigate → fix, run by the agent. Today we make that loop work on your agent. New in Arize AX 4 / 25 Interface text: 04 / 25
- 4:55
loop, uh, and that's sort of the loop I'm gonna show you guys how to build today.
The failures that cost you are invisible at launch
A shopping agent can pass evals and staging yet return gifts over the requested budget or purchase the wrong item. The failure may only surface when a user complains.
Text in image
arize THE PROBLEM The failures that cost you are invisible at launch A shopper asks for "a birthday gift under $30" — gets a list of items over $50. The agent confidently purchases the wrong item because it never verified the product. These don't show up in staging. They show up when users complain. THE GAP It works in eval. It works in staging. You ship it — and the first time you learn it's broken is a Slack message saying "something seems off." New in Arize AX 6 / 25 Interface text: 06 / 25
- 5:02
Cool. So, um, I think the, the big problem I wanna outline here is the failures that cost you are invisible at launch, and that's sort of by definition because if you had caught them, they wouldn't have cost you. You would have shipped the fix already. Um, and I think, like, really what I'm trying to center on here is, you know, your agent works in your experimentation phase. It works with the evals that you set up. You don't really know all the evals that you have to set up before you ship it, otherwise you would have written
- 5:32
it. Um, and something bre- breaks in edge case that your team just wasn't aware about. For example, in this, in this example, a shopper's asking for a birthday gift under thirty bucks. They get a list of items over thirty, fifty dollars. And then what you'll see in production is the agent confidently purchases the wrong item, always does so extremely confidently, uh, because it never verifies the product. And these don't-- again, these don't show up in staging. They show up when users complain. They show up when you get a support ticket. They show up when you get
- 6:02
paged on call at three AM. Um, and that gap, it works in staging, didn't work in production, something sees off, is what I'm trying to close here.
Wonder Toys shopping assistant
Wonder Toys combines five shopping tools with vector search and a text fallback; Arize AX traces every call. The slide flags a hidden bug.
Text in image
arize THE DEMO AGENT The agent: Wonder Toys shopping assistant ARCHITECTURE AGENT OpenAI Agents SDK · gpt-5.4-mini ↓ API FastAPI backend · Next.js frontend ↓ SEARCH ChromaDB — vector, text fallback ↓ TRACE Arize AX — every call traced 5 TOOLS search_products find toys by query, age, price, category get_product_detail full info for one product purchase_product place an order check_order_status look up an existing order cancel_order cancel a placed order PRE-INSTALLED Arize skills already in .agents/skills/ — trace, signal, evaluator, experiment. It looks clean. There's a bug hiding in it. Interface text: New in Arize AX 08 / 25 8 / 25
- 6:15
Cool. So, uh, real quick just to ground our kind of talk in an example, we have this thing called the Wonder Toys Assistant. Um, it's a shopping assistant, very simple application, has a few tools available to it to search products, get product details, get the information required for one particular product, purchase that product, uh, check an order status, and cancel an order. Um, we have the OpenAI Agents SDK running on top of this, a fast
- 6:45
API backend, uh, search with a vector search and a text fallback, and it's traced to Arize, so you can actually see the source of truth for what that agent is doing in production. And just to give you a little kind of like sneak preview of what that agent is, very simple. You know, you kind of have like a web interface for Wonder Toys, and I'll zoom in a bit so you guys can see. Um, I can click on ask about the product.
Wonder Toys home screen
The home screen pairs suggested questions with product cards and category links, giving shoppers ways to start a conversation or browse toys.
Text in image
Wonder Toys Welcome to Wonder Toys! I can help you find toys, answer questions about products, place orders, and check order status. What toys do you have for a 5-year-old? Show me STEM toys I need a gift for a toddler POPULAR RIGHT NOW Color Match – Clas... $7.99 ★★★★★ 4.8 Soft Plush Elephant $19.99 ★★★★★ 4.8 Sweet Path Classic $12.99 ★★★★★ 4.6 Nesting & Stacking ... $8.99 ★★★★★ 4.7 Modeling Dough M... $18.99 ★★★★★ 4.7 BROWSE BY CATEGORY Arts & Crafts Bath & Toys Board & Games Building Card & Games Classic & Retro Collectibles Construction vehicles Developmental Dolls Educational Electronic Figurines Garden & Nature Outdoor Performance Plush Pretend & Play Puzzles Science kits Specialty Sports Stem Travel Vehicles Interface text: localhost:3000 Demo Shopper Sign out 1
7:05Demo milestone
- 7:16
Ask the agent to
- 7:21
purchase the product, and don't worry, no real credit card's involved in this demo at all.
Wonder Toys product request
Wonder Toys initially cannot find Color Match – Classic by name, then displays a $7.99 product card with an Add to Cart button after the user asks to purchase it.
Text in image
Tell me about Color Match – Classic I couldn’t find a product with that exact name. If you want, I can search for it by keywords like: • color match • matching game • classic Or I can help you browse similar toys. please purchase this product Sure — I found Color Match – Classic. Color Match – Classic — $7.99 Add to Cart Interface text: Wonder Toys Demo Shopper Sign out Ask about toys, search products, or check your orders... Send localhost:3000
7:24Demo milestone
- 7:26
Cool. Um, so that just grounds us a little bit in the example we'll talk, we'll be talking about today.
The agent: Wonder Toys shopping assistant
Wonder Toys combines an OpenAI agent, FastAPI and Next.js, and ChromaDB with five shopping tools. Arize AX traces every call; the slide flags a hidden bug despite the pre-installed skills.
Text in image
The agent: Wonder Toys shopping assistant arize THE DEMO AGENT ARCHITECTURE AGENT OpenAI Agents SDK · gpt-5.4-mini ↓ API FastAPI backend · Next.js frontend ↓ SEARCH ChromaDB — vector, text fallback ↓ TRACE Arize AX — every call traced 5 TOOLS search_products find toys by query, age, price, category get_product_detail full info for one product purchase_product place an order check_order_status look up an existing order cancel_order cancel a placed order PRE-INSTALLED Arize skills already in .agents/skills/ — trace, signal, evaluator, experiment. It looks clean. There's a bug hiding in it. New in Arize AX 8 / 25 Interface text: Design System Present · New in Arize AX Mark up Comments Edit Tweaks 100% Present Share In this tab Fullscreen New tab
- 7:34
Um, and I think the, the key thing here is we're gonna be going down a few routes to explain some of the new features in Arize. Um, but really it's gonna be four commands and then a bit of clicking around the UI interface. You're gonna Git clone the repo. And don't worry, I'll, I'll flash back to this slide so you guys have enough time to walk through this. You're really just gonna Git clone the repo,
The agent: Wonder Toys shopping assistant
Wonder Toys combines an OpenAI agent, FastAPI and Next.js, and ChromaDB with five shopping tools. Arize AX traces every call; the slide flags a hidden bug despite the pre-installed skills.
Text in image
The agent: Wonder Toys shopping assistant arize THE DEMO AGENT ARCHITECTURE AGENT OpenAI Agents SDK · gpt-5.4-mini ↓ API FastAPI backend · Next.js frontend ↓ SEARCH ChromaDB — vector, text fallback ↓ TRACE Arize AX — every call traced 5 TOOLS search_products find toys by query, age, price, category get_product_detail full info for one product purchase_product place an order check_order_status look up an existing order cancel_order cancel a placed order PRE-INSTALLED Arize skills already in .agents/skills/ — trace, signal, evaluator, experiment. It looks clean. There's a bug hiding in it. New in Arize AX 8 / 25
4 commands and you're running
Clone the repository, add OpenAI and Arize API keys, and run npm run dev to start the demo. The frontend opens at localhost:3000, and interactions send traces to Arize.
Text in image
arize SETUP 4 commands and you're running git clone https://github.com/seldo/aiewf-2026-demo cd aiewf-2026-demo/openai-agents-py-pre-run cp .env.example .env # add OPENAI_API_KEY, ARIZE_SPACE_ID, ARIZE_API_KEY npm run dev # ChromaDB · venv · deps · indexing · backend · frontend One command handles it all — ChromaDB startup, Python venv, deps, product indexing, backend + frontend. Frontend at http://localhost:3000 — sign in with any email, land on a chat UI. OPENAI_API_KEY · platform.openai.com ARIZE keys · free at arize.com ~2 min to running Interface text: New in Arize AX 09 / 25 9 / 25
- 7:57
navigate to it, add your API keys, npm run dev. And your tracing should be set up after you've gotten an API key from Arize, uh, and an OpenAI, OpenAI API key to run the OpenAI Agents SDK. Um, and that's pretty much it. And then once you start clicking around, you'll see that the traces automatically start flowing, uh, into Arize.
Let's break it
A planted price-gate bug lets a birthday-gift request under $30 return more expensive items. The agent presents them confidently without flagging the mistake, raising the question of how to test an unseen edge case.
Text in image
arize FIND THE FAILURE Let's break it "What toys do you have for a 5-year-old?" — relevant results "Show me educational toys for 8-10 year olds" — works fine "Do you have any STEM toys?" — works fine "I'm looking for a birthday gift under $30" — returns items over $30 THE CATCH The agent didn't flag it — it returned a confident list of completely wrong products. How would you even write an eval for an edge case you haven't seen yet? Interface text: New in Arize AX 10 / 25
- 8:22
Cool. So, um, we're gonna get to kind of like how we can actually start breaking this thing. But, um, the specific bug I've put in this repo is around this kind of idea of price gates. Um, so most of the, most of the kind of queries will work, and you guys are welcome to play around with it on your own. Uh, but if you ask for a birthday gift under $30, you're gonna be get- getting items over $33. Very common edge case, one where, you know, a
Let's break it
A planted price-gate bug lets a birthday-gift request under $30 return more expensive items. The agent presents them confidently without flagging the mistake, raising the question of how to test an unseen edge case.
Text in image
arize FIND THE FAILURE Let's break it "What toys do you have for a 5-year-old?" — relevant results "Show me educational toys for 8-10 year olds" — works fine "Do you have any STEM toys?" — works fine "I'm looking for a birthday gift under $30" — returns items over $30 THE CATCH The agent didn't flag it — it returned a confident list of completely wrong products. How would you even write an eval for an edge case you haven't seen yet? Interface text: New in Arize AX 10 / 25
- 8:52
specific gate wasn't triggered. Um, and without the actual kind of context of the coding repo and the context of the traces, it's very hard to find edge cases like this, um, in, you know, sort of an opaque system that's just emitting telemetry data. Um, the agent won't flag it. It'll return a confident list of completely wrong products. How do you actually re- write an eval for an edge case you haven't seen yet? And so the first of the new features that we have in Arize to kind of talk through this
Signal: let's set it watching
Signal monitors production traces in the background, clusters recurring failures by root cause, and writes investigations. While it processes the failing query, the team tries the trace skills already in the repo.
Text in image
arize DETECT · SIGNAL Signal: let's set it watching One toggle in project settings — Signal is enabled. Runs continuously on your production traces, in the background. Clusters failures by root cause, deduplicates across runs. Writes an investigation per issue: severity, entity type, evidence, proposed fix. Embedded screenshot: Signal Issues Issues Found Issues Over Time Detected Issues Financial_Data_Agent passes all tool responses untruncated causing high LLM synthesis latency Overview Evidence Proposed Fix [illegible] MEANWHILE Our failing query is now in the trace log. Signal will take a few minutes — let's see if we can beat it using the trace skills already in the repo. Interface text: New in Arize AX 11 / 25
- 9:22
is called Signal. Uh, Signal is an out-of-the-box agent that lives on the Arize platform in a sandbox, and it's really just one toggle to turn on in project settings. And what Signal does is it continuously runs on your production traces, um, in the background. And so what it'll do is it'll read every trace that's coming in. It'll start to cluster failures by root cause, dedupe those failures across runs if the same failures are occurring. And because it is an
Signal: let's set it watching
Signal monitors production traces in the background, clusters recurring failures by root cause, and writes investigations. While it processes the failing query, the team tries the trace skills already in the repo.
Text in image
arize DETECT · SIGNAL Signal: let's set it watching One toggle in project settings — Signal is enabled. Runs continuously on your production traces, in the background. Clusters failures by root cause, deduplicates across runs. Writes an investigation per issue: severity, entity type, evidence, proposed fix. Embedded screenshot: Signal Issues Issues Found Issues Over Time Detected Issues Financial_Data_Agent passes all tool responses untruncated causing high LLM synthesis latency Overview Evidence Proposed Fix [illegible] MEANWHILE Our failing query is now in the trace log. Signal will take a few minutes — let's see if we can beat it using the trace skills already in the repo. Interface text: New in Arize AX 11 / 25
- 9:51
agent with an entire harness, it can actually call to external tools. So what you can do is you can connect Signal to your GitHub repo. You can connect Signal to downstream data sources like Datadog. And what Signal will do is, sitting on top of your trace data, which is a source of truth for what your agents are actually doing in production, it will then self-reflect and go look at the code for the agent that you're running, go look at downstream data sources like Datadog, and can actually write an
- 10:21
investigation, write a severity and se- and entity type, and, and collect some evidence for that, and actually write a PR for you and open that PR. Um, so you're, you know, kind of flashing back to the beginning of this presentation that I was talking about where we have this evolution of software engineering. We're going really to that phase three, where the agent is consuming its own telemetry data o- off of the observability platform. It's then looking at its own code and then trying to propose a fix based on some evidence that it
The evolution of software engineering
The slide moves from developers reading telemetry and writing code, through prompting an agent, to coding agents reading telemetry directly. This shift calls for programmatic observability interfaces.
Text in image
The evolution of software engineering Who consumes the telemetry? Phase 1 · Software 1.0 Application → telemetry → Observability Platform → reads → Human Dev → writes code → IDE IDE → deploy → Application Human is the sole consumer of telemetry Phase 2 · Software 2.0 Application → telemetry → Observability Platform → reads → Human Dev → prompts → Agentic IDE Coding Agent · Claude Code Agentic IDE → deploy → Application Human prompts, agent writes code inside the IDE Phase 3 · Autonomous Application → telemetry → Observability Platform → reads → Coding Agent Claude Code / Cursor Coding Agent → notifies → Human Coding Agent → deploy → Application [illegible] Agent autonomously consumes telemetry Observability has to evolve from human dashboards to programmatic interfaces that agents can consume. Interface text: arize WHERE THIS IS GOING New in Arize AX 03 / 25 3 / 25
- 10:51
has.
Signal: let's set it watching
Signal monitors production traces in the background, clusters recurring failures by root cause, and writes investigations. While it processes the failing query, the team tries the trace skills already in the repo.
Text in image
arize DETECT · SIGNAL Signal: let's set it watching One toggle in project settings — Signal is enabled. Runs continuously on your production traces, in the background. Clusters failures by root cause, deduplicates across runs. Writes an investigation per issue: severity, entity type, evidence, proposed fix. Embedded screenshot: Signal Issues Issues Found Issues Over Time Detected Issues Financial_Data_Agent passes all tool responses untruncated causing high LLM synthesis latency Overview Evidence Proposed Fix [illegible] MEANWHILE Our failing query is now in the trace log. Signal will take a few minutes — let's see if we can beat it using the trace skills already in the repo. Interface text: New in Arize AX 11 / 25
- 10:57
Cool. So that's, that's the first part, Signal. Um, the second part is really around this idea of skills, and we'll be having a workshop later today that goes really in depth to all the skills that we've released. Uh, but maybe you don't want to have that ma- agent managed on Arize. Maybe you just want to be able to pull data from some store that you have. Uh, what we've also released is this concept of Arize skills. Um, so Arize skills allow your coding agent that you're working in, whether you're using Cursor, Claude Code, uh, Codex, or
- 11:27
forty-plus other coding agents, OpenCode, Hermes, whatever it is, um, and this can actually start to query the traces from Arize. So, um, a good example of that I think is...
- 11:42
Uh, give me one sec.
Checking traces with a coding agent
Claude Code checks 10 traces and 68 spans, then investigates 16 raw “error” strings before reporting no errors in the project.
Text in image
The structure is nested objects, not flat-dotted keys — my earlier trace/kind counts were wrong (status_code is top-level so that part was valid). Let me redo it correctly. Ran 1 shell command 10 traces, 68 spans, all OK. But grep found 16 "error" strings in the raw data — let me see what those are (could be tool output, schema fields, or actual problems in content even if the span status is OK). Ran 2 shell commands All clear — no errors in the wonder-toys-openai-agents-py project. Here's what I found: Summary Check | Result Project | wonder-toys-openai-agents-py (in "AIE Worlds Fair Presentation" space) Spans pulled | 68 (the full project — well under the 500 limit, not truncated) hey; check my traces again for any errors or anything i can fix? cross reference with the code for the agent in this repo and use the arize skills Application logs: npm run dev [illegible] IncludeList([Metadata, Document, Distance]), has_where: false at rust/frontend/src/server.rs:3037 in HTTP request with http.method: POST, http.uri: /api/v2/tenants/default_tenant/databases/default_database/collections/[illegible]/query, http.route: "/api/v2/tenants/{tenant}/databases/{database}/collections/{collection_id}/query", http.version: HTTP/1.1, http.host: localhost:8000, http.user_agent: Chroma Python Client v1.5.9 (https://github.com/chroma-core/chroma), otel.name: "POST /api/v2/tenants/{tenant}/databases/{database}/collections/{collection_id}/query" 2026-06-29T18:13:17.642658Z INFO Cache miss for collection [illegible] at rust/frontend/src/get_collection_with_segments_provider.rs:182 in HTTP request with http.method: POST, http.uri: /api/v2/tenants/default_tenant/databases/default_database/collections/[illegible]/query, http.route: "/api/v2/tenants/{tenant}/databases/{database}/collections/{collection_id}/query", http.version: HTTP/1.1, http.host: localhost:8000, http.user_agent: Chroma Python Client v1.5.9 (https://github.com/chroma-core/chroma), otel.name: "POST /api/v2/tenants/{tenant}/databases/{database}/collections/{collection_id}/query" 2026-06-29T18:13:17.642833Z INFO Cache miss for collection [illegible] at rust/frontend/src/get_collection_with_segments_provider.rs:182 in HTTP request with http.method: POST, http.uri: /api/v2/tenants/default_tenant/databases/default_database/collections/[illegible]/query, http.route: "/api/v2/tenants/{tenant}/databases/{database}/collections/{collection_id}/query", http.version: HTTP/1.1, http.host: localhost:8000, http.user_agent: Chroma Python Client v1.5.9 (https://github.com/chroma-core/chroma), otel.name: "POST /api/v2/tenants/{tenant}/databases/{database}/collections/{collection_id}/query" POST /api/chat 200 in 12.8s (compile: 3ms, render: 12.8s) Interface text: Using Arize skills npm Jump to bottom (ctrl+End) Auto-update failed · Run /doctor ~/projects/aiewf-2026-demo | main | Opus 4.8 | tokens:792 | ctx:6% auto mode on (shift+tab to cycle) /remote-control File explorer Rich Input11:48Demo milestone
- 11:50
I can actually ask my coding agents, this is just Claude Code, but I can ask my coding agents to actually go read my traces once I'm in the repo. And so it'll be able to see what the agent's doing, uh, from the trace data as well as the actual code for that agent at the same time and make some really impactful fixes.
- 12:14
Cool. Uh, so variety of skills. We have skills for creating data sets off of failures. We have cr- skills for, uh, going in and reading traces and doing some of that trace error analysis, um, as well as skills for, you know, actually running evaluators and running tasks on the platform as well. So the first part of this kind of skills workflow is that Claude can actually write the fix for you because it's sitting in your coding repo. Similar to Signal, Signal is running on top. It has access to your GitHub repo. It can go push up a
Claude writes the fix
Claude adds the price filter back to the fallback search. The slide shows a rerun of the under-$30 gift query returning only items under $30.
Text in image
arize FIX · CLAUDE CODE Claude writes the fix Fix the search_products fallback to reapply the price filter tools.py · search_products() — fallback branch - results = text_search(query) # price filter lost + results = text_search(query) + results = _filter_by_price(results, # reapply price + min_price, max_price) RE-RUN “I'm looking for a birthday gift under $30” → only items under $30. One conversation, no manual code spelunking. Now — what did Signal find? New in Arize AX 13 / 25 Interface text: Design System Present - New in Arize AX Mark up Comments Edit + Tweaks 100% Present Share
Claude writes the fix
Claude adds the price filter back to the fallback search. The slide shows a rerun of the under-$30 gift query returning only items under $30.
Text in image
arize FIX · CLAUDE CODE Claude writes the fix Fix the search_products fallback to reapply the price filter tools.py · search_products() — fallback branch - results = text_search(query) # price filter lost + results = text_search(query) + results = _filter_by_price(results, # reapply price + min_price, max_price) RE-RUN “I'm looking for a birthday gift under $30” → only items under $30. One conversation, no manual code spelunking. Now — what did Signal find? New in Arize AX 13 / 25 Interface text: Design System Present - New in Arize AX Mark up Comments Edit + Tweaks 100% Present Share
- 12:44
PR for you.
Signal found it too
Signal and Claude Code independently identified the same missing price constraint. The slide contrasts Signal’s continuous monitoring with Claude Code and Arize Trace’s on-demand investigation.
Text in image
arize VERIFY · SIGNAL Signal found it too SIGNAL — ALWAYS-ON, CONTINUOUS HIGH severity “When vector similarity returns no results, the fallback text search omits min_price/max_price — returning products above the requested price.” entity: retrieval · evidence: failing-query trace IDs = CLAUDE CODE + ARIZE-TRACE — ON-DEMAND root cause tools.py, search_products fallback path strips min_price, max_price — runs the text search with no price constraint. ✓ Same root cause — found independently. Signal would have caught it even if no one was watching. New in Arize AX 14 / 25 Interface text: Design System Present - New in Arize AX Mark up Comments Edit + Tweaks 100% Present Share
- 12:47
Um, and so, so these two paths are sort of equivalent. Um, you have Claude Code with the Arize trace skills. That's kind of like an on-demand method. Let's say you're an engineer working on your product continuously, and you're actually sitting there and trying to improve it, versus Signal, which is the always-on sort of continuous method. It's continuously serving up, uh, you know, new insights, new analysis for you based on your trace data.
What Signal writes for every issue
Signal groups issues by root cause and ranks them by severity and frequency. The high-severity example affects 14 traces: fallback search omits price filters, so the proposed fix reapplies them.
Text in image
What Signal writes — for every issue arize DETECT · SIGNAL RANKED ISSUES · DEDUPED HIGH search_products fallback drops price filter MED purchase_product runs without confirming the product LOW check_order_status latency on cold cache ranked by severity × frequency INVESTIGATION — ISSUE #1 SEVERITY HIGH FREQUENCY 14 traces affected SUMMARY Fallback text search omits min_price / max_price — returns products above the requested price. PROPOSED FIX Reapply the price filter in the fallback branch. WHY IT SCALES Signal clusters by root cause and dedupes across runs — you triage a ranked list of issues, not a firehose of individual traces. New in Arize AX 15 / 25 Interface text: Design System Present - New in Arize AX 27 pages Mark up Comments Edit Tweaks 100% Present Share 15 / 25
- 13:12
Um, and Signal, like I mentioned before, can actually kind of root cause that because it has access to your coding repo. It can write an investigation report, um, and it can actually try to, uh, put up a PR based on that. And it's not just, you know, individual traces it's looking at. It's looking at the entire context of what your agent is actually doing in production.
- 13:34
So, um, real quick, 'cause I wanna get to the meat of this presentation. I'm going to pause here and keep this slide up. Like I mentioned before, we have Laurie in the room and Aparna in the room, um, to help out and run around and kind of help you guys set up. Aparna's right here. I think Laurie is somewhere else in the room, but he'll, he'll be running around. And we got Dat as well from our team. Uh, so real quick, these are literally the, the only four commands you have to run. I would recommend
- 14:04
signing up for a free account at Arize dot com to just get your Arize API key to start seeing some of the traces flowing in, and you will need an OpenAI API key as well for this presentation. So I'll pause here, leave this slide on, and then we can start walking through getting the traces for the Wonder Toy signal, and then what I'll do is we'll walk through how that loop actually starts to get into production. Um, so yeah.
- 14:30
Who needs a key?
- 14:31
And if anyone needs a key, uh, Dat is more than willing to give, give you guys a key.
- 14:53
All right. How's progress going so far, guys?
- 14:58
And if you don't have your laptop, don't worry. I'll be walking through sort of what this flow looks like, uh, if you were to have done it yourself as well.
- 15:07
I'll give you guys maybe two to three more minutes to set up your API keys and to run these commands and clone the repo. I know that the Wi-Fi's been a little wonky today, so won't fault you guys for that.
- 15:20
Just get a free account, you don't get access to any eval.
- 15:25
Uh, no, you do.
- 15:26
It said you needed enterprise.
- 15:28
You still get access to evals. You don't get-- There's a few features like code evals and things like that, yeah.
- 15:39
And for this flow, we'll mostly be showing how Claude Code can do that trace analysis for you through the skills and through Signal. Um, so it won't be, it won't be necessary for this one.
- 15:53
Anyone having any trouble? Anyone need some help? Feel free to raise your hand, and we'll flit around the room.
- 16:05
Sorry?
- 16:10
Here, I'll jump down.
- 16:14
Sorry.
- 16:15
I think it's a Wi-Fi issue. The Git clone is stalling. Yeah.
- 16:19
Ah, okay.
- 16:20
It's just stopping there.
- 16:20
Cool. That's probably a Wi-Fi issue.
Repository structure
The terminal lists the demo repository’s agent, two OpenAI Agents directories, product images, README, and skills lockfile while attendees wait for the repo to load.
Text in image
v24.3.0 ~/projects/aiewf-2026-demo/openai-agents-py-pre-run git:(demo/age-filter-bug) 69 · +9196 -244 (0.042s) cd .. ~/projects/aiewf-2026-demo git:(demo/age-filter-bug) 69 · +9196 -244 (0.053s) ls agent openai-agents-py openai-agents-py-pre-run product-images README.md skills-lock.json ~/projects/aiewf-2026-demo git:(demo/age-filter-bug) 69 · +9196 -244 lt Interface text: admin aie-slides- npm lt Use agent Dismiss Don't show again
16:42Demo milestone
- 16:43
So if you guys are having Wi-Fi issues, let me know when you can start to see, um, the repo pull up. It might take a few seconds, uh, but this is generally the structure of the repo. Um, we have kind of the agent, all the scripts that it needs, um, and a bunch of other things there. So I'll give you guys a few more seconds 'cause I know the Wi-Fi is a little slow. Uh, once we s- get folks starting to get into the repo, uh, please make sure to sign up for an API key as well, uh, and get your OpenAI API key as
- 17:13
well so we can get started.
- 17:34
Should I just keep going? No, just, like, do it alongside them.
- 17:51
Yeah. And just a reminder, all you have to do is run those few commands. So you Git clone, um, the repo. Then you'll CD into that repo, so I'm already in there. Um, then you'll NPM install, and then make sure that you have your env file loaded. This is where you're gonna be putting your three keys.
- 18:13
You
- 18:16
gotta type no dot at the end.
- 18:16
No dot at the end here.
- 18:22
Uh, at the beginning and the end.
- 18:22
Wait. Sorry. Which one?
- 18:29
Standard
- 18:37
Oh, the dot git at the end.
- 18:50
For example. Got it.
- 19:11
Can't change it right now, but, uh, we'll, we'll get it going.
- 19:24
Cool. All right, I'll start-- I'll, I'll leave this up for, for maybe ten more seconds. Um, very simple, make sure you put your env values in. Uh, there may be a bug there as well, um, but just make sure you look for the .env file, input your own keys. Um, just to show you an example,
- 19:43
uh, I will take this off so you don't see my keys, but there are, there are some keys in there as well. Um, and then you npm install and npm run dev. So what you'll then see is that your Wonder Toys interactions, so your interactions with this chatbot, are being traced to Arize. Uh, and in your Arize traces, you'll actually be able to see, um, the things that are coming in through this chatbot, right? So I can see that I started an agent workflow, the Wonder Toys agent then, um,
Toy-shopping chatbot
The chatbot first fails to find Color Match – Classic by name, then displays the game after a purchase request and asks for shipping details. The presenter says these interactions are traced to Arize.
Text in image
Tell me about Color Match – Classic I couldn't find a product with that exact name. If you want, I can search for it by keywords like: • color match • matching game • classic Or I can help you browse similar toys. please purchase this product Sure — I found Color Match – Classic. Color Match – Classic — $7.99 ⭐ 4.8 (15,678 ratings) · Ages 5-14 years · by WonderPlay Games. The world's #1 matching card game. Match colors and numbers, play action cards, and shout when you're down to one card! 2-10 players. 108 cards. Add to Cart To place the order, please send the shipping details: • Full name • Street address • City • State/Province • ZIP/Postal code • Country Once I have that, I can complete the purchase. Ask about toys, search products, or check your orders... Send Interface text: localhost:3000 Wonder Toys - AI Shopping Project Overview - wonder-
19:52Demo milestone
- 20:13
responded, it searched products, uh, it took another turn and actually found the products and returned them to me, uh, and I can kind of see the output of what that agent is doing. So this is like kind of like baseline value prop. Can I trace? Can I understand what's going on? Uh, and then the second part of this that I wanted to show is sort of like some things that Signal can do on top of this. So you have all this trace data, uh, coming in through Arize, and what you actually want to start doing is you wanna enable Signal to actually run on top of your agents.
- 20:45
And I'll show you an example of Signal, uh, in a repo. So Signal runs continuously in the background. It takes a few minutes to actually get started once your traces start flowing in. Uh, so I'll show you-- We're getting hit with a little slow Wi-Fi, but I'll show you an example of Signal already set up and ready to go and sort of how it's, it's sort of doing that loop of self-improvement.
- 21:30
So kind of like I mentioned earlier, traces are flowing in through Arize. Every interaction you do with the app comes in. Um, and then this is sort of what Signal looks like. Um, so Signal has kind of this clustering analysis of different issues that it's seeing across your application. Uh, you'll see sort of this table of, uh, ca- a categorization of what those issues are. Uh, you can see those issues trending over time. Um, and then you can actually go read examples of what that are, right? So
- 22:00
in this example, uh, we can see some serialization issues with Python, um, and that, that is corrupting the agent, um, input. And you'll see that it actually opened a PR for me already. Um, and so I can just one click, go into that PR and start viewing it. Um, there's also this idea of like writing at least the ticket out of that investigation. So you'll see this overview of what, um, this stock agent is doing, fabricating results, for example, uh,
- 22:30
a hypothesis of why that's happening, evidence for that within an individual trace, um, and then a proposed fix for you to go and, and then do yourself. Um, so kind of going back to this concept of like agents consuming the telemetry data. This agent is sitting in the back consuming the telemetry data, actually writing out the evidence and proposing a fix for you. Uh, in this example, we've still, you know, left the code review to a human. Your mind's the limit on how far you wanna take these things. We do recommend you do that as well. Um,
- 23:01
but kind of this idea of like the self-healing software you can see, uh, in action.
- 23:07
How are we feeling on, um, setup? Is-- Was the Wi-Fi able to get up and running for people? Did they-- Were you able to-- No? Okay, cool. I will continue kind of through the presentation, and maybe we can take some time at the end as well to kind of flit around the room, help you out.
Every phase, completed
Signal flags a fallback failure, trace skills pinpoint the code, and Claude Code fixes it. The slide reports independent root-cause verification; the presenter says he did not plan to cover verification today.
Text in image
arize THE LOOP, CLOSED Every phase, completed All four phases in under an hour — and Signal is still running. DETECT Surfaced the fallback failure before a user reported it Signal INVESTIGATE Pulled real traces, pinpointed tools.py line by line Arize trace skills FIX Wrote the fix; re-run confirmed correct results Claude Code VERIFY Signal confirmed the same root cause independently Harness-as-a-Judge · Signal Signal is still watching — continuously, on every new trace Interface text: Design System Present - New In Arize AX Mark up Comments Edit Tweaks 100% Present Share Describe a tweak... New in Arize AX 17 / 25
- 23:24
So kind of the four steps that I wanted to walk through with you guys were, one, detect. So surface the fa- fallback failure before a user had reported it. Two, based on that failure and evals running in your platform, actually investigate what's happening with real traces. Um, and so in our example, there was a bug with the price check. And so what Signal and your skills via Claude Code was able to do was look at the traces, look at
Every phase, completed
Signal flags a fallback failure, trace skills pinpoint the code, and Claude Code fixes it. The slide reports independent root-cause verification; the presenter says he did not plan to cover verification today.
Text in image
arize THE LOOP, CLOSED Every phase, completed All four phases in under an hour — and Signal is still running. DETECT Surfaced the fallback failure before a user reported it Signal INVESTIGATE Pulled real traces, pinpointed tools.py line by line Arize trace skills FIX Wrote the fix; re-run confirmed correct results Claude Code VERIFY Signal confirmed the same root cause independently Harness-as-a-Judge · Signal Signal is still watching — continuously, on every new trace Interface text: Design System Present - New In Arize AX Mark up Comments Edit Tweaks 100% Present Share Describe a tweak... New in Arize AX 17 / 25
- 23:54
what a user is asking within the trace around, you know, the price being under thirty dollars, actually marry that to an investigation of the code path that checks the price filter, and then propose a fix for you. So, um, in my code, there was a bug with how I was actually doing that price filter, and using Signal, using the trace data that's in Arize, and using, uh, the skills via Claude Code, I was able to see that interaction happen with a real user and then fix that, you know, kind of wrong filter in my
- 24:24
code for the agent. Um, and then finally, we have this verify path. Uh, wasn't planning to get to it today because it is a little bit of setup, uh, but happy to kinda talk through that, and then we'll make sure to, uh, kind of return when the Wi-Fi is good as well. So again, just a, just a quick recap on the basics here. Uh, we have this idea of, you know, the core loop running on a real agent which detects the issue around the age filtering bug, investigates, uh, using the Arize trace
- 24:54
skill or through Signal, uh, fixes that issue, Claude actually patching the fallback, uh, and then verify it, uh, which I'll kind of walk through how that works in Arize as well, uh, with Harness as a judge and agent experiments. Um, and then Signal, it'll kind of do this in parallel as well for when you don't wanna be hands-on with the keyboard and want to run this in the background. Uh, Signal will be tuned to your own taxonomy, your own traces. Um, it-- you can also configure it to be set off
- 25:24
alerts. Um, so for example, if you have an alert on a specific eval that needs to stay below, um, you know, one percent trigger rate, you can configure Signal to be triggered off of an alert, um, or, you know, a latency spike or a cost spike or anything like that. Um, you can then have custom Harness judges running in CI on those PRs. Um, you can experiment with those, and then you can kind of optimize things from there. Um, and quick, quick shout-out as well, at two PM, Lori will be
- 25:54
doing a workshop, hopefully the Wi-Fi's a little bit better as well, where we dive deep, deep into some of these workflows as well. Um, so kinda centering us, I talked about this loop. I think we all kind of get the picture of I have an app, it's traced, those traces flow in, and then in addition to the trace data, uh, whether it's Signal, whether it's your own coding agents, having trace data, which is the source of truth for what your agent is doing, as well as your actual repo data, which is the code that defines what your
- 26:24
agent should be doing, um, you get some really, really interesting analysis, um, and the ability to actually make improvements at scale. Um, I think one thing that we didn't talk about yet that I wanna get into is sort of how do we evaluate agents using agents? So this is kind of like an example of pull the traces, you know, look at my code, figure out what's wrong. When you wanna have sort of this more adaptive analysis, we tend to think of it as Harness as a judge. So I think, raise your hands if you guys are familiar with LLM as
- 26:54
a judge, loosely speaking. Cool. Awesome. I think pretty much everyone in the room should be familiar with that at this point. Um, you know, very common pattern. You feed in an input, your LLM as a judge reads the input, uh, whether it's a summarization task or a hallucination eval, whatever it is. It then responds with a set of rails, um, on how well, uh, the sort of the agent did on the response. What we're really moving to is Harness as a judge, and this is for a few different reasons. I think firstly,
- 27:25
LLM as a judge is really a fixed rubric, so when you have a use case in mind and when you have an eval that you know you wanna score on, it works really well at catching those known issues. However, like I just kind of explained with the price filter, you don't always know what those issues are gonna be like before you hit production. And so Harness as a judge is a way to do that adaptive analysis. Um, so observe what the agent is actually doing and adjust what you actually test as the
- 27:55
agent goes. So the criteria kind of emerges from the trajectory. Um, was this the right tool to pick? Were the arguments valid? Was this grounded in the right context that was handed off by another agent, for example? This idea of that unknown unknown. So that price filter was an example where I wouldn't have written maybe a test case for that, um, but an-- Harness as a judge is able to kind of do that adaptive analysis for me with the trace analysis that it has in it. Um, so
- 28:25
this is sort of like verifying in plain language. You're not column mapping, you're not writing boilerplate, you know, sort of eval templates and, and text. The rubric is really just a sentence, and it reads the actual trace tree directly, um, then runs in a sandbox against those real traces. Um, and so I think it gives you a bunch of, you know, tools on top of that because it also runs in an environment. So one example that I see a-- from a lot of customers is around this idea of access checks.
- 28:55
Um, an LLM as a judge might be able to read an input and read an output, but because it has no data outside of that, unless you paste in every user in the system into the system prompt, it doesn't actually know or have context on what that user should be doing or should have access to. Whereas a Harness as a judge, because you can connect it to external systems, it can then go make a call to actually say, "Hey, is this user allowed to do this? Does the trace reflect the access that they had?" Um, and so
- 29:25
Harness as a judge is essentially moving from just a foundational model call that scores an input and then comes out with an output rail to a more agentic flow where you can actually call tools, um, and, and sort of do that unknown unknown catching as well.
- 29:47
Cool. Um, so I think we, we kinda talked about, uh, Harness as a judge. Um, how that would've worked in this scenario is scoring the fix to fail and, and actually deciding whether to pass the example or not. So for example, pass if every return product's price is within the requested min price, max price range. Fail if any result exceeds the budget. Um, and it's actually able to go into the code and check that those filters are working correctly.
- 30:17
Um, and this rubric now will catch this on every subsequent run and not just, uh, you know, an individual test case that you've coded for before. Um, so quickly checking in, we talked about the detect, investigate, and ship to fix. Um, shipping the fix part is kind of what I wanna talk about now with agent experiments. So maybe we've found an issue, we've grounded that in real trace data, we've investigated that, we know why it's
- 30:46
wrong, and we've decided that we have a fix worth shipping. Before you ship the fix, you probably wanna test this on a set of results. And I'm not sure how you guys are doing this today. Maybe you're deploying to a dev environment and then manually testing. Um, what Arize has come out with is agent experiments that allow you to test that in an automated way via CI as well. And so, uh, I don't think any of us are strangers to the concept of, you know, datasets of failure cases that we collect over time. Um, what you can
- 31:16
do with agent experiments is you can actually stand up that dev agent behind an endpoint and hit that with a dataset directly from Arize, and that dataset will be actual traces that your agent has failed on. So in the previous example, I kinda showed you, uh, the Wonder Toys, where maybe I have a few traces asking for, um, birthday gifts under thirty dollars that actually failed. Uh, what I can use Arize Skills to do is actually go create that dataset. So I'll, I'll have like,
- 31:46
call it twenty-five examples of budget constraint searches that failed, right? And then I ran the workflow with Claude Code, uh, I did some trace analysis, I identified the bug in my code, and I have a fix ready as a PR, or maybe Signal did that all for me in the background. How do I actually go hit my agent with that dataset to make sure that the proposed fix is actually validated? And that's sort of what agent experiments allow you to do. With agent experiments and the agent playground in the product, uh, you can
- 32:16
actually run against that fixed version. So you deploy to dev, it stands up behind an endpoint, uh, you stream sort of the, uh, dataset inputs into that agent, and the agent actually responds. And then you can score it with the same judge rubric that you have in production. Uh, so then you can compare kind of side by side what the agent in dev mode is doing with the fix versus what the agent in production did, and look at whether you're actually catching, uh, a- any of those edge cases or not.
- 32:46
So similar to kind of like other dataset experimentation workflows, what you wanna do is you wanna look before the fix and after the fix, look at your judge, which is in this case the price correctness judge, um, look at your examples and queries, look at whether, you know, maybe latency is increasing, cost is increasing, so you have sort of a view of those operational metrics as well, um, and then quantify the results of your experiment. Um, and then, you know, maybe alluding a little bit back to this idea of the self-correcting
- 33:16
loop. A- as a full storyline, you now have an agent that's operating autonomously on your traces, looking to see if issues are arising. So, you know, I can see that the user asked for under thirty dollars. The response was clearly over thirty dollars. Investigating why that's happening with reference to a code base, putting up a fix in the form of a PR for wha- how it thinks it can improve that. In this case, it was a filtering error. And then because our experiments
- 33:46
are actually exposed via skills and API as well, the agent can actually redeploy itself on the new PR and verify itself against a dataset of actual failure cases before it even notifies you or decides that it's worth your human review. So, you know, this full loop happens completely autonomously. The agent will actually go collect the failures, create a dataset for itself, run it against the new version of itself, and prepare a response back to you that
- 34:16
says, "Hey, I think I found this issue. I investigated it. I know why it might be happening. I pushed a PR. I then ran that PR in dev against the dataset of failures I found, and I saw, you know, um, in this case, a thirty-six percent increase in performance on this specific evaluator." And all that information to you-- is available to you in the PR itself for you to then just go read, validate the code, do some code review, and ship. Um, so I think this makes
- 34:46
it like, you know, kinda coming back to this idea of the loop. Um, I think it makes it really, really powerful to go, you know, full circle where you have the detect, investigate, hypothesize, and fix loop, and then you also have the validation loop, right? So how do I actually know that this fix is correct, and it's not just a, you know, an agent in a sandbox saying it's correct? How do I actually go test that on real failures and real, uh, edge cases from my working example? Um, and how do I
- 35:16
actually, you know, start to stand that up and auto-test that for me? Um, and so I think, you know, we'll, we'll come back. We'll, we'll have some time as well to kinda help you guys around, uh, in the rest of the workshop if the Wi-Fi is doing a little bit better. Uh, but I'll, I'll go back to the slide that was previous, um, so that you guys can kind of walk through this yourselves. One, getting the tracing started. If you're not collecting tracing, nothing else works. Um, takes sub-second-- sub-ten minutes on your, on your
- 35:46
own agent. We have instrumentation skills. We have support for all the big agent frameworks, all the foundational model providers. Um, enabling Signal and going deep, uh, or using Claude Code to actually use the Arize skills to do that investigation for you. Uh, verifying it with a harness as a judge, and then measuring that with agent experiments. So before we end, I will go back to this, and we'll, we'll all walk around the room and try to help you guys get this set up. Uh, if you're having trouble with the env files, um, if you just
- 36:16
search through the repo or ask a coding agent where the env file is, um, it will kind of help you do that. Uh, other than that, make sure you Git clone the repo, um, and you get your API keys from the Arize platform, uh, and then you can actually start to see these traces coming together. Um, in the spirit of AIE workshops, I think this really works better when you guys see it and touch it yourselves and actually try to do it. So hopefully the Wi-Fi works, and I'll, I'll come around and hop around. If you do have questions or you're blocked at any point, feel free
- 36:46
to ask me. Odds are someone else in this room is experiencing the same issue, uh, so I'm happy to kinda shout out some advice as we go. All righty. And if you're having trouble installing the skills as well, uh, I have it up on the slides here Uh, very simple, npx skills add Arize-ai/arize-skills. This will give you access to an interactive mode where you can choose what you wanna install, or you can just choose to install all of them as well. Uh, and this includes skills for trace analysis, dataset curation, as well as
- 37:16
evaluators.
- 37:20
So as an example of, like, some of the skills you can run, uh, I can ask Claude Code, "Hey, check my traces for any errors or anything I can fix, cross-reference with the code for the agent in this repo, and use the a- agent skills." And, uh, we will be sending out kind of a recording of this as well as all the resources you need to kinda get started with this. So apologies for the internet. If it doesn't work out today, you guys will be able to get up and running very quick as well. All righty, guys. We gotta close up, uh, our workshop for now. Thank you once
- 37:49
again. I just wanna leave you guys with sort of this detect, investigate, fix, and verify slide, uh, which kinda covers the through line of how, uh, you can get started with Arize and how to create your self-improving agents. And then once again, uh, feel free to connect with me on Twitter or LinkedIn. Uh, and if you want a free year of Arize Pro as well, uh, just use the code on the screen. Uh, we have a free tier that's always free, free to use. For this conference only, we're also giving out Arize Pro for free, which gives you a bit more of the usage
- 38:19
limits, uh, if you guys are running agents with, you know, thousands and thousands of traces. So thank you guys so much. We're running another workshop at 2:00 PM, uh, led by Lori, that'll go into a few more code examples. Hopefully, the Wi-Fi is also a little bit better then. And yeah, we'll get going. Thank you, guys. Appreciate you.