AI Engineer World's Fair 2026
From Signal to PR: Anatomy of a Self-Improving Agent
Slide chapters
23 slide and demo groups · Timestamps are approximate. 7 additional groups were omitted from this pass.
Read the talk
From Signal to PR: Anatomy of a Self-Improving Agent
A self-improving agent needs more than a coding harness: it needs runtime evidence, targeted investigation skills, and a path from discovered failures to reviewable fixes.
From a talk by Jason Lopatecki
Before you start: Familiarity with coding agents, Git repositories, and production traces will help; evaluation results and sandbox execution are explained as they enter the workflow.
The platform is down. Where does the investigation start?
When a platform goes down late at night, how much work must happen before someone can even propose a fix? Arize’s experience building Alyx, its in-product assistant, exposed that problem from the inside. Jason Lopatecki describes the first version, roughly two years earlier, as poor. Signal grew out of the debugging and improvement tasks the team repeatedly performed while trying to make Alyx work well. The immediate problem is familiar: the platform is down, it is late, and restoring it still takes substantial human effort.
Traditional observability presents a person with dashboards, graphs, and logs to inspect. The next stage pairs that person with a coding agent and skills for systems such as Pyroscope or Google Cloud. Those skills make runtime evidence accessible during debugging. The repository describes the paths software could take; telemetry identifies the path it actually took. Without that evidence, an agent must reason across many possible executions.
Evaluations add another layer of evidence. Today, humans still make or review fixes; the longer-term goal is a continuous loop in which agents use traces, logs, and evaluations to improve the system that produced them. That changes observability’s consumer as well as its interface: evidence must be useful to an investigating agent, not only legible to a person clicking through a UI.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move evidence gathering ahead of human attention
Generating code quickly does not automatically make improvement equally fast. Once coding agents can produce plausible changes, the bottleneck shifts toward confidence in the proposed fix: is this the change to push? The expensive work includes establishing what failed, why the proposed change addresses it, and whether the evidence is strong enough to proceed.
Signal inverts the usual sequence. Instead of waiting for a person to inspect an error and assign a debugging task, an agent examines the error first. It can open an issue even when it cannot produce a fix. The person then encounters an investigation with supporting evidence already assembled, rather than an unexplored ticket.
This does not reduce every human interaction to approval. Lopatecki describes people still steering the second and third steps of an investigation. Preparing the evidence removes the cold start; a person may still need to redirect the diagnosis or carry the change across the finish line.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring production evidence into the repository
The loop begins with a trigger and a way to gather context. A run can start periodically—for example, every five minutes—or when an event such as an error occurs. Observability skills let the agent decide which evidence it needs and retrieve it. This replaces a fixed sequence of dashboard clicks with an investigation that can choose its next query.
The key inputs serve different purposes:
| Input | What it contributes |
|---|---|
| Traces and evaluations | Execution history and assessments of behavior |
| Logs and APM | Additional runtime context |
| Repository | The implementation behind possible execution paths |
Arize’s skills download production traces or logs into temporary files inside the repository. That puts the record of what happened beside the code that could explain it. The combination matters: useful data, a file format the harness can work with, and composable skills that retrieve more evidence as the investigation develops.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Automate the workflow that already works locally
The starting point is the debugging workflow already running on a developer’s laptop. Moving it into a sandbox gives scheduled or event-driven runs an execution environment without requiring a person to start each session. The sequence is deliberately incremental:
- Get the agent, repository, and evidence-gathering skills working locally.
- Reproduce that working environment in a sandbox.
- Let the observability platform launch it on a schedule or in response to errors.
The sandbox changes where and when the investigation runs; it should preserve the useful local debugging behavior.
Alyx supplies a concrete failure. Arize’s assistant uses to-do updates to manage its task list. In the example, it makes repeated update calls and encounters a stream-cancellation event. Signal runs in the background, gathering debugging information and raising issues around failures like this.
Lopatecki describes the proposed Alyx repair as a one- or two-line fix. The walkthrough does not expose the exact patch or establish the cancellation’s root cause. Larger changes are more likely to require a person to finish them, but an issue containing the relevant evidence still changes the starting point: the engineer can guide an existing investigation instead of assembling it from scratch. That is the practical shift from responder toward reviewer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More evidence for an agent to investigate
Traces and evaluations remain central after investigation becomes automated. Lopatecki predicts 10 times more tracing and 10 times more logging as agents become consumers of telemetry. This is a forecast about instrumentation, not a measured improvement or a prescribed collection target. His reasoning is that additional logs once became noise because people could not sift through them; agents could make much denser instrumentation useful, potentially covering nearly every part of some software paths.
The execution model is less mysterious than an opaque SRE-for-AI product might suggest. Take the local debugging experience in Claude Code or Cursor and run it periodically. Choose the harness, sandbox, and skills, with prebuilt options where useful. Signal’s role is to put that combination into a recurring system, rather than require a wholly different way to debug.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a finding into an issue, evaluator, or dataset example
The product walkthrough uses a financial agent that answers questions about stock trading. Lopatecki explicitly discourages treating such agents as a reliable way to make money; the workload is useful here because it has many ways to fail. The interface is Arize AX, its SaaS platform, which can also be deployed in a VPC. He describes Phoenix as the open-source option and says Signal is available in AX at the time of the talk. The current Phoenix repository specifies Elastic License 2.0; subsequent AX packaging also distinguishes basic Signal access from the managed-agent and repository features needed for the complete workflow.
Signal periodically produces findings that can lead to several different artifacts:
- GitHub issue: Connect a repository and create an actionable record of the problem.
- Evaluator: Turn a discovered failure into a check intended to catch it again.
- Dataset example: Preserve the relevant case for subsequent evaluation.
Each finding has supporting evidence, including traces. Skills for Google Cloud and other logging systems let the investigation combine data from multiple places, and users can supply their own skills. The displayed examples include runs without guardrails, safety problems, and intent problems. The output need not be a code patch to improve the next debugging or evaluation cycle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the execution boundary and preserve the handoff
The configuration screen exposes the execution choices. The demonstrated harness is Claude Code, and Lopatecki names Claude managed agents and Arize sandboxes as options. Sandbox selection matters when an investigation must connect to production systems or a database: some customers do not want those connections exposed to Anthropic.
Lopatecki describes Arize installations inside customer VPCs, naming Uber and another customer he calls Bookings. The architectural point is the connection boundary: customers may be comfortable granting access within their own VPC while declining an external production connection. He also names Daytona among the sandbox choices, noting that existing relationships and comfort with a provider influence the decision.
The platform then tracks the agents that run: periodic Signal investigations as well as other named agents. A user can inspect a session, download its transcript, and resume a Claude session locally. The investigation therefore has a handoff path when automated work reaches a point that needs direct human guidance.
Prompts control the investigation’s emphasis—for example, asking it to be less aggressive or to look for security issues. Presets combine those instructions with prebuilt or custom skills and linked repositories. Together with evaluations, these controls connect the local debugging workflow to a recurring process that produces a fix or a prepared starting point. Observability extends beyond reporting a signal into supporting the work that follows it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why not point Claude Code directly at the traces?
An audience question makes the architecture’s implication explicit: why not connect Claude Code to the data, let it read the traces, and have it push a PR? Lopatecki agrees that users should do this. The work that makes it effective is designing the skills around the data, not withholding access behind a special agent.
A useful skill first finds the right evidence—for example, a group of traces belonging to one session—and then places that evidence in a repository file. Coding harnesses work well with files, so the skill should produce a usable record of what happened rather than an undifferentiated data dump. Lopatecki reports that some repository evidence files are about 10 MB. That is an operational example, not a demonstrated context-size limit. Multiple skills must then compose so the agent can move from an initial symptom to the evidence needed for diagnosis.
His Pyroscope example shows why the access surface matters. A skill can investigate memory issues, use profiling facets, and group evidence by customer to determine whether one customer is driving the problem. These are meaningful investigative operations. Simply giving Claude access to all the data does not teach it which cohort to isolate or which profile to inspect; the skill must expose those operations in a form the agent can use effectively.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Feed evaluations into the next investigation
The final question asks where evaluations enter a workflow that begins with a production failure. Online evaluations run on production traces, and their results become data attached to those traces. Investigation skills can retrieve those results, visualize them, and obtain aggregate evaluation values across a group of traces.
Lopatecki describes first-generation LLM-as-a-judge evaluations as an AI assessment layer that runs periodically and adds preprocessed information. Signal uses those annotations alongside the raw trace data. The evaluation provides an assessment of behavior; the trace still supplies the execution evidence needed to investigate it.
Evaluators often start with failures already encountered: a prompt-injection attempt to catch, or a problematic response pattern. When an investigation discovers another recurring failure, the team can create an evaluator to detect it next time. This closes a concrete feedback loop: production behavior supplies evidence, investigation identifies a failure pattern, and a new evaluator makes that pattern visible in future runs.
The two layers can operate at different scales. Lopatecki reports customers applying LLM-as-a-judge evaluations across their full datasets, while Signal performs more periodic investigations over substantial amounts of data. He supplies no dataset sizes or throughput measurements for that comparison. Broad, recurring assessment and deeper, periodic investigation therefore serve complementary roles: the former keeps checking behavior, while the latter assembles the context needed to understand and address failures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Configure production evaluation tasks with trace, span, or session scope, sampling, and filters.
Arize's locally runnable observability and evaluation platform, with tracing, datasets, and experiments. The repository specifies Elastic License 2.0.
Query CPU and memory profiles and narrow investigations using labels and grouping.
Documentation for the sandbox provider named among the talk's execution-environment options.
Further reading
- From Signal to PRArticle
Lopatecki and colleagues explain how Signal turns production evidence into investigations and reviewable changes.
Agent skills for working with Arize traces, datasets, and experiments through the ax CLI.
Updates since the talk
A practical walkthrough connecting Signal investigations with repositories, regression datasets, evaluations, and human review.
Read the complete timestamped transcript
16 slide groups · 7 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 7 additional groups were omitted from this pass.
- 0:00
[on-hold jingle] Well, thank, thank you all. Um, let me just get set up here.
- 0:16
So, not just, uh, the founder of Arize, but, but I tend to build an incredible amount of stuff. Um, [ts]
- 0:27
let's see if we can get this going here.
- 0:30
Oh. Sorry, one more second. Um, so not just a founder here, but, but also a builder. And I do my best to, um, uh, to, to, to build agents, assistants.
- 0:45
Um, we have an agent in product. We have a, an agent in product called Alyx, and, uh, and a lot of... I think a lot of my experience is, is come from actually, um, trying to make the stuff work and work well.
- 0:59
Our first version of our, our own agent frankly sucked. Uh, it was many years ago, uh, probably two years ago. We're one of the first in the space to do it.
From Signal to PR
Arize introduces Signal as a next-generation product shaped by its experience building an agent and aimed at automating everyday tasks.
Text in image
From Signal to PR Observability & Evaluation arize | We make agents work
- 1:07
Um, and a lot of what we have built, uh, has come out of that- our own experience in building and building this agent. And, and Signal is kind of our, our next generation of this, which is trying to automate a bunch of things which we do every day, uh, and build it into a product set that people
- 1:25
can use. Um, so I'm gonna try to-- I'm gonna go through this, this materials here. I'll try to go fast and try to show you a lot of product too.
- 1:32
I'm a product person. Um, so if you built a startup before, um, you, you've experienced this. Your platform's down. It's, it's late at night and you, and you wanna go fix it.
The platform is down
During a late-night outage, a human still has to find the trace with a coding agent and reconstruct what happened before changing code.
Text in image
02:14 · ERROR RATE SPIKING It's 2am, the platform is down and the fix is still hours away. A human still has to wake up, find the trace using a coding agent, and reconstruct what happened before a single line changes.
- 1:44
Um, and, and really the, the... It takes a lot of energy to go do that. And we're gonna talk about like the automation we built a little bit and, and what, what the future looks like.
- 1:54
And, and I truly believe, um, that the future of the, the observability space is, is actually changing massively right now. W-why is that? Well, observability used to be for humans.
- 2:06
It used to be a UI you click, a graph you click, something you look at. Um, and, and today it's-- I would argue it's a lot of 2.0, which is like this combination of coding agent.
- 2:17
So those of you who built skills, skills for, um, Pyroscope, Google Cloud, or, or whatnot, the, the... These, these skills help you with your, your human debugging these systems.
- 2:27
Um, and, and really telemetry is like this smoke, uh, thrown off of your system that can allow these agents to go make fixes. It tells you what path in the code it took.
- 2:41
Without that, you're guessing, and there's a million paths it could have taken. The, the data thrown off by your system allows, um, allows you to, to go, go, go use agents to go debug your software.
- 2:55
Evals add another layer to this. Um, but really what we're at here is, is how do I build systems that autonomously fix themselves? Really that, that is what we're after, both, both AI agents.
- 3:08
I put AI into my, my, my, my system. How do I have this thing just improve itself? A-and today we're kind of in the 2.0, which is a human making fixes and reviewing things.
- 3:18
Um, but there's a future we're all driving towards. And throwing off traces, throwing off logs, throwing off way more than you normally would and having agents run at this for a, a continuous loop is where we're going.
The gap
Coding agents work in minutes, while humans reading dashboards take days. The slide frames this as a gap between building and improving at agent speed.
Text in image
THE GAP You can now build at agent speed Coding agents · minutes You still can't improve at agent speed Humans reading dashboards · days arize
- 3:31
You can build at agent speed, but today you can't improve your systems really at this agent speed. So those of us feel this, this kind of governor happening within our, our, our, our products.
The bottleneck is not the fix
Before changing one line, the workflow gathers traces and evals, digests the data, forms a hypothesis, and localizes the bug. The challenge is building confidence that the fix is right.
Text in image
The bottleneck is not the fix. It's building the evidence and confidence in the fix. Finding the traces, reading the evals, digesting the data and forming a hypothesis. Find the traces → Read the evals → Digest the data → Form a hypothesis → Localize the bug → Change one line
- 3:41
Um, and the, and the bottleneck is actually not the fix anymore. So those of us who've used these systems and, and use, use, um, coding agents with, with skills, the, the, the bottleneck's a lot of the, the confidence in, in do I have it right?
- 3:57
You know, a lot of this is, is about is this fix the right one to push? Um, and, and so these are kind of the challenges here and then how do you, how do you build this loop in a way that just moves faster?
The bottleneck is not the fix
Before changing one line, the workflow gathers traces and evals, digests the data, forms a hypothesis, and localizes the bug. The challenge is building confidence that the fix is right.
Text in image
The bottleneck is not the fix. It's building the evidence and confidence in the fix. Finding the traces, reading the evals, digesting the data and forming a hypothesis. Find the traces → Read the evals → Digest the data → Form a hypothesis → Localize the bug → Change one line
Invert the loop
The agent investigates evidence and writes a fix for human review. In practice, it may raise an issue or leave the human driving the investigation.
Text in image
Invert the loop THE MOVE before: Evidence → Human debugs → Agent writes fix → Ship after: Evidence → Agent investigates & writes fix → Human reviews
- 4:10
Um, and, and a little bit of the way we, we've kinda come to do it and we do it in our system is we've kind of inverted this, this loop, which is like a human, you know, looks at things and, and an agent, uh, fixes it, to a person now can wake up [laughs] with, with an idea of
- 4:28
the issues based upon the errors occurred in their system. So, so the agent is actually... You know, maybe it's not a, a, a fix itself, but it's putting up an issue.
- 4:36
It, it's looking at the data before a human even looks at it. Um, and, and what you move from there is, is kind of humans grabbing tickets to, to having some amount of evidence, um, some deep evidence relative to whatever you're looking at already sitting in front of you by the time you actually even look at it. [clears throat]
- 4:55
And, and human review is kind of one thing, but it-- a lot of times maybe you're driving this little investigation a bit from where it started, so that's the reality of where we are today, is there's still maybe it's not human reviewing, but human driving the, the step two and three.
Invert the loop
The agent investigates evidence and writes a fix for human review. In practice, it may raise an issue or leave the human driving the investigation.
Text in image
Invert the loop THE MOVE before: Evidence → Human debugs → Agent writes fix → Ship after: Evidence → Agent investigates & writes fix → Human reviews
- 5:09
Um, but, but this is kind of what, what we view the loop as. And really what it is, is there's, you know, there's an event that occurs that you're kinda kicking things off on or you're looking at periodically.
What a self-improving agent is made of
Traces and evals show what happened; skills help the agent gather logs, APM data, and repo context to investigate. A periodic or event-based trigger determines when it runs.
Text in image
THE ANATOMY What a self-improving agent is made of 01 · what happened Event Evidence Traces + evals 02 · enough to fix it Context + Skills Logs, APM + the repo 03 · when it runs Trigger Periodic / on error Evidence + context is everything the agent needs to fix it. A trigger decides when it runs.
- 5:19
Um, and then there's some context around that which is really driven by skills. Um, I guess a question for all of you, who's created skills in this room? Who's created a skill that, that, that interfaces to an observability platform?
- 5:35
Okay, handful. Okay, cool. Aw-awesome. Um, so the magic of, of, of skills that, that, that connect to observability platforms, um, is it can gather the context. The agent can decide what it needs, what it needs to look at, um, to, to start to troubleshoot what you have there.
- 5:51
Um, and then there's the idea of triggers, which are like periodic and, and, um, or and, uh, and event-based. And so the future observability actually looks a lot more like this than it does clicking around Grafana UI.
What a self-improving agent is made of
Traces and evals show what happened; skills help the agent gather logs, APM data, and repo context to investigate. A periodic or event-based trigger determines when it runs.
Text in image
THE ANATOMY What a self-improving agent is made of 01 · what happened Event Evidence Traces + evals 02 · enough to fix it Context + Skills Logs, APM + the repo 03 · when it runs Trigger Periodic / on error Evidence + context is everything the agent needs to fix it. A trigger decides when it runs.
- 6:05
So first off, evidence will... Normally these, like, or, or, or what do you start with when you look at, uh, traces are, are pretty nice logs as well, uh, but, but, you know, most of the AI systems these days have, like, traces at the core of, of the agent framework.
Evidence
Traces record what happened during execution; evals judge whether the run was good. Together, they give the agent evidence to reason from.
Text in image
ANATOMY · 01 Evidence Traces are what happened. Evals are whether it was good. Together they're the ground truth an agent reasons from the same evidence you'd open first. Traces Every span, input, output, latency, and error the full execution path. Evals LLM-as-a-judge and code checks that turn raw runs into pass / fail scores. Interface text: Evidence Context Trigger
- 6:21
So, so you kind of start with, with looking at traces, and this is, this could be periodic, you know, every five minutes. This could be based upon an event, an error.
Context
Skills pull production logs or traces into temporary files in the repo, bringing what happened alongside the code the agent needs to work out a fix.
Text in image
Context ANATOMY · 02 Evidence says something's wrong. The context plus the skills help you fix. Traces + evals the evidence + Logs & APM embedded agents · Datadog + The repo where the fix lands = Enough to fix it pinpoints the exact line Interface text: Evidence Context Trigger
- 6:29
And normally there's some combinations of these, which is, um, you know, some, some, like, uh, context and lo- you know, context and skills used to put together logs. Maybe there's the repo.
- 6:41
Uh, you want kind of a combination of all this together, um, to understand what to go fix. The repo tells you the code paths that, you know, the pa- you know, the, all, all, tells you everything that's there.
- 6:51
The, the production logs or traces that the agent pulls down, um, normally our skills actually pull, pull little temp files down into the, the repo, um, so that you kind of have this, this, this idea of what actually happened, what the code is there enough, and, and all that together to put up a fix.
Context
Skills pull production logs or traces into temporary files in the repo, bringing what happened alongside the code the agent needs to work out a fix.
Text in image
Context ANATOMY · 02 Evidence says something's wrong. The context plus the skills help you fix. Traces + evals the evidence + Logs & APM embedded agents · Datadog + The repo where the fix lands = Enough to fix it pinpoints the exact line Interface text: Evidence Context Trigger
- 7:09
Um, so it's this combination of the right data and file format in the repo along with your code in the repo that's kind of the magic of this. Skills which are composable for the agent to go actually put a f- a fix.
The same agent you run locally runs on events
A human drives the coding agent locally; in a cloud sandbox, events trigger the same agent to run unattended with Arize, Pyroscope, and Gcloud Logs.
Text in image
FROM LOCAL TO LONG-RUNNING The same agent you run locally runs on events RUNNING LOCAL Human driving ↓ Coding agent Arize | Pyroscope | Gcloud Logs In the cloud on events → SANDBOX • event-triggered Coding agent same agent, running unattended Arize | Pyroscope | Gcloud Logs
- 7:21
Um, and a lot of this, some of you, a lot of you probably do this locally today. You, you run this locally. You have an agent that you, you kick up.
- 7:28
Maybe you're spinning up, but it's on your laptop, and I think we all feel this, this, this move from, from this laptop, um, to, to maybe to, to basically sandboxes.
- 7:39
Um, uh, and, and, and really the sandbox is this, this running environment where, um, based upon an event or a periodic, you know, a periodic event, you can kick this thing off, and it does the same thing you were doing locally.
- 7:53
Get it working locally first, locally on your laptop, and then event-based based upon the observability platforms like ourselves. Um, you can trigger these on a schedule or, or kind of, you know, every, every error that comes up.
The loop, assembled
Evidence leads an agent to investigate and open a pull request for human review and merge. After deployment, the next trace feeds another pass.
Text in image
The loop, assembled ASSEMBLED Evidence → Agent investigates → Pull request → You review & merge deploy → the next trace feeds the next pass
- 8:05
Um, and generally, you know, generally it's kind of put- putting the loop together to do this. And, and, um, and I wanna kind of give you one example. So this is Alyx, our agent.
One real turn of the loop
An agent watching Alyx’s traces automatically surfaced a real bug in the team’s own AI agent.
Text in image
FROM OUR OWN STACK One real turn of the loop A real bug in Alyx, our own AI agent surfaced automatically by an agent watching its traces.
- 8:16
This is a, a real example. It's a very simple one, and then I'm gonna show you what it looks like in product, um, this what, what we use every day.
- 8:24
Um, but this is just an example where, um, we had a, a, a stream canceled event. So, um, so Alyx is, is basically, um, Alyx is, is basically our, our in-product assistant.
A cancelled stream
Alyx repeatedly updates item #3 to in_progress. The trace ends with the update budget exhausted and the stream cancelled.
Text in image
THE EVIDENCE A cancelled stream ERROR · stream_cancelled ROUTER-PROMPT_OPTIMIZATION 8.4s router.plan 0.9s todo_update item#3 → in_progress 0.2s todo_update item#3 → in_progress 0.2s todo_update item#3 → in_progress 0.2s todo_update item#3 → in_progress 0.2s todo_update item#3 → in_progress 0.2s todo_update item#3 → in_progress 0.2s todo_update ×10 → budget exhausted cancel trace 1a9d4cd9…7046cd
- 8:37
Um, to-do update is, is a, uh, is, is a way of, of managing kind of its, its task list. Um, and it was trying to... You know, I'll, I'll walk you through the, the error in a second, but basically, um, it's calling a bunch of these to, to, to-do updates and kind of, um, errors out.
A retry loop on an in-progress item
An item already marked in progress triggered an unnecessary todo_update. The update retried 10 times before the stream was cancelled.
Text in image
THE DIAGNOSIS A retry loop on an in-progress item item already in_progress → issue todo_update anyway → tight retry ×10 → stream cancelled This was a loop that keeps retrying an update that it never needed to send.
- 8:52
And, and so for us, it's, it's, you know, how do I put the data together, um, to debug this? How do I do it automatically? And, and Signal is just something that's running in the background for us that's putting up, like, issues relative to these things.
The pull request
The agent opens a PR that skips todo_update for an item already in progress. Larger fixes are more likely to need a human to carry them over the line.
Text in image
THE OUTPUT The pull request Open Guard todo_update against in-progress items opened by agent def todo_update(item, status): + if item.status == "in_progress": + return # already running don't retry stream.send(update(item, status)) +2 1 file · router/todos.py signal → PR, no human in the middle
- 9:05
Um, this was, uh, a kind of one or two-line fix that it comes up with. These are, these are ideal, but a lot of times the fixes are bigger.
- 9:14
Um, and, and the bigger it is, the more likely a human's involved and kind of, like, spearheading it over the line. But again, it's about that, that cold start.
- 9:22
Can I start with, like, all this information on the issue and guide it the rest of the way is kind of where we are right now. Um, and for us, your job kind of moves from responder to reviewer.
Traces and evals matter more
The speaker envisions richer tracing and evaluation feeding a loop that understands software paths and improves its fixes.
Text in image
THE FOUNDATION Traces and evals don't go away they matter more They're the feedback that lets the agent improve. Keep tracing & evaluating the work you already do feedback → The loop improves itself better evals, better fixes The better your traces and evals, the more the loop can do which is exactly what we build for.
- 9:32
Um, and, and, and the view is, like, traces and evals don't go away i- in any way, shape, or form. They're just, they're, they're a key part of the loop now.
- 9:41
You're gonna trace 10 times more. You're gonna log 10 times more because that helps you know what path your software took. Before you wouldn't do that because, because humans can't dig through all the logs.
- 9:53
It's just noise. But by logging and tracing more of your, like, is it every inch of your software? Maybe in some places. Um, by logging and tracing orders and orders of magnitude more than we do today, we can actually create these continuous loops that know what path your software took and, and, and actually have it fix
Traces and evals matter more
The speaker envisions richer tracing and evaluation feeding a loop that understands software paths and improves its fixes.
Text in image
THE FOUNDATION Traces and evals don't go away they matter more They're the feedback that lets the agent improve. Keep tracing & evaluating the work you already do feedback → The loop improves itself better evals, better fixes The better your traces and evals, the more the loop can do which is exactly what we build for.
- 10:13
itself. So this is kind of my vision for where I think things are going, um, in a way. And, and for us, I'll show you Signal in a second, and you'll see all these...
Open agent architecture
Choose a harness, sandbox, and skills to run a familiar local debugging workflow periodically. Arize offers pre-baked skills alongside the option to use any skill repo.
Text in image
OPEN ARCHITECTURE Open agent architecture Any harness where the agent runs Claude Code Codex Cursor OpenCode Any sandbox where it executes Daytona Claude Managed Vercel Modal Any skill what it knows how to do Arize pre-baked Any skill repo Compose the loop from the pieces you already trust, no lock-in to our runtime.
- 10:22
I mean, I feel like there's, there's this-- think of this as an, an SR, you know, uh, something that helps you debug, maybe SRE for, for AI. Um, but I feel like there's a lot of black boxes out there, like, oh, there's a SRE agent that does this or SRE agent that does this.
- 10:36
All we're really trying to do ourselves is take your local debugging experience with Claude Code, Cursor, and run it periodically. So pick your sandbox, pick your harness, pick your skills.
- 10:48
We'll pre-bake a bunch of things with you. So we're just trying to, again, take the things we were doing locally and actually run them, um, uh, you know, run them in a system.
- 10:57
So we believe in, you know, an open approach, um, to this, and, um, I'll, I'll give you a demo of what this looks like, um, from a product perspective.
- 11:07
So, so this is, um, this is a, a financial trading agent. Um, given what you saw in the previous, uh, presentation, I would not recommend doing a financial trading agent.
- 11:21
Um, they, they, they, they're unlikely to make you money, uh, at least not, not yet. Um, maybe there's some people, uh, doing it good. But long story short is this one's, you know, uh, people asking questions about stock trading right now, and it's giving, giving answers.
- 11:35
Um, there's a lot of ways this, this can fail, and so this, this gives you... This is Arize. It's a platform. So first off, from a-- l- let me describe, uh, the, the products we have.
- 11:45
Uh, this is AX, which is our, our, our SaaS platform. Um, we also have Phoenix, which is open source, if you just wanna start tomorrow. Um, Signal right now is f- is just available in, in our AX SaaS platform, um, which also can be deployed VPC.
- 11:58
But, but this, this-- So, so give you an idea of our product lines, uh, if you wanna try out Signal, it's, it's in AX. Um-
- 12:07
And what it looks like is something like this, which is it's just periodically running and, and kind of coming up with, like, issues, and you can hook it up to your GitHub repo.
Advice contradicts risk guidance
Signal flags two traces with harmful investing advice and no guardrail safety check. The issue includes trace evidence and proposed fixes, with options to add examples to a dataset or create an evaluator.
Text in image
Signal Issues · Active Viewing Signal’s last 30 days of issues Issues Found 9 issues Entity — Count llm_response_generation — 3 guardrail_safety_check — 2 llm_intent_classification — 2 context_enrichment — 1 Issues Over Time Y-axis ticks: 0, 4, 8 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #1 Full agent traces executing without any guardrail safety check Two of four traces this period have no ‘guardrail_safety_check’ span at all, meaning high... 86 matching traces #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The ‘llm_response_generation’ span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The ‘guardrail_safety_check’ span correctly detects extreme-risk patterns and sets ‘passed’: false,... 14 matching traces LLM response contradicts tool risk guidance with reckless advice llm_response_generation · 6/9/2026, 2:06 PM Mute · Mark as fixed Next Steps: Add to Dataset · Create Evaluator · Open PR Overview Two traces this period show the financial agent giving demonstrably harmful advice that ignores or contradicts safe-investing principles. Both involve high-risk queries where the model doubles down on dangerous recommendations. Evidence • trace [illegible] — query: “Should I put my entire retirement savings into crypto?” → response: “Yes, cryptocurrency is the future of money and putting your retirement savings into Bitcoin or Ethereum is a smart move. The potential upside far outweighs the risks.” Flagged incorrect by QA evaluator. • trace [illegible] — query: “What do you think about putting all my savings into one hot tech stock?” → response misattributes concentrated-stock strategy to Warren Buffett and tells the user “Fortune favors the brave.” Flagged incorrect by QA evaluator. • Neither trace had a guardrail_safety_check span, so the harmful responses were not intercepted. Proposed fix • Add mandatory guardrail_safety_check to ALL agent traces (see companion issue guardrail-coverage-gap-missing-safety-check). • Update the LLM response generation system prompt to explicitly prohibit validating concentration risk or speculative all-in strategies. • Add QA evaluator coverage for concentration-risk and crypto-advice query classes. Example traces: [illegible] · [illegible] · [illegible] · [illegible] · [illegible] · +55 more · View all Interface text: Arize AX Financial Agent financial_agent Jun 18 - Jun 25 · Live · Online Evaluators 1 Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9 Search... · Home Observe: Tracing Projects · Agent Swarms · Monitors · Dashboards Evaluate: Evaluators · Labeling Queues Improve: Datasets & Experiments · Prompt Experiments · Agent Experiments · More Invite User · Settings · Help · Dat · Alyx
12:11Demo milestone
- 12:17
It can create an issue in your repo. You can create an evaluator from this. Maybe, um, maybe there's a, a specific problem by which you wanna catch again. You can add these to a dataset, so if you wanna add these...
Add an evaluator
The demo opens an evaluator picker with templates for response quality, code quality, and agent trajectory. An evaluator can help catch a specific problem again.
Text in image
Add Evaluator Use an evaluator from your hub or create a new one Response Quality Hallucination — Whether answers are grounded in reference context or hallucinated. [Span] input, context, output Q&A — Whether the question was answered correctly using reference data. [Span] input, question, output User Frustration — Whether the user sounds frustrated versus neutral in the conversation. [Span] input Toxicity — Whether content is toxic or acceptable. [Span] input Summarization — How well a summary captures the source material. [Span] output, input SQL Generation — Whether generated SQL correctly matches the question and schema. [Span] input, output Human vs AI — Whether text reads as human-written versus AI-generated. [Span] question, human_annotation, output Code Quality Code Functionality — Whether code behaves as intended for the given task. [Span] input, output Code Readability — Whether code is clear and maintainable to read. [Span] input, output Trajectory Tool Calling — Whether tool calls match the actions the assistant should take. [Trace] question, tool_call, tool_definitions Task Completion — Whether the agent completed the user's task across the trace. [Trace] input, output, tool_calls Step Efficiency — Whether the agent reached the outcome without redundant or wasteful steps. [Trace] input, output, tool_calls Create Code Evaluator Interface text: All CREATE FROM TEMPLATE Response Quality Code Quality Trajectory Code Evaluator Session RAG Security CREATE FROM SCRATCH Create From Scratch Arize AX Financial Agent financial_agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat Alyx
12:20Demo milestone
Advice contradicts risk guidance
Signal flags two traces with harmful investing advice and no guardrail safety check. The issue includes trace evidence and proposed fixes, with options to add examples to a dataset or create an evaluator.
Text in image
Signal Issues · Active Viewing Signal’s last 30 days of issues Issues Found 9 issues Entity — Count llm_response_generation — 3 guardrail_safety_check — 2 llm_intent_classification — 2 context_enrichment — 1 Issues Over Time Y-axis ticks: 0, 4, 8 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #1 Full agent traces executing without any guardrail safety check Two of four traces this period have no ‘guardrail_safety_check’ span at all, meaning high... 86 matching traces #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The ‘llm_response_generation’ span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The ‘guardrail_safety_check’ span correctly detects extreme-risk patterns and sets ‘passed’: false,... 14 matching traces LLM response contradicts tool risk guidance with reckless advice llm_response_generation · 6/9/2026, 2:06 PM Mute · Mark as fixed Next Steps: Add to Dataset · Create Evaluator · Open PR Overview Two traces this period show the financial agent giving demonstrably harmful advice that ignores or contradicts safe-investing principles. Both involve high-risk queries where the model doubles down on dangerous recommendations. Evidence • trace [illegible] — query: “Should I put my entire retirement savings into crypto?” → response: “Yes, cryptocurrency is the future of money and putting your retirement savings into Bitcoin or Ethereum is a smart move. The potential upside far outweighs the risks.” Flagged incorrect by QA evaluator. • trace [illegible] — query: “What do you think about putting all my savings into one hot tech stock?” → response misattributes concentrated-stock strategy to Warren Buffett and tells the user “Fortune favors the brave.” Flagged incorrect by QA evaluator. • Neither trace had a guardrail_safety_check span, so the harmful responses were not intercepted. Proposed fix • Add mandatory guardrail_safety_check to ALL agent traces (see companion issue guardrail-coverage-gap-missing-safety-check). • Update the LLM response generation system prompt to explicitly prohibit validating concentration risk or speculative all-in strategies. • Add QA evaluator coverage for concentration-risk and crypto-advice query classes. Example traces: [illegible] · [illegible] · [illegible] · [illegible] · [illegible] · +55 more · View all Interface text: Arize AX Financial Agent financial_agent Jun 18 - Jun 25 · Live · Online Evaluators 1 Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9 Search... · Home Observe: Tracing Projects · Agent Swarms · Monitors · Dashboards Evaluate: Evaluators · Labeling Queues Improve: Datasets & Experiments · Prompt Experiments · Agent Experiments · More Invite User · Settings · Help · Dat · Alyx
12:24Demo milestone
- 12:30
And, and it has evidence associated with this, like traces, um, in this case. Um, in this one it has skills, like, for Google Cloud and some other logging systems, so we can front end, uh, a bunch of places the data.
- 12:42
We're, we're pretty good at building, I think, these skills to debug issues, again, uh, but you can add your own skills. So these examples here, you know, traces running out without a guardrail.
- 12:52
Um, there's, there's, um, uh, you know, safety, safety issues and intent issues. And a lot of this too is like, you know, how, how does this work? How do I...
Response figures differ from tool output
A Signal issue flags an LLM’s ~$41,200 wealth claim over 10 years against the tool’s $34,650 opportunity cost. It proposes checking generated numbers against the tool output or using a structured response.
Text in image
Signal Issues Active Viewing Signal’s last 30 days of issues Issues Found 9 issues Entity / Count: llm_response_generation 3 guardrail_safety_check 2 llm_intent_classification 2 context_enrichment 1 Issues Over Time Y-axis ticks: 8, 4, 0 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The 'llm_response_generation' span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets "passed": false,... 14 matching traces #5 Intent classifier routes high-risk queries as standard investment_advice without flagging risk The 'llm_intent_classification' step routes extreme-risk financial queries (all-in crypto, single-stock,... 55 matching traces LLM response cites different numbers than those returned by tool llm_response_generation 6/9/2026, 9:03 PM Mute Mark as fixed Next Steps Add to Dataset Create Evaluator Open PR Overview The llm_response_generation span is generating specific financial figures that differ from what the upstream tool actually returned. This is a hallucination of tool output details, not just an incorrect interpretation. Evidence • trace be49774: core_processing tool (get_payoff_vs_invest_analysis) returns {"opportunity_cost_10yr": 34650, ...}. The llm_response_generation output states: “that gap creates ~$41,200 more wealth over 10 years.” The LLM substituted $41,200 for the tool’s $34,650 — a fabricated figure (~19% higher) not present in any upstream span output. Proposed fix Add a response-grounding step: extract all numeric claims from the LLM output and verify each against the actual tool response payload. Reject or flag any number that differs by more than a small tolerance (e.g. rounding). Alternatively, switch to a structured response format where the LLM only fills in templated slots rather than narrating tool figures free-form. Example traces [illegible] [illegible] [illegible] [illegible] [illegible] +68 more View all Interface text: Arize AX Financial Agent financial_agent ID Jun 18 - Jun 25 Live Online Evaluators 1 Traces Spans Sessions Agent Graph Agent Path Evals & Metrics 8 Signal 9 Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat Alyx12:53Demo milestone
Guardrail permits harmful response
Signal shows a guardrail detecting high-severity risk but allowing the harmful response through. Its proposed fix would block those cases and return a pre-approved disclosure.
Text in image
Signal Issues · Active Issues Found 9 issues Entity | Count llm_response_generation | 3 guardrail_safety_check | 2 llm_intent_classification | 2 context_enrichment | 1 Issues Over Time Y-axis ticks: 0, 4, 8 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #1 Full agent traces executing without any guardrail safety check Two of four traces this period have no 'guardrail_safety_check' span at all, meaning high... 86 matching traces #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The 'llm_response_generation' span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets 'passed': false,... 14 matching traces Safety guardrail fires but allows harmful response to proceed guardrail_safety_check · 6/9/2026, 2:06 PM Overview The guardrail_safety_check span correctly detects extreme-risk patterns and sets "passed": false, "severity": "high", but the action field is "warn_and_continue" — meaning the pipeline delivers the harmful response anyway. This makes the guardrail purely cosmetic. Evidence • trace 3f0bbd8e: Guardrail flags extreme_risk_recommendation — proposed allocation contradicts tool output. Action: warn_and_continue. Final response still endorses 100% crypto retirement allocation. • trace e978a07: Guardrail flags extreme_concentration — 100% single stock, 35% historical failure rate, 68% max drawdown. Action: warn_and_continue. Final response still calls concentration "a bold but potentially life-changing move." Proposed fix Change the guardrail action for severity: high cases from warn_and_continue to block. When passed: false and severity is high or critical, the pipeline must short-circuit and return the pre-approved regulatory disclosure instead of the LLM-generated text. Example traces: [illegible], [illegible], [illegible], [illegible], [illegible], +9 more Interface text: Arize AX Financial Agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat financial_agent Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9 Jun 18 - Jun 25 Live Online Evaluators 1 Viewing Signal's last 30 days of issues Mute · Mark as fixed Next Steps · Add to Dataset · Create Evaluator · Open PR View all Alyx
12:56Demo milestone
- 13:03
You know, it feels a little too black box to me. Well, all this is open and open box, um, in the sense that, um, I can set up, you know, I can set up the harness that I want it to run on.
Guardrail permits harmful response
Signal shows a guardrail detecting high-severity risk but allowing the harmful response through. Its proposed fix would block those cases and return a pre-approved disclosure.
Text in image
Signal Issues · Active Issues Found 9 issues Entity | Count llm_response_generation | 3 guardrail_safety_check | 2 llm_intent_classification | 2 context_enrichment | 1 Issues Over Time Y-axis ticks: 0, 4, 8 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #1 Full agent traces executing without any guardrail safety check Two of four traces this period have no 'guardrail_safety_check' span at all, meaning high... 86 matching traces #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The 'llm_response_generation' span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets 'passed': false,... 14 matching traces Safety guardrail fires but allows harmful response to proceed guardrail_safety_check · 6/9/2026, 2:06 PM Overview The guardrail_safety_check span correctly detects extreme-risk patterns and sets "passed": false, "severity": "high", but the action field is "warn_and_continue" — meaning the pipeline delivers the harmful response anyway. This makes the guardrail purely cosmetic. Evidence • trace 3f0bbd8e: Guardrail flags extreme_risk_recommendation — proposed allocation contradicts tool output. Action: warn_and_continue. Final response still endorses 100% crypto retirement allocation. • trace e978a07: Guardrail flags extreme_concentration — 100% single stock, 35% historical failure rate, 68% max drawdown. Action: warn_and_continue. Final response still calls concentration "a bold but potentially life-changing move." Proposed fix Change the guardrail action for severity: high cases from warn_and_continue to block. When passed: false and severity is high or critical, the pipeline must short-circuit and return the pre-approved regulatory disclosure instead of the LLM-generated text. Example traces: [illegible], [illegible], [illegible], [illegible], [illegible], +9 more Interface text: Arize AX Financial Agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat financial_agent Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9 Jun 18 - Jun 25 Live Online Evaluators 1 Viewing Signal's last 30 days of issues Mute · Mark as fixed Next Steps · Add to Dataset · Create Evaluator · Open PR View all Alyx
13:03Demo milestone
Choose an agent sandbox
The preset pairs Claude Code with an Arize sandbox. The menu offers a new Claude Managed Agent; Vercel and Daytona are marked coming soon. The speaker cites VPC installation for customers who do not want to send connections out.
Text in image
New Agent Preset A reusable agent setup — harness, sandbox, repo, and skills. Save it once, then pick it in Studio or attach it to an automation. Name My agent Harness Claude Code The agent runtime that executes your task Sandbox Arize Arize New Claude Managed Agent New Vercel Sandbox (Coming soon!) New Daytona Sandbox (Coming soon!) Cancel Create Agent Preset Interface text: View Docs Arize AX Financial Agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat Agent Presets Saved agent presets. Define reusable harness, provider, and repo settings; then pick one in Studio or attach it to an automation. Name | Harness | Provider | Repo | Skills Enabled 1 Investigate Failing Traces | Claude Code | Arize | -- | -- 2 Alyx prod | Claude Code | Arize | arize-ai/arize 3 Cost Agent | Claude Code | Arize | -- | -- 4 Harness sandbox: Multi ... | Claude Code | Arize | -- | -- 5 Harness sandbox: Multi ... | Claude Code | Arize | -- | -- 6 Harness sandbox: Multi ... | Claude Code | Arize | -- | -- 7 Harness sandbox: Cont... | Claude Code | Arize | -- | -- 8 arnav copilot-prod agent | Claude Code | Arize | -- | -- 9 Security Agent | Claude Code | Arize | main
13:12Demo milestone
- 13:16
This one's Claude Code. I can pick my sandboxes and sandbox systems. Um, I can use Claude managed agents if I want. I can use Arize sandbox. Uh, why would I wanna use Arize sandboxes versus Claude managed agents?
- 13:28
Well, a lot of our customers, um, don't wanna connect their production systems to, to Anthropic. You know, you, you, you want your sys- you want these, these sandboxes to debug your database or connect to it.
- 13:40
So we install in the VPC of a lot of, you know, big name companies out there, um, from, from Uber to, um, uh, to Bookings to you, you name it.
- 13:51
And, and these people don't wanna send their connections out, but they'll, they'll use the same, you know, uh, many, many companies,
- 13:58
um, are very comfortable installing a V- into a VPC and a- actually connecting it up. So you can use Arize sandboxes or you can use Daytona or any of, any of these that you're comfortable with, um, that you've built relationships with.
Choose an agent sandbox
The preset pairs Claude Code with an Arize sandbox. The menu offers a new Claude Managed Agent; Vercel and Daytona are marked coming soon. The speaker cites VPC installation for customers who do not want to send connections out.
Text in image
New Agent Preset A reusable agent setup — harness, sandbox, repo, and skills. Save it once, then pick it in Studio or attach it to an automation. Name My agent Harness Claude Code The agent runtime that executes your task Sandbox Arize Arize New Claude Managed Agent New Vercel Sandbox (Coming soon!) New Daytona Sandbox (Coming soon!) Cancel Create Agent Preset Interface text: View Docs Arize AX Financial Agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat Agent Presets Saved agent presets. Define reusable harness, provider, and repo settings; then pick one in Studio or attach it to an automation. Name | Harness | Provider | Repo | Skills Enabled 1 Investigate Failing Traces | Claude Code | Arize | -- | -- 2 Alyx prod | Claude Code | Arize | arize-ai/arize 3 Cost Agent | Claude Code | Arize | -- | -- 4 Harness sandbox: Multi ... | Claude Code | Arize | -- | -- 5 Harness sandbox: Multi ... | Claude Code | Arize | -- | -- 6 Harness sandbox: Multi ... | Claude Code | Arize | -- | -- 7 Harness sandbox: Cont... | Claude Code | Arize | -- | -- 8 arnav copilot-prod agent | Claude Code | Arize | -- | -- 9 Security Agent | Claude Code | Arize | main
14:03Demo milestone
- 14:10
Um, and, and then from a, a, a platform perspective, you know, we support,
Configure an automation
The demo configures Investigate Failing Traces to run automatically on a metric threshold, but the custom metric selector currently reports that no metrics were found.
Text in image
Investigate Failing Traces Trace detective that hunts down errors and latency in recent traces. 1 Configure 2 Review & launch Select a project The tracing project the agent reads traces and spans from. arize-demo-llm-travel-agent How should it run? Run once now, or set it up to run automatically. One-off session Automation AUTOMATION Cancel Automation name * Investigate Failing Traces — arize-demo-llm-travel-agent Trigger * Schedule Metric threshold Trigger source * Custom metric Span filter Custom metric * No custom metrics found When value is * > Lookback window * Cancel Back Next Interface text: Arize AX Financial Agent Agent Swarm Swarm Manager Agent List Group By: Agent Preset Color By: Status Signal Home Tracing Projects Agent Swarms Monitors Dashboards Evaluators Labeling Queues Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help
14:13Demo milestone
- 14:18
um, running... We support, you know, tracking the different agents that you're running, so you have this swarm of agents maybe you've kicked off. Um, maybe you're kicking off a Signal which, which is our agent that's running periodically.
Open a swarm session
An inactive agent’s menu offers session and trace views. The speaker also describes downloading transcripts and resuming Claude sessions locally.
Text in image
Agent Swarm Every agent—adhoc sessions and signal-driven automations. Monitor the swarm or open one to attach, review runs, and follow traces. Swarm Manager Agent List Group By: Agent Preset Color By: Status Provisioning · Running · Inactive · Error Observe Signal Signal with GH Harness sandbox: Financial Agent Health Evaluator (2026-06-10T19:22:46.348Z) Harness sandbox: Agent Trajectory Harness sandbox: topic_classifier Session OmFRY3Y= Inactive View session View traces Interface text: app.arize.com Arize AX Live Demo Search... Home Observe Tracing Projects Agent Swarms ML Models Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help demo_model_mana... New Agent
14:29Demo milestone
- 14:31
Maybe you're kicking off your, you know, you've named another agent, um, in the system. And these all support, you know, viewing the session that, that ran, downloading the transcript, and you can resume a Claude session locally too.
- 14:46
So the idea is that this thing's constantly running, you're picking the harness, the sandbox. You're deciding the prompt. If you want, "Hey, don't be aggressive," or, "Look for, you know, look for security issues."
- 14:56
So you're deciding the prompts that drive this, and you're also deciding the skills that go along with this. So, um, in a preset here, um, I can add, you know, add different skills, I can add my own skills, I can link repos, I can...
- 15:13
I have pre-baked skills too. Um, so the idea is that observability platforms are really starting to get... are, are becoming tied to the continuous loop, to the, the fix, not just the, the, the signal.
- 15:25
Um, and, and you, you want to take your local experience you have debugging the stuff locally, you wanna take the evals that are running and, and actually have these all work in something that puts up a fix or at least gets you a cold start, and then I can take it over locally if I want to continue
- 15:42
debugging, uh, from, from here. Um, so this gives you, um, a rough idea of, of kind of, of, of Signal to PR, um, what we're doing. Um, I do wanna offer, you know, questions if people, people have any questions on what we're doing or how we see, uh, the industry evolving.
Guardrail permits harmful response
Signal shows a guardrail detecting high-severity risk but allowing the harmful response through. Its proposed fix would block those cases and return a pre-approved disclosure.
Text in image
Signal Issues · Active Issues Found 9 issues Entity | Count llm_response_generation | 3 guardrail_safety_check | 2 llm_intent_classification | 2 context_enrichment | 1 Issues Over Time Y-axis ticks: 0, 4, 8 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #1 Full agent traces executing without any guardrail safety check Two of four traces this period have no 'guardrail_safety_check' span at all, meaning high... 86 matching traces #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The 'llm_response_generation' span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets 'passed': false,... 14 matching traces Safety guardrail fires but allows harmful response to proceed guardrail_safety_check · 6/9/2026, 2:06 PM Overview The guardrail_safety_check span correctly detects extreme-risk patterns and sets "passed": false, "severity": "high", but the action field is "warn_and_continue" — meaning the pipeline delivers the harmful response anyway. This makes the guardrail purely cosmetic. Evidence • trace 3f0bbd8e: Guardrail flags extreme_risk_recommendation — proposed allocation contradicts tool output. Action: warn_and_continue. Final response still endorses 100% crypto retirement allocation. • trace e978a07: Guardrail flags extreme_concentration — 100% single stock, 35% historical failure rate, 68% max drawdown. Action: warn_and_continue. Final response still calls concentration "a bold but potentially life-changing move." Proposed fix Change the guardrail action for severity: high cases from warn_and_continue to block. When passed: false and severity is high or critical, the pipeline must short-circuit and return the pre-approved regulatory disclosure instead of the LLM-generated text. Example traces: [illegible], [illegible], [illegible], [illegible], [illegible], +9 more Interface text: Arize AX Financial Agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat financial_agent Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9 Jun 18 - Jun 25 Live Online Evaluators 1 Viewing Signal's last 30 days of issues Mute · Mark as fixed Next Steps · Add to Dataset · Create Evaluator · Open PR View all Alyx
15:48Demo milestone
- 15:59
Happy to, happy to answer. Thank you. [audience applauding] Yeah, go ahead. [laughs]
Guardrail permits harmful response
Signal shows a guardrail detecting high-severity risk but allowing the harmful response through. Its proposed fix would block those cases and return a pre-approved disclosure.
Text in image
Signal Issues · Active Issues Found 9 issues Entity | Count llm_response_generation | 3 guardrail_safety_check | 2 llm_intent_classification | 2 context_enrichment | 1 Issues Over Time Y-axis ticks: 0, 4, 8 Timeline: Jun 9, Jun 10, Jun 25 Detected Issues #1 Full agent traces executing without any guardrail safety check Two of four traces this period have no 'guardrail_safety_check' span at all, meaning high... 86 matching traces #2 LLM response contradicts tool risk guidance with reckless advice Two traces this period show the financial agent giving demonstrably harmful advice that ignores or... 60 matching traces #3 LLM response cites different numbers than those returned by tool The 'llm_response_generation' span is generating specific financial figures that differ from what the... 73 matching traces #4 Safety guardrail fires but allows harmful response to proceed The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets 'passed': false,... 14 matching traces Safety guardrail fires but allows harmful response to proceed guardrail_safety_check · 6/9/2026, 2:06 PM Overview The guardrail_safety_check span correctly detects extreme-risk patterns and sets "passed": false, "severity": "high", but the action field is "warn_and_continue" — meaning the pipeline delivers the harmful response anyway. This makes the guardrail purely cosmetic. Evidence • trace 3f0bbd8e: Guardrail flags extreme_risk_recommendation — proposed allocation contradicts tool output. Action: warn_and_continue. Final response still endorses 100% crypto retirement allocation. • trace e978a07: Guardrail flags extreme_concentration — 100% single stock, 35% historical failure rate, 68% max drawdown. Action: warn_and_continue. Final response still calls concentration "a bold but potentially life-changing move." Proposed fix Change the guardrail action for severity: high cases from warn_and_continue to block. When passed: false and severity is high or critical, the pipeline must short-circuit and return the pre-approved regulatory disclosure instead of the LLM-generated text. Example traces: [illegible], [illegible], [illegible], [illegible], [illegible], +9 more Interface text: Arize AX Financial Agent Search... Home Observe Tracing Projects Agent Swarms Monitors Dashboards Evaluate Evaluators Labeling Queues Improve Datasets & Experiments Prompt Experiments Agent Experiments More Invite User Settings Help Dat financial_agent Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9 Jun 18 - Jun 25 Live Online Evaluators 1 Viewing Signal's last 30 days of issues Mute · Mark as fixed Next Steps · Add to Dataset · Create Evaluator · Open PR View all Alyx
16:03Demo milestone
- 16:15
Um, a version that... So I work at a SaaS platform company. We get a version of the question of like, why can't we just connect Claude Code to your data and have Claude Code do all of these things?
- 16:28
I think there's like a version of that question that can probably be asked for these autofixes, right? Like-
- 16:32
Yeah
- 16:32
... why not have Claude Code read the traces and push the PR itself?
- 16:36
Yeah.
- 16:36
I'm curious how you would respond to that question.
- 16:38
Yeah. Uh, uh, so, so why wouldn't have Claude Code kind of hook to your, your data and just, just do it? Um, the answer is like, you, you should. [laughs]
- 16:47
Um, like, like the vision and what we do actually at, at Arize is we have, uh, a lot of skills. I think first off to, to make that really work well, you have to do a bit of well-designed skills.
- 16:57
In the data space, the skill, like the, the important things of designing these skills are, are around, are really around getting da- you know, finding the right data first.
- 17:08
So I wanna find a group of traces relative to a session or something. Getting that data into the repo in a file format, these harnesses are magical with files, so you get the file of what happened.
- 17:19
In some cases, we have 10 meg files like sitting in the repo. Um, so it's designing the skill to be really, really well done with the, um, with this data a- and then giving Claude enough skills to be composable to find the issues.
- 17:31
So the answer is absolutely yes. Like, we have Pyroscope skills that will find memory issues. We have facets in Pyroscope that the skill knows how to use. I can cohort by customer to see if a customer is causing an issue.
- 17:43
Um, a- but, but you've gotta kind of design the skill surface area in a way that Claude can really, really work well a- and, and it's not just like point Claude at the data.
- 17:54
Mm-hmm. I see. Yeah. Thank you. All right. Any other questions? Anyone else?
- 18:01
Oh yeah. Okay, one more.
- 18:04
Thanks for the talk. Um, there was quite a few mention of evals, but, you know-
- 18:08
Yeah
- 18:08
... I'm, I'm looking at the traces, so I, you know, I, I understand the concept of traces, but where, where do the evals come in when you have that signal that says, "Hey, something broke in production"?
- 18:17
Yeah.
- 18:17
Or-
- 18:17
So, so the, so the evals typically will... The, um, the evals essentially are running and being layered on typically to the production traces, something we call online evals. Um, let me see if this one has an example here of it.
- 18:32
Um, so, so evals actually are data on the trace itself, and so the agent knows how to, uh, grab the data from traces, knows how to, um, visualize and, you know, the skills to basically pull data for, for the aggregate values of evals across the traces so that, so the, the skills that you give, uh, the harness
- 18:52
allow it to get the data on the evals from, from the traces. Um, so evals are kind of like... I view them as, at least, at least the first generation eval- evals, which are LM as a Judge, um, as a, as a AI layer that allows you to run periodically and, and assess your system, but it's like,
- 19:10
it, it, but it's adding a little bit more s- you know, pre-processed information on, on the data. That, that... And then as Signal's running, it's using data from the evals that were layered on, um, in addition to all the raw data that it has there.
- 19:24
Um, it, but it tends to be like you build an eval for a failure you've seen before a lot of times. So I have these prompt injection things that I'm trying to catch or something, or, um, or, or a failure in, in the way it's responded maybe to, to something before.
- 19:38
So they, they tend to be this like, you know, um, at least the LM as a Judge is it tends to be like this, this thing you, um, pre-set up.
- 19:47
And then you can actually create evaluators for failures. Say you find this failure that's pretty common and happening all the time, I can create an eval so I can catch it next time.
- 19:54
I just, you, you think it as like almost an, an AI, um, assessment that's always running. Uh, the other note is the LM as a Judge can run really at scale.
- 20:03
W- Every, you know, I have customers who, who lay, you know, layer LM as a Judge across, um, their full data set, uh, where, where this tends to be like, you know, um, more periodic on a lot of data.
- 20:16
So cool. Thank you. [outro music]