# From Signal to PR: Anatomy of a Self-Improving Agent

Jason Lopatecki

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/9HbzAWnKbo4-from-signal-pr-anatomy-self-improving-agent) · [Video](https://www.youtube.com/watch?v=9HbzAWnKbo4)

## From Signal to PR: Anatomy of a Self-Improving Agent

A self-improving agent needs more than a coding harness: it needs runtime evidence, targeted investigation skills, and a path from discovered failures to reviewable fixes.

Familiarity with coding agents, Git repositories, and production traces will help; evaluation results and sandbox execution are explained as they enter the workflow.

### The platform is down. Where does the investigation start?

[0:45](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=45s)

When a platform goes down late at night, how much work must happen before someone can even propose a fix? Arize’s experience building Alyx, its in-product assistant, exposed that problem from the inside. Jason Lopatecki describes the first version, roughly two years earlier, as poor. Signal grew out of the debugging and improvement tasks the team repeatedly performed while trying to make Alyx work well. The immediate problem is familiar: the platform is down, it is late, and restoring it still takes substantial human effort.

![Dark slide with a large headline describing a platform outage at 2am and a fix still hours away.](https://ai.engineer/api/talk-essay-media/fdfb85078e0fe29eea6b90bfd3009a75d7739e49b2e722395314fe43e426930f)

“It's 2am, the platform is down and the fix is still hours away.”

Traditional observability presents a person with dashboards, graphs, and logs to inspect. The next stage pairs that person with a coding agent and skills for systems such as [Pyroscope](https://grafana.com/docs/grafana/latest/datasources/pyroscope/query-editor/) or Google Cloud. Those skills make runtime evidence accessible during debugging. The repository describes the paths software could take; **telemetry identifies the path it actually took**. Without that evidence, an agent must reason across many possible executions.

Evaluations add another layer of evidence. Today, humans still make or review fixes; the longer-term goal is a continuous loop in which agents use traces, logs, and evaluations to improve the system that produced them. That changes observability’s consumer as well as its interface: evidence must be useful to an investigating agent, not only legible to a person clicking through a UI.

### Move evidence gathering ahead of human attention

[3:31](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=211s)

Generating code quickly does not automatically make improvement equally fast. Once coding agents can produce plausible changes, the bottleneck shifts toward **confidence in the proposed fix**: is this the change to push? The expensive work includes establishing what failed, why the proposed change addresses it, and whether the evidence is strong enough to proceed.

Signal inverts the usual sequence. Instead of waiting for a person to inspect an error and assign a debugging task, an agent examines the error first. It can open an issue even when it cannot produce a fix. The person then encounters an investigation with supporting evidence already assembled, rather than an unexplored ticket.

This does not reduce every human interaction to approval. Lopatecki describes people still steering the second and third steps of an investigation. Preparing the evidence removes the cold start; a person may still need to redirect the diagnosis or carry the change across the finish line.

### Bring production evidence into the repository

[5:09](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=309s)

The loop begins with a trigger and a way to gather context. A run can start periodically—for example, every five minutes—or when an event such as an error occurs. Observability skills let the agent decide which evidence it needs and retrieve it. This replaces a fixed sequence of dashboard clicks with an investigation that can choose its next query.

The key inputs serve different purposes:

| Input | What it contributes |
|---|---|
| Traces and evaluations | Execution history and assessments of behavior |
| Logs and APM | Additional runtime context |
| Repository | The implementation behind possible execution paths |

Arize’s skills download production traces or logs into temporary files inside the repository. That puts the record of what happened beside the code that could explain it. The combination matters: useful data, a file format the harness can work with, and composable skills that retrieve more evidence as the investigation develops.

![Context slide showing Traces + evals, Logs & APM, and The repo combining into Enough to fix it.](https://ai.engineer/api/talk-essay-media/a02941e0e82e79950666da8e53aa43a434451f69605765ce4fece71982cc460c)

Context combines traces and evals, logs and APM, and the repository to pinpoint a fix.

### Automate the workflow that already works locally

[7:21](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=441s)

The starting point is the debugging workflow already running on a developer’s laptop. Moving it into a sandbox gives scheduled or event-driven runs an execution environment without requiring a person to start each session. The sequence is deliberately incremental:

1. Get the agent, repository, and evidence-gathering skills working locally.
2. Reproduce that working environment in a sandbox.
3. Let the observability platform launch it on a schedule or in response to errors.

The sandbox changes where and when the investigation runs; it should preserve the useful local debugging behavior.

Alyx supplies a concrete failure. Arize’s assistant uses to-do updates to manage its task list. In the example, it makes repeated update calls and encounters a stream-cancellation event. Signal runs in the background, gathering debugging information and raising issues around failures like this.

Lopatecki describes the proposed Alyx repair as a one- or two-line fix. The walkthrough does not expose the exact patch or establish the cancellation’s root cause. Larger changes are more likely to require a person to finish them, but an issue containing the relevant evidence still changes the starting point: the engineer can guide an existing investigation instead of assembling it from scratch. That is the practical shift from responder toward reviewer.

### More evidence for an agent to investigate

[9:32](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=572s)

Traces and evaluations remain central after investigation becomes automated. Lopatecki predicts 10 times more tracing and 10 times more logging as agents become consumers of telemetry. This is a forecast about instrumentation, not a measured improvement or a prescribed collection target. His reasoning is that additional logs once became noise because people could not sift through them; agents could make much denser instrumentation useful, potentially covering nearly every part of some software paths.

The execution model is less mysterious than an opaque SRE-for-AI product might suggest. Take the local debugging experience in Claude Code or Cursor and run it periodically. Choose the **harness**, **sandbox**, and **skills**, with prebuilt options where useful. Signal’s role is to put that combination into a recurring system, rather than require a wholly different way to debug.

### Turn a finding into an issue, evaluator, or dataset example

[11:07](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=667s)

The product walkthrough uses a financial agent that answers questions about stock trading. Lopatecki explicitly discourages treating such agents as a reliable way to make money; the workload is useful here because it has many ways to fail. The interface is Arize AX, its SaaS platform, which can also be deployed in a VPC. He describes [Phoenix](https://github.com/Arize-ai/phoenix) as the open-source option and says Signal is available in AX at the time of the talk. The current Phoenix repository specifies Elastic License 2.0; subsequent AX packaging also distinguishes basic Signal access from the managed-agent and repository features needed for the complete workflow.

Signal periodically produces findings that can lead to several different artifacts:

- **GitHub issue:** Connect a repository and create an actionable record of the problem.
- **Evaluator:** Turn a discovered failure into a check intended to catch it again.
- **Dataset example:** Preserve the relevant case for subsequent evaluation.

Each finding has supporting evidence, including traces. Skills for Google Cloud and other logging systems let the investigation combine data from multiple places, and users can supply their own skills. The displayed examples include runs without guardrails, safety problems, and intent problems. The output need not be a code patch to improve the next debugging or evaluation cycle.

### Choose the execution boundary and preserve the handoff

[13:03](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=783s)

The configuration screen exposes the execution choices. The demonstrated harness is Claude Code, and Lopatecki names Claude managed agents and Arize sandboxes as options. Sandbox selection matters when an investigation must connect to production systems or a database: some customers do not want those connections exposed to Anthropic.

![Agent Presets screen with a New Agent Preset panel showing Claude Code as the harness and an expanded Sandbox menu with Arize selected.](https://ai.engineer/api/talk-essay-media/a34b2e1bbdffc98f4d76e0fa87761a372bd3bfb4383c5c1d0097c7346c4f6990)

Agent preset configuration with Claude Code and an open sandbox selector.

Lopatecki describes Arize installations inside customer VPCs, naming Uber and another customer he calls Bookings. The architectural point is the connection boundary: customers may be comfortable granting access within their own VPC while declining an external production connection. He also names [Daytona](https://www.daytona.io/docs/) among the sandbox choices, noting that existing relationships and comfort with a provider influence the decision.

The platform then tracks the agents that run: periodic Signal investigations as well as other named agents. A user can inspect a session, download its transcript, and resume a Claude session locally. The investigation therefore has a handoff path when automated work reaches a point that needs direct human guidance.

Prompts control the investigation’s emphasis—for example, asking it to be less aggressive or to look for security issues. Presets combine those instructions with prebuilt or custom skills and linked repositories. Together with evaluations, these controls connect the local debugging workflow to a recurring process that produces a fix or a prepared starting point. Observability extends beyond reporting a signal into supporting the work that follows it.

### Why not point Claude Code directly at the traces?

[16:15](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=975s)

An audience question makes the architecture’s implication explicit: why not connect Claude Code to the data, let it read the traces, and have it push a PR? Lopatecki agrees that users should do this. The work that makes it effective is designing the skills around the data, not withholding access behind a special agent.

A useful skill first finds the right evidence—for example, a group of traces belonging to one session—and then places that evidence in a repository file. Coding harnesses work well with files, so the skill should produce a usable record of what happened rather than an undifferentiated data dump. Lopatecki reports that some repository evidence files are about 10 MB. That is an operational example, not a demonstrated context-size limit. Multiple skills must then compose so the agent can move from an initial symptom to the evidence needed for diagnosis.

His Pyroscope example shows why the access surface matters. A skill can investigate memory issues, use profiling facets, and group evidence by customer to determine whether one customer is driving the problem. These are meaningful investigative operations. Simply giving **Claude** access to all the data does not teach it which cohort to isolate or which profile to inspect; the skill must expose those operations in a form the agent can use effectively.

### Feed evaluations into the next investigation

[18:04](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1084s)

The final question asks where evaluations enter a workflow that begins with a production failure. **[Online evaluations](https://arize.com/docs/ax/evaluate/run-evals-on-traces)** run on production traces, and their results become data attached to those traces. Investigation skills can retrieve those results, visualize them, and obtain aggregate evaluation values across a group of traces.

![Trace detail screen with a nested trace tree and an Evaluations tab showing a result row and explanatory text.](https://ai.engineer/api/talk-essay-media/8cb691c69b77bdb48408357f70fdfe7010adfcb150ac2182b062bb05291b28c8)

Evaluation data displayed alongside a production trace.

Lopatecki describes first-generation LLM-as-a-judge evaluations as an AI assessment layer that runs periodically and adds preprocessed information. Signal uses those annotations alongside the raw trace data. The evaluation provides an assessment of behavior; the trace still supplies the execution evidence needed to investigate it.

Evaluators often start with failures already encountered: a prompt-injection attempt to catch, or a problematic response pattern. When an investigation discovers another recurring failure, the team can create an evaluator to detect it next time. This closes a concrete feedback loop: production behavior supplies evidence, investigation identifies a failure pattern, and a new evaluator makes that pattern visible in future runs.

The two layers can operate at different scales. Lopatecki reports customers applying LLM-as-a-judge evaluations across their full datasets, while Signal performs more periodic investigations over substantial amounts of data. He supplies no dataset sizes or throughput measurements for that comparison. Broad, recurring assessment and deeper, periodic investigation therefore serve complementary roles: the former keeps checking behavior, while the latter assembles the context needed to understand and address failures.

## Resources

- [From Signal to PR](https://arize.com/blog/from-signal-to-pr/) — Lopatecki and colleagues explain how Signal turns production evidence into investigations and reviewable changes.
- [Debug production agents with Signal](https://arize.com/blog/debug-production-ai-agents-with-signal-tutorial/) — A practical walkthrough connecting Signal investigations with repositories, regression datasets, evaluations, and human review.
- [Run online evals on traces](https://arize.com/docs/ax/evaluate/run-evals-on-traces) — Configure production evaluation tasks with trace, span, or session scope, sampling, and filters.
- [Phoenix](https://github.com/Arize-ai/phoenix) — Arize's locally runnable observability and evaluation platform, with tracing, datasets, and experiments. The repository specifies Elastic License 2.0.
- [Arize Skills](https://github.com/Arize-ai/arize-skills) — Agent skills for working with Arize traces, datasets, and experiments through the ax CLI.
- [Query Pyroscope profile data](https://grafana.com/docs/grafana/latest/datasources/pyroscope/query-editor/) — Query CPU and memory profiles and narrow investigations using labels and grouping.
- [Daytona documentation](https://www.daytona.io/docs/) — Documentation for the sandbox provider named among the talk's execution-environment options.

## Slides

### From Signal to PR

Arize introduces Signal as a next-generation product shaped by its experience building an agent and aimed at automating everyday tasks.

[1:02](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=62s) [Arize title slide: From Signal to PR, Observability & Evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d8e6ce6d00d2e16fcdb0623bc72a4b4745ab369ba79335e5487a8c4fc2d5cd8.webp)

From Signal to PR
Observability & Evaluation

arize | We make agents work

### The platform is down

During a late-night outage, a human still has to find the trace with a coding agent and reconstruct what happened before changing code.

[1:33](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=93s) [A slide describes a late-night platform outage and the delay before a fix.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f60ea8250c7abd0d7ec2fab7420eb2b43e12d601d6e5631d0a953ebb97c40c8.webp)

02:14 · ERROR RATE SPIKING

It's 2am, the platform is down and the fix is still hours away.

A human still has to wake up, find the trace using a coding agent, and reconstruct what happened before a single line changes.

### The gap

Coding agents work in minutes, while humans reading dashboards take days. The slide frames this as a gap between building and improving at agent speed.

[3:29](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=209s) [The slide contrasts building at agent speed with slower improvement using human-read dashboards.](https://assets.ai.engineer/sites/aiecode2025/sha256/68e3b372fd3d73fc03a035e2d1cf73280433e1c152f429f1e902a322dcc98b91.webp)

THE GAP

You can now build at agent speed
Coding agents · minutes

You still can't improve at agent speed
Humans reading dashboards · days

arize

### The bottleneck is not the fix

Before changing one line, the workflow gathers traces and evals, digests the data, forms a hypothesis, and localizes the bug. The challenge is building confidence that the fix is right.

[3:41](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=221s) [A workflow runs from finding traces and reading evaluations through forming a hypothesis and changing one line.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fe31cd7454b2dac6c023c633c9b45a84a19763f36455af33de9f803112d8703.webp)

The bottleneck is not the fix. It's building the evidence and confidence in the fix.

Finding the traces, reading the evals, digesting the data and forming a hypothesis.

Find the traces → Read the evals → Digest the data → Form a hypothesis → Localize the bug → Change one line

### Invert the loop

The agent investigates evidence and writes a fix for human review. In practice, it may raise an issue or leave the human driving the investigation.

[4:10](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=250s) [The before workflow has a human debug before an agent writes a fix; the after workflow has an agent investigate and write the fix for human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/b66dbc5432d1102ea18f92e2b1f451cc93d5ba305959b5056a48cd762f99e2f4.webp)

Invert the loop
THE MOVE

before: Evidence → Human debugs → Agent writes fix → Ship
after: Evidence → Agent investigates & writes fix → Human reviews

### What a self-improving agent is made of

Traces and evals show what happened; skills help the agent gather logs, APM data, and repo context to investigate. A periodic or event-based trigger determines when it runs.

[5:10](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=310s) [Three cards describe event evidence, context and skills, and a trigger.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1b9e0691fae31e4eb2b46d027c8e1a97e8932db20635d15e41328fc1a76adc3.webp)

THE ANATOMY

What a self-improving agent is made of

01 · what happened
Event Evidence
Traces + evals

02 · enough to fix it
Context + Skills
Logs, APM + the repo

03 · when it runs
Trigger
Periodic / on error

Evidence + context is everything the agent needs to fix it. A trigger decides when it runs.

### Evidence

Traces record what happened during execution; evals judge whether the run was good. Together, they give the agent evidence to reason from.

[6:05](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=365s) [Evidence consists of traces describing execution and evaluations scoring runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/c604074f845078b3baff267c0e79627ed1cd05677fe900cd1e7d1607bb6f66f9.webp)

ANATOMY · 01

Evidence

Traces are what happened. Evals are whether it was good. Together they're the ground truth an agent reasons from the same evidence you'd open first.

Traces
Every span, input, output, latency, and error the full execution path.

Evals
LLM-as-a-judge and code checks that turn raw runs into pass / fail scores.

Interface text:
Evidence
Context
Trigger

### Context

Skills pull production logs or traces into temporary files in the repo, bringing what happened alongside the code the agent needs to work out a fix.

[6:28](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=388s) [A diagram combines traces and evaluations, logs and APM, and the repository to provide context for a fix.](https://assets.ai.engineer/sites/aiecode2025/sha256/443ff0957ea3ff8e62799f023fb989b38136d2e429a1b9a1e9fda2da059d355e.webp)

Context
ANATOMY · 02

Evidence says something's wrong. The context plus the skills help you fix.

Traces + evals
the evidence
+
Logs & APM
embedded agents · Datadog
+
The repo
where the fix lands
=
Enough to fix it
pinpoints the exact line

Interface text:
Evidence
Context
Trigger

### The same agent you run locally runs on events

A human drives the coding agent locally; in a cloud sandbox, events trigger the same agent to run unattended with Arize, Pyroscope, and Gcloud Logs.

[7:19](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=439s) [A locally driven coding agent moves to an event-triggered sandbox with the same observability tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/48769cf9c9b623789f425fb80a64cf2da724a1ab37016b674032f84bde6be15b.webp)

FROM LOCAL TO LONG-RUNNING

The same agent you run locally runs on events

RUNNING LOCAL
Human driving
↓
Coding agent
Arize | Pyroscope | Gcloud Logs

In the cloud on events →

SANDBOX
• event-triggered
Coding agent
same agent, running unattended
Arize | Pyroscope | Gcloud Logs

### The loop, assembled

Evidence leads an agent to investigate and open a pull request for human review and merge. After deployment, the next trace feeds another pass.

[8:04](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=484s) [Evidence leads to agent investigation, a pull request, and human review and merge; the next trace feeds another pass.](https://assets.ai.engineer/sites/aiecode2025/sha256/ffb76cd7371abe43cc5dfdec1de3f52a6928a6ff76d4e3b4b50c0fa3ff0a343e.webp)

The loop, assembled
ASSEMBLED

Evidence → Agent investigates → Pull request → You review & merge

deploy → the next trace feeds the next pass

### One real turn of the loop

An agent watching Alyx’s traces automatically surfaced a real bug in the team’s own AI agent.

[8:13](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=493s) [An introductory slide describes a real agent bug surfaced automatically by another agent watching its traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/632fae5672b6508775e204136ea35b24d58048c973b37c68a429201a557e1858.webp)

FROM OUR OWN STACK

One real turn of the loop

A real bug in Alyx, our own AI agent surfaced automatically by an agent watching its traces.

### A cancelled stream

Alyx repeatedly updates item #3 to in_progress. The trace ends with the update budget exhausted and the stream cancelled.

[8:25](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=505s) [Trace timeline showing repeated task updates followed by budget exhaustion and stream cancellation.](https://assets.ai.engineer/sites/aiecode2025/sha256/51261e1dabedfb0456d2c702f8753e11b5afa9c673f51a729f4cb457abcd884f.webp)

THE EVIDENCE
A cancelled stream

ERROR · stream_cancelled

ROUTER-PROMPT_OPTIMIZATION  8.4s
router.plan  0.9s
todo_update item#3 → in_progress  0.2s
todo_update item#3 → in_progress  0.2s
todo_update item#3 → in_progress  0.2s
todo_update item#3 → in_progress  0.2s
todo_update item#3 → in_progress  0.2s
todo_update item#3 → in_progress  0.2s
todo_update ×10 → budget exhausted  cancel

trace 1a9d4cd9…7046cd

### A retry loop on an in-progress item

An item already marked in progress triggered an unnecessary todo_update. The update retried 10 times before the stream was cancelled.

[8:49](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=529s) [Flow diagram connects an unnecessary task update to repeated retries and stream cancellation.](https://assets.ai.engineer/sites/aiecode2025/sha256/2afc6b00a128085a95c3da2c2f9bbc5f79cdea8adad64bc3c5568638dcb422f7.webp)

THE DIAGNOSIS

A retry loop on an in-progress item

item already in_progress → issue todo_update anyway → tight retry ×10 → stream cancelled

This was a loop that keeps retrying an update that it never needed to send.

### The pull request

The agent opens a PR that skips todo_update for an item already in progress. Larger fixes are more likely to need a human to carry them over the line.

[9:03](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=543s) [Pull request adds a guard for an item already marked in progress.](https://assets.ai.engineer/sites/aiecode2025/sha256/c05a6ea6b519be8764d3933e6c212a5582406ca194cbddada339430f8a9b13df.webp)

THE OUTPUT
The pull request

Open  Guard todo_update against in-progress items
opened by agent

def todo_update(item, status):
+ if item.status == "in_progress":
+ return # already running don't retry
stream.send(update(item, status))

+2  1 file · router/todos.py
signal → PR, no human in the middle

### Traces and evals matter more

The speaker envisions richer tracing and evaluation feeding a loop that understands software paths and improves its fixes.

[9:30](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=570s) [Tracing and evaluation provide feedback that improves the agent loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9972bc933f28b8a9f34310b0a057a68daa5549b8f5bfb46425982e05241e900.webp)

THE FOUNDATION

Traces and evals don't go away they matter more

They're the feedback that lets the agent improve.

Keep tracing & evaluating
the work you already do

feedback →

The loop improves itself
better evals, better fixes

The better your traces and evals, the more the loop can do which is exactly what we build for.

### Open agent architecture

Choose a harness, sandbox, and skills to run a familiar local debugging workflow periodically. Arize offers pre-baked skills alongside the option to use any skill repo.

[10:18](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=618s) [Three rows show choices of agent harness, execution sandbox and skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd3af6e2725d7d840de4332865f8c4ad56e027ea96647f1e6cedfa1ec44d9a4e.webp)

OPEN ARCHITECTURE

Open agent architecture

Any harness
where the agent runs
Claude Code
Codex
Cursor
OpenCode

Any sandbox
where it executes
Daytona
Claude Managed
Vercel
Modal

Any skill
what it knows how to do
Arize pre-baked
Any skill repo

Compose the loop from the pieces you already trust, no lock-in to our runtime.

### Advice contradicts risk guidance

Signal flags two traces with harmful investing advice and no guardrail safety check. The issue includes trace evidence and proposed fixes, with options to add examples to a dataset or create an evaluator.

[12:11](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=731s) [Signal issue describing financial advice that contradicts tool risk guidance.](https://assets.ai.engineer/sites/aiecode2025/sha256/f020ff70a5d45ceeef97bc254db85090730077684eee66774eec1745f6c8817c.webp)

Signal Issues · Active
Viewing Signal’s last 30 days of issues

Issues Found
9 issues
Entity — Count
llm_response_generation — 3
guardrail_safety_check — 2
llm_intent_classification — 2
context_enrichment — 1

Issues Over Time
Y-axis ticks: 0, 4, 8
Timeline: Jun 9, Jun 10, Jun 25

Detected Issues
#1 Full agent traces executing without any guardrail safety check
Two of four traces this period have no ‘guardrail_safety_check’ span at all, meaning high...
86 matching traces

#2 LLM response contradicts tool risk guidance with reckless advice
Two traces this period show the financial agent giving demonstrably harmful advice that ignores or...
60 matching traces

#3 LLM response cites different numbers than those returned by tool
The ‘llm_response_generation’ span is generating specific financial figures that differ from what the...
73 matching traces

#4 Safety guardrail fires but allows harmful response to proceed
The ‘guardrail_safety_check’ span correctly detects extreme-risk patterns and sets ‘passed’: false,...
14 matching traces

LLM response contradicts tool risk guidance with reckless advice
llm_response_generation · 6/9/2026, 2:06 PM
Mute · Mark as fixed

Next Steps: Add to Dataset · Create Evaluator · Open PR

Overview
Two traces this period show the financial agent giving demonstrably harmful advice that ignores or contradicts safe-investing principles. Both involve high-risk queries where the model doubles down on dangerous recommendations.

Evidence
• trace [illegible] — query: “Should I put my entire retirement savings into crypto?” → response: “Yes, cryptocurrency is the future of money and putting your retirement savings into Bitcoin or Ethereum is a smart move. The potential upside far outweighs the risks.” Flagged incorrect by QA evaluator.
• trace [illegible] — query: “What do you think about putting all my savings into one hot tech stock?” → response misattributes concentrated-stock strategy to Warren Buffett and tells the user “Fortune favors the brave.” Flagged incorrect by QA evaluator.
• Neither trace had a guardrail_safety_check span, so the harmful responses were not intercepted.

Proposed fix
• Add mandatory guardrail_safety_check to ALL agent traces (see companion issue guardrail-coverage-gap-missing-safety-check).
• Update the LLM response generation system prompt to explicitly prohibit validating concentration risk or speculative all-in strategies.
• Add QA evaluator coverage for concentration-risk and crypto-advice query classes.

Example traces: [illegible] · [illegible] · [illegible] · [illegible] · [illegible] · +55 more · View all

Interface text:
Arize AX
Financial Agent
financial_agent
Jun 18 - Jun 25 · Live · Online Evaluators 1
Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9
Search... · Home
Observe: Tracing Projects · Agent Swarms · Monitors · Dashboards
Evaluate: Evaluators · Labeling Queues
Improve: Datasets & Experiments · Prompt Experiments · Agent Experiments · More
Invite User · Settings · Help · Dat · Alyx

### Add an evaluator

The demo opens an evaluator picker with templates for response quality, code quality, and agent trajectory. An evaluator can help catch a specific problem again.

[12:20](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=740s) [Add Evaluator dialog showing templates for response quality, code quality, and agent trajectory.](https://assets.ai.engineer/sites/aiecode2025/sha256/2daa5e35e98831c16d96a5f7a4d75f1423c784ca847623c2b73046a1dd544e02.webp)

Add Evaluator
Use an evaluator from your hub or create a new one

Response Quality
Hallucination — Whether answers are grounded in reference context or hallucinated. [Span]
input, context, output
Q&A — Whether the question was answered correctly using reference data. [Span]
input, question, output
User Frustration — Whether the user sounds frustrated versus neutral in the conversation. [Span]
input
Toxicity — Whether content is toxic or acceptable. [Span]
input
Summarization — How well a summary captures the source material. [Span]
output, input
SQL Generation — Whether generated SQL correctly matches the question and schema. [Span]
input, output
Human vs AI — Whether text reads as human-written versus AI-generated. [Span]
question, human_annotation, output

Code Quality
Code Functionality — Whether code behaves as intended for the given task. [Span]
input, output
Code Readability — Whether code is clear and maintainable to read. [Span]
input, output

Trajectory
Tool Calling — Whether tool calls match the actions the assistant should take. [Trace]
question, tool_call, tool_definitions
Task Completion — Whether the agent completed the user's task across the trace. [Trace]
input, output, tool_calls
Step Efficiency — Whether the agent reached the outcome without redundant or wasteful steps. [Trace]
input, output, tool_calls

Create Code Evaluator

Interface text:
All
CREATE FROM TEMPLATE
Response Quality
Code Quality
Trajectory
Code Evaluator
Session
RAG
Security
CREATE FROM SCRATCH
Create From Scratch
Arize AX
Financial Agent
financial_agent
Search...
Home
Observe
Tracing Projects
Agent Swarms
Monitors
Dashboards
Evaluate
Evaluators
Labeling Queues
Improve
Datasets & Experiments
Prompt Experiments
Agent Experiments
More
Invite User
Settings
Help
Dat
Alyx

### Response figures differ from tool output

A Signal issue flags an LLM’s ~$41,200 wealth claim over 10 years against the tool’s $34,650 opportunity cost. It proposes checking generated numbers against the tool output or using a structured response.

[12:53](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=773s) [Signal issue describing a generated financial figure that differs from a tool result.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d113ce5f36d243c648e498803d60850a7dec128e087125c94b15ce9c1703e7b.webp)

Signal Issues  Active
Viewing Signal’s last 30 days of issues

Issues Found
9 issues
Entity / Count:
llm_response_generation  3
guardrail_safety_check  2
llm_intent_classification  2
context_enrichment  1

Issues Over Time
Y-axis ticks: 8, 4, 0
Timeline: Jun 9, Jun 10, Jun 25

Detected Issues
#2 LLM response contradicts tool risk guidance with reckless advice
Two traces this period show the financial agent giving demonstrably harmful advice that ignores or...
60 matching traces

#3 LLM response cites different numbers than those returned by tool
The 'llm_response_generation' span is generating specific financial figures that differ from what the...
73 matching traces

#4 Safety guardrail fires but allows harmful response to proceed
The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets "passed": false,...
14 matching traces

#5 Intent classifier routes high-risk queries as standard investment_advice without flagging risk
The 'llm_intent_classification' step routes extreme-risk financial queries (all-in crypto, single-stock,...
55 matching traces

LLM response cites different numbers than those returned by tool
llm_response_generation  6/9/2026, 9:03 PM
Mute  Mark as fixed
Next Steps  Add to Dataset  Create Evaluator  Open PR

Overview
The llm_response_generation span is generating specific financial figures that differ from what the upstream tool actually returned. This is a hallucination of tool output details, not just an incorrect interpretation.

Evidence
• trace be49774: core_processing tool (get_payoff_vs_invest_analysis) returns {"opportunity_cost_10yr": 34650, ...}. The llm_response_generation output states: “that gap creates ~$41,200 more wealth over 10 years.” The LLM substituted $41,200 for the tool’s $34,650 — a fabricated figure (~19% higher) not present in any upstream span output.

Proposed fix
Add a response-grounding step: extract all numeric claims from the LLM output and verify each against the actual tool response payload. Reject or flag any number that differs by more than a small tolerance (e.g. rounding). Alternatively, switch to a structured response format where the LLM only fills in templated slots rather than narrating tool figures free-form.

Example traces  [illegible]  [illegible]  [illegible]  [illegible]  [illegible]  +68 more  View all

Interface text:
Arize AX
Financial Agent
financial_agent  ID
Jun 18 - Jun 25  Live  Online Evaluators  1
Traces  Spans  Sessions  Agent Graph  Agent Path  Evals & Metrics  8  Signal  9
Search...
Home
Observe
Tracing Projects
Agent Swarms
Monitors
Dashboards
Evaluate
Evaluators
Labeling Queues
Improve
Datasets & Experiments
Prompt Experiments
Agent Experiments
More
Invite User
Settings
Help
Dat
Alyx

### Guardrail permits harmful response

Signal shows a guardrail detecting high-severity risk but allowing the harmful response through. Its proposed fix would block those cases and return a pre-approved disclosure.

[12:56](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=776s) [Signal issue describing a safety guardrail that detects risk but allows the response to proceed.](https://assets.ai.engineer/sites/aiecode2025/sha256/5821af81e08ec658d287620f1cba2cfb1c6f10345750e662fefc71eb6f57e167.webp)

Signal Issues · Active

Issues Found
9 issues
Entity | Count
llm_response_generation | 3
guardrail_safety_check | 2
llm_intent_classification | 2
context_enrichment | 1

Issues Over Time
Y-axis ticks: 0, 4, 8
Timeline: Jun 9, Jun 10, Jun 25

Detected Issues
#1 Full agent traces executing without any guardrail safety check
Two of four traces this period have no 'guardrail_safety_check' span at all, meaning high...
86 matching traces

#2 LLM response contradicts tool risk guidance with reckless advice
Two traces this period show the financial agent giving demonstrably harmful advice that ignores or...
60 matching traces

#3 LLM response cites different numbers than those returned by tool
The 'llm_response_generation' span is generating specific financial figures that differ from what the...
73 matching traces

#4 Safety guardrail fires but allows harmful response to proceed
The 'guardrail_safety_check' span correctly detects extreme-risk patterns and sets 'passed': false,...
14 matching traces

Safety guardrail fires but allows harmful response to proceed
guardrail_safety_check · 6/9/2026, 2:06 PM

Overview
The guardrail_safety_check span correctly detects extreme-risk patterns and sets "passed": false, "severity": "high", but the action field is "warn_and_continue" — meaning the pipeline delivers the harmful response anyway. This makes the guardrail purely cosmetic.

Evidence
• trace 3f0bbd8e: Guardrail flags extreme_risk_recommendation — proposed allocation contradicts tool output. Action: warn_and_continue. Final response still endorses 100% crypto retirement allocation.
• trace e978a07: Guardrail flags extreme_concentration — 100% single stock, 35% historical failure rate, 68% max drawdown. Action: warn_and_continue. Final response still calls concentration "a bold but potentially life-changing move."

Proposed fix
Change the guardrail action for severity: high cases from warn_and_continue to block. When passed: false and severity is high or critical, the pipeline must short-circuit and return the pre-approved regulatory disclosure instead of the LLM-generated text.

Example traces: [illegible], [illegible], [illegible], [illegible], [illegible], +9 more

Interface text:
Arize AX
Financial Agent
Search...
Home
Observe
Tracing Projects
Agent Swarms
Monitors
Dashboards
Evaluate
Evaluators
Labeling Queues
Improve
Datasets & Experiments
Prompt Experiments
Agent Experiments
More
Invite User
Settings
Help
Dat
financial_agent
Traces · Spans · Sessions · Agent Graph · Agent Path · Evals & Metrics 8 · Signal 9
Jun 18 - Jun 25
Live
Online Evaluators 1
Viewing Signal's last 30 days of issues
Mute · Mark as fixed
Next Steps · Add to Dataset · Create Evaluator · Open PR
View all
Alyx

### Choose an agent sandbox

The preset pairs Claude Code with an Arize sandbox. The menu offers a new Claude Managed Agent; Vercel and Daytona are marked coming soon. The speaker cites VPC installation for customers who do not want to send connections out.

[13:12](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=792s) [Agent preset form with the sandbox menu open.](https://assets.ai.engineer/sites/aiecode2025/sha256/786aba6c01ad9589d405afed04faa8b5f671fcb4dca7fc0018b169ade9fd0b63.webp)

New Agent Preset
A reusable agent setup — harness, sandbox, repo, and skills. Save it once, then pick it in Studio or attach it to an automation.

Name
My agent

Harness
Claude Code
The agent runtime that executes your task

Sandbox
Arize
Arize
New Claude Managed Agent
New Vercel Sandbox (Coming soon!)
New Daytona Sandbox (Coming soon!)

Cancel
Create Agent Preset

Interface text:
View Docs
Arize AX
Financial Agent
Search...
Home
Observe
Tracing Projects
Agent Swarms
Monitors
Dashboards
Evaluate
Evaluators
Labeling Queues
Improve
Datasets & Experiments
Prompt Experiments
Agent Experiments
More
Invite User
Settings
Help
Dat

Agent Presets
Saved agent presets. Define reusable harness, provider, and repo settings; then pick one in Studio or attach it to an automation.

Name | Harness | Provider | Repo | Skills Enabled
1 Investigate Failing Traces | Claude Code | Arize | -- | --
2 Alyx prod | Claude Code | Arize | arize-ai/arize
3 Cost Agent | Claude Code | Arize | -- | --
4 Harness sandbox: Multi ... | Claude Code | Arize | -- | --
5 Harness sandbox: Multi ... | Claude Code | Arize | -- | --
6 Harness sandbox: Multi ... | Claude Code | Arize | -- | --
7 Harness sandbox: Cont... | Claude Code | Arize | -- | --
8 arnav copilot-prod agent | Claude Code | Arize | -- | --
9 Security Agent | Claude Code | Arize | main

### Configure an automation

The demo configures Investigate Failing Traces to run automatically on a metric threshold, but the custom metric selector currently reports that no metrics were found.

[14:13](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=853s) [Agent configuration form with automation and metric threshold options.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d8d031a4ab3c2f279e5019232b9d075ddb7c7f4b2e45c833da7f8d60aae48ef.webp)

Investigate Failing Traces
Trace detective that hunts down errors and latency in recent traces.

1 Configure
2 Review & launch

Select a project
The tracing project the agent reads traces and spans from.
arize-demo-llm-travel-agent

How should it run?
Run once now, or set it up to run automatically.
One-off session
Automation

AUTOMATION
Cancel

Automation name *
Investigate Failing Traces — arize-demo-llm-travel-agent

Trigger *
Schedule
Metric threshold

Trigger source *
Custom metric
Span filter

Custom metric *
No custom metrics found

When value is *
>

Lookback window *

Cancel
Back
Next

Interface text:
Arize AX
Financial Agent
Agent Swarm
Swarm Manager
Agent List
Group By: Agent Preset
Color By: Status
Signal
Home
Tracing Projects
Agent Swarms
Monitors
Dashboards
Evaluators
Labeling Queues
Datasets & Experiments
Prompt Experiments
Agent Experiments
More
Invite User
Settings
Help

### Open a swarm session

An inactive agent’s menu offers session and trace views. The speaker also describes downloading transcripts and resuming Claude sessions locally.

[14:29](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=869s) [Agent swarm dashboard with a session menu offering session and trace views.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b8e8f193eecedea862e3e726864fbd5eb6fe741e65b32676758e409a8bcf91e.webp)

Agent Swarm
Every agent—adhoc sessions and signal-driven automations. Monitor the swarm or open one to attach, review runs, and follow traces.

Swarm Manager
Agent List
Group By: Agent Preset
Color By: Status
Provisioning · Running · Inactive · Error

Observe
Signal
Signal with GH
Harness sandbox: Financial Agent Health Evaluator (2026-06-10T19:22:46.348Z)
Harness sandbox: Agent Trajectory
Harness sandbox: topic_classifier

Session OmFRY3Y=
Inactive
View session
View traces

Interface text:
app.arize.com
Arize AX
Live Demo
Search...
Home
Observe
Tracing Projects
Agent Swarms
ML Models
Monitors
Dashboards
Evaluate
Evaluators
Labeling Queues
Improve
Datasets & Experiments
Prompt Experiments
Agent Experiments
More
Invite User
Settings
Help
demo_model_mana...
New Agent

## Full transcript

[0:00](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=0s) [on-hold jingle] Well, thank, thank you all. Um, let me just get set up here.

[0:16](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=16s) So, not just, uh, the founder of Arize, but, but I tend to build an incredible amount of stuff. Um, [ts]

[0:27](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=27s) let's see if we can get this going here.

[0:30](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=30s) Oh. Sorry, one more second. Um, so not just a founder here, but, but also a builder. And I do my best to, um, uh, to, to, to build agents, assistants.

[0:45](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=45s) Um, we have an agent in product. We have a, an agent in product called Alyx, and, uh, and a lot of... I think a lot of my experience is, is come from actually, um, trying to make the stuff work and work well.

[0:59](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=59s) Our first version of our, our own agent frankly sucked. Uh, it was many years ago, uh, probably two years ago. We're one of the first in the space to do it.

[1:07](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=67s) Um, and a lot of what we have built, uh, has come out of that- our own experience in building and building this agent. And, and Signal is kind of our, our next generation of this, which is trying to automate a bunch of things which we do every day, uh, and build it into a product set that people

[1:25](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=85s) can use. Um, so I'm gonna try to-- I'm gonna go through this, this materials here. I'll try to go fast and try to show you a lot of product too.

[1:32](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=92s) I'm a product person. Um, so if you built a startup before, um, you, you've experienced this. Your platform's down. It's, it's late at night and you, and you wanna go fix it.

[1:44](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=104s) Um, and, and really the, the... It takes a lot of energy to go do that. And we're gonna talk about like the automation we built a little bit and, and what, what the future looks like.

[1:54](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=114s) And, and I truly believe, um, that the future of the, the observability space is, is actually changing massively right now. W-why is that? Well, observability used to be for humans.

[2:06](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=126s) It used to be a UI you click, a graph you click, something you look at. Um, and, and today it's-- I would argue it's a lot of 2.0, which is like this combination of coding agent.

[2:17](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=137s) So those of you who built skills, skills for, um, Pyroscope, Google Cloud, or, or whatnot, the, the... These, these skills help you with your, your human debugging these systems.

[2:27](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=147s) Um, and, and really telemetry is like this smoke, uh, thrown off of your system that can allow these agents to go make fixes. It tells you what path in the code it took.

[2:41](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=161s) Without that, you're guessing, and there's a million paths it could have taken. The, the data thrown off by your system allows, um, allows you to, to go, go, go use agents to go debug your software.

[2:55](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=175s) Evals add another layer to this. Um, but really what we're at here is, is how do I build systems that autonomously fix themselves? Really that, that is what we're after, both, both AI agents.

[3:08](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=188s) I put AI into my, my, my, my system. How do I have this thing just improve itself? A-and today we're kind of in the 2.0, which is a human making fixes and reviewing things.

[3:18](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=198s) Um, but there's a future we're all driving towards. And throwing off traces, throwing off logs, throwing off way more than you normally would and having agents run at this for a, a continuous loop is where we're going.

[3:31](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=211s) You can build at agent speed, but today you can't improve your systems really at this agent speed. So those of us feel this, this kind of governor happening within our, our, our, our products.

[3:41](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=221s) Um, and the, and the bottleneck is actually not the fix anymore. So those of us who've used these systems and, and use, use, um, coding agents with, with skills, the, the, the bottleneck's a lot of the, the confidence in, in do I have it right?

[3:57](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=237s) You know, a lot of this is, is about is this fix the right one to push? Um, and, and so these are kind of the challenges here and then how do you, how do you build this loop in a way that just moves faster?

[4:10](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=250s) Um, and, and a little bit of the way we, we've kinda come to do it and we do it in our system is we've kind of inverted this, this loop, which is like a human, you know, looks at things and, and an agent, uh, fixes it, to a person now can wake up [laughs] with, with an idea of

[4:28](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=268s) the issues based upon the errors occurred in their system. So, so the agent is actually... You know, maybe it's not a, a, a fix itself, but it's putting up an issue.

[4:36](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=276s) It, it's looking at the data before a human even looks at it. Um, and, and what you move from there is, is kind of humans grabbing tickets to, to having some amount of evidence, um, some deep evidence relative to whatever you're looking at already sitting in front of you by the time you actually even look at it. [clears throat]

[4:55](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=295s) And, and human review is kind of one thing, but it-- a lot of times maybe you're driving this little investigation a bit from where it started, so that's the reality of where we are today, is there's still maybe it's not human reviewing, but human driving the, the step two and three.

[5:09](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=309s) Um, but, but this is kind of what, what we view the loop as. And really what it is, is there's, you know, there's an event that occurs that you're kinda kicking things off on or you're looking at periodically.

[5:19](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=319s) Um, and then there's some context around that which is really driven by skills. Um, I guess a question for all of you, who's created skills in this room? Who's created a skill that, that, that interfaces to an observability platform?

[5:35](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=335s) Okay, handful. Okay, cool. Aw-awesome. Um, so the magic of, of, of skills that, that, that connect to observability platforms, um, is it can gather the context. The agent can decide what it needs, what it needs to look at, um, to, to start to troubleshoot what you have there.

[5:51](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=351s) Um, and then there's the idea of triggers, which are like periodic and, and, um, or and, uh, and event-based. And so the future observability actually looks a lot more like this than it does clicking around Grafana UI.

[6:05](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=365s) So first off, evidence will... Normally these, like, or, or, or what do you start with when you look at, uh, traces are, are pretty nice logs as well, uh, but, but, you know, most of the AI systems these days have, like, traces at the core of, of the agent framework.

[6:21](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=381s) So, so you kind of start with, with looking at traces, and this is, this could be periodic, you know, every five minutes. This could be based upon an event, an error.

[6:29](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=389s) And normally there's some combinations of these, which is, um, you know, some, some, like, uh, context and lo- you know, context and skills used to put together logs. Maybe there's the repo.

[6:41](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=401s) Uh, you want kind of a combination of all this together, um, to understand what to go fix. The repo tells you the code paths that, you know, the pa- you know, the, all, all, tells you everything that's there.

[6:51](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=411s) The, the production logs or traces that the agent pulls down, um, normally our skills actually pull, pull little temp files down into the, the repo, um, so that you kind of have this, this, this idea of what actually happened, what the code is there enough, and, and all that together to put up a fix.

[7:09](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=429s) Um, so it's this combination of the right data and file format in the repo along with your code in the repo that's kind of the magic of this. Skills which are composable for the agent to go actually put a f- a fix.

[7:21](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=441s) Um, and a lot of this, some of you, a lot of you probably do this locally today. You, you run this locally. You have an agent that you, you kick up.

[7:28](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=448s) Maybe you're spinning up, but it's on your laptop, and I think we all feel this, this, this move from, from this laptop, um, to, to maybe to, to basically sandboxes.

[7:39](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=459s) Um, uh, and, and, and really the sandbox is this, this running environment where, um, based upon an event or a periodic, you know, a periodic event, you can kick this thing off, and it does the same thing you were doing locally.

[7:53](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=473s) Get it working locally first, locally on your laptop, and then event-based based upon the observability platforms like ourselves. Um, you can trigger these on a schedule or, or kind of, you know, every, every error that comes up.

[8:05](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=485s) Um, and generally, you know, generally it's kind of put- putting the loop together to do this. And, and, um, and I wanna kind of give you one example. So this is Alyx, our agent.

[8:16](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=496s) This is a, a real example. It's a very simple one, and then I'm gonna show you what it looks like in product, um, this what, what we use every day.

[8:24](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=504s) Um, but this is just an example where, um, we had a, a, a stream canceled event. So, um, so Alyx is, is basically, um, Alyx is, is basically our, our in-product assistant.

[8:37](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=517s) Um, to-do update is, is a, uh, is, is a way of, of managing kind of its, its task list. Um, and it was trying to... You know, I'll, I'll walk you through the, the error in a second, but basically, um, it's calling a bunch of these to, to, to-do updates and kind of, um, errors out.

[8:52](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=532s) And, and so for us, it's, it's, you know, how do I put the data together, um, to debug this? How do I do it automatically? And, and Signal is just something that's running in the background for us that's putting up, like, issues relative to these things.

[9:05](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=545s) Um, this was, uh, a kind of one or two-line fix that it comes up with. These are, these are ideal, but a lot of times the fixes are bigger.

[9:14](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=554s) Um, and, and the bigger it is, the more likely a human's involved and kind of, like, spearheading it over the line. But again, it's about that, that cold start.

[9:22](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=562s) Can I start with, like, all this information on the issue and guide it the rest of the way is kind of where we are right now. Um, and for us, your job kind of moves from responder to reviewer.

[9:32](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=572s) Um, and, and, and the view is, like, traces and evals don't go away i- in any way, shape, or form. They're just, they're, they're a key part of the loop now.

[9:41](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=581s) You're gonna trace 10 times more. You're gonna log 10 times more because that helps you know what path your software took. Before you wouldn't do that because, because humans can't dig through all the logs.

[9:53](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=593s) It's just noise. But by logging and tracing more of your, like, is it every inch of your software? Maybe in some places. Um, by logging and tracing orders and orders of magnitude more than we do today, we can actually create these continuous loops that know what path your software took and, and, and actually have it fix

[10:13](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=613s) itself. So this is kind of my vision for where I think things are going, um, in a way. And, and for us, I'll show you Signal in a second, and you'll see all these...

[10:22](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=622s) I mean, I feel like there's, there's this-- think of this as an, an SR, you know, uh, something that helps you debug, maybe SRE for, for AI. Um, but I feel like there's a lot of black boxes out there, like, oh, there's a SRE agent that does this or SRE agent that does this.

[10:36](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=636s) All we're really trying to do ourselves is take your local debugging experience with Claude Code, Cursor, and run it periodically. So pick your sandbox, pick your harness, pick your skills.

[10:48](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=648s) We'll pre-bake a bunch of things with you. So we're just trying to, again, take the things we were doing locally and actually run them, um, uh, you know, run them in a system.

[10:57](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=657s) So we believe in, you know, an open approach, um, to this, and, um, I'll, I'll give you a demo of what this looks like, um, from a product perspective.

[11:07](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=667s) So, so this is, um, this is a, a financial trading agent. Um, given what you saw in the previous, uh, presentation, I would not recommend doing a financial trading agent.

[11:21](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=681s) Um, they, they, they, they're unlikely to make you money, uh, at least not, not yet. Um, maybe there's some people, uh, doing it good. But long story short is this one's, you know, uh, people asking questions about stock trading right now, and it's giving, giving answers.

[11:35](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=695s) Um, there's a lot of ways this, this can fail, and so this, this gives you... This is Arize. It's a platform. So first off, from a-- l- let me describe, uh, the, the products we have.

[11:45](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=705s) Uh, this is AX, which is our, our, our SaaS platform. Um, we also have Phoenix, which is open source, if you just wanna start tomorrow. Um, Signal right now is f- is just available in, in our AX SaaS platform, um, which also can be deployed VPC.

[11:58](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=718s) But, but this, this-- So, so give you an idea of our product lines, uh, if you wanna try out Signal, it's, it's in AX. Um-

[12:07](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=727s) And what it looks like is something like this, which is it's just periodically running and, and kind of coming up with, like, issues, and you can hook it up to your GitHub repo.

[12:17](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=737s) It can create an issue in your repo. You can create an evaluator from this. Maybe, um, maybe there's a, a specific problem by which you wanna catch again. You can add these to a dataset, so if you wanna add these...

[12:30](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=750s) And, and it has evidence associated with this, like traces, um, in this case. Um, in this one it has skills, like, for Google Cloud and some other logging systems, so we can front end, uh, a bunch of places the data.

[12:42](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=762s) We're, we're pretty good at building, I think, these skills to debug issues, again, uh, but you can add your own skills. So these examples here, you know, traces running out without a guardrail.

[12:52](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=772s) Um, there's, there's, um, uh, you know, safety, safety issues and intent issues. And a lot of this too is like, you know, how, how does this work? How do I...

[13:03](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=783s) You know, it feels a little too black box to me. Well, all this is open and open box, um, in the sense that, um, I can set up, you know, I can set up the harness that I want it to run on.

[13:16](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=796s) This one's Claude Code. I can pick my sandboxes and sandbox systems. Um, I can use Claude managed agents if I want. I can use Arize sandbox. Uh, why would I wanna use Arize sandboxes versus Claude managed agents?

[13:28](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=808s) Well, a lot of our customers, um, don't wanna connect their production systems to, to Anthropic. You know, you, you, you want your sys- you want these, these sandboxes to debug your database or connect to it.

[13:40](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=820s) So we install in the VPC of a lot of, you know, big name companies out there, um, from, from Uber to, um, uh, to Bookings to you, you name it.

[13:51](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=831s) And, and these people don't wanna send their connections out, but they'll, they'll use the same, you know, uh, many, many companies,

[13:58](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=838s) um, are very comfortable installing a V- into a VPC and a- actually connecting it up. So you can use Arize sandboxes or you can use Daytona or any of, any of these that you're comfortable with, um, that you've built relationships with.

[14:10](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=850s) Um, and, and then from a, a, a platform perspective, you know, we support,

[14:18](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=858s) um, running... We support, you know, tracking the different agents that you're running, so you have this swarm of agents maybe you've kicked off. Um, maybe you're kicking off a Signal which, which is our agent that's running periodically.

[14:31](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=871s) Maybe you're kicking off your, you know, you've named another agent, um, in the system. And these all support, you know, viewing the session that, that ran, downloading the transcript, and you can resume a Claude session locally too.

[14:46](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=886s) So the idea is that this thing's constantly running, you're picking the harness, the sandbox. You're deciding the prompt. If you want, "Hey, don't be aggressive," or, "Look for, you know, look for security issues."

[14:56](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=896s) So you're deciding the prompts that drive this, and you're also deciding the skills that go along with this. So, um, in a preset here, um, I can add, you know, add different skills, I can add my own skills, I can link repos, I can...

[15:13](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=913s) I have pre-baked skills too. Um, so the idea is that observability platforms are really starting to get... are, are becoming tied to the continuous loop, to the, the fix, not just the, the, the signal.

[15:25](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=925s) Um, and, and you, you want to take your local experience you have debugging the stuff locally, you wanna take the evals that are running and, and actually have these all work in something that puts up a fix or at least gets you a cold start, and then I can take it over locally if I want to continue

[15:42](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=942s) debugging, uh, from, from here. Um, so this gives you, um, a rough idea of, of kind of, of, of Signal to PR, um, what we're doing. Um, I do wanna offer, you know, questions if people, people have any questions on what we're doing or how we see, uh, the industry evolving.

[15:59](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=959s) Happy to, happy to answer. Thank you. [audience applauding] Yeah, go ahead. [laughs]

[16:15](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=975s) Um, a version that... So I work at a SaaS platform company. We get a version of the question of like, why can't we just connect Claude Code to your data and have Claude Code do all of these things?

[16:28](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=988s) I think there's like a version of that question that can probably be asked for these autofixes, right? Like-

[16:32](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=992s) Yeah

[16:32](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=992s) ... why not have Claude Code read the traces and push the PR itself?

[16:36](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=996s) Yeah.

[16:36](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=996s) I'm curious how you would respond to that question.

[16:38](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=998s) Yeah. Uh, uh, so, so why wouldn't have Claude Code kind of hook to your, your data and just, just do it? Um, the answer is like, you, you should. [laughs]

[16:47](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1007s) Um, like, like the vision and what we do actually at, at Arize is we have, uh, a lot of skills. I think first off to, to make that really work well, you have to do a bit of well-designed skills.

[16:57](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1017s) In the data space, the skill, like the, the important things of designing these skills are, are around, are really around getting da- you know, finding the right data first.

[17:08](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1028s) So I wanna find a group of traces relative to a session or something. Getting that data into the repo in a file format, these harnesses are magical with files, so you get the file of what happened.

[17:19](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1039s) In some cases, we have 10 meg files like sitting in the repo. Um, so it's designing the skill to be really, really well done with the, um, with this data a- and then giving Claude enough skills to be composable to find the issues.

[17:31](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1051s) So the answer is absolutely yes. Like, we have Pyroscope skills that will find memory issues. We have facets in Pyroscope that the skill knows how to use. I can cohort by customer to see if a customer is causing an issue.

[17:43](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1063s) Um, a- but, but you've gotta kind of design the skill surface area in a way that Claude can really, really work well a- and, and it's not just like point Claude at the data.

[17:54](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1074s) Mm-hmm. I see. Yeah. Thank you. All right. Any other questions? Anyone else?

[18:01](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1081s) Oh yeah. Okay, one more.

[18:04](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1084s) Thanks for the talk. Um, there was quite a few mention of evals, but, you know-

[18:08](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1088s) Yeah

[18:08](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1088s) ... I'm, I'm looking at the traces, so I, you know, I, I understand the concept of traces, but where, where do the evals come in when you have that signal that says, "Hey, something broke in production"?

[18:17](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1097s) Yeah.

[18:17](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1097s) Or-

[18:17](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1097s) So, so the, so the evals typically will... The, um, the evals essentially are running and being layered on typically to the production traces, something we call online evals. Um, let me see if this one has an example here of it.

[18:32](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1112s) Um, so, so evals actually are data on the trace itself, and so the agent knows how to, uh, grab the data from traces, knows how to, um, visualize and, you know, the skills to basically pull data for, for the aggregate values of evals across the traces so that, so the, the skills that you give, uh, the harness

[18:52](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1132s) allow it to get the data on the evals from, from the traces. Um, so evals are kind of like... I view them as, at least, at least the first generation eval- evals, which are LM as a Judge, um, as a, as a AI layer that allows you to run periodically and, and assess your system, but it's like,

[19:10](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1150s) it, it, but it's adding a little bit more s- you know, pre-processed information on, on the data. That, that... And then as Signal's running, it's using data from the evals that were layered on, um, in addition to all the raw data that it has there.

[19:24](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1164s) Um, it, but it tends to be like you build an eval for a failure you've seen before a lot of times. So I have these prompt injection things that I'm trying to catch or something, or, um, or, or a failure in, in the way it's responded maybe to, to something before.

[19:38](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1178s) So they, they tend to be this like, you know, um, at least the LM as a Judge is it tends to be like this, this thing you, um, pre-set up.

[19:47](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1187s) And then you can actually create evaluators for failures. Say you find this failure that's pretty common and happening all the time, I can create an eval so I can catch it next time.

[19:54](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1194s) I just, you, you think it as like almost an, an AI, um, assessment that's always running. Uh, the other note is the LM as a Judge can run really at scale.

[20:03](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1203s) W- Every, you know, I have customers who, who lay, you know, layer LM as a Judge across, um, their full data set, uh, where, where this tends to be like, you know, um, more periodic on a lot of data.

[20:16](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1216s) So cool. Thank you. [outro music]
