The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize

Loading slides…

Read the talk

The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents

Selected presentation frame from The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize at 184 secondsOpen full source frame
The displayed refund exchange moves from order forty to the customer’s correction, “No, fourteen.”

Fuad Ali shows how a refund call can appear to recover in text while still failing in audio and execution—and how linked traces, audio evaluations and replayed experiments make those failures easier to find and fix.

From a talk by Fuad Ali

At a glance

Ideas worth remembering

  • A correction in the transcript does not prove that the action changed: the refund example still processed order forty instead of fourteen.

  • Common audio conventions make provider events queryable; session IDs connect those events into a timeline of speech and tool activity.

  • Evaluate audible behavior and executed outcomes together. Tone, response latency, interruptions and task success answer different questions.

  • Attach evaluations to relevant spans so failures can be filtered, inspected and used to trigger investigations.

  • The proposed self-healing workflow tests development fixes by replaying failed traces and gives a human reviewer comparison evidence before approval.

A readable transcript can hide a broken conversation

Voice agents can fail in ways that their transcripts barely register. A response arrives late. The assistant talks over a correction. A number sounds close enough to another number that the wrong action follows. Fuad Ali, a product manager at Arize AI working on experimentation, opens with this debugging problem: voice needs observations that preserve how a conversation unfolded, alongside what its participants said.

The stakes extend from customer support to drive-through ordering. Ali jokes about getting six hundred cheeseburgers at McDonald’s at two in the morning, but the underlying problem is ordinary: spoken requests lead to actions, and a plausible response does not establish that the action was right. Latency, turn-taking and transcription drift each create a separate way for that interaction to break.

The central example is an AI-generated refund call. The customer asks for order fourteen. The agent announces a refund for order forty. The customer corrects it: “No, fourteen.” A final confirmation follows. Read as consecutive lines, the exchange invites a reassuring interpretation: the agent heard the correction and completed the intended refund.

The call’s timing undermines that interpretation. There are 2.4 seconds of dead air before the first response, the agent talks over the caller, and fourteen is misheard. Its flat, robotic tone adds another audible problem. Text preserves the words of the correction without preserving whether the agent gave that correction room to change the interaction. The refund example therefore starts with a useful distinction: an apparent conversational recovery is something to investigate, rather than proof of a successful action.

0:120:42

Keep audio, transcript and execution together

Debugging the call requires three views in the same place: playable audio, the transcript and the execution trace. The session must include multiple user and assistant turns, so a pause or interruption can be understood in context. Arize AX’s session view brings these observations together with latency and evaluation results.

Each observation answers a different question:

  • Audio and turns: Who spoke, how did they sound, and where did speech overlap or stop?
  • Execution spans: What inputs, outputs and tool calls occurred during the interaction?
  • Metrics and evaluations: How long did the response take, what did audio tokens cost, and which interruption or quality checks failed?

The links between these observations matter as much as their presence. A list of events leaves the engineer to reconstruct the conversation; a linked trace lets the engineer inspect a particular turn alongside the work that produced it.

This carries familiar agent-development practices into voice. Coding agents already have tool calls, handoffs, sub-agents, harnesses and evaluations. Voice adds the need to preserve audible behavior and timing within that same debugging workflow. An interruption is part of the interaction’s behavior, even when every sentence looks reasonable on its own.

3:404:10

Normalize provider events, then stitch them into a session

OpenInference supplies the conventions underneath this view. Its open-source, OpenTelemetry-compliant semantic conventions give generative-AI activity a common representation. For real-time audio, the engineering work includes mapping WebSocket events into recognizable concepts: session lifecycle, audio input, transcripts, conversation items, output, response lifecycle and token counts.

A shared schema makes events from providers such as OpenAI Realtime and Google Gemini Live queryable in the same way. Instead of writing a different parser for each vendor’s event stream, the instrumenter normalizes the supported provider events. That helps both during an incident—when filtering and sorting must find the relevant interaction among many traces—and during a provider change, when engineers need to keep checking the same system behavior. The conventions can also send observations to another backend, including Grafana; this tracing approach does not require Arize’s analysis interface.

Normalization alone does not reconstruct a call. A session ID stitches turns and WebSocket events into one timeline, with tags identifying user and assistant activity. Playing audio directly from that trace makes it possible to inspect who spoke when, where the agent paused, where either participant was cut off, and when a tool ran.

How do vendor-specific events become a conversation an engineer can inspect? The diagram separates two jobs: common conventions make events comparable, while session identity connects them into a call. The resulting view preserves both the conversation timeline and its execution relationships.

A weather-agent preview illustrates the practical payoff. The session view places a weather tool call alongside the audio, so a tool failure can be investigated at the point where it affected the conversation. The engineer can follow the session’s trace tree rather than reconcile disconnected records from different systems.

From provider events to a replayable session

Provider event streamsReal-time events from OpenAI Realtime or Google Gemini Live.
OpenInference conventionsRepresent audio, transcripts, lifecycle events and token counts in a common queryable schema.
Session timelineA session ID stitches events and turns together; tags identify user and assistant.
Audio beside the trace treeInspect pauses, cutoffs and tool calls within the whole conversation.
Semantic conventions normalize the events; session identity connects turns, audio and tool activity for inspection.
Read the diagram as text
  • Provider event streams. Real-time events from OpenAI Realtime or Google Gemini Live.
  • OpenInference conventions. Represent audio, transcripts, lifecycle events and token counts in a common queryable schema.
  • Session timeline. A session ID stitches events and turns together; tags identify user and assistant.
  • Audio beside the trace tree. Inspect pauses, cutoffs and tool calls within the whole conversation.
  • Provider event streams → OpenInference conventions: Normalize provider events.
  • OpenInference conventions → Session timeline: Group by session identity.
  • Session timeline → Audio beside the trace tree: Replay and inspect linked activity.
5:406:10

Check the action, and evaluate the audio itself

Returning to the refund call, the session view changes the diagnosis. The first turn exposes the 2.4-second wait for audio. The third turn exposes the assistant speaking over the caller. The tool call establishes the consequential outcome: the refund was processed for order forty, rather than fourteen. The customer’s correction existed in the conversation, but the executed action still targeted the wrong order.

These observations separate failures that a single success label would blur. The call had a responsiveness problem, a turn-taking problem and an incorrect transaction. Scrubbing the audio beside the tool activity lets an engineer locate each one. It also prevents the agent’s final confirmation from serving as the only evidence that the task succeeded.

Audio evaluations turn those observations into repeatable checks:

  • Tone and sentiment: Classify emotion from the actual audio, including positive, neutral or negative tone. Words alone can miss audible frustration.
  • Response latency: Check time to first audio against the service-level expectations of a customer-service interaction.
  • Conversation accuracy: Flag interruptions and transcription drift, including cases where spoken content and its textual representation diverge.
  • Task success: Check whether the interaction accomplished the requested task, rather than relying on a fluent confirmation.

Audio-capable models can judge the underlying recording. A transcript-only LLM judge has no direct access to the pauses, overlap or vocal delivery that these checks need. Access to audio supplies the missing evidence; the talk does not establish the accuracy of the resulting sentiment classifiers.

Some checks need evidence beyond either audio or text. An “agent as a judge” can use tools and third-party systems to gather evaluation context. Ali’s example is a role-based access-control check: did the agent access the system it was supposed to have access to? This makes evaluation an investigation with its own tool use, rather than only a model’s judgment of a conversation.

8:409:10

Attach scores to the spans that need investigation

Arize’s skills provide a natural-language route to configuring an audio evaluator. The example request asks Claude to create a sentiment evaluation that catches frustrated tone on audio spans. Underneath that request, the CLI queries spans, analyzes traces, filters for audio-specific spans, creates and deploys the evaluator, and applies it to the system’s traces and spans.

The operational work continues after evaluator creation. Incoming interactions need scoring, and engineers need to query the results at scale. Ali frames querying billions of spans with sub-second latency and running live evaluations as reasons to use a platform rather than build the whole workflow yourself. These are the scale requirements he describes, not a benchmark demonstrated in the recording.

Putting evaluation outputs on the relevant spans keeps a score beside the audio and execution that explain it. Engineers can filter directly for failures and inspect them without searching a separate evaluation system. That placement also gives monitors something concrete to act on: a failed evaluation can trigger an investigation agent to examine misrepresented tool calls, uncovered edge cases or a missing tool.

11:3912:10

Replay failed interactions against a proposed fix

The ending expands debugging into an observe, evaluate, improve loop. Observations preserve the interaction; evaluations attach judgments to it; fixes change the system; another experiment checks whether the change helped. Arize’s newly released agent experiments are described as “Postman with tracing”: a way to exercise an endpoint while retaining the trace needed to understand its behavior.

Ali then sketches a future workflow for an SRE agent. Live audio is traced and scored. The agent investigates failures, proposes a fix, stands up the changed endpoint in development and replays the failed traces against it. A fresh trace checks whether latency decreased. This complete self-healing workflow is presented as a future vision, rather than a demonstrated autonomous repair in the recording.

The hypothetical repair has a specific cause: tool results were too large, so the agent received too much information and its context became overloaded. The proposed change truncates tool results. Replaying the failed interactions against the development endpoint would then test the claimed latency improvement, and a pull request would present the problem, change and report card for human approval or rejection. This is a separate example from the refund call; it does not establish what caused that call’s 2.4-second delay.

What turns a proposed repair into something a reviewer can assess? The cycle below keeps replay between implementation and approval. The important relationship is that the original failure supplies the test case, while a new trace supplies evidence about the changed endpoint.

The immediate starting point is smaller: add the OpenInference instrumenter, trace audio, check time to first audio and interruptions, and evaluate whether calls accomplish their intended tasks. Tone and latency evaluations are useful first checks. Ali closes by asking for the same engineering rigor applied to coding agents and agents that execute financial actions—even when the voice agent is only taking a cheeseburger order. The task may sound mundane; the system still needs to hear the request and execute the right action.

The proposed repair loop keeps failed traces as test cases

Trace and evaluate callsCapture incoming audio interactions and score them.
Investigate failuresAn SRE agent examines traces and hypothesizes a cause.
Change the development endpointThe example proposes truncating oversized tool results.
Replay failed tracesTrace the changed endpoint and compare latency.
Review the pull requestA person approves or rejects the proposed repair and its report card.
In Ali’s future workflow, investigation leads to a development fix, replay produces comparison evidence, and a person reviews the resulting pull request.
Read the diagram as text
  • Trace and evaluate calls. Capture incoming audio interactions and score them.
  • Investigate failures. An SRE agent examines traces and hypothesizes a cause.
  • Change the development endpoint. The example proposes truncating oversized tool results.
  • Replay failed traces. Trace the changed endpoint and compare latency.
  • Review the pull request. A person approves or rejects the proposed repair and its report card.
  • Trace and evaluate calls → Investigate failures: Failed interactions.
  • Investigate failures → Change the development endpoint: Propose and implement a repair.
  • Change the development endpoint → Replay failed traces: Test the changed endpoint.
  • Replay failed traces → Review the pull request: Present comparison evidence.
  • Review the pull request → Trace and evaluate calls: Continue the improvement loop.
13:3914:09

Resources

Transcript download
Full transcript

Loading transcript…