Engineering voice agents: Latency, quality, and scale

What would it take for an AI to handle a phone call without making the caller wait, repeat themselves, or correct its mistakes? Rishabh Bhargava approaches that question from infrastructure: he introduces himself as Together AI’s voice AI lead and previously co-founder and CEO of Refuel, acquired by Together. Together supplies training compute and inference infrastructure; Bhargava describes a platform serving more than a million developers and hundreds of companies, including Cursor and Decagon. The engineering path runs from today’s multi-model pipelines through their individual components, deployment trade-offs, and emerging speech-to-speech systems.

Consider calling customer support to check an order or change a reservation, or calling a doctor’s office to book an appointment for a family member. The task may be straightforward; getting through can still mean waiting on hold. Bhargava points to billions of human-handled calls each year as an opportunity for agents to do useful work.

Voice also opens a different interface to computers. People learn to speak before they learn to read, and the same conversational habit can extend from ChatGPT’s advanced voice mode to talking directly to Cursor or Claude Code. The opportunity in 2026 is to make these interactions rich and dependable enough for everyday use.

Real-time conversation leaves little room for delay. Bhargava gives roughly 300 ms as a human conversational response heuristic, says AI delays above 500 ms become noticeable, and warns that one-to-two-second pauses can cause callers to hang up. These are his design heuristics, rather than measured abandonment thresholds for a specified workload. Their practical implication is clear: latency belongs in the product requirements from the beginning.

Speed alone cannot complete a call. The other requirements are equally consequential:

• Intelligence: Follow complicated instructions, resolve ambiguity, and call tools that connect the agent to real-world actions.
• Naturalness: Speak the caller’s language, use an appropriate accent, pronounce names correctly, and convey emotion suited to the situation.
• Reliability: Preserve the experience as a one-caller demonstration grows to hundreds, thousands, or ten thousand concurrent calls.

Voice is an AND problem: a fast agent that cannot do the job, or a capable agent that becomes unreliable under load, still fails the caller.

Bhargava describes the cascading architecture as the dominant production approach at the time of the talk. An orchestrator such as Pipecat, LiveKit, or a homegrown system coordinates the streaming path:

1. Receive audio chunks from the caller.
2. Feed those chunks into speech-to-text to produce a transcript.
3. Pass text to an LLM, which decides whether to call a tool and what response to produce.
4. Feed the response text into text-to-speech.
5. Stream generated audio chunks back to the caller.

Each component has a distinct job, but the caller experiences their combined behavior as one conversation.

Speech-to-text is the agent’s first opportunity to get the conversation wrong. Its basic quality measure, word error rate, compares the generated transcript with a reference. Bhargava cites approximately 6% WER for state-of-the-art models on unspecified open benchmarks, with results varying by use case. Aggregate accuracy is only part of the requirement: a mistaken customer name or drug name can pass through the LLM and emerge unchanged in synthesized speech. Downstream components cannot reliably recover information that transcription has already lost.

The latency measure is time to complete a transcript: after the user stops speaking, how long until the finished text is ready for the LLM? Bhargava reports approximately 100 ms P90 transcript-completion latency for some models running on Together. He does not specify the models, workload, hardware, or endpointing conditions, so this is a reported serving result rather than a deployment guarantee.

Before measuring that interval, the system must decide whether the user has actually finished. A pause may mark the end of a turn, or it may mean the caller is thinking and will continue. Starting the agent’s response too early turns a recognition decision into an interruption. Turn detection remains partly unsolved, and both recognition and turn-taking need to work for the languages customers actually use.

Recognition architecture is also moving from batch processing toward native streaming. Whisper is the canonical batch-oriented example: Bhargava describes its training on 30-second clips and the work needed to adapt it to conversation. That window does not require an application to wait for 30 seconds of new speech; wrappers can chunk audio, pad it with silence, make repeated calls, and stitch the results together. The cost is additional orchestration around a model not originally organized around small incoming increments.

The alternative he describes is a recent, unnamed NVIDIA model whose encoder supports variable look-ahead, from roughly 80 ms to about a second. The encoder also caches activations. As new audio frames arrive, it reuses earlier computation instead of repeatedly performing the same heavy work.

The distinction is architectural: streaming is part of how the encoder operates, rather than only a behavior assembled around batch calls.

For the LLM, the critical streaming measure is time to first token, or TTFT. The first generated text lets the next stage begin preparing speech; waiting for the whole response would waste that opportunity. Bhargava’s rough target is 200–300 ms TTFT, which he associates with models in the 8–30 billion parameter range. This is a model-selection heuristic, not a limit independent of serving hardware, prompt length, or load.

The trade-off runs in both directions. A larger model may consume too much of the latency budget before speech synthesis even starts. A smaller model may respond quickly but fail at the instructions and tool calls that make the agent useful. The right choice is the smallest model that meets the workflow’s capability requirements within the available time.

Text-to-speech has two separate speed requirements. Time to first audio, or TTFA, measures the delay from receiving text to producing the first audio chunk that can be streamed back. Real-time factor, or RTF, measures whether generation can keep pace with playback:

In Bhargava’s example, five seconds of processing produces ten seconds of audio, giving an RTF of 0.5. An RTF below one means generation is faster than playback, helping avoid buffering once speech begins. A low startup delay and sufficient sustained generation speed solve different parts of the experience.

Quality still requires listening. Objective measures help, but Bhargava recommends hearing samples from the specific models and voices being considered and deciding whether they deliver the experience customers should have. Evaluate naturalness across voices, exact pronunciation of customer and product names, and emotional control. Some models expose tags for emotions such as happy, angry, or sad; language coverage remains another independent requirement.

Component metrics must fit inside a system-wide latency and cost budget. Bhargava’s rough ordering for both is the LLM first, followed by text-to-speech, then speech-to-text. Those engine measurements describe how long models take to produce outputs. They do not include all the network time spent moving requests and results between an orchestrator and models in different data centers.

Autoscaling also needs to respect conversational time. Scale up earlier and more aggressively than an asynchronous service: queueing that might be acceptable for a background job becomes dead air on a call. Scale-down has the opposite constraint. Even when traffic falls at night or on weekends, existing conversations may hold stateful, long-lived connections to model servers. Drain those conversations to completion instead of arbitrarily terminating the pods that serve them.

Deployment location serves two purposes. Putting the system near callers reduces travel time for their audio. Supporting the required regions also allows the service to meet data-residency requirements, including those relevant to European deployments. Capacity planning and regional placement therefore belong to the voice architecture, not just its hosting configuration.

Once model engines are fast, their physical placement can become a substantial part of the remaining delay. Bhargava’s illustrated setup uses roughly 100–200 ms engine times to first token or audio, with 75 ms of network latency between models and the orchestrator. He uses U.S. West–Europe as an example of meaningful distance and notes that networking conditions can push latency higher.

The proposed change is to colocate the models, and potentially the orchestrator, in the same data center or building. In Bhargava’s illustrated optimized pipeline, reducing network latency from 75 ms to 5 ms produces about a 30% reduction in overall latency. The slide compares total time to first audio of approximately 710 ms without colocation and 500 ms with colocation. Those totals belong to this example; they are not a general percentage improvement for moving any voice system into one region.

This is why detailed observability matters even after the model choices look settled. A system can have good engine metrics and still spend an avoidable portion of the caller’s wait crossing networks. At conversational timescales, even ten milliseconds is worth being able to locate.

A different architecture replaces the speech-to-text, LLM, and text-to-speech chain with one speech-to-speech model. It still has to follow complicated instructions and call functions, but fewer model boundaries need orchestration. Bhargava points to the OpenAI Realtime API and NVIDIA’s Nemotron 3 VoiceChat; the latter was an early-access model at the time of the talk.

Bhargava reports that instruction-following and tool-calling weaknesses have limited adoption in the production workflows he discusses. Teams may try a speech-to-speech model, spend time prompt engineering around its failures, and eventually return to a pipeline. His expectation is that these capabilities will improve. The attraction is not only fewer components: avoiding a text-only intermediate representation can preserve tone, emotion, and hesitation for the model’s next decision.

Full duplex means the model can generate audio while it is still receiving audio. That supports backchannels such as “I see” and “Uh-huh” while the customer continues speaking. It also creates a more direct route to handling interruptions and barge-ins, which require additional coordination in a cascading pipeline.

The architectural promise is substantial, but dependable voice interfaces still require engineering around the model.

The first audience question asks what evaluation score makes voice-driven function calling production-ready. In a pipeline, assuming recognition and synthesis are satisfactory, evaluate the LLM’s tool calls much as in a text application. Separate structural validity—whether the output is parsable and has the required form—from semantic correctness—whether it calls the right tool with the right arguments. Bhargava recommends tool-call structure very close to 100% correct; the required semantic-correctness threshold depends on the use case.

When a small model is fast enough but insufficiently reliable at the workflow, increasing model size is not the only option. Bhargava reports customers fine-tuning smaller LLMs on use-case-specific data to improve tool calling while staying within the latency-driven model-size budget.

A follow-up question makes colocation concrete for cloud users. Imagine an agent server in London calling an LLM endpoint hosted in the United States: each exchange must cross the Atlantic and return. Bhargava uses OpenAI as a hypothetical provider in this example, not as a verified statement about every endpoint’s location. Running an open-source model in the agent’s own data center replaces that long-distance exchange with local data-center traffic. Physical distance matters alongside the rest of the network configuration.

Another audience member describes a concrete failure: an agent offers an unauthorized 8% discount. Adding a classifier or guardrail can catch the problem, but where does that component belong, and what happens to latency? The three-model chain is only a reference architecture. Production systems often introduce additional decisions around it.

Two placements serve different purposes:

• Before the main LLM: A classifier can route a request to the model responsible for refunds or the one responsible for order tracking.
• After LLM generation: An output guardrail can check the proposed response before the system produces it for the caller.

Each added component needs an explicit latency allocation, an SLA, and independent scaling where necessary. Bhargava offers no shortcut that makes those extra checks free.

The audience member sharpens the problem: once the agent has spoken the unauthorized offer, a later classifier cannot take it back. An apology or attempted revocation is a poor substitute for preventing the claim. Bhargava agrees that the check must happen before invoking text-to-speech. For the discount example, the ordering can be expressed with a small Python gate: a proposed offer reaches speech generation only after authorization succeeds.

from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class DiscountOffer:
    percent: Decimal

async def speak_discount(
    offer: DiscountOffer,
    authorize: Callable[[DiscountOffer], Awaitable[bool]],
    synthesize: Callable[[str], Awaitable[bytes]],
) -> bytes | None:
    if not await authorize(offer):
        return None
    return await synthesize(
        f"I can offer you a {offer.percent}% discount."
    )

For DiscountOffer(Decimal("8")), a failed authorization returns no audio and never calls synthesize. The response remains a proposal until the check passes; the function illustrates the pre-speech boundary, while the authorization policy is supplied separately.

Bhargava then describes a thinker–talker pattern for separating conversational responsiveness from heavier work:

1. A small conversational LLM receives the transcript and gives a short acknowledgment, such as “Let me think about it.”
2. It delegates through a tool call to a larger model with the relevant instructions, tools, and potentially more guardrails.
3. The larger model produces the substantive response, which feeds text-to-speech.

This keeps the conversational role distinct from the deeper task execution. It also adds components whose reliability and timing must be observable individually.

The final question returns to speech-to-speech: removing the transcript from the main response path does not remove the need for logs, auditability, or evaluation. One option is to run transcription alongside the audio-native model. That gives operators a text view of incoming and outgoing speech while audio generation continues.

Evaluation must also follow the new architecture. Separate text-to-text and text-to-speech tests no longer map cleanly onto a model that listens and speaks directly. Longer, full-duplex conversations become the unit of evaluation, with metrics focused on the complete interaction. Bhargava’s final clarification is operational: this evaluation generally runs above the base inference API. The model supplies inference; the surrounding system still has to establish whether the conversation worked.