5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo

A voice agent can sound great in development and start failing as soon as real callers arrive. Venky B, founder of Plivo, opens with that familiar transition. Plivo began building voice and SMS APIs in 2011; he reports that its platform carries over a billion voice calls each month. That is the telephony scale behind these observations, rather than a count of AI-agent conversations. 0:34

Plivo's agent offering sits above its own SIP trunking and audio-streaming infrastructure. It includes a programmable speech pipeline and a no-code visual studio. The programmable offering combines separate components rather than using a single model that takes speech in and produces speech out.

The first implementation usually connects speech-to-text, an LLM and text-to-speech, with turn detection deciding when the conversation should advance. LiveKit and Pipecat are examples of frameworks used to orchestrate those pieces. Each component can appear fast enough in isolation, and the proof of concept can work. Production exposes the gaps between them: waiting too long, misreading an identifier, collecting an invalid value or saying a correct answer badly.

Time to first audio measures the interval between the caller finishing and the agent starting to speak. Venky reports that teams often aim for less than 550 milliseconds, while many deployments land between 750 and 1,200 milliseconds. Beyond 1.2 seconds, he sees callers begin hanging up. These are production observations from his experience, rather than a universal threshold for every conversation. 5:15

That pause forces a three-way decision between cost, intelligence and latency. Longer model reasoning can improve an answer, but it also keeps the caller waiting. The speaking model therefore usually needs thinking turned off. Better instruction following and tool calling still help; gains that depend on spending extra time reasoning are difficult to use on this immediate response path.

The LLM is the largest latency contributor in the pipeline Venky describes. His frontier-model examples have a median time to first token of roughly 450–500 milliseconds on a good day, with P90 or P95 reaching 1.2–1.3 seconds. Those percentiles describe the slower end of the response distribution. Time to first token is only one part of time to first audio: the system still has to turn generated text into audible speech. A satisfactory median can conceal pauses that make the conversation frustrating.

Fast token generation does not automatically solve the wait for the first token. Cerebras and Groq enter the discussion as high-throughput options, but Venky describes dedicated capacity as expensive and, at the time of his experience, booked twelve months ahead. That commitment creates another risk: the team must choose infrastructure for a model it expects to keep using despite rapid model changes.

Self-hosted open-source models are Plivo's answer to the LLM latency problem. Running models on its own GPUs adds operational work, but Venky describes this as a practical way to target consistently less than 300 milliseconds at the model layer while balancing cost and capability. That target remains separate from the complete caller-to-agent audio delay. 9:27

Venky names Qwen 3.5 and Gemma 4 as the models his team evaluated. He says both worked well for English and reports Gemma as 2.5–3× better in their multilingual token-fertility evaluations—how many tokens it takes to generate one word in a language. He connects this to faster word generation under otherwise equal conditions. This is his team's reported comparison; the supplied material does not specify the exact checkpoints, language set or evaluation configuration, or provide independently reproduced results. 10:28

The mechanism connects tokenization to the caller's wait. If producing a word requires more tokens, the model must generate more pieces before it has produced the same spoken content. Under otherwise equal conditions, fewer tokens per word can mean faster word generation; that is the explanation Venky gives for the multilingual contrast he observed. Tokens per second alone therefore misses an important part of voice performance: how quickly those tokens become words in the caller's language.

Model size depends on what the agent must do:

• Generic conversation: Venky describes mixture-of-experts models using a three- or four-billion shorthand as useful out of the box. The talk does not specify checkpoints or whether that shorthand refers to active parameters. Active and total parameter counts differ for MoE models: the supplied Gemma 4 model card lists 25.2B total and 3.8B active parameters for its MoE model, so the shorthand should not be read as a 3–4B total model size for GPU hosting. The drawback comes when fine-tuning: his team has found that modifying these models can damage their behavior.
• Industry-specific work: For applications that need deeper adaptation, such as healthcare, his starting point is an eight- or twelve-billion model. The selection criteria include fast generation, good instruction following and a high tool-calling success rate.
• Separate speaking and action models: Another architecture uses a small conversational model, perhaps three billion, for talking and a larger model for tool calls. It assigns the more demanding action work to a model chosen for that capability.

Plivo runs two flavors: fine-tuned models for specific industries and an out-of-the-box mixture-of-experts model for more general uses. The useful decision comes before fine-tuning: establish whether the model follows the required instructions and calls tools successfully. A model that already does those jobs may need changes to the surrounding pipeline more urgently than changes to its weights.

Even a strong transcription engine needs help on real calls. Venky contrasts word error rates of 4–6 percent on known evaluation sets with double-digit rates on noisy calls involving accents and domain vocabulary. The errors concentrate in consequential places: proper nouns, jargon, missing phone-number digits and omitted pieces of long addresses. 13:35

Code-switched speech creates a different kind of failure. English words can arrive written in the script used for Hindi; Hindi can arrive written in Latin characters. The LLM may continue in the incoming script, and the speech synthesizer can then pronounce the response incorrectly. The mismatch begins in transcription but changes both the generated text and the eventual audio.

Three mechanisms clean up the input before it reaches the main conversational model:

• Dynamic keyword boosting: Boost relevant proper nouns during the phase of the call where they are expected. Keeping a large keyword list active throughout the call can encourage unwanted substitutions; changing the list with the call state narrows what the recognizer should listen for.
• Contextual post-processing: An LLM with domain context can interpret a suspicious transcription. Venky's example is an E inside a phone number, where 3 is a plausible correction. The field-collection step that follows must decide whether to confirm that interpretation or ask again.
• Transliteration: An LLM or a neural transliteration engine can normalize the script of multilingual transcripts before downstream generation.

The goal is a consistent cleaned transcript regardless of which transcription engine produced it. Recognition supplies an imperfect reading of the audio; the application supplies the context needed to interpret that reading. Cleanup alone is insufficient, though: a plausible string still needs to become a valid value.

Data collection is a voice UX problem. Venky estimates that 50–60 percent of agents struggle badly here. His remedy borrows from Python dataclasses, Pydantic, TypeScript's Zod and ordinary form fields: decide what kind of value the application needs before asking the caller for it. Plivo reports collection accuracy improving from roughly 30 percent to 95 percent, later citing 95–97 percent without fine-tuning. The evaluation population and scoring method are unspecified, so those figures describe Plivo's reported experience rather than a transferable benchmark. 18:00

Follow the phone-number example through that change. As free text, a transcript containing an E is simply a string the LLM has to interpret. As a phone-number field, it has allowed characters and an expected length. The E now creates a detectable validation error. The agent can propose 3 and confirm it with the caller, or reject the value and request a repetition. The visible change is a different next action: the suspicious character triggers a correction conversation instead of passing silently into collected data.

Where does the field definition change the phone-number flow? The diagram follows the same E through validation and the two recovery paths. The field supplies the rules that expose the error. A likely correction then remains a proposal for the caller to confirm; the alternative is to collect the value again.

Other fields need their own collection behavior:

• Names: A difficult name may require letter-by-letter spelling and confirmation. Repeating the same recognition attempt does not supply the structure that spelling does.
• Relative dates: “Next week, Wednesday, eight” needs the current date to resolve the calendar day and clarification to distinguish 8 AM from 8 PM. Treating it as a datetime field makes the missing information explicit. Tool calls can do much of the date-resolution work alongside the LLM.

The same decomposition changes evaluation. Test phone-number collection, name spelling and datetime resolution at the field level, as small units of behavior. A failed field test identifies the broken collection step directly, instead of requiring hundreds of complete conversations to reveal it. Those checks target collection reliability; they do not establish that the rest of the call behaves correctly. 20:51

Call state ties the approach together. At each point, the agent knows which field it is collecting and what that field permits. This gives recognition, interpretation, validation and confirmation a narrower job. Plivo's reported gains came from structuring that context, rather than endlessly adjusting prompt wording and hoping the model would become more obedient.

Correct generated text can still make bad audio. A normalization layer between the LLM and text-to-speech gives the application control over how the answer is spoken. Passing model output directly to the synthesizer leaves formatting, entity pronunciation and reading conventions to the engine. 22:24

The output layer has several separate jobs:

• Remove display formatting: Strip emojis and Markdown before synthesis. Venky notes that orchestration frameworks can handle this with configuration flags, which still need to be set.
• Specify pronunciation: Use custom dictionaries for proper nouns, brands and acronyms.
• Slow down for entities: Reduce speaking speed to 0.8× or 0.7× when reading an email, phone number or a name letter by letter, so the caller can hear the individual parts.
• Normalize reading conventions: Prepare emails, currencies and dates in the application's own layer rather than depending entirely on a TTS engine's interpretation.

This also makes switching synthesizers less disruptive. If the first provider is unavailable, or the application changes providers, its own normalization rules can remain in place. Each engine may still need pronunciation configuration, but the application retains responsibility for preparing the content it wants spoken.

Venky's first pronunciation check is deliberately personal: can the agent say his surname, Balasubramanian, and his company's name, Plivo? These are concrete words the product should handle, rather than an abstract voice-quality score. For a customer-facing platform, the same control should be available to customers, so they can specify the names and terms their own callers need to hear.

The ending returns to the timing layer around the pipeline: end-of-turn detection, barge-in and backchanneling. These concern when the agent takes its turn, how the caller interrupts it and how conversational acknowledgments fit into the exchange. Generating a good sentence quickly does not settle those interaction decisions. 25:06

Venky closes with a specific architectural claim: a dedicated speech-to-speech model is not required to support these behaviors; Plivo has found ways to implement them in a pipeline. He moves through the final slides without explaining the detection or interruption algorithms, so this establishes an implementation option rather than a recipe for reproducing it. The modular pipeline remains viable, provided conversation control receives attention alongside recognition, generation and synthesis.