Voice AI: Your Bot Isn't Special

What still breaks when you take a capable chat agent and give it a voice? Faster, cheaper LLMs make complex conversations more practical, but speaking introduces another set of failure modes. Generative text-to-speech can sound realistic while hallucinating audio, mispronouncing a name, or mangling an identifier. Meanwhile, the growing supply of voice infrastructure raises a practical engineering question: which parts are worth owning?

In this 2025 talk, Nik Caryotakis describes SuperDial's preference for reliability over realism. At the time, his concern with speech-to-speech models was that they could generate sounds that were not useful speech or imitate voices, undermining predictable conversations. A voice MVP could already be impressive; making it dependable enough to perform work was the harder problem.

SuperDial handles a familiar administrative burden: calling an insurance company. Its customers are mid-to-large healthcare administration businesses. They design a conversation script containing the questions they need answered, then submit calls through CSV files, an API, or integrations with electronic health record systems. Structured results come back within the next few hours or the next day.

The customer specifies whom to call and what to ask. Behind that simple contract, execution includes scheduling and escalation:

1. Wait until the relevant office or call center is open.
2. Attempt the call with the voice bot.
3. Bring in a human when the bot needs help, or send the call to a fallback team after unsuccessful attempts.
4. Return the answers in a structured format.

Human fallback is disclosed to customers and is part of the service's value. They are buying completed work, not a guarantee that every conversation will be handled exclusively by a bot.

Completed calls also improve later attempts. SuperDial updates its knowledge of office hours for each phone number and learns from phone-tree traversal. Separately, it randomly samples calls for audits. The learning loop improves future execution; the audit loop checks whether sensitive healthcare work is being performed correctly.

The embedded demonstration begins after the bot has navigated the phone tree and reached a human representative. The caller introduces itself as Sarah, identifies the provider's office, supplies a member ID, and gives a CPT code. The provider and member details and the CPT code are withheld.

The representative finds a case for that code. Its status is pending, and the insurer has not received the clinical documents. That is already a useful result: the call has identified both the current state and the missing material. The case number is withheld.

The caller then asks for the representative's name and a reference number. The representative supplies a name, says the pending case number can serve as the call reference, and provides the fax destination for the clinical documents. Those identifying details are also withheld. The caller thanks the representative, and the call ends. Nothing spectacular happens; the useful work is collecting the information needed to move the case forward.

That ordinariness is the success condition. A healthcare administration team needs routine calls to finish reliably, not to demonstrate an unusually expressive voice. Caryotakis reports that SuperDial had saved over 100,000 hours of human phone-calling time by the talk, and forecasts millions more during 2025. The forecast is distinct from the reported savings. He reports a team of four engineers building the web application, EHR integrations, and voice bot while onboarding customers and supporting new conversational use cases.

Voice engineering extends application work into a real-time, multimodal system. Engineers handle MP3s and audio bytes alongside transcripts, and integrate transcription, speech-generation, and speech-to-speech models. Latency becomes a product constraint, and asynchronous Python becomes an everyday concern. The application must also fit an existing social interaction: the person answering an insurance call expects a recognizable business conversation.

SuperDial's working principles are to say the right thing at the right time and keep improving the service while it is running. The hardest customer-specific work is designing scripts and conversations. Horizontal voice infrastructure can supply much of the machinery around them. The differentiation is in conversational content and vertical integrations—the parts that make an agent's work useful to a particular business. Voice realism, interruption handling, and turn-taking matter, but they are not sufficient differentiation on their own.

Moving quickly also creates responsibilities. Voice systems can work unevenly across accents and dialects, and realistic voices can become unsettling when they say inappropriate things. Caryotakis frames gaps in US AI regulation as placing responsibility on engineers and leaders. That should not be read as an absence of applicable rules: the FCC had already classified AI-generated voices as artificial voices under TCPA restrictions. His practical design principle is to choose accessible, collaborative tooling so a diverse set of stakeholders can participate from the beginning.

SuperDial began with a quickly assembled transcription → LLM → text-to-speech pipeline. It was enough to get started, but the problems it exposed were not all new. Voice interfaces have a history stretching through phone trees and assistants such as Alexa and Siri. Caryotakis sought advice from conversation designer Cathy Pearl, whom he describes as working on Gemini UX at the time. Her experience connected that older design discipline to generative voice applications.

The major shift is from prescriptive to descriptive development:

Describing a goal removes the need to map every possible branch, but it does not remove the need to design the interaction. The generated conversation still has to accomplish the task.

One immediate choice is whether to ask open questions or constrain the other person to a list of answers. In SuperDial's existing insurance workflows, broad questions often work better: let the representative provide information, then adapt to what they say. Trying to force an established business conversation into a narrow menu can discard useful information.

Pearl's recommendation was to hire a conversation designer. A lightweight way to begin applying that discipline is a table read:

1. Write a sample conversation.
2. Have one person play the bot and another play the user.
3. Read the exchange aloud and notice the gaps and awkward turns.

Spoken rehearsal exposes problems that a plausible-looking transcript can hide. SuperDial wanted to spend more time on this work, but first had to address the technical debt in its custom orchestration framework.

Adopting Pipecat, the open-source voice orchestration framework maintained by Daily, reduced that burden. Extensibility mattered because SuperDial needed behavior such as call transfers. Hosting control mattered too: Caryotakis says some calls can last about an hour and a half. Self-hosting let the team choose how to deploy and scale for that workload.

The benefit was not simply replacing one framework with another. Resolving orchestration problems gave the team more room to work on conversations—the part of the application that varied with each customer's needs.

SuperDial also chose to own the OpenAI-style endpoint exposed to its voice tools. Keeping that interface under its control made it possible to route requests behind the endpoint to different models, including models suited to latency-sensitive work. The interface could remain useful to the surrounding tooling while the model choice changed.

Generative responses pass through TensorZero. Caryotakis emphasizes its structured, typed LLM endpoints and the ability to experiment in production. That places the model gateway between the conversational application and the underlying model choices, rather than scattering those choices throughout the voice system.

For logging and observability, SuperDial self-hosts Langfuse, using it for anomaly detection, evaluations, and datasets. Healthcare data and HIPAA requirements motivate the hosting decisions. Self-hosting is one part of that compliance strategy; it does not by itself establish compliance.

Healthcare calls often authenticate someone through a name, date of birth, and long member identifier. These are not details a voice system can approximate. A member ID may be a roughly twelve-character sequence that must survive generation, speech synthesis, telephone transmission, and another person's interpretation.

LLM output, TTS input, and recorded speech are three different things. The text generated by the language model may need transformation before it reaches the speech engine. Even correctly prepared TTS input does not guarantee that the recording contains the intended speech. Caryotakis makes the issue personal with his surname: a voice application built for him should pronounce it correctly.

Rime's pronunciation controls provide a way to specify pronunciation explicitly. For spelling, a spell function can introduce useful grouping, pauses, and breaks into long sequences. These controls address different needs: pronouncing a name as a word and communicating the individual characters of an identifier. Current Rime documentation is model-specific and should not be treated as the exact syntax or model version shown in the 2025 demonstration.

The final check is listening. SuperDial reviews recordings as well as transcripts because the delivered artifact is audio bytes. A transcript alone cannot establish whether a name sounded right or whether the spacing of a spoken identifier was understandable. Speech-to-speech models make these explicit, rule-based interventions more complicated because the text boundary is less straightforward to control.

Even the bot's name can create unnecessary work. When the company was called SuperBill, the team named its bot Billy. The playful connection to the brand turned into repeated confusion over the phone. The lesson is to test the persona in the spoken medium early, including whether people can hear and recognize its name without a correction exchange.

For a new implementation, Caryotakis recommends starting with existing infrastructure rather than rebuilding the stack. Once it is running, instrument latency throughout it. In particular, track time to first byte for each processor: a slow first output can leave the other person waiting even when the eventual response is correct.

In an asynchronous Python pipeline, a small wrapper can record when a byte-producing processor first emits nonempty output while preserving the stream:

from collections.abc import AsyncIterator, Callable
from time import perf_counter

async def measure_ttfb(
    name: str,
    start_stream: Callable[[], AsyncIterator[bytes]],
    record: Callable[[str, float], None],
) -> AsyncIterator[bytes]:
    started = perf_counter()
    first_byte_seen = False

    async for chunk in start_stream():
        if chunk and not first_byte_seen:
            record(name, perf_counter() - started)
            first_byte_seen = True
        yield chunk

Starting the timer before opening the stream includes the wait for initial output. Each processor gets its own name and measurement; an empty stream does not produce a misleading successful first-byte measurement. This illustrates the per-processor instrumentation principle, rather than an end-to-end conversational latency score.

Two additional choices determine whether the system can improve and survive failures:

• An accuracy upgrade path. SuperDial uses Deepgram for speech-to-text. Caryotakis values the option of working with the provider on fine-tuning when greater transcription accuracy is needed. He presents this as an available improvement path, not a completed experiment with measured gains.
• Fallbacks for each component. An OpenAI outage can disrupt many concurrent calls at once. Prepare alternatives throughout the stack; TensorZero is one tool he identifies for configuring model fallbacks. These infrastructure fallbacks complement the human escalation path used to complete customer work.

For end-to-end testing, the telephone connection provides a useful external boundary. It exercises the bot through the interface it must actually use, rather than testing only an internal prompt or transcript. Caryotakis describes progressively richer test setups:

1. Prerecorded playback: create a phone number that plays an MP3 and check whether the bot can handle it.
2. A simulated phone tree: construct a menu system and test whether the bot can navigate it.
3. Generative conversations: use services such as Coval or the service named Vocera in the talk to let the bot converse with another bot.

These test different demands: basic audio interaction, navigation, and responses to generated conversational behavior.

The purpose of the stack is to preserve attention for the distinctive conversation. Good infrastructure choices leave room to solve the details that turn a plausible voice agent into a useful worker: the right question, the correctly spoken identifier, and the completed call. As new models arrive, the architecture should let the team adopt them quickly while retaining the ability to adopt them safely.