AI Engineer Summit 2025
Voice AI: Your Bot Isn't Special
Read the talk
Voice AI: Your Bot Isn't Special
A convincing voice is only the beginning: reliable healthcare calls depend on conversation design, precise spoken identifiers, observable infrastructure, and a path to completed answers.
From a talk by Nik Caryotakis
Before you start: Familiarity with LLM applications and the roles of speech-to-text and text-to-speech will help; the Python example also uses asynchronous iteration.
From a chat agent to a reliable conversation
What still breaks when you take a capable chat agent and give it a voice? Faster, cheaper LLMs make complex conversations more practical, but speaking introduces another set of failure modes. Generative text-to-speech can sound realistic while hallucinating audio, mispronouncing a name, or mangling an identifier. Meanwhile, the growing supply of voice infrastructure raises a practical engineering question: which parts are worth owning?
In this 2025 talk, Nik Caryotakis describes SuperDial's preference for reliability over realism. At the time, his concern with speech-to-speech models was that they could generate sounds that were not useful speech or imitate voices, undermining predictable conversations. A voice MVP could already be impressive; making it dependable enough to perform work was the harder problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The product is a completed set of answers
SuperDial handles a familiar administrative burden: calling an insurance company. Its customers are mid-to-large healthcare administration businesses. They design a conversation script containing the questions they need answered, then submit calls through CSV files, an API, or integrations with electronic health record systems. Structured results come back within the next few hours or the next day.
The customer specifies whom to call and what to ask. Behind that simple contract, execution includes scheduling and escalation:
- Wait until the relevant office or call center is open.
- Attempt the call with the voice bot.
- Bring in a human when the bot needs help, or send the call to a fallback team after unsuccessful attempts.
- Return the answers in a structured format.
Human fallback is disclosed to customers and is part of the service's value. They are buying completed work, not a guarantee that every conversation will be handled exclusively by a bot.
Completed calls also improve later attempts. SuperDial updates its knowledge of office hours for each phone number and learns from phone-tree traversal. Separately, it randomly samples calls for audits. The learning loop improves future execution; the audit loop checks whether sensitive healthcare work is being performed correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A deliberately ordinary prior-authorization call
The embedded demonstration begins after the bot has navigated the phone tree and reached a human representative. The caller introduces itself as Sarah, identifies the provider's office, supplies a member ID, and gives CPT code 81243. The provider and member details are withheld.
The representative finds a case for that code. Its status is pending, and the insurer has not received the clinical documents. That is already a useful result: the call has identified both the current state and the missing material. The case number is withheld.
The caller then asks for the representative's name and a reference number. The representative supplies a name, says the pending case number can serve as the call reference, and provides the fax destination for the clinical documents. Those identifying details are also withheld. The caller thanks the representative, and the call ends. Nothing spectacular happens; the useful work is collecting the information needed to move the case forward.
That ordinariness is the success condition. A healthcare administration team needs routine calls to finish reliably, not to demonstrate an unusually expressive voice. Caryotakis reports that SuperDial had saved over 100,000 hours of human phone-calling time by the talk, and forecasts millions more during 2025. The forecast is distinct from the reported savings. He reports a team of four engineers building the web application, EHR integrations, and voice bot while onboarding customers and supporting new conversational use cases.
Suggest correction
What the voice AI engineer owns
Voice engineering extends application work into a real-time, multimodal system. Engineers handle MP3s and audio bytes alongside transcripts, and integrate transcription, speech-generation, and speech-to-speech models. Latency becomes a product constraint, and asynchronous Python becomes an everyday concern. The application must also fit an existing social interaction: the person answering an insurance call expects a recognizable business conversation.
SuperDial's working principles are to say the right thing at the right time and keep improving the service while it is running. The hardest customer-specific work is designing scripts and conversations. Horizontal voice infrastructure can supply much of the machinery around them. The differentiation is in conversational content and vertical integrations—the parts that make an agent's work useful to a particular business. Voice realism, interruption handling, and turn-taking matter, but they are not sufficient differentiation on their own.
Moving quickly also creates responsibilities. Voice systems can work unevenly across accents and dialects, and realistic voices can become unsettling when they say inappropriate things. Caryotakis frames gaps in US AI regulation as placing responsibility on engineers and leaders. That should not be read as an absence of applicable rules: the FCC had already classified AI-generated voices as artificial voices under TCPA restrictions. His practical design principle is to choose accessible, collaborative tooling so a diverse set of stakeholders can participate from the beginning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Design the conversation out loud
SuperDial began with a quickly assembled transcription → LLM → text-to-speech pipeline. It was enough to get started, but the problems it exposed were not all new. Voice interfaces have a history stretching through phone trees and assistants such as Alexa and Siri. Caryotakis sought advice from conversation designer Cathy Pearl, whom he describes as working on Gemini UX at the time. Her experience connected that older design discipline to generative voice applications.
The major shift is from prescriptive to descriptive development:
| Approach | Developer specifies | Central difficulty |
|---|---|---|
| Prescriptive | Branches and possible conversation paths | Anticipating the paths |
| Descriptive | Desired behavior and goals | Getting reliable generated behavior |
Describing a goal removes the need to map every possible branch, but it does not remove the need to design the interaction. The generated conversation still has to accomplish the task.
One immediate choice is whether to ask open questions or constrain the other person to a list of answers. In SuperDial's existing insurance workflows, broad questions often work better: let the representative provide information, then adapt to what they say. Trying to force an established business conversation into a narrow menu can discard useful information.
Pearl's recommendation was to hire a conversation designer. A lightweight way to begin applying that discipline is a table read:
- Write a sample conversation.
- Have one person play the bot and another play the user.
- Read the exchange aloud and notice the gaps and awkward turns.
Spoken rehearsal exposes problems that a plausible-looking transcript can hide. SuperDial wanted to spend more time on this work, but first had to address the technical debt in its custom orchestration framework.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use orchestration that leaves room for the application
Adopting Pipecat, the open-source voice orchestration framework maintained by Daily, reduced that burden. Extensibility mattered because SuperDial needed behavior such as call transfers. Hosting control mattered too: Caryotakis says some calls can last about an hour and a half. Self-hosting let the team choose how to deploy and scale for that workload.
The benefit was not simply replacing one framework with another. Resolving orchestration problems gave the team more room to work on conversations—the part of the application that varied with each customer's needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Own the model interface and observe the work
SuperDial also chose to own the OpenAI-style endpoint exposed to its voice tools. Keeping that interface under its control made it possible to route requests behind the endpoint to different models, including models suited to latency-sensitive work. The interface could remain useful to the surrounding tooling while the model choice changed.
Generative responses pass through TensorZero. Caryotakis emphasizes its structured, typed LLM endpoints and the ability to experiment in production. That places the model gateway between the conversational application and the underlying model choices, rather than scattering those choices throughout the voice system.
For logging and observability, SuperDial self-hosts Langfuse, using it for anomaly detection, evaluations, and datasets. Healthcare data and HIPAA requirements motivate the hosting decisions. Self-hosting is one part of that compliance strategy; it does not by itself establish compliance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The text is not the delivered speech
Healthcare calls often authenticate someone through a name, date of birth, and long member identifier. These are not details a voice system can approximate. A member ID may be a roughly twelve-character sequence that must survive generation, speech synthesis, telephone transmission, and another person's interpretation.
LLM output, TTS input, and recorded speech are three different things. The text generated by the language model may need transformation before it reaches the speech engine. Even correctly prepared TTS input does not guarantee that the recording contains the intended speech. Caryotakis makes the issue personal with his surname: a voice application built for him should pronounce it correctly.
Rime's pronunciation controls provide a way to specify pronunciation explicitly. For spelling, a spell function can introduce useful grouping, pauses, and breaks into long sequences. These controls address different needs: pronouncing a name as a word and communicating the individual characters of an identifier. Current Rime documentation is model-specific and should not be treated as the exact syntax or model version shown in the 2025 demonstration.
The final check is listening. SuperDial reviews recordings as well as transcripts because the delivered artifact is audio bytes. A transcript alone cannot establish whether a name sounded right or whether the spacing of a spoken identifier was understandable. Speech-to-speech models make these explicit, rule-based interventions more complicated because the text boundary is less straightforward to control.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Names, latency, upgrade paths, and failures
Even the bot's name can create unnecessary work. When the company was called SuperBill, the team named its bot Billy. The playful connection to the brand turned into repeated confusion over the phone. The lesson is to test the persona in the spoken medium early, including whether people can hear and recognize its name without a correction exchange.
For a new implementation, Caryotakis recommends starting with existing infrastructure rather than rebuilding the stack. Once it is running, instrument latency throughout it. In particular, track time to first byte for each processor: a slow first output can leave the other person waiting even when the eventual response is correct.
In an asynchronous Python pipeline, a small wrapper can record when a byte-producing processor first emits nonempty output while preserving the stream:
python
from collections.abc import AsyncIterator, Callable
from time import perf_counter
async def measure_ttfb(
name: str,
start_stream: Callable[[], AsyncIterator[bytes]],
record: Callable[[str, float], None],
) -> AsyncIterator[bytes]:
started = perf_counter()
first_byte_seen = False
async for chunk in start_stream():
if chunk and not first_byte_seen:
record(name, perf_counter() - started)
first_byte_seen = True
yield chunk
Starting the timer before opening the stream includes the wait for initial output. Each processor gets its own name and measurement; an empty stream does not produce a misleading successful first-byte measurement. This illustrates the per-processor instrumentation principle, rather than an end-to-end conversational latency score.
Two additional choices determine whether the system can improve and survive failures:
- An accuracy upgrade path. SuperDial uses Deepgram for speech-to-text. Caryotakis values the option of working with the provider on fine-tuning when greater transcription accuracy is needed. He presents this as an available improvement path, not a completed experiment with measured gains.
- Fallbacks for each component. An OpenAI outage can disrupt many concurrent calls at once. Prepare alternatives throughout the stack; TensorZero is one tool he identifies for configuring model fallbacks. These infrastructure fallbacks complement the human escalation path used to complete customer work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test through the telephone
For end-to-end testing, the telephone connection provides a useful external boundary. It exercises the bot through the interface it must actually use, rather than testing only an internal prompt or transcript. Caryotakis describes progressively richer test setups:
- Prerecorded playback: create a phone number that plays an MP3 and check whether the bot can handle it.
- A simulated phone tree: construct a menu system and test whether the bot can navigate it.
- Generative conversations: use services such as Coval or the service named Vocera in the talk to let the bot converse with another bot.
These test different demands: basic audio interaction, navigation, and responses to generated conversational behavior.
The purpose of the stack is to preserve attention for the distinctive conversation. Good infrastructure choices leave room to solve the details that turn a plausible voice agent into a useful worker: the right question, the correctly spoken identifier, and the completed call. As new models arrive, the architecture should let the team adopt them quickly while retaining the ability to adopt them safely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source framework for realtime voice agents and multimodal applications, maintained by Daily and the community.
Open-source platform combining an LLM gateway with evaluation, observability and experimentation.
Documentation for deploying Langfuse observability on your own infrastructure.
Phonetic notation and pronunciation controls for Rime's Mist-family speech models.
Spell letters, numbers and mixed identifiers with pauses between character groups.
Connect voice and chat agents to run simulated conversations and evaluate behavior.
Updates since the talk
Deepgram's later explanation of enterprise speech-model selection, adaptation and custom-model options.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hey everyone, uh, I'm Nick.
- 0:19
I'm an engineer at SuperDial, and first of all, big thanks to the organizers. This event has been awesome. I've had a blast talking to you guys, connecting with you guys, and hearing all these great talks.
- 0:29
Um, somehow I'm one of the few voice AI talks today and this weekend, so I have a lot to cover. We're gonna dive right in. If you're new to voice AI, I hope I can ver-- provide a nice little framework to think about this very fast-moving space.
- 0:41
And if you're building with voice AI already, I'll be sharing some little anecdotes from our own scaling journey that I hope will help yours as well. So voice AI in twenty twenty-five, extremely exciting.
- 0:54
We're seeing these new smart, really fast, really affordable LLMs that are supporting a lot more complex conversational use cases. Uh, but you still kinda need some tricks to take your chat agent and turn it into a voice agent.
- 1:09
We have these low latency, really realistic, super generative text-to-speech models, but sometimes we have audio hallucinations, and we have to deal with things like pronunciation and spelling.
- 1:20
With all the new things that people are building, there's this explosion in voice AI infrastructure, and tooling, and evaluation systems, and a big question becomes, what's actually worth owning?
- 1:30
And the big one on everyone's mind are these new speech-to-speech or voice-to-voice models. Uh, and our take is that for a lot of production applications, they're not quite yet ready.
- 1:42
And a big reason for that is they start to output things that aren't actually speech, aren't actually, uh,
- 1:49
things that you can use to build a reliable conversation. And this-- we saw this when they first came out. They were, like, imitating people's voices. And from the start, that's why we've kinda been favoring, uh, reliability over this sort of realism.
- 2:03
So today I'm gonna talk about how we at SuperDial approach agents as a service, how we think about the voice AI engineer, and the last mile problem. So once you have your little voice, uh, MVP, all the challenges that you're gonna face trying to actually make it reliable and put it to work.
- 2:21
So at SuperDial, we're in the business of phone calls, specifically one of the most annoying phone calls ever, that phone call to your insurance company. So for mid to large-sized healthcare administration businesses, we sell the SuperDial platform.
- 2:38
And with SuperDial, you can build your script, so design the sort of conversation, ask all the questions that you need to get answered over the phone. You send us your calls via CSV, API, or we also integrate with a lot of EHR software systems.
- 2:54
And then, you know, within the next couple of hours, in the next day, we send you back your results in a structured format. And this makes for a really interesting agentic contract that we sort of have with our customers.
- 3:06
So from their perspective, they're paying for results. They tell us who to call, which questions to ask, and we tell them the answers. Internally, we have a little agentic loop set up so that,
- 3:18
uh, we go out, we wait for these offices to be open, we wait for, um, you know, the call centers to open, so we can actually make these calls.
- 3:26
We will attempt to make the call with our voice bot, and then if our voice bot needs to bring in a human to complete the call or cannot complete the call after a certain number of attempts, then we send it to a fallback team.
- 3:37
And this is something that, of course, we're transparent with with our customers. In fact, it's a benefit to them because it's kind of inevitable with these healthcare phone calls that sometimes you need to bring in a human.
- 3:48
So with us, they know that no matter what happens, the call will get made. Whether or not it gets made with a human or a bot doesn't matter to them.
- 3:55
They get their answers reliably and in a structured format.
- 3:59
Uh, and with all these calls, we try to do our best to learn from them. So we'll update the sort of office hours for the given phone number we're calling and learn from the sort of phone tree traversal that we just tried, so that when we call it again, we get even better at that sort of call.
- 4:14
And because these are sensitive healthcare phone calls, we wanna make sure our system always works. So randomly, we'll take out some of these calls, audit them, make sure everything's working.
- 4:24
Uh, for a quick little demo, this is actually a prior authorization call. Uh, this is after the point where we've traversed the little phone tree by clicking the right buttons, and now we're talking to a human and trying to get some questions answered for a customer.
- 4:39
May I know your first name?
- 4:42
Hi, this is Sarah.
- 4:45
Are you calling from a doctor's office or from a facility?
- 4:49
I'm calling from [redacted] provider's office.
- 4:53
Do you have a member ID or a case number?
- 4:57
The member ID is [redacted].
- 5:03
What is the CPT code?
- 5:06
The CPT codes are [REDACTED:generic_id].
- 5:12
Okay, hold on. So there's a case on file, uh, that was initiated for the code [REDACTED:generic_id]. It is pending. So this case number is [redacted]
- 5:27
. And we have not received any clinicals for this case yet.
- 5:34
Okay. What is your name again, and what is the reference number for this call?
- 5:39
My first name is [redacted]. You may have the pending case number as a call reference number, and the fax number on where to send the clinicals is [redacted]
- 5:50
.
- 5:50
Thanks so much for your help.
- 5:54
You're welcome. Thanks for calling. Have a great day.
- 5:58
So that's it. Uh, if that call was really boring to you, thanks-
- 6:04
If that call was really boring, that's kind of just how these things go. A boring call for us is an excellent call 'cause it turns out a lot of work is boring. [laughing]
- 6:14
Uh, so with this system, we've been able to save over a hundred thousand hours of human phone calling time, and we're on track to save millions more in twenty twenty-five.
- 6:21
And what's really incredible about voice AI today is that we did this with a really lean team of four engineers. So building the whole full stack web application, these EHR integrations, the bot you just saw, all while bringing on new customers, supporting new conversational use cases really quickly.
- 6:38
And a big part of why that was possible was because we really all embrace this role of a voice AI engineer. So let's kind of uncover what's unique about a voice AI engineer today and what hats they may be wearing.
- 6:51
So starting from Switch's, like, original graph, we can kinda see that a voice AI engineer is gonna deal with multimodal data. So MP3s, audio bytes, in addition to transcripts.
- 7:01
You're dealing with transcription models, voice models, speech-to-speech, all that sort of thing. The application you're building, it's in real time. Latency all of a sudden matters so much more.
- 7:12
You're gonna be dealing with async in Python a lot more than you probably wanted to be doing. And the product constraint here is almost always gonna be a voice conversation.
- 7:21
So people have really high expectations of how these sorts of conversation goes. Uh, for us, like, we're slotting ourselves into an existing, uh, sort of like business interaction, and people expect us to be conversational and fit into that use case.
- 7:37
So to grapple with all these challenges, we kind of have two sayings at SuperDial that we've been saying over the past year and a half. Say the right thing at the right time, and build this plane while we fly it.
- 7:49
So the trickiest part, uh, for us is customizing all these scripts and all these use cases for each customer individually, and then we really rely on this kind of like horizontal voice AI stack to help us out with all those other problems.
- 8:06
And this is kind of how we think about the voice AI engineer today and its unique roles. And in the larger context, we're really at this inflection point where it's so easy to build out an MVP for these sorts of applications that ultimately what is going to make your voice bot unique isn't its voice or its interruption
- 8:26
handling or how realistic it sounds or how it does turn-taking. Ultimately, it's gonna be in the conversational content and the design there and the vertical integrations around it that make your agents' work actually valuable.
- 8:40
And if you're like me, and your favorite classes in college were the AI ethics ones, everything I just said about moving fast, building with generative AI could raise a few red, uh, or like raise some alarms.
- 8:54
So it's not hard to imagine how voice AI apps specifically could be biased against people with certain accents, people with certain dialects, or be really spooky when they sound so real and then say weird things.
- 9:07
So in the US, we both like enjoy and suffer from a lack of AI regulation, and that leaves the onus ultimately on the AI engineers and leaders in this room to think about these sorts of problems.
- 9:20
This is not gonna be like a talk on like AI safety and ethics, but I think for voice AI specifically with how it's such like a new modality of interaction with artificial intelligence today, I think it's really important how we go about building it.
- 9:35
So for AI engineers, when we go about making tooling and infrastructure choices, uh, remember that like developing AI should be really accessible and collaborative, and the work that AI does should be for everyone.
- 9:49
And a key part in making sure that's the case is choosing tooling and infrastructure so that a really diverse set of stakeholders can be involved in that process from the start.
- 9:58
So with the role of the voice AI engineer kind of scoped out now, let's dive into some of the last mile problems in voice AI that we've been dealing with.
- 10:07
So when we started out, we had a really scrapped together pipeline of like a transcription model and an LLM and then a text-to-speech model. Uh, this was awesome to get started at [chuckles], but you know, we faced a lot of problems very quickly.
- 10:21
And a lot of what we were learning was not new at all. So though the voice agents we see today are better than ever, voice UI itself is not that new.
- 10:30
So when we were just getting started, uh, around a year and a half ago, I had the chance to speak to Cathy Pearl, who is a close family friend and has been working on, uh, the UX of Gemini.
- 10:41
She's been in the conversation design game for like twenty years or something. Uh, and back in the day, like voice UI was lots of phone tree design, and then it becomes th- these Alexa and Siri-type things, and now we're just in this whole new world.
- 10:56
But a lot of the principles remain the same. And one of the biggest things that's changed with developing voice UI is the shift from prescriptive to descriptive development. So we no longer prescribe what we want our bot to do over the course of the conversation by mapping out every possible direction that it could go.
- 11:16
Instead, we describe what we want to do and then kind of pray to the generative gods that it happens. [laughing]
- 11:23
And for this, you know, there's a lot of things I'd talk about with conversation design, but it comes up really quickly when that becomes your main interface. One thing for us is when we're asking these questions, you know, should we be really open-ended with it or kind of constrain the user into selecting from a list of choices?
- 11:39
And for us, because these are existing conversations, we find it's often better to just go general, hope the call center representative gives us a ton of information, and then instead of trying to prevent them from saying the wrong thing, we try to adapt to whatever they say.
- 11:55
So Cathy's recommendation was hire a conversation designer. If you're thinking about these sorts of problems, there are experts in this. And if you're just a voice AI engineer and you wanna get started in this kind of thinking, a great recommendation is to do little- Table reads.
- 12:13
So have one person pretend to be the bot and the other person pre- pretend to be a user, and the sort of like transcript that you may write out by hand, immediately
- 12:23
the sort of gaps and awkwardness of it comes out when you say these things out loud. So knowing all these things, we were really excited to work on our conversations, but we had-- kind of had to deal with the tech debt, debt of the orchestration framework that we had built.
- 12:37
So we really hit our stride when we started using Pipecat for our voice AI orchestration. This is an open source framework maintained by the guys at Daily. It's really easy to extend and hack upon, which is important for our use case when we need to do transfers and stuff.
- 12:53
Um, and we make really long phone calls. These can be like an hour and a half long. So a big decision for us in choosing Pipecat was that we can self-host it and deploy it and scale it how we want.
- 13:05
So with some of our like voice orchestration headaches dealt with, we really wanted to get back to focusing on our conversations. And everything in this slide for us is really not unique to voice UI, uh, and AI.
- 13:19
So I'm gonna kinda speed over it. Two interesting decisions we've made here, because we just have, you know, an LLM in the backbone. Uh, we chose to own our own OpenAI endpoint.
- 13:30
We find this leads to a better interface with a lot of these new voice AI tools. So behind our OpenAI endpoint, we can kind of route to different models that are maybe more, uh, latency sensitive.
- 13:44
For all of our generative responses, we route them through this tool called TensorZero. TensorZero is relatively new. They have this nice framing of LLMs. Uh, if that quote interests you, I recommend you look them up and talk to them.
- 13:57
They're awesome. Uh, this is like a little open source tool, so you can do whatever you want with it. They give us kind of structured and typed LLM end-endpoints that we can then experiment with in production.
- 14:08
So that's our gateway to our LLM. And then all of our logging and observability, we self-host LaneFuse, and we self-host these things also because these are like healthcare calls.
- 14:19
We have to be HIPAA compliant. That's often an easiest-- an easier way to deal with, you know, the rapid growth of this space. So there we do like anomaly detection, evals, and datasets.
- 14:30
So with a good plan in place for our LLM sort of work, another big challenge is our text-to-speech system. So when you make these sorts of phone calls, your password is basically your name, your date of birth, and then your member ID or something, which is like a twelve-digit long string of characters that you have to be
- 14:49
able to communicate over the phone. And something we quickly realized was that what our LLM is outputting is not necessarily what we wanna shove through our text-to-speech engine, and neither of those things may actually match what's in the recording.
- 15:04
So a little example of this, and this is like a personal last mile, is that if you're building me a personal voice UI application, it should say my last name correctly.
- 15:14
So my last name is pronounced Caryotakis. Most people and most models will say Caryotakis. But with a lot of new tools out there, this is the syntax this company called Rhyme uses.
- 15:26
You can spell out the exact sort of pronunciations you want, and then for things like spelling, where you may have kind of an intuition for like the sort of pauses and breaks you might wanna use to say a really long word, you can use something like this little spell function.
- 15:42
Um, and then with all this stuff, like, because this is outputting audio bytes, we usually review recordings to make sure that this all sounds okay, in addition to checking the transcripts.
- 15:53
And to start wrapping things up, I have a couple little mini last mile problems that we've had to deal with. Oh, and, you know, with voice-to-voice models, all this sort of rule-based stuff gets a little more complicated.
- 16:05
So some little mini ones. Uh, we used to be called SuperBill, and we called our bot Billy 'cause we thought that was a fun name. Turns out that's an awful name over the phone because we would constantly have these conversations where people were like, "Hey, nice to meet you, Billy."
- 16:21
And we would say, "It's Billy, not Billy." [laughing]
- 16:25
So yeah, think about your persona a lot. Dial that in early.
- 16:31
Uh, if you're just starting, don't build from scratch. What's gonna make your bot unique is the conversation, and there are so many new tools out there like Pipecat that you can use to get a quick jump start.
- 16:41
Track latency everywhere. Time to first byte for each of your little processors is the new most important metric, and it's something you always kind of have to keep an eye on.
- 16:50
Uh, upgrade paths. This is a big one for us when we need to make sure we have really high transcription accuracy. So we use Deepgram for our speech-to-text engine, and we know that whenever we kinda wanna improve that part of our system, we can work with them to fine-tune a better model.
- 17:08
Have fallbacks ready. It really sucks when OpenAI goes down for a little bit, and all of a sudden, all the concurrent conversations you have are just down the drain.
- 17:17
So have fallbacks ready for each part of your stack. It's really easy to set that up with something like TensorZero. There are lots of other tools that'll help you figure that out.
- 17:26
And then end-to-end testing. This is pretty unique for voice UI and-- or voice AI. Uh, it seems like people are kind of settling on telephony as a boundary layer to test your bot with, like, an external service.
- 17:41
We do a couple different things. The easiest test for us is to create a kind of fake phone number that just plays an MP3. If your bot can't talk to an MP3, then you probably have bigger problems.
- 17:51
Next, we can kind of create, uh, a simulated voice tree with like different, uh, like phone tree building tools and have our bot pseudo-navigate it. And then there's lots of generative services like Koval and Vocera, where you can have your bot talk to another bot.
- 18:07
So some takeaways for a, a what I call a vertical voice AI, AI engineer. Choose your stack wisely. The better decision you makes-- you make here, it will allow you to focus on the things that are really truly unique to your conversational experience.
- 18:22
Laser focus on the last mile because this is where ultimately you can provide a lot of value and put your agents to work. And then ride the wave. There's so much new stuff happening in this space, and whenever new models come out, you wanna be able to use them quickly, and you also won't-- wanna be able to
- 18:38
use them safely. So thank you very much. I'm excited to talk to you all and hear about what's so special about your conversations. [clapping] [upbeat music]