AI Engineer Europe 2026
Engineering voice agents: Latency, quality, and scale
Read the talk
Engineering voice agents: latency, quality, and scale
A useful voice agent must understand the caller, act correctly, sound natural, and respond before the conversation stalls—all while keeping live calls reliable at scale.
From a talk by Rishabh Bhargava
Before you start: Familiarity with LLMs, tool calling, and streaming APIs will help; no speech-model background is required.
From waiting on hold to talking to computers
What would it take for an AI to handle a phone call without making the caller wait, repeat themselves, or correct its mistakes? Rishabh Bhargava approaches that question from infrastructure: he introduces himself as Together AI’s voice AI lead and previously co-founder and CEO of Refuel, acquired by Together. Together supplies training compute and inference infrastructure; Bhargava describes a platform serving more than a million developers and hundreds of companies, including Cursor and Decagon. The engineering path runs from today’s multi-model pipelines through their individual components, deployment trade-offs, and emerging speech-to-speech systems.
Consider calling customer support to check an order or change a reservation, or calling a doctor’s office to book an appointment for a family member. The task may be straightforward; getting through can still mean waiting on hold. Bhargava points to billions of human-handled calls each year as an opportunity for agents to do useful work.
Voice also opens a different interface to computers. People learn to speak before they learn to read, and the same conversational habit can extend from ChatGPT’s advanced voice mode to talking directly to Cursor or Claude Code. The opportunity in 2026 is to make these interactions rich and dependable enough for everyday use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four requirements have to hold at once
Real-time conversation leaves little room for delay. Bhargava gives roughly 300 ms as a human conversational response heuristic, says AI delays above 500 ms become noticeable, and warns that one-to-two-second pauses can cause callers to hang up. These are his design heuristics, rather than measured abandonment thresholds for a specified workload. Their practical implication is clear: latency belongs in the product requirements from the beginning.
Speed alone cannot complete a call. The other requirements are equally consequential:
- Intelligence: Follow complicated instructions, resolve ambiguity, and call tools that connect the agent to real-world actions.
- Naturalness: Speak the caller’s language, use an appropriate accent, pronounce names correctly, and convey emotion suited to the situation.
- Reliability: Preserve the experience as a one-caller demonstration grows to hundreds, thousands, or ten thousand concurrent calls.
Voice is an AND problem: a fast agent that cannot do the job, or a capable agent that becomes unreliable under load, still fails the caller.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The cascading pipeline
Bhargava describes the cascading architecture as the dominant production approach at the time of the talk. An orchestrator such as Pipecat, LiveKit, or a homegrown system coordinates the streaming path:
- Receive audio chunks from the caller.
- Feed those chunks into speech-to-text to produce a transcript.
- Pass text to an LLM, which decides whether to call a tool and what response to produce.
- Feed the response text into text-to-speech.
- Stream generated audio chunks back to the caller.
Each component has a distinct job, but the caller experiences their combined behavior as one conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognize the words—and the end of the turn
Speech-to-text is the agent’s first opportunity to get the conversation wrong. Its basic quality measure, word error rate, compares the generated transcript with a reference. Bhargava cites approximately 6% WER for state-of-the-art models on unspecified open benchmarks, with results varying by use case. Aggregate accuracy is only part of the requirement: a mistaken customer name or drug name can pass through the LLM and emerge unchanged in synthesized speech. Downstream components cannot reliably recover information that transcription has already lost.
The latency measure is time to complete a transcript: after the user stops speaking, how long until the finished text is ready for the LLM? Bhargava reports approximately 100 ms P90 transcript-completion latency for some models running on Together. He does not specify the models, workload, hardware, or endpointing conditions, so this is a reported serving result rather than a deployment guarantee.
Before measuring that interval, the system must decide whether the user has actually finished. A pause may mark the end of a turn, or it may mean the caller is thinking and will continue. Starting the agent’s response too early turns a recognition decision into an interruption. Turn detection remains partly unsolved, and both recognition and turn-taking need to work for the languages customers actually use.
Recognition architecture is also moving from batch processing toward native streaming. Whisper is the canonical batch-oriented example: Bhargava describes its training on 30-second clips and the work needed to adapt it to conversation. That window does not require an application to wait for 30 seconds of new speech; wrappers can chunk audio, pad it with silence, make repeated calls, and stitch the results together. The cost is additional orchestration around a model not originally organized around small incoming increments.
The alternative he describes is a recent, unnamed NVIDIA model whose encoder supports variable look-ahead, from roughly 80 ms to about a second. The encoder also caches activations. As new audio frames arrive, it reuses earlier computation instead of repeatedly performing the same heavy work.
| Recognition approach | Handling incoming audio | Work between updates |
|---|---|---|
| Batch model adapted for streaming | Chunk, pad, call, stitch | Wrapper coordinates repeated inference |
| Native streaming encoder | Advance through short audio increments | Reuse cached activations |
The distinction is architectural: streaming is part of how the encoder operates, rather than only a behavior assembled around batch calls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit the LLM inside the response budget
For the LLM, the critical streaming measure is time to first token, or TTFT. The first generated text lets the next stage begin preparing speech; waiting for the whole response would waste that opportunity. Bhargava’s rough target is 200–300 ms TTFT, which he associates with models in the 8–30 billion parameter range. This is a model-selection heuristic, not a limit independent of serving hardware, prompt length, or load.
The trade-off runs in both directions. A larger model may consume too much of the latency budget before speech synthesis even starts. A smaller model may respond quickly but fail at the instructions and tool calls that make the agent useful. The right choice is the smallest model that meets the workflow’s capability requirements within the available time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start speaking quickly, then keep up
Text-to-speech has two separate speed requirements. Time to first audio, or TTFA, measures the delay from receiving text to producing the first audio chunk that can be streamed back. Real-time factor, or RTF, measures whether generation can keep pace with playback:
In Bhargava’s example, five seconds of processing produces ten seconds of audio, giving an RTF of 0.5. An RTF below one means generation is faster than playback, helping avoid buffering once speech begins. A low startup delay and sufficient sustained generation speed solve different parts of the experience.
Quality still requires listening. Objective measures help, but Bhargava recommends hearing samples from the specific models and voices being considered and deciding whether they deliver the experience customers should have. Evaluate naturalness across voices, exact pronunciation of customer and product names, and emotional control. Some models expose tags for emotions such as happy, angry, or sad; language coverage remains another independent requirement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Budget and scale the complete conversation
Component metrics must fit inside a system-wide latency and cost budget. Bhargava’s rough ordering for both is the LLM first, followed by text-to-speech, then speech-to-text. Those engine measurements describe how long models take to produce outputs. They do not include all the network time spent moving requests and results between an orchestrator and models in different data centers.
Autoscaling also needs to respect conversational time. Scale up earlier and more aggressively than an asynchronous service: queueing that might be acceptable for a background job becomes dead air on a call. Scale-down has the opposite constraint. Even when traffic falls at night or on weekends, existing conversations may hold stateful, long-lived connections to model servers. Drain those conversations to completion instead of arbitrarily terminating the pods that serve them.
Deployment location serves two purposes. Putting the system near callers reduces travel time for their audio. Supporting the required regions also allows the service to meet data-residency requirements, including those relevant to European deployments. Capacity planning and regional placement therefore belong to the voice architecture, not just its hosting configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the network becomes the next optimization
Once model engines are fast, their physical placement can become a substantial part of the remaining delay. Bhargava’s illustrated setup uses roughly 100–200 ms engine times to first token or audio, with 75 ms of network latency between models and the orchestrator. He uses U.S. West–Europe as an example of meaningful distance and notes that networking conditions can push latency higher.
The proposed change is to colocate the models, and potentially the orchestrator, in the same data center or building. In Bhargava’s illustrated optimized pipeline, reducing network latency from 75 ms to 5 ms produces about a 30% reduction in overall latency. The slide compares total time to first audio of approximately 710 ms without colocation and 500 ms with colocation. Those totals belong to this example; they are not a general percentage improvement for moving any voice system into one region.
This is why detailed observability matters even after the model choices look settled. A system can have good engine metrics and still spend an avoidable portion of the caller’s wait crossing networks. At conversational timescales, even ten milliseconds is worth being able to locate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a single speech-to-speech model changes
A different architecture replaces the speech-to-text, LLM, and text-to-speech chain with one speech-to-speech model. It still has to follow complicated instructions and call functions, but fewer model boundaries need orchestration. Bhargava points to the OpenAI Realtime API and NVIDIA’s Nemotron 3 VoiceChat; the latter was an early-access model at the time of the talk.
Bhargava reports that instruction-following and tool-calling weaknesses have limited adoption in the production workflows he discusses. Teams may try a speech-to-speech model, spend time prompt engineering around its failures, and eventually return to a pipeline. His expectation is that these capabilities will improve. The attraction is not only fewer components: avoiding a text-only intermediate representation can preserve tone, emotion, and hesitation for the model’s next decision.
Full duplex means the model can generate audio while it is still receiving audio. That supports backchannels such as “I see” and “Uh-huh” while the customer continues speaking. It also creates a more direct route to handling interruptions and barge-ins, which require additional coordination in a cascading pipeline.
| Concern | Cascading pipeline | Speech-to-speech |
|---|---|---|
| Model coordination | Separate recognition, LLM, synthesis | One audio-native model |
| Speech nuance | Can disappear in transcription | Available in the audio input |
| Overlapping speech | Requires orchestration | Supported by full-duplex designs |
| Instructions and tools | Choose a suitable text LLM | Still a limiting capability in Bhargava’s account |
The architectural promise is substantial, but dependable voice interfaces still require engineering around the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate tool calls and make location concrete
The first audience question asks what evaluation score makes voice-driven function calling production-ready. In a pipeline, assuming recognition and synthesis are satisfactory, evaluate the LLM’s tool calls much as in a text application. Separate structural validity—whether the output is parsable and has the required form—from semantic correctness—whether it calls the right tool with the right arguments. Bhargava recommends tool-call structure very close to 100% correct; the required semantic-correctness threshold depends on the use case.
When a small model is fast enough but insufficiently reliable at the workflow, increasing model size is not the only option. Bhargava reports customers fine-tuning smaller LLMs on use-case-specific data to improve tool calling while staying within the latency-driven model-size budget.
A follow-up question makes colocation concrete for cloud users. Imagine an agent server in London calling an LLM endpoint hosted in the United States: each exchange must cross the Atlantic and return. Bhargava uses OpenAI as a hypothetical provider in this example, not as a verified statement about every endpoint’s location. Running an open-source model in the agent’s own data center replaces that long-distance exchange with local data-center traffic. Physical distance matters alongside the rest of the network configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check the answer before it becomes speech
Another audience member describes a concrete failure: an agent offers an unauthorized 8% discount. Adding a classifier or guardrail can catch the problem, but where does that component belong, and what happens to latency? The three-model chain is only a reference architecture. Production systems often introduce additional decisions around it.
Two placements serve different purposes:
- Before the main LLM: A classifier can route a request to the model responsible for refunds or the one responsible for order tracking.
- After LLM generation: An output guardrail can check the proposed response before the system produces it for the caller.
Each added component needs an explicit latency allocation, an SLA, and independent scaling where necessary. Bhargava offers no shortcut that makes those extra checks free.
The audience member sharpens the problem: once the agent has spoken the unauthorized offer, a later classifier cannot take it back. An apology or attempted revocation is a poor substitute for preventing the claim. Bhargava agrees that the check must happen before invoking text-to-speech. For the discount example, the ordering can be expressed with a small Python gate: a proposed offer reaches speech generation only after authorization succeeds.
python
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from decimal import Decimal
@dataclass(frozen=True)
class DiscountOffer:
percent: Decimal
async def speak_discount(
offer: DiscountOffer,
authorize: Callable[[DiscountOffer], Awaitable[bool]],
synthesize: Callable[[str], Awaitable[bytes]],
) -> bytes | None:
if not await authorize(offer):
return None
return await synthesize(
f"I can offer you a {offer.percent}% discount."
)
For DiscountOffer(Decimal("8")), a failed authorization returns no audio and never calls synthesize. The response remains a proposal until the check passes; the function illustrates the pre-speech boundary, while the authorization policy is supplied separately.
Bhargava then describes a thinker–talker pattern for separating conversational responsiveness from heavier work:
- A small conversational LLM receives the transcript and gives a short acknowledgment, such as “Let me think about it.”
- It delegates through a tool call to a larger model with the relevant instructions, tools, and potentially more guardrails.
- The larger model produces the substantive response, which feeds text-to-speech.
This keeps the conversational role distinct from the deeper task execution. It also adds components whose reliability and timing must be observable individually.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Audit and evaluate the whole conversation
The final question returns to speech-to-speech: removing the transcript from the main response path does not remove the need for logs, auditability, or evaluation. One option is to run transcription alongside the audio-native model. That gives operators a text view of incoming and outgoing speech while audio generation continues.
Evaluation must also follow the new architecture. Separate text-to-text and text-to-speech tests no longer map cleanly onto a model that listens and speaks directly. Longer, full-duplex conversations become the unit of evaluation, with metrics focused on the complete interaction. Bhargava’s final clarification is operational: this evaluation generally runs above the base inference API. The model supplies inference; the surrounding system still has to establish whether the conversation worked.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python framework for assembling streaming voice agents from speech, language-model and transport components.
Documentation for building realtime agents with Python or Node.js and connecting media to AI pipelines.
Speech-recognition code and weights, including transcription and lower-level decoding examples.
Overview of the 12B full-duplex speech-to-speech model and its early-access program.
Current official documentation entry point for realtime audio applications.
May 2025 acquisition announcement documenting Refuel's founders and its move into Together AI.
Further reading
English streaming ASR model card covering cached encoder context, configurable audio chunks, deployment examples and benchmark conditions.
Original study measuring question-response timing in natural conversations across ten languages.
Read the complete timestamped transcript
- 0:00
[on hold music] All right. Well, folks, thanks for being here.
- 0:16
Excited to chat more about how to engineer voice agents, you know, high quality, low latency at scale. First, maybe a little bit about me. My name is Rishabh. Uh, I work at a company called Together AI.
- 0:26
I lead the voice AI team there. Uh, prior to Together, I was the co-founder, CEO of a company called Refuel that was acquired by Together last year. But generally, been building AI and machine learning infrastructure for about a decade.
- 0:38
Um, for folks who maybe don't know about Together, Together is building the AI native cloud. What that really means is for companies that are looking to train models and need access to reliable compute, or you want to do inference at scale, uh, we're probably a very good fit for you.
- 0:52
Uh, we work with... You know, there's a million-plus developers. We closely work with hundreds of companies and, you know, very proud to be working with companies like Cursor and Decagon.
- 1:01
Okay, here is the agenda for today. So we're gonna first talk about, you know, why are we talking about voice, although the previous speaker alluded to a lot of interesting things that he's doing with voice.
- 1:11
But why does voice really matter? What does it actually take to build voice agents at scale? What are the challenges? We'll talk about this pipeline architecture, which is becoming the dominant way to build agents in production today.
- 1:22
Um, we'll go a little-- we'll deep dive a little bit. We'll look at all the components. We'll look at the system and trade-offs. And then finally, we'll chat about maybe what might be the next generation of building voice agents in the coming months and years.
- 1:34
Okay, so starting with why voice matters. Well,
- 1:38
there's billions of phone calls a year that are still handled by humans. I am pretty sure all of us have the experience of calling customer support, asking about the status of our order, looking to change the reservation.
- 1:50
We've also probably had the experience of calling a doctor's office because we've got to g- book an appointment for ourself or a loved one, and pretty much everybody here probably has had the experience of being on hold.
- 2:00
Now, it would be amazing for AI agents to be able to handle some of these calls for us. But really, one of the more exciting directions is, frankly, voice is just this brand-new interface to interact with systems and computers.
- 2:13
Look, humans, we learn how to talk before we learn how to read, right? So this comes very naturally to us. And obviously, we're seeing this with ChatGPT's advanced voice mode, and we're seeing this with folks who are starting to directly talk to Cursor, talk to Claude Code in order to get their work done.
- 2:30
And this is, frankly, just the beginning. And one of the exciting pieces of 2026 is, you know, building these rich, high-quality conversations. This is not the domain of science fiction or research anymore.
- 2:42
This is primarily an engineering problem today. Now, why is it hard? Well, there's a few things that you've got to solve first. First, building voice AI and building voice agents, this has to be real-time.
- 2:55
You know, when humans are having a conversation, we respond to each other's cues in something like three hundred milliseconds. And so if you're talking to an AI and it's taking more than five hundred milliseconds to respond, you'll start to notice.
- 3:08
If it takes a second, if it takes two seconds, people will just hang up. So you've got to get latency down. The next thing that matters is you want it to be a reasonably smart call.
- 3:19
You want to get the work, the job done. And so for real-world complex workflows, you know, the instructions are complicated. Uh, there's a lot of ambiguity. Uh, you have to be good at tool calling because that's the way you give agents access to the real world.
- 3:34
So you have a baseline level of intelligence that you've got to meet. The third piece that you've got to solve for is the voice has to be natural enough.
- 3:41
It has to sound pleasant enough. Um, and this means a lot of different things. It means, you know, can it talk to you in your own language with the right accent, potentially?
- 3:49
Uh, can it pronounce your name? Can it deliver the right emotion that is needed for a particular situation? A lot of the things fall into that bucket. And finally, you know, you could stitch together like a nice demo with one person calling, but what happens when you're doing a hundred calls, a thousand calls, ten thousand calls concurrently?
- 4:08
Reliability really starts to matter. And this is an and problem. You have to solve every single one of them at the same time, or you're gonna be in a little bit of a trouble.
- 4:20
So at least today, the dominant way of building these voice agents is this pipeline architecture or this cascading architecture, which attempts to solve all of the problems that I outlined earlier.
- 4:32
Now, there's a few boxes that are going on, but you know, conceptually, it's relatively simple to understand, which is audio chunks from an end user that are being streamed in, potentially to an agent orchestrator, something like a Pipecat or LiveKit or something that is homegrown.
- 4:46
And then essentially, this audio is being fed into a speech-to-text system that converts it into text. That's being fed into an LLM that then decides, do I do a tool call?
- 4:55
What is the output? Produces text that is then fed into a text-to-speech model, which starts to produce audio chunks that are then streamed back to the end user. So that's a rough architecture.
- 5:06
Uh, let's maybe look at like each of the components, uh, one by one, the components that matter here. The first is speech-to-text, you know, very much like the ears of your agent here.
- 5:17
Um, the, the performance metrics that matter here, the first is quality, word error rate. Now, you know, depending on use case, the numbers might look different, but state-of-the-art models are typically in the six percent word error rate on open benchmarks.
- 5:30
What that really means is the transcript that is produced by your model, comparing it with a reference transcript, that six percent of words have an error in them. Now, you can imagine why this might matter, right?
- 5:41
Because if you don't get the transcript right, you don't get somebody's name right, you don't get the name of, let's say, a drug right, essentially, there's no way to fix this.
- 5:50
Your LLM will make the-- will carry forward the mistake. The TTS model will carry forward the mistake. So you have to get it right for the important keywords. And then the second metric that matters often, which is latency-driven, is time to complete a transcript.
- 6:03
The way to understand this is when somebody completes an utterance, they stop speaking, how many milliseconds does it take for you to complete the transcript and have that be ready for the LLM?
- 6:13
And so as an example, for some of the models that we run on Together, where we get sort of consistently P90 of like a hundred milliseconds, which is pretty fast.
- 6:22
Aside from just raw performance, there's a few other capabilities that matter. Turn detection, very important, still somewhat unsolved problem. Frankly, that could be a twenty-minute talk in itself. But really the best way to understand this is you've got people who are, you know, they're talking, maybe they pause for a second.
- 6:38
But do you actually know, does that pause mean their turn has ended? Are they gonna continue talking? Because really the last thing that you want here is for you to-- for the agent to start, you know, sending audio back and talking at this person even though their turn hasn't ended.
- 6:53
We don't enjoy this in human conversations, and we will certainly not enjoy this in AI conversations.
- 6:59
Depending on who your customers are, language matters, and so being able to do this for a wide variety of language and getting it right there, it's important. And the final piece that I'll mention, this is somewhat new, is, um, we're sort-- also starting to see architectures, model architectures that are streaming native.
- 7:16
Um, a little sidebar, we won't spend too much time on this, but there's an architectural evolution for speech-to-text models that is in progress, which is going from batch models to stream models.
- 7:26
Whisper is the sort of canonical model, came out a few years ago. Um, it was trained on thirty-second audio clips. Uh, thirty seconds is way too much. You can't wait thirty seconds to start to do transcription.
- 7:37
So people have had to build all sorts of complicated logic around models like Whisper to do chunking and to pad it with silences, and then they make multiple calls, stitch that together to produce the final transcript in st- in streaming mode.
- 7:50
But recently, and this is a fairly new model from the NVIDIA team, instead you have the encoder, uh, of the model have two interesting characteristics. The first is it's trained with different amounts of look-ahead time, so it only looks at perhaps eighty millisecond or maybe up to a second of audio instead of thirty seconds.
- 8:08
Um, and it's also able to cache these activations so that as you kind of make small steps, uh, in audio frames, you're actually only doing the heavy computation once.
- 8:17
So again, uh, just kinda stepping out, but, you know, it's an interesting direction that we're seeing to be able to handle streaming conversations for these voice agent use cases.
- 8:27
Okay, so that's speech-to-text. Uh, jumping into the next part of the pipeline, which is LLMs, you know, very much the brains of, of your agent. Um, the performance metric that matters here first and foremost is streaming latency.
- 8:40
And so, you know, TTFT is the metric here. And a rough kind of, uh, you know, metric is like it would-- it's usually pretty good if you can get to three hundred milliseconds of TTFT, um, because, you know, you want to start producing tokens, start feeding that into the TTS model as fast as possible.
- 8:57
That number, two hundred to three hundred millisecond, has implications for what models you can use. And so a good size model typically ends up being in this eight to thirty billion range.
- 9:06
Uh, if you go any bigger, uh, you'll burn through your latency budgets. If you go too small, that has implications for the intelligence of the model and frankly, the tool calling that is needed, which are both pretty critical if you want to build a voice agent that does meaningful stuff in the world.
- 9:24
Okay, text-to-speech. This is very much, you know, the voice of your agent. There's a few interesting things on, uh, um, on sort of performance and capabilities. Performance, again, you know, the trend continues.
- 9:34
What is the time to first audio, right? As you get a transcript, how long does it take to produce the first audio chunk that can start be, start to be streamed back?
- 9:43
And aside from TTFA, um, what does the real-time factor look like? Real-time factor is, and this is generally the case for most TTS models, but what it means is how much, um, how much audio can you produce in certain number of seconds of processing time.
- 9:58
So if you can produce ten seconds of audio in five seconds, your, your RDF is point five. And so you typically want that to be less than one so that you're not buffering.
- 10:07
Um, quality is one of the hard ones with TTS because, you know, there are some objective measures, but frankly, nothing quite beats listening to audio samples for the models, for the voices that you care about, and getting a feel for is this the right, uh, experience that you want your end customers to have.
- 10:26
Some of the other capabilities, you know, it's, you know, naturalness across a different, a number of different voices, being able to pronounce things exactly right, whether it's, you know, customer names, whether it's product names, um, being able to have some amount of control over emotions.
- 10:42
And so you might see TTS models that allow you to add these different little tags which says, you know, this is happy or angry or sad, and it's the start, but these models are getting pretty good, uh, at emotional control.
- 10:53
And of course, coverage over language continues to matter.
- 10:57
Okay, so those are the main components. But just to kind of zoom out a little bit, you know, this is still-- all of these components are part of this larger architecture, which is multiple models, uh, being, being orchestrated.
- 11:10
And so the-- there's a few things that we should always kind of keep in mind, which is first, you know, there's a latency and cost budget across these models that we're thinking about.
- 11:19
A rough rubric is the LLM is going to take up a majority of it, followed by TTS, followed by speech-to-text, both from a latency and a cost perspective. Um, and so again, just kind of rough, uh, rules to kind of think about.
- 11:32
And one piece that we didn't mention, and we'll come back to this in a second, is so far a lot of the, the numbers that we're looking at is just engine latency.
- 11:40
How much time does it take the model to produce an output? But actually, when you're calling models that might be sitting in different data centers, there's network latency as well, and that starts to have an impact.
- 11:50
But again, we'll come to that in, um, in one more slide.
- 11:55
Autoscaling is also, you know, somewhat interesting and tricky to get right for, for agent systems. Of course, you wanna be doing autoscaling to scale up. As demand goes up, scale down.
- 12:05
Potentially, you know, night times or weekends, you want to scale down seamlessly. Um- Scaling up, you know, uh, what we've typically seen is people are much more aggressive about scaling up because the last thing you want is, you know, requests to be slowed down or backed up.
- 12:18
So you typically might autoscale earlier than you might do with somewhat more asynchronous systems. And scaling down is also tricky because you might actually have these stateful long-lived connections to your models, and so you can't just arbitrarily kill a pod.
- 12:33
You might want to wait for conversations to kind of fully finish. So some interesting nuances with autoscaling.
- 12:39
And finally, you know, it's like, you know, global deployments are important because you want your models, uh, you want your system to be as close to your end users to shave off latency as much as possible.
- 12:49
And of course, you know, if you're building sort of models in Europe or in places where residency matters, you want to be making sure that you have the ability to deploy wherever you, uh, we absolutely need.
- 13:01
I know I referenced colocation, um, but here's one way to kind of understand this problem. So the chart on the left-hand side, this is very much, you know, a very optimized, uh, sort of system where you're doing a pretty good job with your speech-to-text and text-to-speech and LLM models, where the engine latency is in exactly the right
- 13:21
ballpark. You're doing a hundred to two hundred milliseconds of time to first token or audio. But you might actually end up having your models being sufficiently far away from your agent orchestrator that it's taking seventy-five milliseconds of network latency.
- 13:36
Seventy-five milliseconds is really not that much. Like, you know, even in, I think like, you know, U.S. West to Europe would certainly be seventy-five milliseconds, but, you know, depending on networking, it can be much higher as well.
- 13:48
And so an interesting kind of direction that we're seeing folks go is, how can you colocate all your models and potentially your agent orchestrator to either be in the same data center or be very, very close to each other?
- 14:01
How can you get them literally in the same building? Because that drop from seventy-five milliseconds to five basically gets you a thirty percent reduction in already a fairly optimized voice agent setup, right?
- 14:13
So some of these things, um, especially again, with real-time systems, uh, is just pretty important to have like fairly deep observability and, you know, every ten milliseconds matters.
- 14:24
Okay. So hopefully that's an interesting picture on sort of this pipeline architecture. Um, but that's not the only way people do it. One of the other kind of directions that is becoming interesting is a pure speech-to-speech model.
- 14:38
And so instead of having speech-to-text followed by LLM, followed by text-to-speech, where you're sort of coordinating and orchestrating across a number of different models, um, it's just way simpler if you could have a pure speech-to-speech model that still is responsible for function calling, um, still handles all the complicated instructions, uh, but just a single model doing it.
- 14:59
And of course, you know, for folks who've played around with OpenAI's real-time API, they have a single model behind the scenes. Uh, NVIDIA recently launched a model called VoiceChat, again, very similar ideas.
- 15:10
The reason why most of these models are not used in production too much is because they h- they still have trouble with instruction following and tool calling. So, you know, the real-world experience often looks like you'll try them, um, and then you'll spend a lot of time just prompt engineering and hoping to kind of fix issues and,
- 15:27
you know, eventually move to a pipeline architecture. But as these models get better, which I'm confident they will, um, it has some pretty incredible benefits because suddenly you don't lose anything about the nuances of speech when that speech is getting converted into text.
- 15:44
So the model will natively understand what was the tone, what was the emotion, uh, was the user hesitant. That, that stuff will still remain with the model to make the next kind of decision.
- 15:55
And this type of model also allows for, um, sort of more full duplex comm-- like communication, which basically means that the model can start producing audio while it's still receiving audio.
- 16:06
And this means that, you know, as you're-- as a customer is talking to this model, you can backchannel. You can say, you know, "I see," or "Uh-huh," like similar to what a human conversation would look like.
- 16:15
And these models become much better at handling interruptions and margins, which again, with the pipeline architecture, you have to do a lot more complicated engineering workaround. So, um,
- 16:27
hopefully this kind of points in the direction of like, you know, what might be coming in the future. But frankly, we have a lot of engineering work ahead for all of these kind of voice interfaces that still have to be built.
- 16:37
Um, if you want to learn more about what Together does, you know, uh, here are a couple of links. We're hiring. We also have a booth G1 downstairs. Um, happy to chat more and happy to answer questions. [applause]
- 16:56
Um, for voice-to-function calling use cases, what sort of, um, evals do you use? And then what score do you need to get on those evals in order to have something that's good enough for production?
- 17:08
Um, that's-- Like the classic answer is like, it depends, y- you know. But, um, I think in terms of evals, like, you know, there's of, of course, like component-by-component evals.
- 17:18
So if we're talking about sort of like the pipeline architecture, um, and assuming that speech-to-text is good, text-to-speech is good, and the only thing that we're caring about is sort of function calling or tool calling evals on the LLM, then it's very similar to how one might do evals for tool calling for LLMs broadly.
- 17:34
Which is, you know, was the, um, uh, was the tool call correct? Was the output actually parsable? There's a bunch of those. One would imagine that, you know, you'd want the, uh, the tool call structure to at least be very close to a hundred percent.
- 17:47
Uh, um, and then the correctness, again, it depends a little bit on what does the use case kind of eventually demand. One of the other things that we are seeing is especially to get around-- to make models better at tool calling, and because we have to stay within that LLM budget of like, you know, the models have
- 18:04
to be relatively small, we do see customers fine-tune smaller LLMs with their kind of use case specific data so that they can get tool calling quality to go up while remaining a model that is relatively small
- 18:21
Um, sorry.
- 18:22
Yeah.
- 18:23
When you mentioned about colocation, so in that, what I understand from that is you're using-- you're decreasing the latency, network latency.
- 18:31
Yeah. Um, it's literally because the, the machines are closer to each other. So in a-
- 18:36
What does that mean in like, uh, if I'm using cloud providers or...
- 18:39
Uh, good question. So for example, um, uh, you know, let's say you're using-- let's say you're building a voice agent here in London, right? You have servers here, but you're using perhaps OpenAI's models for the LLM.
- 18:52
And now odds are that OpenAI servers might be somewhere in the US. So literally, the data has the-- the network hop has to be from your server here all the way there and back.
- 19:02
Uh, instead, if you had-- if you were able to run, let's say, an open source model in the data center that you're running, uh, your voice agent, now it's basically gonna be intra, uh, sort of, uh, d-- like, uh, the data center rather than going, um, let's say, over the Atlantic.
- 19:17
That's one way to kind of think about like... And just distance literally kind of like has that big of an impact, aside from other kind of networking related concerns.
- 19:25
Cool. Thank you.
- 19:27
Yes.
- 19:28
Uh, one more question. Um, so we've had, uh, uh, when trying these voice pipelines issue that we had to introduce some guardrailing. So like let's say we have classifier model in between, uh, which just checks that the model is not offering eight percent discount, which is not, uh, which is not, uh, authorized to do.
- 19:46
Um, how does it fit into the pipeline-
- 19:49
Yeah
- 19:50
... and how you do that without compromising the latency and the, the experience?
- 19:55
Yeah, it's a great question. So, uh, you know, the question is like, what if we have other models like guardrail models or other classifiers in the mix? How does this fit into this architecture?
- 20:04
Uh, you're absolutely right. Like this is the most kind of basic reference architecture that one might have. But in many production settings, there's actually multiple models that might be in the mix.
- 20:14
The guardrail or the classifier, we definitely see people kind of like start to introduce that right before the main LLM as well, because maybe you want to check, is this something that, um, something that goes to an LLM that handles refunds versus something that handles order tracking, perhaps.
- 20:28
And so you might have a classifier there. Guardrails at the end of the LLM generation before you produce a response, that also makes sense. And so often this ends up growing as the, um, as the sort of like scope of what you're hoping to achieve grows.
- 20:42
Um, and it puts real pressure on sort of like latency concerns and so forth. So, you know, no easy answers except that it becomes, you know, one more or two more components to think about, have very clear sort of guidelines and SLAs on, you know, how much budget can you really kind of associate with them, and then
- 20:58
sort of independently scaling them as needed. But unfortunately, no, no easy answers.
- 21:04
Yeah. Especially because when you have things like, uh, like the a- the agent already-
- 21:09
Yeah
- 21:09
... answered with an answer which was already not what something it should be, but the classifier, the guardrails for example-
- 21:15
Yeah
- 21:15
... catches later, you can't take back things that are spoken. You might have to do something, "Hey, sorry, I shouldn't have said that. Uh, I need to revoke that," or something.
- 21:23
So this is really a problem I see there.
- 21:25
Yeah. I, I think that's spot on. Like catching all of those before the TTS model gets invoked is, is certainly important. Uh, one of the other things, patterns that at least like we've seen is sort of this thinker talker pattern, where you might have a small LLM that is handling, um, all of the conversation.
- 21:42
And so as it gets sort of, uh, text from speech-to-text, it produces a response, and the response might look like, "Let me think about it," or, "Let me get back to you."
- 21:50
And then it basically issues one big tool-- one tool call to a much bigger model that then has, you know, much, you know, has better instructions, has all the tools associated, maybe more guardrails.
- 22:01
And then it produces a much cleaner response that, y- you know, the, the model is much more-- the, the architecture is like, you know, you're much more comfortable with.
- 22:08
And so that gets fed into the TTS model, uh, to produce a response. But, you know, in some ways, this is kind of the-- this is the kind of beauty of like all of these architectures is, you know, the components are, you know, you can-- you'll only add components, and so it kind of, you know, pushes more
- 22:23
on, um, sort of reliability, having like, you know, sort of detailed kind of observability on every single component.
- 22:32
Yes.
- 22:34
Hi. Um, I think I, I was just, um, curious about the, the upcoming voice-to-voice, uh, conversations coming out. And I know that this is upcoming, but, um, at the end of the day, all the surrounding infrastructure that we have in the, you know, through Trump, while we have the conversational systems now and now, I guess, with the,
- 22:53
um, pipeline approach, uh, how do we-- like how do you do evals in that model and observability? Like we still need to transcribe everything.
- 23:02
Yeah. Uh, very good question. So how does sort of observability, logging, evals change? Um, so i- in some ways-- so some parts of it can still remain the same.
- 23:13
So sometimes what you might have is, aside from pure speech-to-speech, you might actually have a, a, a transcription model that is running so that you can at least seeing the transcription, uh, at the same time as the audio is being generated.
- 23:24
So that gives you some amount of auditability in terms of what audio is coming in and what audio is being produced. But yes, you know, some evals are going to change.
- 23:33
There is no, um, there's no sort of real concept of sort of text-to-speech anymore. There's no concept of pure kind of text-to-text here anymore. And so the evals become much more full duplex kind of conversation evals, which is like a much longer conversation and then, you know, evals and metrics, uh, that are sort of focused on that
- 23:51
entire conversation.
- 23:53
Right. But what I mean is like, is the nature of those models like able to, uh, output the correct respects of the, the conversation in a way that you can evaluate?
- 24:02
Is it that native to API or is that something that you do on top?
- 24:06
Typically, a lot of the eval stuff would happen on top, on top of the base kind of inference API.
- 24:12
I think, I think we're running out of time, but... Okay. Um, well, thank you everybody for the-- for, for being here. [clapping] [upbeat music]