AI Engineer World's Fair 2025
Optimizing inference for voice models in production
Read the talk
Optimizing voice inference beyond tokens per second
Orpheus TTS shows how LLM runtimes, audio decoding, batching and client connections work together to deliver more concurrent speech streams with less startup delay.
From a talk by Philip Kiely
Before you start: Familiarity with autoregressive token generation, GPU inference and streaming HTTP will help with the runtime and client examples.
What can LLM tooling do for a speech model?
How do you make a text-to-speech model faster in production—and keep the client from adding the latency back? Philip Kiely approaches that problem through his work at Baseten, a model inference platform serving production workloads for AI startups and enterprises. The running example is Orpheus TTS: inspect its architecture, choose useful performance metrics, optimize its runtime, and then preserve those gains through deployment.
The starting simplification is that much of this work resembles LLM optimization. Kiely explicitly calls the idea that everything is an LLM wrong but useful. Transformer-based systems span neighboring tasks, including embeddings and transcription with Whisper; diffusion-based image generation presents a different optimization problem. For TTS models derived directly from language models, the practical benefit is access to an established ecosystem of inference tools.
Orpheus makes that connection concrete. Kiely chooses Canopy Labs’ model because it is open source, he likes its output, and he admires its creators. Its backbone is Llama 3.2 3B, and the Hugging Face configuration shown in the talk identifies the architecture as LlamaForCausalLM. That makes familiar Llama optimizations applicable to speech generation.
The configuration still has speech-specific requirements. Kiely points to a larger vocabulary for speech tokens, including laughter, and extended context lengths using RoPE scaling. An optimized runtime must preserve support for those features; recognizing the backbone is only the beginning of checking compatibility.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reach playback speed, then optimize first audio and concurrency
Voice serving changes what counts as a useful response. An LLM benchmark may emphasize time to first token, but a speech application needs enough output to begin doing something audible. That shifts attention toward time to first byte, sometimes even time to first sentence. Token generation speed still matters, alongside throughput: how many requests the service can handle at once.
For a general Llama deployment, the ambition might be hundreds or a thousand tokens per second. A real-time speech stream has a more specific requirement. Kiely gives approximately 83 tokens per second as Orpheus’s real-time playback requirement. Once a stream can sustain playback, extra decoding speed is less valuable than reducing the initial wait or serving more connections on the same hardware.
| Metric | Voice-serving objective |
|---|---|
| Time to first byte or sentence | Start useful audio sooner |
| Tokens per second | Sustain real-time playback |
| Simultaneous streams | Serve more agents per GPU |
After meeting playback speed, prioritize startup latency and concurrency. The collection of generated voices in Kiely’s example represents the agents a service needs to support. The deployment goal is to fit them onto one GPU—or a fraction of one—without making their conversations wait.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize token generation and audio decoding together
The first optimization uses Orpheus’s Llama architecture: run it with TensorRT-LLM. Kiely reports that it can outperform vLLM in many cases, while acknowledging a more complicated initial developer experience. Baseten uses it extensively; the tradeoff is more setup work in exchange for the runtime performance available once the engine is configured.
Next comes FP8 quantization on Hopper GPUs, including quantization of the KV cache. Reducing precision shrinks the representation used during inference. Small models can suffer from quantization, but Kiely reports good results with this Orpheus implementation even with the cache quantized.
Speech generation also has an audio path that a text-only LLM does not. Generated tokens must pass through audio decoding before they become playable output. Baseten uses the decoding component of SNAC, runs it on the GPU, and applies torch.compile together with PyTorch inference mode. The optimization boundary therefore includes both next-token prediction and conversion into audio.
The PyTorch pattern is to compile the decoder used for inference and invoke it without autograd tracking. Given a decoder module and its code tensors, that wrapper can look like this:
python
import torch
def prepare_decoder(decoder: torch.nn.Module):
compiled_decoder = torch.compile(decoder.eval().cuda())
@torch.inference_mode()
def decode(codes):
gpu_codes = [code.cuda(non_blocking=True) for code in codes]
return compiled_decoder(gpu_codes)
return decode
Compiling the decoder matters because speeding up token generation alone can leave the next stage consuming the saved time.
The remaining work connects those stages: make token batching function throughout the pipeline and support the streaming protocols clients need. Kiely describes engine settings for FP8 with KV-cache quantization and FP8 context FMHA on Hopper. These settings belong to the demonstrated engine configuration; their exact configuration syntax depends on the runtime version.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Batch the decoder without delaying the stream
The audio-decoding example uses dynamic batching. It collects work into a batch, then dispatches on a timeout rather than waiting indefinitely for more work. The demonstrated decoder batching timeout is 15 milliseconds. A larger batch can improve throughput, but collecting more work introduces a latency tradeoff.
This is distinct from the token-level continuous batching familiar from LLM serving. At the time of the demonstration, that continuous batching was not available in this decoding path; dynamic batching was the implemented approximation. The distinction matters because an optimized language-model scheduler does not automatically optimize the downstream decoder’s scheduling.
Profiling then revealed a less obvious limit: Kiely reports that this Orpheus implementation was often CPU-bound. Both next-token prediction and audio decoding ran on the GPU, yet both loops returned to the CPU at points along the way. Those CPU interactions could cap simultaneous streams. Additional CPU resources could therefore help concurrency even when the expensive model operations were already accelerated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure simultaneous streams and first-byte latency
The first test of these changes is how many streams the deployment can serve concurrently. Kiely compares the optimized system with an off-the-shelf, standard implementation, while explicitly acknowledging that vLLM can also deliver good performance. The comparison concerns the implementations being tested, not an inherent limit of vLLM.
Kiely reports the following optimized capacity on an H100 MIG:
| Traffic pattern | Simultaneous streams |
|---|---|
| Variable traffic | 16 |
| Constant traffic | 24 |
He describes the instance as half an H100. More precisely, his companion article specifies 40 GB of memory and three of seven compute partitions. The attraction is Hopper performance and FP8 support without allocating a full 80 GB GPU to a three-billion-parameter model.
Using Baseten’s list prices at the time, Kiely estimates a few cents per conversation-hour at sufficient volume. Utilization is central to that claim: the cost advantage over a per-token API depends on having enough traffic to use the available concurrent capacity.
In the talk, Kiely reports time to first byte as low as 150 milliseconds in Baseten’s real-world TensorRT testing on MIGs and H100s. The companion article separates those hardware results: 200 milliseconds on MIG and below 150 milliseconds on a full H100. The spoken figure should therefore not be treated as an established MIG result. In either case, TTS startup latency covers one component of the voice pipeline, not the time from a user speaking to an agent answering.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the client from erasing the runtime gains
Copying an engine configuration into production does not reproduce the benchmark by itself. With small models and multimodal systems, infrastructure and client behavior can dominate the remaining latency. Kiely’s example is straightforward: save a couple hundred milliseconds in the runtime, then add them back by routing a California request to New York or establishing a fresh session on every call.
A quickstart client is designed to make the first result easy. It might use Python requests, stream the response to the local machine, and play it with FFmpeg. But the demonstrated simple pattern sends requests sequentially and creates a new session each time. It adds connection overhead and never exercises the server’s ability to handle concurrent streams.
A production or load-testing client needs the behavior of a benchmark harness:
- Use a multiprocess pool to distribute client work.
- Create and reuse a session within each worker process.
- Issue concurrent streaming requests at a load that exercises the server’s capacity.
Session reuse here does not mean sharing one live aiohttp session across processes. Each worker owns its session and reuses its connections for the requests it handles. That preserves connection reuse while allowing the client to generate enough concurrent traffic.
Both client examples use HTTP streaming. A larger voice application may instead be built with LiveKit Agents or Pipecat, with WebSockets or gRPC as its transport. Kiely names both protocols as supported options. The transport and framework are part of integrating the inference service into the application, not a replacement for efficient request handling.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Budget latency across listening, thinking and talking
There is more to improve inside the voice model itself: fine-tuning, custom voices, zero-shot voice cloning, and removing static or popping at the ends of messages. Kiely names those as further work rather than developing them here. But even a polished TTS service handles only the talking stage of an agent that must also listen and think.
Connecting those stages efficiently is a separate engineering task. Running successive models in the same data center reduces network overhead between them. Avoiding unnecessary DNS-level hairpin routing removes round trips that do no useful inference work. As the pipeline gains a chunking algorithm or an interruption model, the number of handoffs can grow beyond the three conceptual stages.
In Kiely’s illustrative budget, 10 milliseconds of hairpin overhead at each of four or five steps adds 40–50 milliseconds, potentially 10% of the latency SLA. Small per-hop costs accumulate into a noticeable share of the response budget. Runtime optimization remains valuable, but infrastructure placement and client implementation determine how much of that improvement reaches the person waiting to hear the agent speak.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Canopy Labs' speech model repository with streaming inference, emotion prompting and fine-tuning examples.
Neural audio codec with hierarchical discrete tokens, pretrained models and encoding and decoding examples.
NVIDIA's library and runtimes for optimized language-model inference on NVIDIA GPUs.
Framework for building realtime voice agents and connecting speech and language-model components.
Open-source framework for voice agents and realtime multimodal applications.
Further reading
Philip Kiely's companion explanation of Orpheus serving benchmarks, runtime optimizations and session reuse.
Python client example for concurrent streaming requests to a deployed Baseten Orpheus endpoint.
Read the complete timestamped transcript
- 0:00
[upbeat music] Uh, hello everyone.
- 0:15
Thank you so much for being here, for, uh, sticking around for this talk. Um, I'm gonna be talking about optimizing inference for voice models in production. I'm going to be talking mostly about the runtime component, but also just a little bit on the infrastructure side.
- 0:30
Um, just a quick introduction. Um, I'm Philip from Baseten. Baseten is a model inference platform. Uh, we run production workloads for a wide variety of AI native startups and enterprises.
- 0:41
Um, I'm based in here in SF. Um, I actually just moved here. It's really awesome. My favorite part about being in SF is much better sports teams than I had in Chicago.
- 0:50
Um, and uh, one of my favorite voice models is Orpheus TTS, um, which we're gonna be talking about a whole bunch today. Um, quick agenda. So we're gonna talk about TTS model architecture, like what is a text-to-speech model actually when you look on the config in Hugging Face.
- 1:07
Uh, what sort of performance metrics are we looking at? What sort of optimization techniques can we do to make the model better? Um, how do we measure whether or not we've succeeded?
- 1:16
And then finally, what can we do on the infrastructure and client code to not shoot ourselves in the foot after doing a ton of runtime work and then just adding all that latency back by not doing our client code correctly.
- 1:29
So architecture. This is one of the things I've been learning this year, which has been pretty great, uh, to, to realize. It's made life a lot simpler at the runtime level.
- 1:39
Now, this is wrong. Like the, the, the thing up here, uh, that, that I'm gonna say is that like everything is an LLM, uh, that is, that is wrong, but it's useful.
- 1:48
Um, there's kind of like two types of models. There's autoregressive transformers models that are LLM or very LLM adjacent. Um, you see this in embeddings, you see this in transcription with stuff like Wispa.
- 1:59
TTS, um, is another example. You also have the more like diffuser image type models, um, which is like a very different optimization problem. But something that's cool is because TTS models are so architecturally similar to LLMs or in many cases derive directly from LLMs, we can access the rich ecosystem of LLM tooling and use it to make
- 2:19
TTS models better. So the TTS model that we're going to be using and as an example all day, um, is Orpheus TTS. We're using it for two reasons. Okay, three reasons.
- 2:31
The two reasons are 'cause it's open source and it's really good. And also I think, uh, Elias and Amoo and everyone at Canopy Labs is really awesome. So, uh, that's the, that's the third reason we're, we're talking about their model.
- 2:42
But it's a Llama 3.2 3B backbone. So like if you look at this is the little like config from Hugging Face copy and pasted onto the screen. It's a Llama Focausal LLM architecture.
- 2:54
And so because of that we can do like all of our normal Llama stuff to this model and make it faster. Um, they did a couple things. I mean they did a bunch of things to make it work.
- 3:03
But a couple things that are relevant here. There is a larger vocab size because you need all the speech specific tokens like laugh and stuff. Um, and then they also extended the context lengths with slope scaling.
- 3:14
So we gotta make sure everything we do supports that. So performance metrics, like what do we wanna actually do here?
- 3:22
Uh, we, we think about LLM metrics, um, a little bit here. Uh, we, we just look at them a little bit differently. So in LLMs you talk about time to first token.
- 3:31
Now we're talking about time to first byte or sometimes even time to first sentence. Uh, we, we need a little bit more of a useful output, um, from the model before we really start feeling good about our response time.
- 3:42
We do think about tokens per second, although we're gonna think about it differently, which I'll explain later. And we mostly think about throughput, which is, you know, how many requests are we able to serve at a given time.
- 3:52
So on that, you know, goals perspective, if you ask me like, "Hey Philip, how do you wanna optimize Llama in general?" I'll like, I'll say, "Well, we want a lot of TPS.
- 4:01
We want hundred, we want five hundred TPS. We want a thousand tokens per second. We want as many tokens per second as we can get." With voice models, you actually don't necessarily need that.
- 4:11
In many cases, you only want as many tokens per second as you need for a real-time stream. For Orpheus, that's like eighty-three tokens per second, which for like a three billion parameter LLM is nothing.
- 4:22
Um, but what we actually wanna do instead is we want to, once we hit that mark, start optimizing for time to first byte so that our latency's really good and start optimizing for concurrency so that we can get more connections and spend less on GPUs.
- 4:39
So our goal in general, if all of these very nice and definitely not AI generated people, all the different like voices that our, that our model is capable of creating, these are all the voice agents that we're running.
- 4:50
How can we make all of these people fit on one or even less than one GPU? That's the goal. So how do we do it?
- 5:00
Bunch of ways. Um, so first off, it's an LLM. Um, if you are running an LLM with like vLLM, for example, um, you can generally in many cases get better performance with TensorRT-LLM.
- 5:12
Uh, TensorRT is something that we've been using at Baseten a lot. I like to joke that I'm the unofficial marketing department for TensorRT-LLM because of how much I talk about it.
- 5:21
Uh, but it really is fast. It can be a little bit complicated from a developer experience perspective to get up and running with it. But once you are up and running, um, it, it works really well.
- 5:30
Uh, we can also just like quantize the model. Um, even though it's small, you can always make it faster by making it smaller. Uh, with Hop architecture, we quantize this model to FP8 pretty successfully.
- 5:41
I know usually quantizing really small models like this can lead to performance degradation, but for this model it's working pretty well in FP8 even when we quantize the KV cache
- 5:51
And then a lot of the other runtime stuff is actually more like audio specific than it is LLM specific. So one of the big challenges that we don't have with LLMs, which are just parsing nice convenient bits of text back and forth, is you have your audio, you have your audio codec, you have your decoding, all that
- 6:06
kind of stuff. So we use SNAC, which I was very disappointed to learn is not an actual tasty snack, um, but an audio decoder. And we actually use torch.compile.
- 6:16
Um, and torch.compile you might be used to running, uh, on, you know, a, a model, um, co-compiling your model weights to make your runtime faster. We're actually using the same kind of system with torch.compile and with, um, PyTorch inference mode on the audio decoder and running that on the GPU.
- 6:35
Um, we make sure that all the token batching, um, token level batching works well throughout the entire pipeline and support multiple streaming protocols. Um, yeah, so these are the engine settings that you would need.
- 6:47
Um, you've got the, you know, quantization type of FP8KV, the FP8 context, uh, FMHA, um, in order to, you know, support the, uh, support the, um, Hopo architecture and the, uh, quantization there.
- 7:05
Um, and here's a quick code sample of... I got a little ahead of my slides, I guess. Here's a little quick code sample of the audio decoding. Um, so we are basically, you know, batching.
- 7:16
Um, usually we would talk about continuous batching when we're talking about LLM optimization. We wanna package all those tokens together. In this case, we are doing dynamic batching, um, so we're trying to pack as much into a batch as we can, but every fifteen milliseconds we're gonna, uh, shoot it out.
- 7:33
Um, you've got that timeout set up here. Um, if you wanna trade off, um, for a little bit of latency for more throughput, you can make that batch bigger.
- 7:41
Um, so yeah. We don't have token level continuous batching yet here, but we do have dynamic batching, which is gonna get you pretty close. Um, and because of this, actually, something that's, that I was surprised about, uh, when we profiled this is that our TTS imple-implementation with Orpheus is actually in many cases CPU bound, which is kind
- 8:00
of where you wanna be. Um, you can throw more CPUs at a resource, uh, pretty, pretty efficiently. Um, even though the next token prediction and the audio co-decoding are both on the GPU, um, both of those loops hit the CPU at different points, um, and that can actually be the bottleneck in the number of simultaneous streams that
- 8:17
we're able to create. So how'd we do? Like, I just showed you a lot of code and talked through it really quickly without really getting into depth. Uh, that could all just be smoke and mirrors.
- 8:28
Uh, let's see, let's see if it's actually any faster. Um, so again, the number one thing is gonna be simultaneous streams because you want to be able to be very cost efficient and use fewer GPU resources to serve a, you know, large amount of traffic.
- 8:44
And in this case, a, a base implementation, I don't necessarily wanna like call anyone out because there's a lot of really good ways to run this model. Um, you can get really good performance with vLLM, um, but this is just kinda like the off the shelf, um, just take it, run it completely standard, uh, implementation.
- 9:02
So with variable traffic, we're able to support sixteen simultaneous streams and with constant traffic, twenty-four simultaneous streams on, um, an H100 MIG. So this is actually half an H100 GPU.
- 9:13
It's a SKU that we do a lot because it's really good for these small models where you want the Hopu performance, the Hopu architecture uplift in TensorRT-LLM, the FP8 support, but you don't wanna pay for like an entire eighty gigabyte GPU for just a three billion parameter model.
- 9:29
So, you know, we're, we're seeing much better, um, mu-much better concurrency. So if you kinda like price that out with like our list prices and stuff, um, you can get, you know, a few cents per hour of, of conversation, um, which is going to be, you know, substantially better than if you...
- 9:45
If you have the volume for it, it's gonna be substantially better than paying for a sort of like per token type API. But okay, so maybe it's cheap at scale, but is it fast?
- 9:57
Yes, it's fast. Um, so with the, you know, with the TOT implementation on the MIGs and on the H100s, we can actually get all the way down to a hundred fifty millisecond time to first bytes, um, in like real world, uh, testing that we've done.
- 10:12
Now that we'll, we'll talk in a minute, like that doesn't mean your whole pipeline is that fast. That's just like one part of the pipeline. Uh, but it's, it's important because, you know, you definitely don't want to be spending a lot of time waiting around for that first token.
- 10:27
So to kind of transition into that, that discussion of like what can go wrong here. Like you have this graph and you have this, you know, nice, uh, config that I had up here, and you're like, "All right, cool.
- 10:40
I'm gonna take this, I'm gonna put it in production, and I'm gonna see the results that, uh, that he put up on screen and it's gonna work great." And the answer is no, it's not.
- 10:49
It's a little bit harder than that. Um, so the thing is like non-runtime factors when we get, especially with these small models and with these multimodal systems can actually be like way more important than your runtime.
- 11:03
Um, and that's your lat- your infrastructure and your client code because, you know, I, I showed here... All right, maybe, maybe I got it, you know, I, I cut the runtime in, in half, um, from, from the base implementation.
- 11:16
I, I saved a couple hundred milliseconds. Very easy to add those couple of hundred milliseconds back and well beyond that by, you know, sending my query to New York instead of California or by having to establish a session every time I, uh, you know, run my client code.
- 11:32
Um, so a few like pitfalls to avoid. Um, number one, like if you go in, you know, a, a model library or something and we're just trying to get you started very quickly, um, with this, with this kind of inference sample, um, it's basically gonna be, hey, use requests, make a stream, stream it to your local computer
- 11:51
and start, you know, playing it on FFmpeg or something. The issue is that here, like the requests are gonna be sent sequentially and you need to create a new session every time.
- 12:01
Um, that takes time. So if you're using this in production, you want a code sample. Uh, by the way, this is all up on, uh, GitHub. Um, you'll want a code sample that looks a lot more like a benchmarking script where you're using a, um, multi-process pool.
- 12:17
Um, you're sharing the, you're sharing the session between all of these different requests, and you're actually sending traffic with the concurrency that allows you to, you know, saturate this benchmark with, you know, the, the multiple concurrent requests.
- 12:34
Finally, both of these code samples, they do sit on top of HTTP and HTTP streaming. Um, in many cases, if you're implementing voice pipelines, you're gonna use something like LiveKit or Pipecat or something, and you're also potentially going to be using a different protocol.
- 12:49
You're gonna be using something like WebSockets or gRPC, which we do have support for. Um, and finally, I wanted to leave you on the thought that these, uh, you know, these models are only one part of a voice agent pipeline.
- 13:02
So, like, we can spend a lot more than 15 minutes actually talking about, like, the very detailed, uh, implementation mechanics of making your voice model faster. Of, you know, we, we haven't even touched on stuff like fine-tuning the model, um, you know, custom voices, zero-shot voice cloning, um, being able to, you know, remove static and popping at
- 13:23
the end of messages. There's, there's, there's a lot of work to do just on the voice part, but it really only is one-third of the problem. When I think about voice agents, I think about three parts: listening, thinking, talking.
- 13:35
Um, and the most important thing here is, again, while you can have great runtimes, the infrastructure to connect these three together is really what's going to determine your latency.
- 13:45
Being able to go from one model to have the next one running in the same data center with, you know, minimal, like, minimal network overhead in between the two.
- 13:57
Uh, even things as simple as not having to go off and do a halo to pin, um, at the DNS level and come back. If that saves you ten milliseconds on every step and your voice pipeline has this and, you know, a chunking algorithm, it's got an interruption model, and so a- you end up having four or
- 14:12
five steps, well, they'll-- Just halo pinning alone is costing you forty or fifty milliseconds, and that can be ten percent of your SLA for, for a voice model. So yeah, uh, that's, that's my, that's my main point here, is that as much fun as it is to talk about the runtime stuff and as much work as we
- 14:30
do there, the, the infrastructure and the client implementation is equally important, if not more so. Anyway, thank-- Uh, so yeah, that's the, that's the review. Um, thank you all for coming through.
- 14:41
Uh, I have a-- We're doing an, an event next week at, uh, [REDACTED:location], which is gonna be pretty fun. I'm gonna be talking in more detail about building some, uh, systems with open source models, and there's also gonna be a lot of steak.
- 14:54
So definitely come on through, uh, if you're interested and, um, I'm on, I'm on Twitter, I'm on LinkedIn, so it's Baseten. Um, hit me up if you have any questions about this or anything else model performance.
- 15:04
Thank you so much, and I'll let you go eight seconds early. [outro music]