Optimizing inference for voice models in production

How do you make a text-to-speech model faster in production—and keep the client from adding the latency back? Philip Kiely approaches that problem through his work at Baseten, a model inference platform serving production workloads for AI startups and enterprises. The running example is Orpheus TTS: inspect its architecture, choose useful performance metrics, optimize its runtime, and then preserve those gains through deployment.

The starting simplification is that much of this work resembles LLM optimization. Kiely explicitly calls the idea that everything is an LLM wrong but useful. Transformer-based systems span neighboring tasks, including embeddings and transcription with Whisper; diffusion-based image generation presents a different optimization problem. For TTS models derived directly from language models, the practical benefit is access to an established ecosystem of inference tools.

Orpheus makes that connection concrete. Kiely chooses Canopy Labs’ model because it is open source, he likes its output, and he admires its creators. Its backbone is Llama 3.2 3B, and the Hugging Face configuration shown in the talk identifies the architecture as LlamaForCausalLM. That makes familiar Llama optimizations applicable to speech generation.

The configuration still has speech-specific requirements. Kiely points to a larger vocabulary for speech tokens, including laughter, and extended context lengths using RoPE scaling. An optimized runtime must preserve support for those features; recognizing the backbone is only the beginning of checking compatibility.

Voice serving changes what counts as a useful response. An LLM benchmark may emphasize time to first token, but a speech application needs enough output to begin doing something audible. That shifts attention toward time to first byte, sometimes even time to first sentence. Token generation speed still matters, alongside throughput: how many requests the service can handle at once.

For a general Llama deployment, the ambition might be hundreds or a thousand tokens per second. A real-time speech stream has a more specific requirement. Kiely gives approximately 83 tokens per second as Orpheus’s real-time playback requirement. Once a stream can sustain playback, extra decoding speed is less valuable than reducing the initial wait or serving more connections on the same hardware.

After meeting playback speed, prioritize startup latency and concurrency. The collection of generated voices in Kiely’s example represents the agents a service needs to support. The deployment goal is to fit them onto one GPU—or a fraction of one—without making their conversations wait.

The first optimization uses Orpheus’s Llama architecture: run it with TensorRT-LLM. Kiely reports that it can outperform vLLM in many cases, while acknowledging a more complicated initial developer experience. Baseten uses it extensively; the tradeoff is more setup work in exchange for the runtime performance available once the engine is configured.

Next comes FP8 quantization on Hopper GPUs, including quantization of the KV cache. Reducing precision shrinks the representation used during inference. Small models can suffer from quantization, but Kiely reports good results with this Orpheus implementation even with the cache quantized.

Speech generation also has an audio path that a text-only LLM does not. Generated tokens must pass through audio decoding before they become playable output. Baseten uses the decoding component of SNAC, runs it on the GPU, and applies torch.compile together with PyTorch inference mode. The optimization boundary therefore includes both next-token prediction and conversion into audio.

The PyTorch pattern is to compile the decoder used for inference and invoke it without autograd tracking. Given a decoder module and its code tensors, that wrapper can look like this:

import torch


def prepare_decoder(decoder: torch.nn.Module):
    compiled_decoder = torch.compile(decoder.eval().cuda())

    @torch.inference_mode()
    def decode(codes):
        gpu_codes = [code.cuda(non_blocking=True) for code in codes]
        return compiled_decoder(gpu_codes)

    return decode

Compiling the decoder matters because speeding up token generation alone can leave the next stage consuming the saved time.

The remaining work connects those stages: make token batching function throughout the pipeline and support the streaming protocols clients need. Kiely describes engine settings for FP8 with KV-cache quantization and FP8 context FMHA on Hopper. These settings belong to the demonstrated engine configuration; their exact configuration syntax depends on the runtime version.

The audio-decoding example uses dynamic batching. It collects work into a batch, then dispatches on a timeout rather than waiting indefinitely for more work. The demonstrated decoder batching timeout is 15 milliseconds. A larger batch can improve throughput, but collecting more work introduces a latency tradeoff.

This is distinct from the token-level continuous batching familiar from LLM serving. At the time of the demonstration, that continuous batching was not available in this decoding path; dynamic batching was the implemented approximation. The distinction matters because an optimized language-model scheduler does not automatically optimize the downstream decoder’s scheduling.

Profiling then revealed a less obvious limit: Kiely reports that this Orpheus implementation was often CPU-bound. Both next-token prediction and audio decoding ran on the GPU, yet both loops returned to the CPU at points along the way. Those CPU interactions could cap simultaneous streams. Additional CPU resources could therefore help concurrency even when the expensive model operations were already accelerated.

The first test of these changes is how many streams the deployment can serve concurrently. Kiely compares the optimized system with an off-the-shelf, standard implementation, while explicitly acknowledging that vLLM can also deliver good performance. The comparison concerns the implementations being tested, not an inherent limit of vLLM.

Kiely reports the following optimized capacity on an H100 MIG:

He describes the instance as half an H100. More precisely, his companion article specifies 40 GB of memory and three of seven compute partitions. The attraction is Hopper performance and FP8 support without allocating a full 80 GB GPU to a three-billion-parameter model.

Using Baseten’s list prices at the time, Kiely estimates a few cents per conversation-hour at sufficient volume. Utilization is central to that claim: the cost advantage over a per-token API depends on having enough traffic to use the available concurrent capacity.

In the talk, Kiely reports time to first byte as low as 150 milliseconds in Baseten’s real-world TensorRT testing on MIGs and H100s. The companion article separates those hardware results: 200 milliseconds on MIG and below 150 milliseconds on a full H100. The spoken figure should therefore not be treated as an established MIG result. In either case, TTS startup latency covers one component of the voice pipeline, not the time from a user speaking to an agent answering.

Copying an engine configuration into production does not reproduce the benchmark by itself. With small models and multimodal systems, infrastructure and client behavior can dominate the remaining latency. Kiely’s example is straightforward: save a couple hundred milliseconds in the runtime, then add them back by routing a California request to New York or establishing a fresh session on every call.

A quickstart client is designed to make the first result easy. It might use Python requests, stream the response to the local machine, and play it with FFmpeg. But the demonstrated simple pattern sends requests sequentially and creates a new session each time. It adds connection overhead and never exercises the server’s ability to handle concurrent streams.

A production or load-testing client needs the behavior of a benchmark harness:

1. Use a multiprocess pool to distribute client work.
2. Create and reuse a session within each worker process.
3. Issue concurrent streaming requests at a load that exercises the server’s capacity.

Session reuse here does not mean sharing one live aiohttp session across processes. Each worker owns its session and reuses its connections for the requests it handles. That preserves connection reuse while allowing the client to generate enough concurrent traffic.

Both client examples use HTTP streaming. A larger voice application may instead be built with LiveKit Agents or Pipecat, with WebSockets or gRPC as its transport. Kiely names both protocols as supported options. The transport and framework are part of integrating the inference service into the application, not a replacement for efficient request handling.

There is more to improve inside the voice model itself: fine-tuning, custom voices, zero-shot voice cloning, and removing static or popping at the ends of messages. Kiely names those as further work rather than developing them here. But even a polished TTS service handles only the talking stage of an agent that must also listen and think.

Connecting those stages efficiently is a separate engineering task. Running successive models in the same data center reduces network overhead between them. Avoiding unnecessary DNS-level hairpin routing removes round trips that do no useful inference work. As the pipeline gains a chunking algorithm or an interruption model, the number of handoffs can grow beyond the three conceptual stages.

In Kiely’s illustrative budget, 10 milliseconds of hairpin overhead at each of four or five steps adds 40–50 milliseconds, potentially 10% of the latency SLA. Small per-hop costs accumulate into a noticeable share of the response budget. Runtime optimization remains valuable, but infrastructure placement and client implementation determine how much of that improvement reaches the person waiting to hear the agent speak.