Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing

What does it take to host Orpheus inside a real-time voice stack? Generating a convincing voice is only part of the job. The deployment also has to move audio through a live session. Neil Dwyer approaches that problem through Gabber’s experience operating its own inference infrastructure.

The background is real-time media engineering. At Bebo, Neil built game-streaming infrastructure and an ML pipeline that watched Fortnite gameplay and triggered effects for kills or victories. That meant working with GStreamer, WebRTC, and RTMP. After Uber, he and his brother Jack built a multiplayer gaming startup aimed at bringing sophisticated multiplayer simulation and voice to web games.

Using LiveKit in that startup led Neil to build a LiveKit SDK and then join the company. He describes writing the first code for LiveKit Agents, a platform that grew out of LiveKit’s involvement with GPT Voice. Gabber carries that experience into hosting models for interactive conversations.

Gabber supplies infrastructure for real-time AI personas: voice, memory, tool calling, and, at the time of the talk, planned video inputs. Its focus is consumer applications rather than call centers, customer support, or AI sales representatives. Neil’s forecast was that synchronous AI experiences would become as commonplace as websites and apps within two to five years; that expectation shaped the product’s APIs and SDKs.

The applications include AI companions, game NPCs, therapists, personal trainers, and children’s toys. But their economics are uneven. Neil reports that some end-to-end voice platforms cost upwards of $5 per hour. Companion apps could sometimes support that expense because users were accustomed to buying credits and spending them during conversations. Most other consumer applications needed interaction costs much closer to free.

That pushed Gabber toward running models on its own GPUs. In the team’s assessment, earlier open voice models were useful for asynchronous generation but did not offer the streaming performance it needed. Orpheus became its practical opening: the team deployed it on an H100, and a viral tweet from Jack brought substantial inbound interest. Neil describes the launch as a turning point for the company.

Orpheus starts from a Llama 3B foundation. Neil describes pretraining on approximately 100,000 hours of voice, alongside text data intended to retain language understanding. Instead of producing only text, the model generates tokens for SNAC, an audio codec. Decoding those tokens produces 24 kHz audio.

Gabber uses approximately 85 SNAC tokens per second of audio as its planning budget, with a generation target of 90–100 tokens per second. The distinction is between audio duration and the rate at which the GPU can produce its representation. Generation must stay ahead of playback. If it falls behind, the listener hears gaps. The token budget is a rounded operational estimate, not a claim that every second of every utterance has an identical token count.

Consumer personas need both recognizable voices and expressive delivery. Gabber’s one-shot cloning results were not good enough for that goal. Neil speculates that stronger zero-shot behavior might emerge with million-plus-hour training datasets, but presents no established threshold. The team was not going to close that pretraining gap itself, so it chose low-rank adaptation, or LoRA, to make individual voice clones.

The demonstration uses Jack’s voice, with LoRA rank 16, alpha 32, and essentially all projection layers targeted. Neil says better customer examples existed but were not included. The source recording expresses frustration about a missed school pickup, including groans—material that tests more than neutral narration.

This particular clone used approximately 10 minutes of poor-quality data and five training epochs; Neil preferred approximately 30 minutes and explicitly called the result overfit. The generated sample changes the situation to a greeting followed by “I'm kind of sick.” It demonstrates a new utterance in the cloned voice, rather than replaying the school-pickup recording.

Neil’s assessment is mixed: knowing his brother’s voice closely makes the imperfect likeness jarring, but he hears useful expressiveness. In particular, the sickness line sounds sad to him. That is the desired behavior for a persona: delivery responds to language cues as well as explicit expressive tokens. It remains a subjective demonstration with acknowledged training limitations, not a measured fidelity result.

Four factors shape the voice response delay: time to first token, generation throughput, network latency, and what Gabber calls head-of-line silence. The last was its largest surprise. Neil attributes this Orpheus-specific behavior to leading silence in the voice-actor recordings used for fine-tuning: the model learned to generate silence before speech.

In Gabber’s Tara example, the generated audio begins with approximately 600 ms of silence. Its L40S setup was producing approximately 100 tokens per second. Using the rounded audio-token budget explains why an apparently simple playback filter cannot solve the whole problem:

Those 600 ms describe audio duration; approximately half a second describes the time spent generating it. Gabber already filtered the silence before playback, but the GPU still had to produce the silent tokens before reaching speech. Neil estimated that filtering alone saved only about 10% in this near-real-time setup. Removing silence after generation does not recover the compute time spent generating it.

Fine-tuning offered a more direct intervention: train the unwanted opening silence away. For a customer LoRA clone, Neil reports approximately 100 ms P50 latency in the opening-silence comparison, recovering roughly half a second. This is a result for that comparison, not a measured end-to-end conversation latency. The waveform’s marked opening interval makes the change concrete: the model reaches useful speech sooner, rather than merely having its output trimmed afterward.

The recovered time matters because response generation fits inside a larger turn-taking process. End-of-turn detection is imperfect, so Gabber adds a short snooze period after it suspects the human has finished. That waiting period can also be productive:

1. Detect a possible end to the human’s turn and begin the snooze period.
2. Start the response LLM during that period.
3. Send text to Orpheus after two sentences, or when the response finishes if earlier. The extra context helps emotional delivery.
4. Aim to have the first audio packet ready before the snooze period ends.

When that works, response preparation overlaps time the system was already spending deciding whether it should speak.

Neil anticipates a 0.5–1 second snooze period as endpointing improves, and treats approximately 1.5 seconds as a subjective upper threshold for acceptable response delay. These are conversational design judgments, not universal perceptual limits. Saving time in TTS gives the LLM more room to generate, which matters especially because Gabber lets customers bring their own LLMs and cannot control their speed.

The serving system had three linked requirements:

• Concurrent generation: run multiple requests on the same GPU to lower cost.
• Multiple adapters: allow different voice LoRAs to share a batch.
• One entry point: put a single load balancer in front of models serving different languages.

vLLM’s LoRA batching support supplied the central inference capability. But batching alone did not solve the throughput problem: Gabber found FP16 Orpheus slower than real time on an L40S, while it could run on an H100.

The next step was dynamic FP8 quantization. It handled scaling without the team having to prepare a calibration dataset. Gabber reported the following throughput on L40S hardware with FP8 and a batch size of 10:

Both cleared the approximate real-time floor, although the adapted voices had less headroom. These are Gabber’s deployment measurements, not general performance guarantees for vLLM. Gabber describes the resulting margins as workable; the talk’s $1/hour framing is not accompanied by a complete cost derivation.

Routing becomes part of latency management once every voice has an adapter. Gabber describes LoRAs of approximately 100–200 MB, depending on hyperparameters. A request should ideally reach a server that already has its adapter in memory. Sticky sessions also support streaming text input: the LLM may still be writing when the voice model needs to start speaking. Long generations, such as storytelling, strengthen the reason to keep the session on the same GPU.

Gabber uses a conventional consistent hash ring to distribute this traffic:

1. Hash each server multiple times, creating virtual nodes distributed around the ring.
2. Hash the LoRA generation’s routing identity using the same algorithm.
3. Select the nearby server on the ring.

The benefit appears when membership changes. Removing a server remaps only part of the traffic instead of redistributing everything, limiting the amount of adapter movement needed.

A popular clone can also be spread across more servers. The talk describes appending to the LoRA routing identity so additional placements can be added or removed as demand changes, without specifying the exact key scheme. This extends adapter placement beyond a single server while preserving the goal of session affinity.

The deployed connection path separates client transport from GPU selection. A WebRTC backend terminates client connections and uses WebSockets to communicate with GPU servers. The GPU servers consult Redis for coordination. Redis is a pragmatic choice here: Gabber acknowledges that it may not be the eventual scaling solution, without proposing a replacement.

A session reaches its intended GPU through a short routing procedure:

1. The WebRTC backend connects to an arbitrary GPU server.
2. That server asks Redis which GPU should handle the request.
3. It proxies the request to the selected GPU through another TCP connection.

The extra hop is acceptable in this deployment because the GPUs share a data center and private networking. That locality is part of the design: the proxy is a low-latency internal connection, not another trip across the public internet.

A small team can operate this stack by composing existing projects. Canopy Labs’ Orpheus builds on Llama and SNAC; Gabber’s WebRTC infrastructure uses LiveKit; vLLM provides the inference capabilities behind batching, adapters, and quantization. Gabber still had to solve the application-specific problems—expressive clones, generated silence, and session routing—but did not have to build every layer. Gabber presents that experience as evidence that small teams can host voice models themselves, opening room for consumer applications that were difficult to support at higher interaction costs.