Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing

[upbeat music] I'm Neil. [laughing]

That's, that's Jack in the, in the front there, but it's, it's just me. Um, so yeah, we're really just gonna talk about our experience, um, ho-hosting Orpheus, uh, inference, um, for our real, real-time stack.

So I'm Neil. Um, the CTO at a company called Gabber, a small startup. Uh, but I spent a lot of my career doing real-time media stuff, so sending audio and video around the internet.

Um, started at a company called Bebo, um, was ultimately acquired by Amazon, but, uh, there I was doing a lot of-- We did, like, a game streaming app, kind of like OBS.

Um, built, uh, a lot of the streaming infrastructure there. Built a, a ML pipeline to watch people play video games. Um, so they would watch people play "Fortnite" and put some cool effects on the screen when, when they got a kill or a victory or something.

Um, so spent a lot of time in, like, the GStreamer trenches and with WebRTC and RTMP and all that stuff. Um, took a detour, worked at Uber for a couple years, uh, then left that.

Um, did a, a multiplayer gaming, uh, startup with my brother Jack here. Um, so doing-- basically trying to bring, like, triple A style multiplayer to, to web games. So a lot of real...

And with voice and stuff too. So it, it's a lot of real-time media slash real-time, um, you know, simulation kind of stuff there. Um, and then, yeah, we, uh, didn't do s-super good job there and, uh, shut, shut that company down and, um, we were using LiveKit.

I made a LiveKit, uh, SDK, and that, uh, segued to me working at LiveKit. I think a lot of people have probably heard of LiveKit in this room. Um, and yeah, the second half of my time at LiveKit w-- I, uh, was spent doing the LiveKit Agents platform.

So that's like the platform that was kind of born out of, um, LiveKit's involvement with GPT Voice. Um, so yeah, wrote the first line of code on that and wor-worked on that.

Um, and then yeah, left LiveKit and did another startup with my brother, um, Gabber. Um, so that's what we're doing now. So Gabber is real-time f-- uh, in-infer for real-time, basically AI personas.

Um, so, you know, we have some core building blocks like voice, memory, um, video inputs coming soon, tool calling. Kind of like the usual suspects, I guess. But our focus is really on the consumer apps.

Um, you know, we, we see the enter-- like, the replacing human use cases pretty often. Um, like the call center use cases, customer support, AI SDR, that, that kind of stuff.

Um, but our interest is really in the, the consumer space. We think, um, these kind of like real-time synchronous AI experiences are gonna be as ubiquitous as, as websites and apps in the next kind of like two to five years.

So that's our focus and we-- that's how we try and differentiate in terms of opinion into our product and our SDKs and APIs and stuff. Um,

uh, here are some of the use cases we're seeing. Um, these are also kind of like the usual suspects. AI girlfriends was the first one. Um, that is like, uh...

I'll, I'll get to why that's the first one, I guess. But, um, other ones are like AI NPCs, uh, AI therapists, AI personal trainers, AI toys for kids. I think that you saw that in a couple, a couple sessions ago.

These use cases, like we're seeing a lot of different use cases, and I saw it at LiveKit too, and it got me really, really excited about, about this stuff.

But, um, AI girlfriends was, was the first one, mainly because, um, everything's so expensive. Um, uh, some, some of these voice platforms, it's, you know, end-to-end upwards of $5 an hour.

Uh, and that doesn't really work for like 90% of the consumer, consumer apps. Um, but AI girlfriends it works 'cause like the users are paying like, um... It's like usually like a credit system.

Like you buy credits, and you use the app, and it uses credit. So i- they're more comfortable with that, with that kind of spend. But most consumer use cases, they need something pretty close to free.

Um, so we knew that, uh-- And at the time, we were not hosting any, any voice models. We, we-- But we knew we had to. Uh, we knew that the only way to really get this to, to execute on our vision of putting these experiences everywhere, um, we had to start bringing more things in-house and running on

our own GPUs. Um, so at the time, open source, there weren't a lot of good open source voice models. Um, uh, there were a lot of good ones for asynchronous, uh, use cases, so generating voice slower than real-time.

Um, but there weren't any really good, like, real-time streaming ones until, uh, uh, Orpheus. Uh, Orpheus was the first really good one, um, that, uh, w-was kind of like ready to go.

So, um, Orpheus came out, and we're like, "Okay, this is our time to shine." Uh, we immediately like put it on an H100, um, hosted it, um, went viral with Jack's tweet, and, uh, got a ton of top of, top of funnel.

Um, and yeah, that was kind of like the starting point. It's like our, our company. There's like before Orpheus and after Orpheus, our company kind of changed.

Um, so a little background on what Orpheus is. Uh, it's a voice model, but it, it started as a Llama 3 billion. Um, it was trained on, uh, pre-trained on like 100,000 hours of, uh, voice, uh, data and text data as well to make sure it kept its understanding of kind of like language.

Uh, and, and it was trained to output audio tokens. They're called SNAC tokens, so that's another open source project, SNAC, which is a, a audio codec. Um, and so it's trained to output the 24 kHz version of, of SNAC tokens.

Um, those SNAC tokens are then decoded, and then you get audio. You get 24 kHz audio. Um, important thing to note here is it's about 85 SNAC tokens for one second of audio.

So, um, Orpheus, w-wherever you're hosting it, it has to, it has to be, uh, a, a throughput of about 85. I mean, you want like 90 to 100 tokens per second to keep up with real-time.

Otherwise, you get ga-gaps, obviously, in the audio, and it sounds bad. Um, other things that were important to us because we're going after the consumer use cases, um, was cloning.

Um, so our clones need to be emotive and, and high fidelity. Um, and one-shot cloning doesn't work that well. Um, that's more true for Orpheus 'cause it, it only had 100,000 hours of pre-trained data.

Um, whereas I think some of the zero-shot Emergent behavior comes out at, like, a million plus hours. So, and w-we're scrappy. I think you can tell by, like, our design here, um- [laughing] ...

that we're, like, pretty scrappy, right? We weren't gonna fill that gap. So,

um, so we went with low-rank fine-tunes for our clones. Um, so here's, uh, an example. So this is, um, a low-rank fine-tune. We have some better ones. This isn't, like, the best example, but they're customers, so I didn't wanna put it in the thing here.

So we just cloned Jack's voice, um, like yesterday and used 16 rank Alpha 32, basically all the projections. Um, here's the source audio. Let's see if it'll let me restart it.

"You forgot to pick up our child from school. Ugh! Ugh! The school called me in the middle of a meeting."

You-- Oops. Um, so that's the source. Uh, and then here's the result of a, of a fine-tune. So, um, let me manage expectations here. Um- [laughing] ... this wa- this was, like,

pretty bad data, like, 10, 10 minutes of data. You really want, like, 30 minutes. It w-- So I had to overfit, so I trained on, like, like, five epochs.

Um, it's pretty overfit, but y-you'll see it, like, still sounds okay.

"Hey, ha, how are you? I'm kind of sick." This is a longer generation. Let's see if it sounds okay.

Yeah. So it's not bad. Um, but, you know, I sp- my whole life, or most of my... I'm the older brother, so most of my life, so I know his voice very well.

Um, so- [laughing] ... so it's draw- it's drawing to me, but, um, cool, cool thing is, like, yeah, it's trained to do these tokens, which is important for consumer. Um, uh, so, and it's pretty emotive.

Like, when it said, "I'm kinda sick," it sounded pretty sad, so it picks up on the language cues as well.

Um, other thing that's really important, obviously, for all voice use cases, not just, um, not just consumer, is latency. Um, so there, there's four things that really affect latency.

Uh, time to first token is, is one of them. Um, tokens per second is one of them. Um,

I'll get into why that is later. Um, but what we found in network latency is another one. But we found the, the most, uh, biggest cause of latency was what we're calling head-of-line silence.

Um, this is somewhat specific to the Orpheus model. This isn't gonna be true for all models. Um,

but head-of-line silence is basically that, uh, some-somewhere in the fine-tune of Orpheus, um, the data had a lot of silence at the beginning, um, because it was voice actors that they hired, and they train, they, and they, like, took those scripts and train, fine-tune a model from it.

Um, so this is, like, the default Orpheus voice, uh, or one of the ones that came with it called Tara, and it has 600 milliseconds of latency at the beginning.

And they, they probably had other good reasons for, like, adding silence at the beginning, um, uh, but this is a lot, right? So six-600 milliseconds of silence. Um, we actually found that...

Oh, so 600 milliseconds of silence, we're running on L40S machines, uh, as of now. Um, they can do about 100 tokens a second. So 600 milliseconds is, uh, almost half a second of, of silence.

So even-- We're, we are filtering out the silence. Like, we're not just playing that audio back to the user. But because it takes a while to generate those tokens, we're adding, like, basically half a second of, of latency just on the, on wasted compute pretty much.

Um, so yeah, even filtering out the silence, you're only, like, saving 10% there 'cause you're just barely faster than real-time. Um, we're scrappy again, so we're running on L40s.

Um, but what we found was interesting is that we could actually just fine-tune the silence away. So, um, this is an example of a clone that we did, a LoRA fine-tune of a customer's clone, and the latency is, is basically, like, 100 milliseconds, like P50.

Um, so much better, like half a second basically for free.

Uh, and that matters, uh, because these real-time... You, you kinda have a latency budget on the, on the real-time application. So the way these work is y-you know, the human talks, and then at some point you decide, um, is the human done talking?

Those models are not perfect, so you typically add, like, a snooze period at the end of that. But during that snooze period, you can still do work. Um, so what we do is we kick off the LLM.

Um, the way we have our Orpheus stack set up is we start generating audio after two sentences, um, or if it's done, but two sentences typically, uh, which gives it enough context to, like, capture the emotions.

Um, so all that to say is if we generate the first audio packet within that snooze period, then we're kind of, like, in the money on, on latency, in our latency budget.

Um, now these end-pointing models are gonna get better, so you know, that snooze period's gonna go down to, like, half a second to a second is probably, like, the sweet spot.

But one and a half seconds is, um, kind of the threshold, I think, for... A-anything above that sounds pretty bad. Um, and anything kind of equal to or below that is, like, acceptable.

Um, so yeah, that half a second mattered a lot 'cause it gives our LLM more time to, um, create tokens, and because we're letting customers bring their own LLMs, um, we're, it's somewhat out of our control.

Um, so the next big category here is infrastructure. Um, again, we're, we're scrappy, so we really needed, uh, s-something that, um, was robust and, uh, not too complicated. Um, and we needed batch inference.

So we needed batch inf-inference obviously to save money, so we need to run, um, multiple, uh, generations on the, in the same batch or in the, on the same GPU concurrently.

And we also needed multiple LoRAs, uh, to be running in the same batch on the same GPU. Um, and we wanted one load balancer in front of everything. We're spinning up multiple different models for different languages.

So we all, all wanted this to sort of be like a black box that just sort of worked. Um,

uh, so vLLM to the rescue. It supports all those things. Um, so vLLM, um, can do batch inference with LoRAs, which is really, really awesome. Um, this is-- Unfortunately, the FP16 model was slower than real-time on an L40s.

It worked on an H100, but it was slower than real-time. But again, vLLM to the rescue. Um, they support FP8 dynamic quantization, which requires basically zero work. Um, it just works automatically.

It does all the, um, scaling and everything automatically, so you don't have to, like, train the calibration data into, uh, uh, your own quant. It just works, um, and it's amazing.

So that brought us up to 105 tokens a second on the non-fine-tune, uh, voices and 95 tokens a second on the LoRA.

Uh, voices with a batch of 10, um, which it, we're, yeah, well, well in the money, uh, in terms of margins and things like that. So that's nice. Um, part of the infrastructure is, is of course load balancing.

Um, so, you know, LoRAs are, depending on what your hyperparameters are, uh, they're between 100 and 200 megabytes. Um, so you wanna make sure you end up on a server that has a LoRA and memory and, and, and things like that.

We also wanted to support, um... So that's where, like, sticky session comes in here. Um, uh, and yeah, latency low, I guess. Um, but we also wanted to support streaming input, um, uh, mainly because the LLM often, you know, might not be done by the time you wanna start producing audio.

But we also wanted to support arbitrarily long, um, generations, so, like, storytelling, things like that. Um, so we, we have, um... So that, that's another reason why, uh, it, I guess, this load balancing problem is interesting 'cause you wanna make sure you end up on the same GPU a-across the whole session.

Uh, so we went with, uh, pretty much, like, a by-the-book consistent hash ring setup. Um, so if you've seen hash rings before, this is not that interesting. But basically, the way it works is you hash the servers, um, multiple times.

So you want it called virtual nodes, so it distributes around this hash ring. Um, and then when a LoRA w- you know, generation starts, you hash that with the same hashing algorithm.

You pick the nearest server to that, and it just works. And re- and the reason this is chosen is 'cause you can, like, remove a server, and it, it doesn't, um, re- re-load balance, like, everything.

It just, only a few, um, I guess, migrations are needed. Um, the other nice thing about this strategy is if a clone gets very popular, um, it's pretty easy to handle that.

You can just, uh, append, um, to the LoRA. So you can just-- The more popular a LoRA is, you can just add it to more servers and upscale and down-scale that, uh, very elegantly without really a ton of engineering work.

Um, so yeah, at the high level, it looks something like this. Um, we have our WebRTC backend that kinda, like, terminates the client connections. Then we use WebSockets, um, to our GPUs, and then the GPUs are talking to Redis.

Redis is not the best, um, the best choice, uh, but if we scale beyond needing Redis, uh, for this kinda thing, um, we can just solve that with piles of money, I guess.

Um, but yeah, the way it works here is you, uh, start a session. Uh, the WebRTC backend just connects to any GPU. Then it asks, uh, Redis, "Hey, what GPU is this request supposed to be on?"

And then it just proxies it with another TCP connection to the correct GPU, um, which is fine 'cause these GPUs are in the same data center, private networking. Um, so low latency TCP, that's totally fine with-within the same network.

Um, so that's, that's pretty much it. I mean, the conclusion here is, you know, we're, we're pretty scrappy. Um, and we were able to host voice models on GPUs and handle that infrastructure, so you can too.

Um, open source is there. And, um, yeah, it-- I think it's gonna unlock a, a ton of cool use cases. Um, shout-outs. Uh, shout out Swix. Um, he's a supporter of ours, um, and obviously put, put this on or half, half of, half of it, I guess.

But, uh, Swix is awesome. We love him. Um, Canopy La-Labs, uh, who created Orpheus. Um, haven't met them. Would love to if they're here. Um, uh, and then just free open source software in general, like Canopy Labs is built on Llama, which is...

And, and SNAC. So it's this whole ecosystem is greater than the sum of its parts, I guess. And, um, LiveKit. We're LiveKit, uh, alums, so, uh, love, love those guys.

And we're-- Our WebRTC infra is, is built on them. Um, and then vLLM, uh, um, notable open source project.

And yeah, that's it. [outro music]