Optimizing inference for voice models in production

Talk transcript

1. 0:00

   [upbeat music] Uh, hello everyone.

2. 0:15

   Thank you so much for being here, for, uh, sticking around for this talk. Um, I'm gonna be talking about optimizing inference for voice models in production. I'm going to be talking mostly about the runtime component, but also just a little bit on the infrastructure side.

3. 0:30

   Um, just a quick introduction. Um, I'm Philip from Baseten. Baseten is a model inference platform. Uh, we run production workloads for a wide variety of AI native startups and enterprises.

4. 0:41

   Um, I'm based in here in SF. Um, I actually just moved here. It's really awesome. My favorite part about being in SF is much better sports teams than I had in Chicago.

5. 0:50

   Um, and uh, one of my favorite voice models is Orpheus TTS, um, which we're gonna be talking about a whole bunch today. Um, quick agenda. So we're gonna talk about TTS model architecture, like what is a text-to-speech model actually when you look on the config in Hugging Face.

6. 1:07

   Uh, what sort of performance metrics are we looking at? What sort of optimization techniques can we do to make the model better? Um, how do we measure whether or not we've succeeded?

7. 1:16

   And then finally, what can we do on the infrastructure and client code to not shoot ourselves in the foot after doing a ton of runtime work and then just adding all that latency back by not doing our client code correctly.

8. 1:29

   So architecture. This is one of the things I've been learning this year, which has been pretty great, uh, to, to realize. It's made life a lot simpler at the runtime level.

9. 1:39

   Now, this is wrong. Like the, the, the thing up here, uh, that, that I'm gonna say is that like everything is an LLM, uh, that is, that is wrong, but it's useful.

10. 1:48

    Um, there's kind of like two types of models. There's autoregressive transformers models that are LLM or very LLM adjacent. Um, you see this in embeddings, you see this in transcription with stuff like Wispa.

11. 1:59

    TTS, um, is another example. You also have the more like diffuser image type models, um, which is like a very different optimization problem. But something that's cool is because TTS models are so architecturally similar to LLMs or in many cases derive directly from LLMs, we can access the rich ecosystem of LLM tooling and use it to make

12. 2:19

    TTS models better. So the TTS model that we're going to be using and as an example all day, um, is Orpheus TTS. We're using it for two reasons. Okay, three reasons.

13. 2:31

    The two reasons are 'cause it's open source and it's really good. And also I think, uh, Elias and Amoo and everyone at Canopy Labs is really awesome. So, uh, that's the, that's the third reason we're, we're talking about their model.

14. 2:42

    But it's a Llama 3.2 3B backbone. So like if you look at this is the little like config from Hugging Face copy and pasted onto the screen. It's a Llama Focausal LLM architecture.

15. 2:54

    And so because of that we can do like all of our normal Llama stuff to this model and make it faster. Um, they did a couple things. I mean they did a bunch of things to make it work.

16. 3:03

    But a couple things that are relevant here. There is a larger vocab size because you need all the speech specific tokens like laugh and stuff. Um, and then they also extended the context lengths with slope scaling.

17. 3:14

    So we gotta make sure everything we do supports that. So performance metrics, like what do we wanna actually do here?

18. 3:22

    Uh, we, we think about LLM metrics, um, a little bit here. Uh, we, we just look at them a little bit differently. So in LLMs you talk about time to first token.

19. 3:31

    Now we're talking about time to first byte or sometimes even time to first sentence. Uh, we, we need a little bit more of a useful output, um, from the model before we really start feeling good about our response time.

20. 3:42

    We do think about tokens per second, although we're gonna think about it differently, which I'll explain later. And we mostly think about throughput, which is, you know, how many requests are we able to serve at a given time.

21. 3:52

    So on that, you know, goals perspective, if you ask me like, "Hey Philip, how do you wanna optimize Llama in general?" I'll like, I'll say, "Well, we want a lot of TPS.

22. 4:01

    We want hundred, we want five hundred TPS. We want a thousand tokens per second. We want as many tokens per second as we can get." With voice models, you actually don't necessarily need that.

23. 4:11

    In many cases, you only want as many tokens per second as you need for a real-time stream. For Orpheus, that's like eighty-three tokens per second, which for like a three billion parameter LLM is nothing.

24. 4:22

    Um, but what we actually wanna do instead is we want to, once we hit that mark, start optimizing for time to first byte so that our latency's really good and start optimizing for concurrency so that we can get more connections and spend less on GPUs.

25. 4:39

    So our goal in general, if all of these very nice and definitely not AI generated people, all the different like voices that our, that our model is capable of creating, these are all the voice agents that we're running.

26. 4:50

    How can we make all of these people fit on one or even less than one GPU? That's the goal. So how do we do it?

27. 5:00

    Bunch of ways. Um, so first off, it's an LLM. Um, if you are running an LLM with like vLLM, for example, um, you can generally in many cases get better performance with TensorRT-LLM.

28. 5:12

    Uh, TensorRT is something that we've been using at Baseten a lot. I like to joke that I'm the unofficial marketing department for TensorRT-LLM because of how much I talk about it.

29. 5:21

    Uh, but it really is fast. It can be a little bit complicated from a developer experience perspective to get up and running with it. But once you are up and running, um, it, it works really well.

30. 5:30

    Uh, we can also just like quantize the model. Um, even though it's small, you can always make it faster by making it smaller. Uh, with Hop architecture, we quantize this model to FP8 pretty successfully.

31. 5:41

    I know usually quantizing really small models like this can lead to performance degradation, but for this model it's working pretty well in FP8 even when we quantize the KV cache

32. 5:51

    And then a lot of the other runtime stuff is actually more like audio specific than it is LLM specific. So one of the big challenges that we don't have with LLMs, which are just parsing nice convenient bits of text back and forth, is you have your audio, you have your audio codec, you have your decoding, all that

33. 6:06

    kind of stuff. So we use SNAC, which I was very disappointed to learn is not an actual tasty snack, um, but an audio decoder. And we actually use torch.compile.

34. 6:16

    Um, and torch.compile you might be used to running, uh, on, you know, a, a model, um, co-compiling your model weights to make your runtime faster. We're actually using the same kind of system with torch.compile and with, um, PyTorch inference mode on the audio decoder and running that on the GPU.

35. 6:35

    Um, we make sure that all the token batching, um, token level batching works well throughout the entire pipeline and support multiple streaming protocols. Um, yeah, so these are the engine settings that you would need.

36. 6:47

    Um, you've got the, you know, quantization type of FP8KV, the FP8 context, uh, FMHA, um, in order to, you know, support the, uh, support the, um, Hopo architecture and the, uh, quantization there.

37. 7:05

    Um, and here's a quick code sample of... I got a little ahead of my slides, I guess. Here's a little quick code sample of the audio decoding. Um, so we are basically, you know, batching.

38. 7:16

    Um, usually we would talk about continuous batching when we're talking about LLM optimization. We wanna package all those tokens together. In this case, we are doing dynamic batching, um, so we're trying to pack as much into a batch as we can, but every fifteen milliseconds we're gonna, uh, shoot it out.

39. 7:33

    Um, you've got that timeout set up here. Um, if you wanna trade off, um, for a little bit of latency for more throughput, you can make that batch bigger.

40. 7:41

    Um, so yeah. We don't have token level continuous batching yet here, but we do have dynamic batching, which is gonna get you pretty close. Um, and because of this, actually, something that's, that I was surprised about, uh, when we profiled this is that our TTS imple-implementation with Orpheus is actually in many cases CPU bound, which is kind

41. 8:00

    of where you wanna be. Um, you can throw more CPUs at a resource, uh, pretty, pretty efficiently. Um, even though the next token prediction and the audio co-decoding are both on the GPU, um, both of those loops hit the CPU at different points, um, and that can actually be the bottleneck in the number of simultaneous streams that

42. 8:17

    we're able to create. So how'd we do? Like, I just showed you a lot of code and talked through it really quickly without really getting into depth. Uh, that could all just be smoke and mirrors.

43. 8:28

    Uh, let's see, let's see if it's actually any faster. Um, so again, the number one thing is gonna be simultaneous streams because you want to be able to be very cost efficient and use fewer GPU resources to serve a, you know, large amount of traffic.

44. 8:44

    And in this case, a, a base implementation, I don't necessarily wanna like call anyone out because there's a lot of really good ways to run this model. Um, you can get really good performance with vLLM, um, but this is just kinda like the off the shelf, um, just take it, run it completely standard, uh, implementation.

45. 9:02

    So with variable traffic, we're able to support sixteen simultaneous streams and with constant traffic, twenty-four simultaneous streams on, um, an H100 MIG. So this is actually half an H100 GPU.

46. 9:13

    It's a SKU that we do a lot because it's really good for these small models where you want the Hopu performance, the Hopu architecture uplift in TensorRT-LLM, the FP8 support, but you don't wanna pay for like an entire eighty gigabyte GPU for just a three billion parameter model.

47. 9:29

    So, you know, we're, we're seeing much better, um, mu-much better concurrency. So if you kinda like price that out with like our list prices and stuff, um, you can get, you know, a few cents per hour of, of conversation, um, which is going to be, you know, substantially better than if you...

48. 9:45

    If you have the volume for it, it's gonna be substantially better than paying for a sort of like per token type API. But okay, so maybe it's cheap at scale, but is it fast?

49. 9:57

    Yes, it's fast. Um, so with the, you know, with the TOT implementation on the MIGs and on the H100s, we can actually get all the way down to a hundred fifty millisecond time to first bytes, um, in like real world, uh, testing that we've done.

50. 10:12

    Now that we'll, we'll talk in a minute, like that doesn't mean your whole pipeline is that fast. That's just like one part of the pipeline. Uh, but it's, it's important because, you know, you definitely don't want to be spending a lot of time waiting around for that first token.

51. 10:27

    So to kind of transition into that, that discussion of like what can go wrong here. Like you have this graph and you have this, you know, nice, uh, config that I had up here, and you're like, "All right, cool.

52. 10:40

    I'm gonna take this, I'm gonna put it in production, and I'm gonna see the results that, uh, that he put up on screen and it's gonna work great." And the answer is no, it's not.

53. 10:49

    It's a little bit harder than that. Um, so the thing is like non-runtime factors when we get, especially with these small models and with these multimodal systems can actually be like way more important than your runtime.

54. 11:03

    Um, and that's your lat- your infrastructure and your client code because, you know, I, I showed here... All right, maybe, maybe I got it, you know, I, I cut the runtime in, in half, um, from, from the base implementation.

55. 11:16

    I, I saved a couple hundred milliseconds. Very easy to add those couple of hundred milliseconds back and well beyond that by, you know, sending my query to New York instead of California or by having to establish a session every time I, uh, you know, run my client code.

56. 11:32

    Um, so a few like pitfalls to avoid. Um, number one, like if you go in, you know, a, a model library or something and we're just trying to get you started very quickly, um, with this, with this kind of inference sample, um, it's basically gonna be, hey, use requests, make a stream, stream it to your local computer

57. 11:51

    and start, you know, playing it on FFmpeg or something. The issue is that here, like the requests are gonna be sent sequentially and you need to create a new session every time.

58. 12:01

    Um, that takes time. So if you're using this in production, you want a code sample. Uh, by the way, this is all up on, uh, GitHub. Um, you'll want a code sample that looks a lot more like a benchmarking script where you're using a, um, multi-process pool.

59. 12:17

    Um, you're sharing the, you're sharing the session between all of these different requests, and you're actually sending traffic with the concurrency that allows you to, you know, saturate this benchmark with, you know, the, the multiple concurrent requests.

60. 12:34

    Finally, both of these code samples, they do sit on top of HTTP and HTTP streaming. Um, in many cases, if you're implementing voice pipelines, you're gonna use something like LiveKit or Pipecat or something, and you're also potentially going to be using a different protocol.

61. 12:49

    You're gonna be using something like WebSockets or gRPC, which we do have support for. Um, and finally, I wanted to leave you on the thought that these, uh, you know, these models are only one part of a voice agent pipeline.

62. 13:02

    So, like, we can spend a lot more than 15 minutes actually talking about, like, the very detailed, uh, implementation mechanics of making your voice model faster. Of, you know, we, we haven't even touched on stuff like fine-tuning the model, um, you know, custom voices, zero-shot voice cloning, um, being able to, you know, remove static and popping at

63. 13:23

    the end of messages. There's, there's, there's a lot of work to do just on the voice part, but it really only is one-third of the problem. When I think about voice agents, I think about three parts: listening, thinking, talking.

64. 13:35

    Um, and the most important thing here is, again, while you can have great runtimes, the infrastructure to connect these three together is really what's going to determine your latency.

65. 13:45

    Being able to go from one model to have the next one running in the same data center with, you know, minimal, like, minimal network overhead in between the two.

66. 13:57

    Uh, even things as simple as not having to go off and do a halo to pin, um, at the DNS level and come back. If that saves you ten milliseconds on every step and your voice pipeline has this and, you know, a chunking algorithm, it's got an interruption model, and so a- you end up having four or

67. 14:12

    five steps, well, they'll-- Just halo pinning alone is costing you forty or fifty milliseconds, and that can be ten percent of your SLA for, for a voice model. So yeah, uh, that's, that's my, that's my main point here, is that as much fun as it is to talk about the runtime stuff and as much work as we

68. 14:30

    do there, the, the infrastructure and the client implementation is equally important, if not more so. Anyway, thank-- Uh, so yeah, that's the, that's the review. Um, thank you all for coming through.

69. 14:41

    Uh, I have a-- We're doing an, an event next week at, uh, [REDACTED:location], which is gonna be pretty fun. I'm gonna be talking in more detail about building some, uh, systems with open source models, and there's also gonna be a lot of steak.

70. 14:54

    So definitely come on through, uh, if you're interested and, um, I'm on, I'm on Twitter, I'm on LinkedIn, so it's Baseten. Um, hit me up if you have any questions about this or anything else model performance.

71. 15:04

    Thank you so much, and I'll let you go eight seconds early. [outro music]