Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Talk transcript

1. 0:00

   [upbeat music] My name is Rohit Talluri.

2. 0:16

   I am a member of the foundation model training and inference team at AWS, and we have an amazing talk, super, super awesome founder here talking about voice AI. So Arjun, why don't you go ahead?

3. 0:26

   Awesome. Well, thanks for the intro, Ro. Uh, I'm Arjun. I'm one of the co-founders of Cartesia AI. Uh, you know, I'll give you a little bit of a spiel about what we do, which is that, you know, we build real-time multimodal intelligence that runs on any device.

4. 0:38

   Um, so maybe, you know, the topic of this conversation will be focused a little bit more on how do we build voice AI for enterprise. As many of you know, voice AI is, you know, definitely up and coming and one of the places where, you know, having interactive models is really critical for being able to get really,

5. 0:53

   uh, good experience, uh, for, uh, your, you know, agents.

6. 0:59

   So just a little bit of a recap, right? When you think about foundation models, you know, we often think about, you know, models that are hosted in the cloud, really large.

7. 1:06

   You know, they're doing things kind of in batch mode, right? You know, you submit a request, and then you might get something back. You know, you're not gonna get it back thirty seconds later, but you might get it back, you know, in five hundred, six hundred milliseconds, and, you know, that's okay, right?

8. 1:17

   Typically, for text, you know, you're not also gonna be reading, uh, at speeds of, uh, like two hundred tokens per second, uh, and so it's okay if, you know, y- there's a little bit of a delay, and then you get high throughput later.

9. 1:28

   When you think about interactive applications like video and voice especially, this is an incredible visceral experience that you're giving to your users. Speed is of the utmost importance, and quality is just table stakes.

10. 1:42

    So at Cartesia, we're really trying to change the paradigm of how we think about foundation models to not just be like in the cloud, what you build for batch op-operations, but actually how you build-- bring it to real time, have it cover multiple modalities, and we'll be talking more about voice today, and have it run anywhere on

11. 1:58

    the wor-- in the world, not just on the cloud, but on any device.

12. 2:03

    So because we're talking a little bit about voice, I wanna maybe give a little bit of a prelude why it's so important to think about speed. Imagine you're having a conversation with the person next to you, uh, and you probably should do that after this talk.

13. 2:14

    But, uh, I think the main thing is that it'd feel really awkward, right? If you were trying to talk to someone, and they responded a second later. So in voice, you don't have seconds to actually give your response back, you have milliseconds.

14. 2:24

    And when you think about voice agents, where you might be calling, you know, customer support on the line trying to triage your problem, you're gonna get pretty annoyed if it's not responding to you and giving you accurate results as soon as possible.

15. 2:35

    So when you think about voice, you have to deal with things like interruptions, you have to think about globalization. These things relate to accents, right, background noises that you might be calling from.

16. 2:44

    And at the end of the day, when you actually wanna deliver this experience, it's incredibly subjective. So there's so many nuances and customizations that you want around voice.

17. 2:54

    So what we've done at Cartesia is really thought about building solutions for voice AI on the modeling side from first principles. We care about three things. The first is quality.

18. 3:04

    The naturalness of the voice must be exquisite and something that is just table stakes for the kinds of experiences that you want to be able to deliver. The second is latency.

19. 3:13

    You want to be able to hear the first bite or the first sound of audio on the other line as soon as possible. This gives your end-to-end agent a lot more time to be able to do things like reason more, right, and have a little bit more slack in the end-to-end system.

20. 3:28

    And third, arguably most important, is controllability. The experience that you wanna exude by having an agent customized for the things that you wanna do is a reflection of your brand.

21. 3:40

    You want the agent to be able to talk about what your company is or what you're trying to sell in the way that you would. So being able to customize the voice AI to be able to do this is critical and paramount to being able to deliver a great experience to your user.

22. 3:55

    So what we've done is we've pioneered a new architecture called state-space models. These are an alternative to transformers, and the main takeaway here is that typically transformers scale quadratically, which means the longer your inputs get, you get quadratic scaling in your memory as well as in your, uh, in your runtime.

23. 4:13

    So, you know, the longer your inputs, the slower your models get. With state-space models or SSMs, generation at inference time is O(1). We maintain a state that you can generate from, and this means that you have perfect very low latency, things that you can't achieve with traditional transformer architectures.

24. 4:31

    What we've pioneered is that these state-space models typically have been, you know, lower performing in the recurrent sense compared to transformers, but we've closed that gap, and our models actually perform not just better from a latency perspective, but also from a quality perspective.

25. 4:47

    So I can give a little bit of a spiel about, you know, why this matters, but I think we're more interested in having more of a conversation-

26. 4:51

    Yeah

27. 4:52

    ... around voice AI. So Ro, why don't you kick us off?

28. 4:54

    Definitely. So just so you all know, this was supposed to be a fireside chat.

29. 4:56

    Yeah.

30. 4:57

    We had some chairs and asking some good questions. So we're gonna open it up to the audience in a little bit. I do have a few questions for you, Arjun.

31. 5:02

    Talked a little bit about latency. We hear voice AI challenges like quality and, let's say, the speed of the models, as mentioned, the way that they're hosted on the edge, et cetera.

32. 5:12

    What are the challenges that, like, your customers are facing? Why are you building for this?

33. 5:16

    Yeah. So, uh, I think the main thing is that, you know, when you do voice AI, and, you know, main model that we have is sonic-2, which is focused on voice generation.

34. 5:24

    That's one part of the puzzle. You have to-- If you're actually trying to build voice agents, you have to hook it up with the LM, you have to hook it up with your speech-to-text model, and the biggest issue honestly is that there's not enough time.

35. 5:35

    You need your LM to actually have a lot of time. You know, these models aren't typically built for, you know, high, uh, or low latency workflows, and so you wanna actually give it the most slack.

36. 5:44

    So latency is, you know, of utmost importance there, and I can talk a little bit about, you know, what we've done to create the fastest model in the world for text-to-speech.

37. 5:52

    Uh, the second I'd say is controllability. So, you know, one thing that we've noticed so much is people use our platform because you're able to get amazing quality around things like voice cloning, accents, being able to capture background noises just so natively in the generations that you want.

38. 6:07

    You know, people think it's like a little bit of an uncanny valley when you're talking to an [REDACTED:origin] and it just sounds perfect. [laughs] They like the little phone noises in the background, little beep boops, right?

39. 6:14

    You know, that's kind of what you like to hear, right? Or what you expect, right, when you're on a phone call. Uh, so I think those are the two main things that I-I'd like to say that, you know, we've done a really amazing job of getting-- nailing those down.

40. 6:23

    Yeah. And you mentioned something interesting there, a couple of the use cases, right? Voice cloning-

41. 6:27

    Right

42. 6:27

    ... the control of the voices, et cetera.

43. 6:29

    Yeah.

44. 6:29

    Can you talk a little bit about the customer use cases you're seeing today of voice AI?

45. 6:32

    Yeah. So, you know, voice AI honestly has penetrated so much, so many markets out here. We look at healthcare, customer support. Uh, a lot of this is actually going into real-time gaming, right?

46. 6:42

    When you're dealing with non-player characters, you want them to be dynamic, interact, you know, with the, uh, players of the game. Um, these are, you know, three of many, many markets that, you know, voice AI has started to grow in.

47. 6:53

    Uh, and you know, I think what's really exciting is that because it's growing so fast, you know, it's just so great to have partners like AWS who are also, you know, investing in that space and being like, "Look, you know, these are things that we need to support naturally," um, and you know, people wanna use our platform

48. 7:06

    for.

49. 7:06

    Yeah. What about human narration, right? I think we hear a lot about, uh, like voice AI taking over different types of industries or these use cases as mentioned.

50. 7:14

    Yeah.

51. 7:14

    Do you see a place for human narratives-- narrators in the future as well?

52. 7:17

    Definitely, yeah. You know, creators are a huge part of the voice AI platform. I think one thing that I think we do a really go-good job of is we actually have like a voice marketplace for creators, right?

53. 7:25

    You know, why is this so important? You know, our goal is not to, you know, replace voice actors. Actually, I, I-- it's not what I wanna do at all.

54. 7:31

    Yeah.

55. 7:31

    I think my main goal is like, how do you give them a platform so that their, you know, their, um, uh, I forgot what-- like their essence, right, you know who they are, their personality can actually be exuded and like, you know, licensed by other people that wanna use the platform.

56. 7:44

    Um, and so, you know, this actually gives, uh, you know, we've had a lot of voice actors onboard our platform in a great way of amplifying them. Uh, and yeah, a lot of use cases are honestly focused on narration as well.

57. 7:52

    Got it. So I have a few more questions. Maybe we'll end with them. Does anybody have questions in the crowd? Anything about voice AI?

58. 8:00

    Yeah.

59. 8:01

    Question. I, I, I fear this might be a dumb question, but I'm gonna ask anyway.

60. 8:04

    No, please.

61. 8:04

    I've just started using Cartesia. It's like it was a game changer-

62. 8:09

    Awesome

63. 8:09

    ... as part of a, uh, Pipecat framework.

64. 8:11

    Yeah. Yeah.

65. 8:12

    Um, because there's an Amazon partnership here, I was just wondering, like Claude doesn't seem to work so well 'cause of the latency.

66. 8:19

    Okay.

67. 8:19

    Is that, is that normal? Uh, uh, what are the models that you usually pair with and is there, is there plans for...

68. 8:26

    Can you repeat the question on your end just so that- Sure, sure ... the log thing. Yeah, yeah. So, uh, the question was, you know, um, we're u-- Cartesia is integrated with Pipecat.

69. 8:34

    You know, Pipecat, Cartesia, we both, you know, work with AWS. Uh, but Claude, you know, still has, you know, pretty high latency when you're trying to, you know, do things end to end.

70. 8:42

    Uh, so you know, this is, you know, a huge, huge problem, right? This is why, you know, the slack that we give you on the TTS side is something, you know, you can account for on the LM side.

71. 8:49

    Uh, I-I'll let Roh, you know, take a, a stab at that question, but I know that like, you know, Claude and, you know, other o- um, other models like, you know, have different, you know, mechanisms where, you know, you can have like a dedicated instance that's running, so you can get like, you know, better latency numbers.

72. 9:02

    Um, you know, but a lot of this is actually just, you know, it's, it's up to, you know, the LM providers often, right, to make these optimizations. I think one thing that, you know, we're really excited about is our goal is to, you know, make real-time AI pervasive, right?

73. 9:13

    Uh, and so, you know, we're really excited about if there's like certain applications that, you know, you're really after, you know, how can we enable you to do that?

74. 9:18

    So happy to chat, you know, after this talk. Yeah.

75. 9:21

    Yeah, of course. And I'll add a couple things to that too. What's really amazing about Cartesia and their background is the development of custom model architectures specifically for this use case, which is voice AI.

76. 9:31

    We spoke a little bit about SSM and linear scaling versus quadratic scaling. One thing from the AWS side, and what I think is really amazing about our design philosophy for our generative AI ecosystem, is we want customers to have optionality in this, in our platform.

77. 9:44

    And if you look at AWS's model, model gardens, we have SageMaker JumpStart and Amazon Bedrock. These model gardens will have a host of different types of models s- for specific types of use cases.

78. 9:54

    And we're on the lookout for the next foundation model provider fun-- like Cartesia, that we can bring into our ecosystem and unlock downstream industries that might be underserved by the existing foundation models today.

79. 10:06

    So that's part of our strategy, part of our philosophy, and I think what, and I'll say it again, with Cartesia, we're really unlocking the voice AI. We need real-time AI.

80. 10:15

    We need, we need the ability to host on edge devices and, uh, we're unlocking a lot of customers with their models. So hopefully that answers a little bit.

81. 10:23

    I'll ask-

82. 10:23

    A little bit.

83. 10:23

    Yeah. We can chat a little bit more after.

84. 10:25

    Yeah.

85. 10:26

    Yeah.

86. 10:27

    Um, yeah, so I have some friends that, uh, that are pretty into video AI research and something they're kinda telling me is that in video AI, one prevailing theory is that, you know, you don't need so much scale of actual video data, but rather the density of information per frame of that video is actually more important.

87. 10:45

    I'm curious if there's something similar in voice data and whether you believe that, you know, to achieve, you know, next-level models, um, you need like a, like the next scale of data or is it more about like the quality and density of that data?

88. 10:58

    Yeah, that's a good question. So I think the question, uh, was more around the lines of like, you know, how does voice AI data, you know, compare to, you know, some of the other mul- multimodal datasets in video?

89. 11:06

    You know, one of the prevailing theories, uh, was that, um, you know, you don't, um, actually need a lot of data. You just need it to be very rich, right?

90. 11:13

    And, you know, if that's the same in voice AI. Uh, I think the short answer to that is, uh, yes, but also no. Um, I think like when you think about scaling for a lot of models, you know, the generative, uh, you know, in generative AI, right?

91. 11:27

    The general philosophy is like, oh, you know, large scale pre-training data, and then you have some kind of like alignment data or like preference data that, you know, you fine-tune on typically.

92. 11:35

    Um, I think that holds true for, uh, a lot of other modalities as well. It actually holds true for video as well. So my background, uh, I guess I didn't say was like, you know, I did my PhD, um, in generative AI at Stanford, was working a lot on like image video models there as well.

93. 11:49

    Um, but in audio it's actually quite interesting because, you know, what people want from preference data is actually so diverse that it's like, you know, you can't capture it purely by like a one, one-stage fine-tuning step.

94. 12:00

    So yeah, you definitely need very rich data. You know, you need high quality data of course, but you also need to pair it, uh, with information that like, you know, many o- many different kind of people will want.

95. 12:09

    So, uh, you know, this is how traditional LMs are trained. I think a lot of the like, you know, large open source, you know, uh, image generation, video generation models also are trained this way.

96. 12:17

    Uh, but honestly just really depends on what you're trying to, trying to get after. But- Sorry, I don't know... I know that's not a satisfying answer, but yeah.

97. 12:24

    Yeah. Um, what's your take on the speech-to-speech models? I know Amazon has one. Seems immature, I would say.

98. 12:32

    Yeah.

99. 12:33

    Um, but is that the future, or, like, what's happening?

100. 12:35

     Yeah, it's a g- it's a great question. Um, you know, I think there are places where, like, speech-to-speech is valuable right now, but I don't think it's at the point where we can actually use it for, you know, production or enterprise-grade, uh, use cases.

101. 12:49

     Um, and, you know, it's great. You know, I think, you know, uh, Amazon released, uh, the speech-to-speech model recently. Um, you know, I think there's a few others that are out there as well.

102. 12:57

     Uh, but I think it's still pretty clear that, like, orchestrated solutions are places that you get a lot of controllability around, like, how you want the different pieces of your system to operate.

103. 13:05

     Um, and, you know, from a latency- latency perspective, I think speech-to-speech models will of course, you know, dominate over time. But, uh, I think that level of controllability is something that needs to be thought of from first principles when building these systems, right?

104. 13:17

     The goal of the system is not to build something really cute. It's to build something that actually functions for real-world use cases, and we might not be there yet, but I- I'm sure, you know, we'll get there over time.

105. 13:25

     Yeah.

106. 13:27

     Yeah?

107. 13:27

     What do you think about local models, and will you have benefits for the future?

108. 13:31

     Hundred percent, yeah. Um, so at Cartesia, like I said, you know, we build for any device. So, uh, we have models that run locally on edge devices. Um, and, you know, I, I think it's actually really important because, you know, I think cloud models will always be there, right?

109. 13:44

     You know, there's certain size models, certain capabilities, like, you know, that you get at, like, certain sizes that you can't just, you know, get, you know, fit on, like, I don't know, very, very edge devices, right?

110. 13:54

     Like, not even, like, laptops, but, like, if you get to, like, smaller phones and things like that. Um, but there's a lot of applications where, you know, edge devices are just critical, right?

111. 14:03

     Uh, I think the question becomes, like, where is a network latency plus cloud speed actually slower than edge? And we've broken that. So running our models on edge are about f- five times faster than if you were to round trip.

112. 14:17

     Um, and so you get, like, just... It's, like, unparalleled latency that you get. Um, and I think that's quite exciting.

113. 14:24

     So we're almost up at time. We'll maybe one more question.

114. 14:27

     Sure, yeah.

115. 14:29

     Uh.

116. 14:29

     Yeah. Um, so what tools do you recommend to monitor basically the agent, right? Because you're gonna be the one that's g- we're gonna call you-

117. 14:36

     Yeah

118. 14:36

     ... if something's broken. It said pause three seconds.

119. 14:39

     Yeah.

120. 14:39

     It's not supposed to say pause three seconds.

121. 14:41

     Right.

122. 14:41

     So it can be, like, the speech-to-text. It can be the LLM. It can be the text-to-speech.

123. 14:45

     Yeah.

124. 14:45

     It can be s- I don't know, something wrong with AWS. But, like, how do you, how do you tell us that, "Hey, it's not us. It's not a regression issue."

125. 14:52

     How do you, like, uh, feed that into it?

126. 14:55

     Yeah. Yeah, it's a good question. Evals are an incredibly important part of any system that you build. Um, how do we tell you that? Uh, I guess you can message us and ask us if it's a problem.

127. 15:05

     Uh, but I think what we've found is, you know, typically it's at the LM stage that we see a lot of these issues. Um, you know, you end up, like, you know...

128. 15:12

     Oftentimes there's, like, some level of formatting that you need to do for outputs that come from speech-to-text to make it, like, viable for your LM and the outputs for your LM to make it, like, you know, viable or ergonomic for your TTS to handle.

129. 15:23

     Uh, but we've actually handled a lot of those edge cases for you. So over time, you know, with Sonic 2, uh, you know, I'm showing a slide here that's, like, you know, showing most on the latency side, but a lot of the capabilities and quality has also improved substantially.

130. 15:34

     So not only did we get our models to be, you know, two and a half times as fast as our initial model, so we can serve at, like, forty millisecond model latency.

131. 15:41

     But a lot of those, you know, edge case issues that LMs often run into, we know that you don't wanna deal with them, so we deal with them. We deal with them for you to making our system more robust.

132. 15:50

     Got it. So one more from me. Like, it's twenty twenty-five now. Think five years in the future, uh, twenty thirty.

133. 15:57

     Yeah.

134. 15:57

     If you look back at what you've done over the pa- those past five years, where is Cartesia at? Where is voice AI at? What are you seeing?

135. 16:03

     Yeah. Um, you know, I think voice AI will be just, like, the de facto norm, right? Uh, I think there... In every industry that you see, you're gonna have some interaction with voice AI, whether it's on the triaging side, the full end-to-end, you know, support side, uh, you know, gaming, right?

136. 16:18

     All of those interactions will, will definitely be, you know, a large part, you know, covered by voice AI. Uh, but I don't think this is the only thing, right?

137. 16:26

     When you think about true interactive models, it's more than just what you can hear, right? It's about how you experience the world around you, right? So, you know, a lot of these people call it, like, world models, right?

138. 16:37

     But I think actually getting these to work in real-time I think will be really exciting, and I- I'm, I'm really looking forward to, like, a world where, you know, we can actually have these kinds of systems work with us as, you know, assistants or co-pilots or, you know, whatever, to actually help us understand in the world in

139. 16:50

     ways that we couldn't imagine originally.

140. 16:52

     Amazing.

141. 16:53

     Yeah.

142. 16:53

     Well, thank you. I think that's all we had today, and really appreciate it.

143. 16:56

     Yeah.

144. 16:56

     And thank you for sharing a little bit with- about Cartesia.

145. 16:58

     Yeah, appreciate it. Thanks, guys. [upbeat music]