AI Engineer
← All AI Engineer talks

AI Engineer World's Fair 2025

Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Arjun Desai· CartesiaRohit Talluri· Amazon Web Services (AWS)17:05

Read the talk

Serving Voice AI Within a Conversational Latency Budget

A voice agent must listen, reason, and speak before a pause becomes awkward. That shared deadline shapes model architecture, voice control, deployment, and failure diagnosis.

From a talk by Arjun Desai and Rohit Talluri

Before you start: Familiarity with speech-to-text, language models, and text-to-speech will help; no prior knowledge of state-space models is required.

010:26 ↗

When waiting changes the interaction

Why does a foundation model that works well for text still make an awkward voice agent? A text interface can tolerate an initial wait followed by fast output. A conversation makes that wait part of the experience. Cartesia’s starting point is to build interactive, multimodal models that can operate in real time and reach devices beyond the cloud.

Desai uses a 500–600 ms initial response delay as an example of what can be acceptable for text. His reference to 200 tokens per second concerns how quickly people read, not a measured model throughput result: users do not necessarily benefit from receiving text faster than they can consume it. Voice and video impose a different requirement. Quality is table stakes; the timing of delivery determines whether the interaction works.

Slide contrasts batch foundation-model workloads with real-time interactive applications, alongside a running Road Runner illustration.
Batch workloads tolerate slow intelligence; interactive applications require real-time intelligence.

That changes the target from a large model serving requests in the cloud to intelligence that responds continuously across modalities and locations. Voice is the immediate application, but real-time operation and deployment on different devices are separate parts of the goal.

0:260:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:26 · section reference included

022:03 ↗

Naturalness, first audio, and control

Imagine talking to the person beside you and waiting a full second for each response. Desai uses that pause to illustrate conversational awkwardness, not to establish a universal latency threshold. In customer-support triage, the same delay compounds the frustration of trying to resolve a problem. The agent must also handle interruptions, different accents, and background noise, while satisfying preferences that vary from listener to listener.

Three requirements follow:

  • Quality: The generated voice needs to sound natural enough for the intended experience.
  • Latency: The first audible response should arrive quickly. Reducing this delay leaves more time elsewhere in the agent for reasoning.
  • Controllability: The voice should express the company’s identity and speak about its products in the intended style.

These requirements interact, but they are not interchangeable. A natural voice can still respond too late, and a fast voice can still sound wrong for the brand.

2:032:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:03 · section reference included

033:55 ↗

Generate from maintained state

Cartesia’s architectural approach uses state-space models, or SSMs, as an alternative to transformers. Desai motivates the choice through a quadratic-scaling comparison: as input sequences grow, conventional attention can require substantially more work and memory. That comparison needs a precise boundary. Quadratic attention costs are not a universal description of every transformer decoding implementation or its memory use.

The SSM mechanism is recurrent: maintain a state representing the processed sequence, then generate from that state. Desai’s O(1) claim means that a generation step need not become more expensive merely because more history has accumulated. It does not mean generating an entire response takes constant time; longer outputs still require more steps. This distinction matters for a sustained conversation, where the interaction keeps growing but the response deadline does not.

Recurrent efficiency alone would not be enough if it reduced voice quality. Desai says Cartesia has closed the quality gap historically associated with recurrent models and improved both latency and quality relative to transformers. The talk supplies no benchmark setup for that comparison, so it remains Cartesia’s reported result rather than a general ranking of the architectures.

3:554:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:55 · section reference included

045:02 ↗

Fast speech generation buys reasoning time

Sonic 2 provides voice generation, not the entire voice agent. A conventional agent connects speech-to-text, a language model, and text-to-speech. Each component consumes part of the time between a user’s turn and the audible answer. The language model may need substantial time to produce a useful response, especially when it was not designed for low-latency interaction.

Faster TTS creates slack for the language model; it does not eliminate the language model’s delay. Desai describes Cartesia’s TTS as the fastest available, although he supplies no comparative benchmark here. The engineering objective is more concrete than that ranking: spend less of the shared response budget turning an answer into sound.

Control extends beyond choosing a pleasant voice. Desai points to voice cloning, accents, and generated background sounds. A perfectly clean agent can feel uncanny on a phone call, while familiar phone noises and small imperfections can make the interaction sound more like what the caller expects. Naturalness therefore depends on context, not simply on removing every audible irregularity.

5:025:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:02 · section reference included

056:29 ↗

From support calls to licensed performances

The applications Desai names include healthcare, customer support, and real-time gaming. Gaming makes the interactive requirement particularly clear: a non-player character should respond dynamically to the player rather than only deliver fixed dialogue. He describes growing demand across these markets and connects it to AWS’s investment in infrastructure for voice workloads, without giving adoption measurements.

Narration raises a different question: what role remains for human performers? Desai describes a voice marketplace in which creators make their voices and personalities available for others to license. His stated aim is to amplify voice actors rather than replace them, and he says narration is already an important use case. The mechanism is licensing a performer’s voice through the platform; the discussion does not establish compensation terms or employment outcomes.

6:296:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:29 · section reference included

068:04 ↗

The slow component still sets a constraint

An audience member using Cartesia through Pipecat reports that Claude introduces too much latency and asks which models pair well with the voice stack. This is a concrete integration problem: improving the speech generator can make the overall agent faster while leaving its largest remaining delay untouched.

Desai returns to the shared budget: time saved in TTS can be spent at the language-model stage. He suggests that dedicated instances may improve latency, but does not provide a configuration, a measured improvement, or a preferred model pairing. Some optimizations remain dependent on the language-model provider.

Talluri’s answer emphasizes model choice. SageMaker JumpStart and Amazon Bedrock offer different models for different workloads, and AWS’s stated strategy includes bringing specialized providers into that ecosystem. For voice applications, the relevant capabilities include real-time responses and deployment on edge devices. This broadens the available design choices; it does not resolve the audience member’s particular Claude configuration.

8:048:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:04 · section reference included

0710:27 ↗

Rich data must represent different preferences

The next question asks whether audio resembles a proposed pattern in video research: perhaps information density matters more than simply collecting more data. Desai’s answer preserves both sides. The familiar recipe of large-scale pretraining followed by alignment or preference fine-tuning remains relevant across modalities, including video, an area he worked on during his Stanford research.

Audio complicates the preference stage because listeners want different things. In Desai’s account, a single fine-tuning stage cannot fully capture that diversity. High-quality recordings and rich data are necessary, but they must be paired with information reflecting the range of experiences people want. The answer is application-dependent: it does not establish a formula in which data richness replaces scale.

10:2710:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:27 · section reference included

0812:24 ↗

Integrated speech versus component control

Speech-to-speech models offer another way to organize the system. Asked about their future, Desai says they already have useful applications but, in his view at the time of this 2025 discussion, were not yet ready for the enterprise use cases he had in mind. The conversation mentions a recently released Amazon model without naming it.

ApproachPrincipal advantage in the discussionDesign concern
Orchestrated STT → LM → TTSControl over individual componentsCoordinate their behavior and delays
Integrated speech-to-speechExpected long-term latency advantagePreserve enterprise control

Separate components let developers determine how each stage behaves. Desai expects speech-to-speech models to win on latency over time, but argues that controllability must be designed into them from the beginning. A convincing interaction is not sufficient unless the system also functions reliably for its real-world task.

12:2412:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:24 · section reference included

0913:27 ↗

Compare local execution with the whole cloud trip

Local models introduce a deployment tradeoff rather than making the cloud obsolete. Desai says Cartesia already has models running on edge devices. Cloud systems still support model sizes and capabilities that may not fit on small devices, particularly phones and hardware more constrained than laptops.

The relevant comparison is local execution against network latency plus cloud execution:

Tlocal<Tnetworkroundtrip+TcloudinferenceT_local < T_network_round_trip + T_cloud_inference

Desai reports that Cartesia’s edge models run approximately five times faster than a cloud round trip. This is a comparison with the network-inclusive path, not a claim that the device performs inference compute five times faster than a cloud accelerator. The model variant, device, network conditions, and measurement protocol are not specified.

13:2713:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:27 · section reference included

1014:29 ↗

When the agent says the instruction aloud

An audience member gives a useful failure case: the agent says “pause three seconds” instead of pausing. Where did the error originate—speech recognition, the language model, speech synthesis, or infrastructure? And how can the developer distinguish an integration problem from a provider regression?

Desai emphasizes evaluations and contacting the provider, but names no monitoring tool. He says many issues Cartesia encounters arise at the language-model stage. More specifically, the boundaries need attention: speech-to-text output may require formatting before the LM consumes it, and LM output may require formatting before TTS can speak it correctly.

One way to make that boundary explicit is to separate speech from actions. In this TypeScript example, the intended pause is an action record, so the TTS input contains only the words meant to be spoken:

typescript

type VoiceAction =
  | { kind: "speak"; text: string }
  | { kind: "pause"; durationMs: number };

const response: VoiceAction[] = [
  { kind: "speak", text: "Let me check that for you." },
  { kind: "pause", durationMs: 3000 },
];

async function deliver(
  actions: VoiceAction[],
  speak: (text: string) => Promise<void>,
  pause: (durationMs: number) => Promise<void>,
): Promise<void> {
  for (const action of actions) {
    if (action.kind === "speak") {
      await speak(action.text);
    } else {
      await pause(action.durationMs);
    }
  }
}

This illustrates a boundary contract, not automatic interpretation of arbitrary LM text. The producer must emit the intended action, and the orchestrator must route it correctly. Evaluating those records separately from the resulting audio makes the failure easier to locate.

Desai says Sonic 2 handles more formatting edge cases internally and has improved in quality as well as latency. The accompanying slide separates two variants: Sonic Turbo is listed below 40 ms time to first audio, while Sonic-2 is listed below 90 ms. It also shows a comparison with the next best alternative at P50, P90, P95, and P99.

Slide titled “Capabilities: Maintaining Low-Latency at Scale” lists latency claims and compares Cartesia with the next best alternative in a bar chart at P50, P90, P95, and P99.
Cartesia’s latency slide lists under 90 ms for Sonic-2 and under 40 ms for Sonic Turbo time to first audio.

Desai reports a 2.5× speed improvement over Cartesia’s initial model and approximately 40 ms model latency. The historical changelog associates the 40 ms figure specifically with Sonic Turbo, matching the slide’s distinction. Neither that figure nor the spoken improvement establishes total conversational latency; the talk does not supply the benchmark conditions or detailed baseline. His broader point is that the newer system also absorbs more of the edge cases developers would otherwise need to handle themselves.

14:2914:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:29 · section reference included

1115:50 ↗

Beyond hearing, with the same real-time requirement

Talluri closes by asking for a five-year view from 2025 to 2030. Desai expects voice AI to become commonplace across industries, including triage, end-to-end support, and gaming. These are forecasts about adoption, not capabilities or market outcomes demonstrated in the session.

The longer-term direction extends beyond speech. Interactive models would help people interpret the world around them through multiple modalities, operating as assistants or copilots. Desai connects this ambition to world models, but the condition remains the one that made voice difficult in the first place: the system must work in real time to participate usefully in the user’s experience.

15:5015:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:50 · section reference included

Resources

From the talk

Further reading

Updates since the talk

Read the complete timestamped transcript
  1. 0:00

    [upbeat music] My name is Rohit Talluri.

  2. 0:16

    I am a member of the foundation model training and inference team at AWS, and we have an amazing talk, super, super awesome founder here talking about voice AI. So Arjun, why don't you go ahead?

  3. 0:26

    Awesome. Well, thanks for the intro, Ro. Uh, I'm Arjun. I'm one of the co-founders of Cartesia AI. Uh, you know, I'll give you a little bit of a spiel about what we do, which is that, you know, we build real-time multimodal intelligence that runs on any device.

  4. 0:38

    Um, so maybe, you know, the topic of this conversation will be focused a little bit more on how do we build voice AI for enterprise. As many of you know, voice AI is, you know, definitely up and coming and one of the places where, you know, having interactive models is really critical for being able to get really,

  5. 0:53

    uh, good experience, uh, for, uh, your, you know, agents.

  6. 0:59

    So just a little bit of a recap, right? When you think about foundation models, you know, we often think about, you know, models that are hosted in the cloud, really large.

  7. 1:06

    You know, they're doing things kind of in batch mode, right? You know, you submit a request, and then you might get something back. You know, you're not gonna get it back thirty seconds later, but you might get it back, you know, in five hundred, six hundred milliseconds, and, you know, that's okay, right?

  8. 1:17

    Typically, for text, you know, you're not also gonna be reading, uh, at speeds of, uh, like two hundred tokens per second, uh, and so it's okay if, you know, y- there's a little bit of a delay, and then you get high throughput later.

  9. 1:28

    When you think about interactive applications like video and voice especially, this is an incredible visceral experience that you're giving to your users. Speed is of the utmost importance, and quality is just table stakes.

  10. 1:42

    So at Cartesia, we're really trying to change the paradigm of how we think about foundation models to not just be like in the cloud, what you build for batch op-operations, but actually how you build-- bring it to real time, have it cover multiple modalities, and we'll be talking more about voice today, and have it run anywhere on

  11. 1:58

    the wor-- in the world, not just on the cloud, but on any device.

  12. 2:03

    So because we're talking a little bit about voice, I wanna maybe give a little bit of a prelude why it's so important to think about speed. Imagine you're having a conversation with the person next to you, uh, and you probably should do that after this talk.

  13. 2:14

    But, uh, I think the main thing is that it'd feel really awkward, right? If you were trying to talk to someone, and they responded a second later. So in voice, you don't have seconds to actually give your response back, you have milliseconds.

  14. 2:24

    And when you think about voice agents, where you might be calling, you know, customer support on the line trying to triage your problem, you're gonna get pretty annoyed if it's not responding to you and giving you accurate results as soon as possible.

  15. 2:35

    So when you think about voice, you have to deal with things like interruptions, you have to think about globalization. These things relate to accents, right, background noises that you might be calling from.

  16. 2:44

    And at the end of the day, when you actually wanna deliver this experience, it's incredibly subjective. So there's so many nuances and customizations that you want around voice.

  17. 2:54

    So what we've done at Cartesia is really thought about building solutions for voice AI on the modeling side from first principles. We care about three things. The first is quality.

  18. 3:04

    The naturalness of the voice must be exquisite and something that is just table stakes for the kinds of experiences that you want to be able to deliver. The second is latency.

  19. 3:13

    You want to be able to hear the first bite or the first sound of audio on the other line as soon as possible. This gives your end-to-end agent a lot more time to be able to do things like reason more, right, and have a little bit more slack in the end-to-end system.

  20. 3:28

    And third, arguably most important, is controllability. The experience that you wanna exude by having an agent customized for the things that you wanna do is a reflection of your brand.

  21. 3:40

    You want the agent to be able to talk about what your company is or what you're trying to sell in the way that you would. So being able to customize the voice AI to be able to do this is critical and paramount to being able to deliver a great experience to your user.

  22. 3:55

    So what we've done is we've pioneered a new architecture called state-space models. These are an alternative to transformers, and the main takeaway here is that typically transformers scale quadratically, which means the longer your inputs get, you get quadratic scaling in your memory as well as in your, uh, in your runtime.

  23. 4:13

    So, you know, the longer your inputs, the slower your models get. With state-space models or SSMs, generation at inference time is O(1). We maintain a state that you can generate from, and this means that you have perfect very low latency, things that you can't achieve with traditional transformer architectures.

  24. 4:31

    What we've pioneered is that these state-space models typically have been, you know, lower performing in the recurrent sense compared to transformers, but we've closed that gap, and our models actually perform not just better from a latency perspective, but also from a quality perspective.

  25. 4:47

    So I can give a little bit of a spiel about, you know, why this matters, but I think we're more interested in having more of a conversation-

  26. 4:51

    Yeah

  27. 4:52

    ... around voice AI. So Ro, why don't you kick us off?

  28. 4:54

    Definitely. So just so you all know, this was supposed to be a fireside chat.

  29. 4:56

    Yeah.

  30. 4:57

    We had some chairs and asking some good questions. So we're gonna open it up to the audience in a little bit. I do have a few questions for you, Arjun.

  31. 5:02

    Talked a little bit about latency. We hear voice AI challenges like quality and, let's say, the speed of the models, as mentioned, the way that they're hosted on the edge, et cetera.

  32. 5:12

    What are the challenges that, like, your customers are facing? Why are you building for this?

  33. 5:16

    Yeah. So, uh, I think the main thing is that, you know, when you do voice AI, and, you know, main model that we have is sonic-2, which is focused on voice generation.

  34. 5:24

    That's one part of the puzzle. You have to-- If you're actually trying to build voice agents, you have to hook it up with the LM, you have to hook it up with your speech-to-text model, and the biggest issue honestly is that there's not enough time.

  35. 5:35

    You need your LM to actually have a lot of time. You know, these models aren't typically built for, you know, high, uh, or low latency workflows, and so you wanna actually give it the most slack.

  36. 5:44

    So latency is, you know, of utmost importance there, and I can talk a little bit about, you know, what we've done to create the fastest model in the world for text-to-speech.

  37. 5:52

    Uh, the second I'd say is controllability. So, you know, one thing that we've noticed so much is people use our platform because you're able to get amazing quality around things like voice cloning, accents, being able to capture background noises just so natively in the generations that you want.

  38. 6:07

    You know, people think it's like a little bit of an uncanny valley when you're talking to an [REDACTED:origin] and it just sounds perfect. [laughs] They like the little phone noises in the background, little beep boops, right?

  39. 6:14

    You know, that's kind of what you like to hear, right? Or what you expect, right, when you're on a phone call. Uh, so I think those are the two main things that I-I'd like to say that, you know, we've done a really amazing job of getting-- nailing those down.

  40. 6:23

    Yeah. And you mentioned something interesting there, a couple of the use cases, right? Voice cloning-

  41. 6:27

    Right

  42. 6:27

    ... the control of the voices, et cetera.

  43. 6:29

    Yeah.

  44. 6:29

    Can you talk a little bit about the customer use cases you're seeing today of voice AI?

  45. 6:32

    Yeah. So, you know, voice AI honestly has penetrated so much, so many markets out here. We look at healthcare, customer support. Uh, a lot of this is actually going into real-time gaming, right?

  46. 6:42

    When you're dealing with non-player characters, you want them to be dynamic, interact, you know, with the, uh, players of the game. Um, these are, you know, three of many, many markets that, you know, voice AI has started to grow in.

  47. 6:53

    Uh, and you know, I think what's really exciting is that because it's growing so fast, you know, it's just so great to have partners like AWS who are also, you know, investing in that space and being like, "Look, you know, these are things that we need to support naturally," um, and you know, people wanna use our platform

  48. 7:06

    for.

  49. 7:06

    Yeah. What about human narration, right? I think we hear a lot about, uh, like voice AI taking over different types of industries or these use cases as mentioned.

  50. 7:14

    Yeah.

  51. 7:14

    Do you see a place for human narratives-- narrators in the future as well?

  52. 7:17

    Definitely, yeah. You know, creators are a huge part of the voice AI platform. I think one thing that I think we do a really go-good job of is we actually have like a voice marketplace for creators, right?

  53. 7:25

    You know, why is this so important? You know, our goal is not to, you know, replace voice actors. Actually, I, I-- it's not what I wanna do at all.

  54. 7:31

    Yeah.

  55. 7:31

    I think my main goal is like, how do you give them a platform so that their, you know, their, um, uh, I forgot what-- like their essence, right, you know who they are, their personality can actually be exuded and like, you know, licensed by other people that wanna use the platform.

  56. 7:44

    Um, and so, you know, this actually gives, uh, you know, we've had a lot of voice actors onboard our platform in a great way of amplifying them. Uh, and yeah, a lot of use cases are honestly focused on narration as well.

  57. 7:52

    Got it. So I have a few more questions. Maybe we'll end with them. Does anybody have questions in the crowd? Anything about voice AI?

  58. 8:00

    Yeah.

  59. 8:01

    Question. I, I, I fear this might be a dumb question, but I'm gonna ask anyway.

  60. 8:04

    No, please.

  61. 8:04

    I've just started using Cartesia. It's like it was a game changer-

  62. 8:09

    Awesome

  63. 8:09

    ... as part of a, uh, Pipecat framework.

  64. 8:11

    Yeah. Yeah.

  65. 8:12

    Um, because there's an Amazon partnership here, I was just wondering, like Claude doesn't seem to work so well 'cause of the latency.

  66. 8:19

    Okay.

  67. 8:19

    Is that, is that normal? Uh, uh, what are the models that you usually pair with and is there, is there plans for...

  68. 8:26

    Can you repeat the question on your end just so that- Sure, sure ... the log thing. Yeah, yeah. So, uh, the question was, you know, um, we're u-- Cartesia is integrated with Pipecat.

  69. 8:34

    You know, Pipecat, Cartesia, we both, you know, work with AWS. Uh, but Claude, you know, still has, you know, pretty high latency when you're trying to, you know, do things end to end.

  70. 8:42

    Uh, so you know, this is, you know, a huge, huge problem, right? This is why, you know, the slack that we give you on the TTS side is something, you know, you can account for on the LM side.

  71. 8:49

    Uh, I-I'll let Roh, you know, take a, a stab at that question, but I know that like, you know, Claude and, you know, other o- um, other models like, you know, have different, you know, mechanisms where, you know, you can have like a dedicated instance that's running, so you can get like, you know, better latency numbers.

  72. 9:02

    Um, you know, but a lot of this is actually just, you know, it's, it's up to, you know, the LM providers often, right, to make these optimizations. I think one thing that, you know, we're really excited about is our goal is to, you know, make real-time AI pervasive, right?

  73. 9:13

    Uh, and so, you know, we're really excited about if there's like certain applications that, you know, you're really after, you know, how can we enable you to do that?

  74. 9:18

    So happy to chat, you know, after this talk. Yeah.

  75. 9:21

    Yeah, of course. And I'll add a couple things to that too. What's really amazing about Cartesia and their background is the development of custom model architectures specifically for this use case, which is voice AI.

  76. 9:31

    We spoke a little bit about SSM and linear scaling versus quadratic scaling. One thing from the AWS side, and what I think is really amazing about our design philosophy for our generative AI ecosystem, is we want customers to have optionality in this, in our platform.

  77. 9:44

    And if you look at AWS's model, model gardens, we have SageMaker JumpStart and Amazon Bedrock. These model gardens will have a host of different types of models s- for specific types of use cases.

  78. 9:54

    And we're on the lookout for the next foundation model provider fun-- like Cartesia, that we can bring into our ecosystem and unlock downstream industries that might be underserved by the existing foundation models today.

  79. 10:06

    So that's part of our strategy, part of our philosophy, and I think what, and I'll say it again, with Cartesia, we're really unlocking the voice AI. We need real-time AI.

  80. 10:15

    We need, we need the ability to host on edge devices and, uh, we're unlocking a lot of customers with their models. So hopefully that answers a little bit.

  81. 10:23

    I'll ask-

  82. 10:23

    A little bit.

  83. 10:23

    Yeah. We can chat a little bit more after.

  84. 10:25

    Yeah.

  85. 10:26

    Yeah.

  86. 10:27

    Um, yeah, so I have some friends that, uh, that are pretty into video AI research and something they're kinda telling me is that in video AI, one prevailing theory is that, you know, you don't need so much scale of actual video data, but rather the density of information per frame of that video is actually more important.

  87. 10:45

    I'm curious if there's something similar in voice data and whether you believe that, you know, to achieve, you know, next-level models, um, you need like a, like the next scale of data or is it more about like the quality and density of that data?

  88. 10:58

    Yeah, that's a good question. So I think the question, uh, was more around the lines of like, you know, how does voice AI data, you know, compare to, you know, some of the other mul- multimodal datasets in video?

  89. 11:06

    You know, one of the prevailing theories, uh, was that, um, you know, you don't, um, actually need a lot of data. You just need it to be very rich, right?

  90. 11:13

    And, you know, if that's the same in voice AI. Uh, I think the short answer to that is, uh, yes, but also no. Um, I think like when you think about scaling for a lot of models, you know, the generative, uh, you know, in generative AI, right?

  91. 11:27

    The general philosophy is like, oh, you know, large scale pre-training data, and then you have some kind of like alignment data or like preference data that, you know, you fine-tune on typically.

  92. 11:35

    Um, I think that holds true for, uh, a lot of other modalities as well. It actually holds true for video as well. So my background, uh, I guess I didn't say was like, you know, I did my PhD, um, in generative AI at Stanford, was working a lot on like image video models there as well.

  93. 11:49

    Um, but in audio it's actually quite interesting because, you know, what people want from preference data is actually so diverse that it's like, you know, you can't capture it purely by like a one, one-stage fine-tuning step.

  94. 12:00

    So yeah, you definitely need very rich data. You know, you need high quality data of course, but you also need to pair it, uh, with information that like, you know, many o- many different kind of people will want.

  95. 12:09

    So, uh, you know, this is how traditional LMs are trained. I think a lot of the like, you know, large open source, you know, uh, image generation, video generation models also are trained this way.

  96. 12:17

    Uh, but honestly just really depends on what you're trying to, trying to get after. But- Sorry, I don't know... I know that's not a satisfying answer, but yeah.

  97. 12:24

    Yeah. Um, what's your take on the speech-to-speech models? I know Amazon has one. Seems immature, I would say.

  98. 12:32

    Yeah.

  99. 12:33

    Um, but is that the future, or, like, what's happening?

  100. 12:35

    Yeah, it's a g- it's a great question. Um, you know, I think there are places where, like, speech-to-speech is valuable right now, but I don't think it's at the point where we can actually use it for, you know, production or enterprise-grade, uh, use cases.

  101. 12:49

    Um, and, you know, it's great. You know, I think, you know, uh, Amazon released, uh, the speech-to-speech model recently. Um, you know, I think there's a few others that are out there as well.

  102. 12:57

    Uh, but I think it's still pretty clear that, like, orchestrated solutions are places that you get a lot of controllability around, like, how you want the different pieces of your system to operate.

  103. 13:05

    Um, and, you know, from a latency- latency perspective, I think speech-to-speech models will of course, you know, dominate over time. But, uh, I think that level of controllability is something that needs to be thought of from first principles when building these systems, right?

  104. 13:17

    The goal of the system is not to build something really cute. It's to build something that actually functions for real-world use cases, and we might not be there yet, but I- I'm sure, you know, we'll get there over time.

  105. 13:25

    Yeah.

  106. 13:27

    Yeah?

  107. 13:27

    What do you think about local models, and will you have benefits for the future?

  108. 13:31

    Hundred percent, yeah. Um, so at Cartesia, like I said, you know, we build for any device. So, uh, we have models that run locally on edge devices. Um, and, you know, I, I think it's actually really important because, you know, I think cloud models will always be there, right?

  109. 13:44

    You know, there's certain size models, certain capabilities, like, you know, that you get at, like, certain sizes that you can't just, you know, get, you know, fit on, like, I don't know, very, very edge devices, right?

  110. 13:54

    Like, not even, like, laptops, but, like, if you get to, like, smaller phones and things like that. Um, but there's a lot of applications where, you know, edge devices are just critical, right?

  111. 14:03

    Uh, I think the question becomes, like, where is a network latency plus cloud speed actually slower than edge? And we've broken that. So running our models on edge are about f- five times faster than if you were to round trip.

  112. 14:17

    Um, and so you get, like, just... It's, like, unparalleled latency that you get. Um, and I think that's quite exciting.

  113. 14:24

    So we're almost up at time. We'll maybe one more question.

  114. 14:27

    Sure, yeah.

  115. 14:29

    Uh.

  116. 14:29

    Yeah. Um, so what tools do you recommend to monitor basically the agent, right? Because you're gonna be the one that's g- we're gonna call you-

  117. 14:36

    Yeah

  118. 14:36

    ... if something's broken. It said pause three seconds.

  119. 14:39

    Yeah.

  120. 14:39

    It's not supposed to say pause three seconds.

  121. 14:41

    Right.

  122. 14:41

    So it can be, like, the speech-to-text. It can be the LLM. It can be the text-to-speech.

  123. 14:45

    Yeah.

  124. 14:45

    It can be s- I don't know, something wrong with AWS. But, like, how do you, how do you tell us that, "Hey, it's not us. It's not a regression issue."

  125. 14:52

    How do you, like, uh, feed that into it?

  126. 14:55

    Yeah. Yeah, it's a good question. Evals are an incredibly important part of any system that you build. Um, how do we tell you that? Uh, I guess you can message us and ask us if it's a problem.

  127. 15:05

    Uh, but I think what we've found is, you know, typically it's at the LM stage that we see a lot of these issues. Um, you know, you end up, like, you know...

  128. 15:12

    Oftentimes there's, like, some level of formatting that you need to do for outputs that come from speech-to-text to make it, like, viable for your LM and the outputs for your LM to make it, like, you know, viable or ergonomic for your TTS to handle.

  129. 15:23

    Uh, but we've actually handled a lot of those edge cases for you. So over time, you know, with Sonic 2, uh, you know, I'm showing a slide here that's, like, you know, showing most on the latency side, but a lot of the capabilities and quality has also improved substantially.

  130. 15:34

    So not only did we get our models to be, you know, two and a half times as fast as our initial model, so we can serve at, like, forty millisecond model latency.

  131. 15:41

    But a lot of those, you know, edge case issues that LMs often run into, we know that you don't wanna deal with them, so we deal with them. We deal with them for you to making our system more robust.

  132. 15:50

    Got it. So one more from me. Like, it's twenty twenty-five now. Think five years in the future, uh, twenty thirty.

  133. 15:57

    Yeah.

  134. 15:57

    If you look back at what you've done over the pa- those past five years, where is Cartesia at? Where is voice AI at? What are you seeing?

  135. 16:03

    Yeah. Um, you know, I think voice AI will be just, like, the de facto norm, right? Uh, I think there... In every industry that you see, you're gonna have some interaction with voice AI, whether it's on the triaging side, the full end-to-end, you know, support side, uh, you know, gaming, right?

  136. 16:18

    All of those interactions will, will definitely be, you know, a large part, you know, covered by voice AI. Uh, but I don't think this is the only thing, right?

  137. 16:26

    When you think about true interactive models, it's more than just what you can hear, right? It's about how you experience the world around you, right? So, you know, a lot of these people call it, like, world models, right?

  138. 16:37

    But I think actually getting these to work in real-time I think will be really exciting, and I- I'm, I'm really looking forward to, like, a world where, you know, we can actually have these kinds of systems work with us as, you know, assistants or co-pilots or, you know, whatever, to actually help us understand in the world in

  139. 16:50

    ways that we couldn't imagine originally.

  140. 16:52

    Amazing.

  141. 16:53

    Yeah.

  142. 16:53

    Well, thank you. I think that's all we had today, and really appreciate it.

  143. 16:56

    Yeah.

  144. 16:56

    And thank you for sharing a little bit with- about Cartesia.

  145. 16:58

    Yeah, appreciate it. Thanks, guys. [upbeat music]