[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12760,
    "text": ">> Okay, hi everyone."
  },
  {
    "startMs": 14520,
    "text": "I'm Nel"
  },
  {
    "startMs": 15520,
    "text": "co-founder and CEO of of Gradio."
  },
  {
    "startMs": 18360,
    "text": "So Gradio is a startup based in Paris."
  },
  {
    "startMs": 22000,
    "text": "Most of our background is from research."
  },
  {
    "startMs": 24440,
    "text": "In particular, we have invented"
  },
  {
    "startMs": 26280,
    "text": "algorithms such as audio LLMs, speech"
  },
  {
    "startMs": 28760,
    "text": "speech-to-speech models, neural codex,"
  },
  {
    "startMs": 31160,
    "text": "and so on and so forth."
  },
  {
    "startMs": 32720,
    "text": "And"
  },
  {
    "startMs": 33720,
    "text": "basically"
  },
  {
    "startMs": 35120,
    "text": "we started from a research project"
  },
  {
    "startMs": 37280,
    "text": "called QTI, a non-profit research lab"
  },
  {
    "startMs": 39520,
    "text": "that has been focusing on voice since"
  },
  {
    "startMs": 41360,
    "text": "day one. So in particular, we released"
  },
  {
    "startMs": 43160,
    "text": "in 2024 the first"
  },
  {
    "startMs": 45320,
    "text": "full duplex speech-to-speech model"
  },
  {
    "startMs": 46800,
    "text": "called Moshi, the first real-time"
  },
  {
    "startMs": 48720,
    "text": "speech-to-speech translation system"
  },
  {
    "startMs": 50120,
    "text": "called Hibiki, and the first"
  },
  {
    "startMs": 52680,
    "text": "TTS model that can run locally on a on a"
  },
  {
    "startMs": 54920,
    "text": "smartphone."
  },
  {
    "startMs": 56200,
    "text": "And basically I will just say a few"
  },
  {
    "startMs": 58280,
    "text": "words about what we do, but we are"
  },
  {
    "startMs": 60520,
    "text": "a model company that trains models for"
  },
  {
    "startMs": 62840,
    "text": "building voice agents and voice"
  },
  {
    "startMs": 64120,
    "text": "applications. So we do TTS, API, and"
  },
  {
    "startMs": 66800,
    "text": "on-device speech-to-text,"
  },
  {
    "startMs": 68040,
    "text": "speech-to-speech translation, and much"
  },
  {
    "startMs": 69680,
    "text": "more to come. What we do is that we"
  },
  {
    "startMs": 72080,
    "text": "train foundation models for audio, and"
  },
  {
    "startMs": 73840,
    "text": "then we can apply them for a lot of"
  },
  {
    "startMs": 75520,
    "text": "different tasks."
  },
  {
    "startMs": 77080,
    "text": "So I go quickly on voice agents because"
  },
  {
    "startMs": 79240,
    "text": "it's the fourth talk"
  },
  {
    "startMs": 81600,
    "text": "about the topic, but basically now we"
  },
  {
    "startMs": 84040,
    "text": "have these"
  },
  {
    "startMs": 86520,
    "text": "voice interfaces that we can use to do a"
  },
  {
    "startMs": 88400,
    "text": "lot of things across a variety of"
  },
  {
    "startMs": 90320,
    "text": "products and types of interactions with"
  },
  {
    "startMs": 92200,
    "text": "NPCs, with customer agents, language"
  },
  {
    "startMs": 95360,
    "text": "learners, coach, and so on and so forth."
  },
  {
    "startMs": 97840,
    "text": "And in this talk I tried to go through"
  },
  {
    "startMs": 100560,
    "text": "the history of this technology and"
  },
  {
    "startMs": 104000,
    "text": "where I see it going in the next years."
  },
  {
    "startMs": 106360,
    "text": "And maybe to start, I think we can take"
  },
  {
    "startMs": 108680,
    "text": "a look at the announcement of Siri back"
  },
  {
    "startMs": 110680,
    "text": "in 2011."
  },
  {
    "startMs": 112920,
    "text": "And you'll see that it's actually, you"
  },
  {
    "startMs": 115560,
    "text": "know, I think it it aged pretty well."
  },
  {
    "startMs": 120280,
    "text": ">> What is the weather like today?"
  },
  {
    "startMs": 126400,
    "text": ">> Here's the forecast for today."
  },
  {
    "startMs": 129479,
    "text": ">> It is THAT EASY."
  },
  {
    "startMs": 131001,
    "text": ">> [cheering]"
  },
  {
    "startMs": 132311,
    "text": "[applause]"
  },
  {
    "startMs": 133440,
    "text": ">> LOTS OF THINGS. We've integrated with"
  },
  {
    "startMs": 134680,
    "text": "the stocks. So, you can ask it about the"
  },
  {
    "startMs": 136280,
    "text": "stock market. Something like"
  },
  {
    "startMs": 138760,
    "text": "How is the NASDAQ doing today?"
  },
  {
    "startMs": 143720,
    "text": ">> NASDAQ composite is down right now at"
  },
  {
    "startMs": 146240,
    "text": "2,321.70."
  },
  {
    "startMs": 149480,
    "text": ">> Again, you can ask this from the lock"
  },
  {
    "startMs": 150880,
    "text": "screen anywhere. Just press the button"
  },
  {
    "startMs": 152080,
    "text": "and ask. You can ask about, you know,"
  },
  {
    "startMs": 153200,
    "text": "the NASDAQ, the Dow."
  },
  {
    "startMs": 155240,
    "text": ">> So,"
  },
  {
    "startMs": 156400,
    "text": "what you just saw is what kind of a"
  },
  {
    "startMs": 158120,
    "text": "voice agent. It was a bit constrained,"
  },
  {
    "startMs": 159680,
    "text": "but it was technically a voice agent."
  },
  {
    "startMs": 161440,
    "text": "And the architecture behind it, so you"
  },
  {
    "startMs": 163200,
    "text": "have seen a thousand times today the STT"
  },
  {
    "startMs": 165160,
    "text": "LLM TTS. Back then, it was even worse,"
  },
  {
    "startMs": 167480,
    "text": "right? So, there was no LLM, obviously."
  },
  {
    "startMs": 170440,
    "text": "So, there was what was called natural"
  },
  {
    "startMs": 172080,
    "text": "language understanding. So, you would go"
  },
  {
    "startMs": 173800,
    "text": "from the transcript and uh try to do"
  },
  {
    "startMs": 177080,
    "text": "basic classification of what is"
  },
  {
    "startMs": 178560,
    "text": "expected, uh what is the app that is"
  },
  {
    "startMs": 180959,
    "text": "supposed to be uh controlled, what is"
  },
  {
    "startMs": 183280,
    "text": "the action to trigger, and so on. So, it"
  },
  {
    "startMs": 185600,
    "text": "was a very complex pipeline and very"
  },
  {
    "startMs": 189000,
    "text": "constrained to very specific use cases."
  },
  {
    "startMs": 191200,
    "text": "So, it was what was called close"
  },
  {
    "startMs": 194200,
    "text": "uh ended um"
  },
  {
    "startMs": 196040,
    "text": "uh dialogue back then."
  },
  {
    "startMs": 198280,
    "text": "Then, the next generation of models, uh"
  },
  {
    "startMs": 201519,
    "text": "I could say, I think was"
  },
  {
    "startMs": 204120,
    "text": "is illustrated pretty well by the OpenAI"
  },
  {
    "startMs": 206560,
    "text": "voice mode, not the advanced voice mode,"
  },
  {
    "startMs": 208040,
    "text": "the original one. In which case, you"
  },
  {
    "startMs": 209840,
    "text": "just have um a conversational experience"
  },
  {
    "startMs": 212320,
    "text": "that is open-ended. So, it's much less"
  },
  {
    "startMs": 214440,
    "text": "constrained than Siri. You can talk"
  },
  {
    "startMs": 215800,
    "text": "about pretty much anything. At the same"
  },
  {
    "startMs": 217760,
    "text": "time, you don't have um"
  },
  {
    "startMs": 219920,
    "text": "as the same agentic capabilities, let's"
  },
  {
    "startMs": 222280,
    "text": "say, in the sense that it cannot really"
  },
  {
    "startMs": 223880,
    "text": "give you the NASDAQ or uh give you the"
  },
  {
    "startMs": 226080,
    "text": "weather."
  },
  {
    "startMs": 231000,
    "text": ">> We want to hear a bedtime story."
  },
  {
    "startMs": 233160,
    "text": "Tell us a story about the super-duper"
  },
  {
    "startMs": 234880,
    "text": "sunflower hedgehog named Larry."
  },
  {
    "startMs": 237400,
    "text": "Start with telling us a little bit about"
  },
  {
    "startMs": 238720,
    "text": "him."
  },
  {
    "startMs": 243440,
    "text": "Larry was a unique hedgehog unlike any"
  },
  {
    "startMs": 245560,
    "text": "other."
  },
  {
    "startMs": 246640,
    "text": "He had bright sunflower petals instead"
  },
  {
    "startMs": 248680,
    "text": "of spines. Residing in Meadowville,"
  },
  {
    "startMs": 251440,
    "text": "Larry spread joy and color wherever he"
  },
  {
    "startMs": 253560,
    "text": "rolled."
  },
  {
    "startMs": 255040,
    "text": ">> So, in that context, basically that was,"
  },
  {
    "startMs": 257200,
    "text": "you know, kind of second generation of"
  },
  {
    "startMs": 258799,
    "text": "models where all the logic, the very"
  },
  {
    "startMs": 260840,
    "text": "complex logic was replaced"
  },
  {
    "startMs": 262960,
    "text": "uh by an LLM."
  },
  {
    "startMs": 264320,
    "text": "And"
  },
  {
    "startMs": 265520,
    "text": "if we compare Siri and the voice mode"
  },
  {
    "startMs": 267919,
    "text": "from OpenAI, uh"
  },
  {
    "startMs": 269800,
    "text": "not the Ad Ons one, Ad Ons one, the"
  },
  {
    "startMs": 271640,
    "text": "original one,"
  },
  {
    "startMs": 272880,
    "text": "uh"
  },
  {
    "startMs": 273800,
    "text": "the big strength is opening the"
  },
  {
    "startMs": 275160,
    "text": "dialogue, but the agentic capabilities"
  },
  {
    "startMs": 277520,
    "text": "were inexistent. The naturalness was,"
  },
  {
    "startMs": 279919,
    "text": "obviously, much better than with Siri."
  },
  {
    "startMs": 281919,
    "text": "The latency was still pretty bad. So, we"
  },
  {
    "startMs": 283480,
    "text": "are talking about several seconds, and"
  },
  {
    "startMs": 284919,
    "text": "that was documented in the in the"
  },
  {
    "startMs": 286800,
    "text": "previous presentation."
  },
  {
    "startMs": 288880,
    "text": "And so, then, you know, we're in this"
  },
  {
    "startMs": 291080,
    "text": "third phase, which is a voice, what we"
  },
  {
    "startMs": 294120,
    "text": "call voice agents. Uh and here I'm going"
  },
  {
    "startMs": 296440,
    "text": "to show a quick demo of uh of a"
  },
  {
    "startMs": 298560,
    "text": "drive-thru bot uh live-coded with our"
  },
  {
    "startMs": 301480,
    "text": "with our voice models."
  },
  {
    "startMs": 303040,
    "text": ">> Hi there. What can I get started for you"
  },
  {
    "startMs": 305280,
    "text": "today?"
  },
  {
    "startMs": 306680,
    "text": ">> Hi, I'm actually in a hurry and I'd like"
  },
  {
    "startMs": 308520,
    "text": "to order a chicken sandwich quickly."
  },
  {
    "startMs": 311360,
    "text": ">> We have the classic, spicy, deluxe, or"
  },
  {
    "startMs": 314160,
    "text": "grilled chicken sandwich. Which one"
  },
  {
    "startMs": 316320,
    "text": "would you like?"
  },
  {
    "startMs": 317200,
    "text": ">> Uh let's go with the classic chicken"
  },
  {
    "startMs": 318680,
    "text": "sandwich."
  },
  {
    "startMs": 319720,
    "text": ">> Anything else for you?"
  },
  {
    "startMs": 321480,
    "text": ">> Uh what else do you have on the menu?"
  },
  {
    "startMs": 324320,
    "text": ">> We have sandwiches, sides, drinks, and"
  },
  {
    "startMs": 327400,
    "text": "desserts. What can I get for you?"
  },
  {
    "startMs": 330080,
    "text": ">> Can I get some mac and cheese, maybe?"
  },
  {
    "startMs": 332840,
    "text": ">> You have a classic chicken sandwich and"
  },
  {
    "startMs": 334680,
    "text": "mac and cheese."
  },
  {
    "startMs": 336120,
    "text": "Your total comes to eight."
  },
  {
    "startMs": 338000,
    "text": ">> So, now you know, unlike what we just"
  },
  {
    "startMs": 340400,
    "text": "saw, it's an actual agent. It's taking"
  },
  {
    "startMs": 342640,
    "text": "actions. It's keeping track of the"
  },
  {
    "startMs": 344560,
    "text": "order. It's then going to make you pay."
  },
  {
    "startMs": 346520,
    "text": "So, it's it's an actual voice agent that"
  },
  {
    "startMs": 348840,
    "text": "can do uh real tasks. So, here instead"
  },
  {
    "startMs": 351480,
    "text": "of having an LLM that is just a"
  },
  {
    "startMs": 353000,
    "text": "conversational interface,"
  },
  {
    "startMs": 354760,
    "text": "we have a real agent that is empowered"
  },
  {
    "startMs": 356720,
    "text": "with tool call, reasoning, planning, and"
  },
  {
    "startMs": 358760,
    "text": "and all this stuff."
  },
  {
    "startMs": 360160,
    "text": "So,"
  },
  {
    "startMs": 362320,
    "text": "what we see now is we have gained back"
  },
  {
    "startMs": 365160,
    "text": "agentic capabilities, and actually they"
  },
  {
    "startMs": 367560,
    "text": "are much more"
  },
  {
    "startMs": 369440,
    "text": "uh powerful and generic than before,"
  },
  {
    "startMs": 372360,
    "text": "while keeping a very good level of uh of"
  },
  {
    "startMs": 375200,
    "text": "naturalness."
  },
  {
    "startMs": 376360,
    "text": "And"
  },
  {
    "startMs": 377560,
    "text": "that's where speech-to-speech LLM came."
  },
  {
    "startMs": 379560,
    "text": "In particular, what we could see here is"
  },
  {
    "startMs": 382200,
    "text": "the latency, it's better with cascaded"
  },
  {
    "startMs": 384680,
    "text": "system, but it's still higher than you"
  },
  {
    "startMs": 387520,
    "text": "will have with human conversation. And"
  },
  {
    "startMs": 389360,
    "text": "as also was explained before, the"
  },
  {
    "startMs": 391240,
    "text": "naturalness is fundamentally limited by"
  },
  {
    "startMs": 392960,
    "text": "the fact that you go through text, so"
  },
  {
    "startMs": 394440,
    "text": "you lose a lot of information about what"
  },
  {
    "startMs": 397000,
    "text": "uh is said, the tone, the emotion of the"
  },
  {
    "startMs": 399160,
    "text": "user, and so on and so forth."
  },
  {
    "startMs": 401000,
    "text": "So, now that we have tackled"
  },
  {
    "startMs": 402200,
    "text": "intelligence and agentic capabilities,"
  },
  {
    "startMs": 404720,
    "text": "speech-to-speech seems like a natural"
  },
  {
    "startMs": 406080,
    "text": "next step for naturalness and latency."
  },
  {
    "startMs": 408120,
    "text": "And so here it's the announcement from"
  },
  {
    "startMs": 409720,
    "text": "the uh OpenAI advanced voice mode."
  },
  {
    "startMs": 413309,
    "text": ">> [clears throat]"
  },
  {
    "startMs": 413720,
    "text": ">> Hey, ChatGPT. I'm Mark. How are you?"
  },
  {
    "startMs": 416800,
    "text": ">> Oh, Mark."
  },
  {
    "startMs": 418640,
    "text": "I'm doing great. Thanks for asking. How"
  },
  {
    "startMs": 420800,
    "text": "about you?"
  },
  {
    "startMs": 422000,
    "text": ">> Hey, so I'm on stage right now. I'm"
  },
  {
    "startMs": 423960,
    "text": "doing a live demo, and frankly I'm"
  },
  {
    "startMs": 425680,
    "text": "feeling a little bit nervous. Can you"
  },
  {
    "startMs": 427240,
    "text": "help me calm my nerves a little bit?"
  },
  {
    "startMs": 429840,
    "text": ">> Oh, you're doing a live demo right now?"
  },
  {
    "startMs": 432200,
    "text": "That's awesome."
  },
  {
    "startMs": 433880,
    "text": "Just"
  },
  {
    "startMs": 434600,
    "text": ">> I think we all remember it was very"
  },
  {
    "startMs": 436000,
    "text": "impressive very impressive release."
  },
  {
    "startMs": 438680,
    "text": "And in that context now, all the steps"
  },
  {
    "startMs": 441360,
    "text": "of STT, LLM, and TTS have been absorbed"
  },
  {
    "startMs": 444160,
    "text": "into a a single one."
  },
  {
    "startMs": 447120,
    "text": "And so now,"
  },
  {
    "startMs": 448520,
    "text": "intelligence, you know, like naturalness"
  },
  {
    "startMs": 450280,
    "text": "is"
  },
  {
    "startMs": 452480,
    "text": "still very good. Uh"
  },
  {
    "startMs": 454280,
    "text": "actually it can be better because it can"
  },
  {
    "startMs": 456120,
    "text": "understand non-linguistic information."
  },
  {
    "startMs": 457919,
    "text": "Latency is really, really nice."
  },
  {
    "startMs": 460200,
    "text": "Honestly, it doesn't make sense to go uh"
  },
  {
    "startMs": 462720,
    "text": "better than that."
  },
  {
    "startMs": 464080,
    "text": "Interestingly and everyone was used any"
  },
  {
    "startMs": 466640,
    "text": "uh speech-to-speech model can uh attest"
  },
  {
    "startMs": 468960,
    "text": "that"
  },
  {
    "startMs": 470040,
    "text": "the intelligence"
  },
  {
    "startMs": 471880,
    "text": "is still much more limited in that"
  },
  {
    "startMs": 474360,
    "text": "context than uh the cascaded"
  },
  {
    "startMs": 476720,
    "text": "counterpart. So, the speech-to-speech"
  },
  {
    "startMs": 478480,
    "text": "models are fundamentally still limited"
  },
  {
    "startMs": 480520,
    "text": "compared to the textual models."
  },
  {
    "startMs": 483200,
    "text": "Another limitation is turn-taking. So,"
  },
  {
    "startMs": 487400,
    "text": "people tend to"
  },
  {
    "startMs": 488840,
    "text": "mix speech-to-speech and full duplex."
  },
  {
    "startMs": 491480,
    "text": "But basically,"
  },
  {
    "startMs": 493200,
    "text": "when you do have a speech-to-speech"
  },
  {
    "startMs": 494360,
    "text": "model like GPT-3 time, it's still based"
  },
  {
    "startMs": 496520,
    "text": "on fundamental turn-taking. In the sense"
  },
  {
    "startMs": 498280,
    "text": "that it's going to segment the"
  },
  {
    "startMs": 499840,
    "text": "conversation into as long as the model"
  },
  {
    "startMs": 501400,
    "text": "is speaking or the model is listening."
  },
  {
    "startMs": 503400,
    "text": "And to give to show you how this can"
  },
  {
    "startMs": 506680,
    "text": "make an interaction unnatural, I'm going"
  },
  {
    "startMs": 508720,
    "text": "to show a a small demo with what is"
  },
  {
    "startMs": 510640,
    "text": "called backchanneling, which is this"
  },
  {
    "startMs": 512240,
    "text": "very human thing that you do when"
  },
  {
    "startMs": 513440,
    "text": "someone talks to you is that you say,"
  },
  {
    "startMs": 514680,
    "text": "\"Mhm, yeah.\" and so on."
  },
  {
    "startMs": 517560,
    "text": ">> Hey, how's it going? Just like to"
  },
  {
    "startMs": 519240,
    "text": "brainstorm a bit about it with you."
  },
  {
    "startMs": 522039,
    "text": ">> Oh, that's a great topic. Yeah, I'd love"
  },
  {
    "startMs": 524080,
    "text": "to help you brainstorm. Are you thinking"
  },
  {
    "startMs": 525839,
    "text": "what"
  },
  {
    "startMs": 527320,
    "text": "Exactly. Yeah, I was thinking"
  },
  {
    "startMs": 528720,
    "text": ">> No, no, I didn't mean to interrupt, you"
  },
  {
    "startMs": 530080,
    "text": "know, I was just saying, \"Yeah.\" like"
  },
  {
    "startMs": 531560,
    "text": "that. You can just keep going. I know,"
  },
  {
    "startMs": 533240,
    "text": "don't mind me. It's just something I"
  },
  {
    "startMs": 534560,
    "text": "typically do."
  },
  {
    "startMs": 536240,
    "text": ">> Uh no worries at all. Yeah, I was just"
  },
  {
    "startMs": 538280,
    "text": "going to say we could break it down into"
  },
  {
    "startMs": 539680,
    "text": "a few aspects. Like"
  },
  {
    "startMs": 541480,
    "text": "Yeah, exactly."
  },
  {
    "startMs": 542240,
    "text": ">> please stop stop interrupting. You know,"
  },
  {
    "startMs": 543680,
    "text": "it's called backchanneling. Humans do it"
  },
  {
    "startMs": 545400,
    "text": "all the time. It shows that you're just"
  },
  {
    "startMs": 546760,
    "text": "following the conversation. That you"
  },
  {
    "startMs": 548120,
    "text": "don't,"
  },
  {
    "startMs": 549000,
    "text": "you know, like interrupt you in your"
  },
  {
    "startMs": 550160,
    "text": "flow. Just just going."
  },
  {
    "startMs": 552400,
    "text": ">> Ah, got it. Thanks for letting me know."
  },
  {
    "startMs": 554480,
    "text": ">> No problem."
  },
  {
    "startMs": 556160,
    "text": "Oh, come on."
  },
  {
    "startMs": 557160,
    "text": ">> Yeah, so you see, you know, it's it's"
  },
  {
    "startMs": 558480,
    "text": "still very annoying. Uh you can have"
  },
  {
    "startMs": 560760,
    "text": "lightning speed latency. Fundamentally,"
  },
  {
    "startMs": 563240,
    "text": "this is"
  },
  {
    "startMs": 564880,
    "text": "uh an issue that can not be resolved"
  },
  {
    "startMs": 567720,
    "text": "when you're using turn-taking. So, here"
  },
  {
    "startMs": 569200,
    "text": "that's the walkie-talkie."
  },
  {
    "startMs": 570880,
    "text": "Um any real-time model today, I mean,"
  },
  {
    "startMs": 573920,
    "text": "now there is a bidirectional one that"
  },
  {
    "startMs": 575880,
    "text": "will come from OpenAI, but it's called"
  },
  {
    "startMs": 577600,
    "text": "half duplex. So, the model is listening"
  },
  {
    "startMs": 579760,
    "text": "or speaking. A human conversation"
  },
  {
    "startMs": 582800,
    "text": "has a constant flow between two people."
  },
  {
    "startMs": 585480,
    "text": "People do back channeling. People"
  },
  {
    "startMs": 587000,
    "text": "interrupt one another, talk on one"
  },
  {
    "startMs": 588800,
    "text": "another, and so on."
  },
  {
    "startMs": 590240,
    "text": "If you have If you're having a relative"
  },
  {
    "startMs": 592120,
    "text": "on the phone, there is up to 20% of the"
  },
  {
    "startMs": 594720,
    "text": "time where you are both speaking at the"
  },
  {
    "startMs": 596120,
    "text": "same time."
  },
  {
    "startMs": 597400,
    "text": "And that makes, you know, this very"
  },
  {
    "startMs": 600160,
    "text": "flexible dynamics in the conversation"
  },
  {
    "startMs": 602080,
    "text": "makes it much more comfortable for"
  },
  {
    "startMs": 603400,
    "text": "humans."
  },
  {
    "startMs": 604440,
    "text": "And so, to understand how"
  },
  {
    "startMs": 607280,
    "text": "we can make a model full duplex, I'll"
  },
  {
    "startMs": 609360,
    "text": "give a very short"
  },
  {
    "startMs": 612000,
    "text": "presentation of how we train such"
  },
  {
    "startMs": 613720,
    "text": "models. So, the way you create a"
  },
  {
    "startMs": 615160,
    "text": "speech-to-speech model half duplex or"
  },
  {
    "startMs": 616920,
    "text": "full duplex is the following one. So,"
  },
  {
    "startMs": 618520,
    "text": "you you start from a text LLM, which is"
  },
  {
    "startMs": 620520,
    "text": "a probabilistic models over over words."
  },
  {
    "startMs": 623320,
    "text": "And instead of predicting the next word"
  },
  {
    "startMs": 624920,
    "text": "based on the past,"
  },
  {
    "startMs": 626480,
    "text": "what you want to do is rather predict"
  },
  {
    "startMs": 628120,
    "text": "the next audio based based on the past"
  },
  {
    "startMs": 630240,
    "text": "audio."
  },
  {
    "startMs": 631360,
    "text": "The issue now is that if you pass a raw"
  },
  {
    "startMs": 633760,
    "text": "audio to your model, which is, you know,"
  },
  {
    "startMs": 635680,
    "text": "a waveform, it's"
  },
  {
    "startMs": 637760,
    "text": "air pressure variations."
  },
  {
    "startMs": 639520,
    "text": "Uh"
  },
  {
    "startMs": 640480,
    "text": "basically, you take this sentence, it's"
  },
  {
    "startMs": 642960,
    "text": "eight words."
  },
  {
    "startMs": 644480,
    "text": "It takes around 3 seconds to pronounce"
  },
  {
    "startMs": 646080,
    "text": "it. And so, at 24 kHz audio, instead of"
  },
  {
    "startMs": 649120,
    "text": "having eight words, the audio form is"
  },
  {
    "startMs": 651000,
    "text": "72,000 time steps that you would need to"
  },
  {
    "startMs": 653640,
    "text": "feed to your LLM. Given that LLMs have"
  },
  {
    "startMs": 656720,
    "text": "quadratic complexity with sequence"
  },
  {
    "startMs": 658440,
    "text": "length, so the complexity is the square"
  },
  {
    "startMs": 660360,
    "text": "of the sequence length. A 10,000 times"
  },
  {
    "startMs": 662480,
    "text": "longer sequence is 100 million times"
  },
  {
    "startMs": 664400,
    "text": "more expensive to to process. So, there"
  },
  {
    "startMs": 666200,
    "text": "is no way you can train an LLM on raw"
  },
  {
    "startMs": 668000,
    "text": "audio. So, the way you address it is by"
  },
  {
    "startMs": 670160,
    "text": "creating neural codecs, or you can also"
  },
  {
    "startMs": 672480,
    "text": "call them audio tokenizers. And"
  },
  {
    "startMs": 674760,
    "text": "basically, it's an encoder that takes an"
  },
  {
    "startMs": 676600,
    "text": "audio and compresses it in a very dense"
  },
  {
    "startMs": 679720,
    "text": "compressed representation, a bit similar"
  },
  {
    "startMs": 681640,
    "text": "to text. And then you have a decoder"
  },
  {
    "startMs": 683640,
    "text": "that can reconstruct high-quality audio"
  },
  {
    "startMs": 685520,
    "text": "from it. So, now"
  },
  {
    "startMs": 687560,
    "text": "you have gone from the audio domain into"
  },
  {
    "startMs": 689600,
    "text": "a abstract representation domain, where"
  },
  {
    "startMs": 691839,
    "text": "you can train an LLM exactly like you"
  },
  {
    "startMs": 693560,
    "text": "would train it on text."
  },
  {
    "startMs": 695560,
    "text": "And the speech-to-speech model from"
  },
  {
    "startMs": 697160,
    "text": "ElevenLabs, as I was showing before,"
  },
  {
    "startMs": 698760,
    "text": "works in this fashion. So, instead of"
  },
  {
    "startMs": 700840,
    "text": "having text tokens into your model, you"
  },
  {
    "startMs": 703480,
    "text": "have audio tokens that represent either"
  },
  {
    "startMs": 705880,
    "text": "the LLM or the user, and you put them"
  },
  {
    "startMs": 707840,
    "text": "one after the other, and the model"
  },
  {
    "startMs": 709440,
    "text": "predicts the audio tokens uh that should"
  },
  {
    "startMs": 712000,
    "text": "be said by the model, being given the"
  },
  {
    "startMs": 713960,
    "text": "context from both sides of the"
  },
  {
    "startMs": 715520,
    "text": "conversation."
  },
  {
    "startMs": 716920,
    "text": "However, you can see that it's still a"
  },
  {
    "startMs": 718520,
    "text": "sequence between user and system, which"
  },
  {
    "startMs": 721600,
    "text": "is still half duplex. So, how did we"
  },
  {
    "startMs": 724000,
    "text": "make the first full duplex model ever?"
  },
  {
    "startMs": 726240,
    "text": "Very simple. We call it multi-stream"
  },
  {
    "startMs": 728720,
    "text": "language models. That's the technology"
  },
  {
    "startMs": 730520,
    "text": "now used also by Thinking Machines for"
  },
  {
    "startMs": 732360,
    "text": "their interaction model, and most likely"
  },
  {
    "startMs": 734440,
    "text": "by the for the by the directional model"
  },
  {
    "startMs": 736720,
    "text": "of OpenAI. Is that instead of having a"
  },
  {
    "startMs": 738800,
    "text": "transformer that models one sequence of"
  },
  {
    "startMs": 740680,
    "text": "tokens, it models two of them, so that"
  },
  {
    "startMs": 743200,
    "text": "both parties can be active at the same"
  },
  {
    "startMs": 745720,
    "text": "time, inactive at the same time, one"
  },
  {
    "startMs": 747520,
    "text": "active and one inactive. And"
  },
  {
    "startMs": 750240,
    "text": "I just show a very quick demo of uh of"
  },
  {
    "startMs": 752880,
    "text": "how it sounds [snorts] like, but that's"
  },
  {
    "startMs": 754560,
    "text": "the release of machine August 2024,"
  },
  {
    "startMs": 757480,
    "text": "uh where we did an announcement live on"
  },
  {
    "startMs": 759640,
    "text": "stage talking to it for the first time."
  },
  {
    "startMs": 761960,
    "text": "And you'll see that the model often"
  },
  {
    "startMs": 763720,
    "text": "guesses the end of the question, answers"
  },
  {
    "startMs": 766520,
    "text": "over the speaker,"
  },
  {
    "startMs": 768320,
    "text": "and both speaking at the same time is"
  },
  {
    "startMs": 770000,
    "text": "not breaking the flow like we saw with"
  },
  {
    "startMs": 772640,
    "text": "GPT. The whole thing is just extremely"
  },
  {
    "startMs": 774520,
    "text": "resilient to the most chaotic uh"
  },
  {
    "startMs": 776880,
    "text": "situations."
  },
  {
    "startMs": 777560,
    "text": ">> So, the planet is serious 22. Can you"
  },
  {
    "startMs": 780440,
    "text": "plot a trajectory course to it, please?"
  },
  {
    "startMs": 782560,
    "text": ">> Yes, sir."
  },
  {
    "startMs": 783760,
    "text": ">> Okay. How long is it going to take us to"
  },
  {
    "startMs": 785440,
    "text": "get there?"
  },
  {
    "startMs": 786080,
    "text": ">> it out. It's approximately 5 months to"
  },
  {
    "startMs": 788560,
    "text": "get there."
  },
  {
    "startMs": 789160,
    "text": ">> Okay, that's that's not too bad. Uh do"
  },
  {
    "startMs": 791760,
    "text": "you think we have all we need on board"
  },
  {
    "startMs": 793120,
    "text": "the ship to start the mission?"
  },
  {
    "startMs": 794640,
    "text": ">> We have everything we need."
  },
  {
    "startMs": 796560,
    "text": ">> So, back then it was even a bit"
  },
  {
    "startMs": 798280,
    "text": "irritating to people because it was"
  },
  {
    "startMs": 799760,
    "text": "interrupting you all the time. But the"
  },
  {
    "startMs": 801680,
    "text": "thing is that you can use you could"
  },
  {
    "startMs": 803400,
    "text": "still use it in extremely noisy"
  },
  {
    "startMs": 804880,
    "text": "environments with a lot of noise, people"
  },
  {
    "startMs": 806800,
    "text": "coughing, and so on. And you know, the"
  },
  {
    "startMs": 808360,
    "text": "flow is just constant. You don't get"
  },
  {
    "startMs": 810040,
    "text": "this very irritating break of the"
  },
  {
    "startMs": 812320,
    "text": "conversational flow. So,"
  },
  {
    "startMs": 815520,
    "text": "these full duplex models, they are the"
  },
  {
    "startMs": 817440,
    "text": "highest level of naturalness you can"
  },
  {
    "startMs": 818880,
    "text": "expect. That's the same conversation"
  },
  {
    "startMs": 820720,
    "text": "with a human."
  },
  {
    "startMs": 822080,
    "text": "The thing is, in with our models, it was"
  },
  {
    "startMs": 824640,
    "text": "even more stupid than"
  },
  {
    "startMs": 826520,
    "text": "speech-to-speech models that were"
  },
  {
    "startMs": 827760,
    "text": "already less intelligent than cascaded"
  },
  {
    "startMs": 829400,
    "text": "systems."
  },
  {
    "startMs": 830520,
    "text": "It's probably fine for some use cases if"
  },
  {
    "startMs": 832320,
    "text": "you just want to have a chit-chat. You"
  },
  {
    "startMs": 833720,
    "text": "know, the model doesn't need to be very"
  },
  {
    "startMs": 835200,
    "text": "intelligent. But make an actual full"
  },
  {
    "startMs": 837200,
    "text": "duplex voice agent,"
  },
  {
    "startMs": 838800,
    "text": "there is no way we can give up on on"
  },
  {
    "startMs": 840400,
    "text": "intelligence just to gain uh"
  },
  {
    "startMs": 841960,
    "text": "speech-to-speech abilities."
  },
  {
    "startMs": 844120,
    "text": "So, how do we finally make models that"
  },
  {
    "startMs": 846200,
    "text": "tackle all these aspects jointly?"
  },
  {
    "startMs": 848920,
    "text": "And I think interestingly, if you if you"
  },
  {
    "startMs": 851000,
    "text": "look at the history I showed, there is a"
  },
  {
    "startMs": 852920,
    "text": "tension between naturalness and"
  },
  {
    "startMs": 854240,
    "text": "intelligence. So, every time we improve"
  },
  {
    "startMs": 856160,
    "text": "naturalness or humanness of the of the"
  },
  {
    "startMs": 858440,
    "text": "models, they were less intelligent than"
  },
  {
    "startMs": 860960,
    "text": "the cascaded system. The cascaded"
  },
  {
    "startMs": 862960,
    "text": "agents, they are basically as smart as"
  },
  {
    "startMs": 864840,
    "text": "the best text models. So, if you have a"
  },
  {
    "startMs": 866840,
    "text": "voice agent that is powered by the"
  },
  {
    "startMs": 868400,
    "text": "latest model from Anthropic or OpenAI,"
  },
  {
    "startMs": 869960,
    "text": "it's going to be extremely smart, have"
  },
  {
    "startMs": 871360,
    "text": "all the same reliability for tool call,"
  },
  {
    "startMs": 873040,
    "text": "and so on. Speech-to-speech has this"
  },
  {
    "startMs": 875200,
    "text": "naturalness"
  },
  {
    "startMs": 876640,
    "text": "aspect. However, you give up"
  },
  {
    "startMs": 878560,
    "text": "intelligence to get that. And the reason"
  },
  {
    "startMs": 880720,
    "text": "why you give up intelligence is remember"
  },
  {
    "startMs": 883560,
    "text": "that the LLM is a model that has a"
  },
  {
    "startMs": 885480,
    "text": "certain number of weights that we call"
  },
  {
    "startMs": 887320,
    "text": "the capacity."
  },
  {
    "startMs": 888839,
    "text": "And if you take a text model and now it"
  },
  {
    "startMs": 891120,
    "text": "not only has to handle text, but it also"
  },
  {
    "startMs": 893320,
    "text": "needs to understand speech and produce"
  },
  {
    "startMs": 894880,
    "text": "speech,"
  },
  {
    "startMs": 895920,
    "text": "it's taking some of its capacity, and"
  },
  {
    "startMs": 898200,
    "text": "this capacity now"
  },
  {
    "startMs": 900040,
    "text": "is taken from the intelligence. So,"
  },
  {
    "startMs": 901680,
    "text": "fundamentally, there is a cost of adding"
  },
  {
    "startMs": 904040,
    "text": "a new modality to a text model that is"
  },
  {
    "startMs": 906280,
    "text": "going to be paid in in intelligence."
  },
  {
    "startMs": 908960,
    "text": "So, where do we go from here? There are"
  },
  {
    "startMs": 910680,
    "text": "two paths that are in front of us, and"
  },
  {
    "startMs": 912560,
    "text": "both are going to be explored at the"
  },
  {
    "startMs": 914320,
    "text": "same time. The first one is scaling the"
  },
  {
    "startMs": 916360,
    "text": "model. So,"
  },
  {
    "startMs": 918400,
    "text": "making your speech-to-speech model"
  },
  {
    "startMs": 919720,
    "text": "bigger, better pre-trained, better"
  },
  {
    "startMs": 922040,
    "text": "post-trained, and so on. We likely"
  },
  {
    "startMs": 924440,
    "text": "progressively increase its intelligence"
  },
  {
    "startMs": 926400,
    "text": "until it it's good enough for a lot of"
  },
  {
    "startMs": 928440,
    "text": "use cases."
  },
  {
    "startMs": 929680,
    "text": "The second one is splitting the model"
  },
  {
    "startMs": 932280,
    "text": "between naturalness and intelligence."
  },
  {
    "startMs": 935280,
    "text": "The first one,"
  },
  {
    "startMs": 936560,
    "text": "I'm not at OpenAI, so I don't know"
  },
  {
    "startMs": 937960,
    "text": "because they don't release their model."
  },
  {
    "startMs": 939520,
    "text": "I guess OpenAI is the path one, so it's"
  },
  {
    "startMs": 942280,
    "text": "a frontier text LLM with a lot of"
  },
  {
    "startMs": 944760,
    "text": "science around post-training, in-"
  },
  {
    "startMs": 946960,
    "text": "instruct tuning to fine-tune it on"
  },
  {
    "startMs": 948400,
    "text": "audio, and teaching it to be quite smart"
  },
  {
    "startMs": 951680,
    "text": "while using audio."
  },
  {
    "startMs": 954120,
    "text": "The nice thing about that is you have a"
  },
  {
    "startMs": 955160,
    "text": "single model to orchestrate, so it's"
  },
  {
    "startMs": 957160,
    "text": "quite easy to deploy."
  },
  {
    "startMs": 959120,
    "text": "Um"
  },
  {
    "startMs": 960640,
    "text": "and"
  },
  {
    "startMs": 962240,
    "text": "one"
  },
  {
    "startMs": 963720,
    "text": "big aspect, however, is that it's a"
  },
  {
    "startMs": 966000,
    "text": "extremely complex and costly process to"
  },
  {
    "startMs": 968440,
    "text": "go from the text model to the"
  },
  {
    "startMs": 969840,
    "text": "speech-to-speech model."
  },
  {
    "startMs": 971560,
    "text": "The second path is to split it. It's an"
  },
  {
    "startMs": 973320,
    "text": "approach that we introduced in one of"
  },
  {
    "startMs": 974960,
    "text": "our recent papers called Moushiraq, and"
  },
  {
    "startMs": 976640,
    "text": "that has been reused by, in particular,"
  },
  {
    "startMs": 978480,
    "text": "the Thinking Machine Interaction models."
  },
  {
    "startMs": 981000,
    "text": "Where, basically, the idea is that now"
  },
  {
    "startMs": 983520,
    "text": "you have two models. The first one is a"
  },
  {
    "startMs": 985360,
    "text": "small, maybe even on-device,"
  },
  {
    "startMs": 987800,
    "text": "full-duplex, extremely natural"
  },
  {
    "startMs": 989360,
    "text": "speech-to-speech interface. And its only"
  },
  {
    "startMs": 991520,
    "text": "role"
  },
  {
    "startMs": 992680,
    "text": "is to keep a very natural"
  },
  {
    "startMs": 994680,
    "text": "conversation and be able to delegate"
  },
  {
    "startMs": 997760,
    "text": "all the thinking, tool calling,"
  },
  {
    "startMs": 1000240,
    "text": "reasoning, agentic capabilities to a"
  },
  {
    "startMs": 1002120,
    "text": "background text model. And so, the way"
  },
  {
    "startMs": 1004080,
    "text": "to see it is you have a background text"
  },
  {
    "startMs": 1006440,
    "text": "LLM"
  },
  {
    "startMs": 1007480,
    "text": "that receives asynchronously queries"
  },
  {
    "startMs": 1009720,
    "text": "from hundreds to thousands of small"
  },
  {
    "startMs": 1011520,
    "text": "voice interfaces and just give them"
  },
  {
    "startMs": 1013240,
    "text": "their text, you know? And, basically,"
  },
  {
    "startMs": 1015640,
    "text": "what we did is um"
  },
  {
    "startMs": 1017440,
    "text": "very small full-duplex model that just"
  },
  {
    "startMs": 1019920,
    "text": "needs to know when it doesn't know, so"
  },
  {
    "startMs": 1021680,
    "text": "that it can delegate to the background"
  },
  {
    "startMs": 1023240,
    "text": "model."
  },
  {
    "startMs": 1024400,
    "text": "And the reason why we believe mostly in"
  },
  {
    "startMs": 1026040,
    "text": "this approach,"
  },
  {
    "startMs": 1027480,
    "text": "um"
  },
  {
    "startMs": 1028880,
    "text": "and I go back to to it later, it's a"
  },
  {
    "startMs": 1032480,
    "text": "our"
  },
  {
    "startMs": 1033640,
    "text": "uh"
  },
  {
    "startMs": 1034520,
    "text": "let's say our culture is more of first"
  },
  {
    "startMs": 1036120,
    "text": "one, the bitter lesson. So, every time"
  },
  {
    "startMs": 1038480,
    "text": "we've been pushing for end-to-end"
  },
  {
    "startMs": 1039880,
    "text": "systems and so on. But now I think the"
  },
  {
    "startMs": 1042040,
    "text": "hybrid approach has two main"
  },
  {
    "startMs": 1044880,
    "text": "advantages. The The first one is cost."
  },
  {
    "startMs": 1046880,
    "text": "So, speech-to-speech models are"
  },
  {
    "startMs": 1048360,
    "text": "notoriously quite expensive."
  },
  {
    "startMs": 1050440,
    "text": "And when you think about it,"
  },
  {
    "startMs": 1052640,
    "text": "it's a loss of money to do chit-chat"
  },
  {
    "startMs": 1055120,
    "text": "with gigantic speech-to-speech models"
  },
  {
    "startMs": 1057680,
    "text": "that can resolve differential equations"
  },
  {
    "startMs": 1059720,
    "text": "and so on. So, it doesn't really"
  },
  {
    "startMs": 1062400,
    "text": "make sense economically to get all your"
  },
  {
    "startMs": 1064520,
    "text": "workflow through this gigantic"
  },
  {
    "startMs": 1066240,
    "text": "multimodal mixture of experts."
  },
  {
    "startMs": 1068680,
    "text": "At the same time, we see that people are"
  },
  {
    "startMs": 1070240,
    "text": "very attached to their ability to"
  },
  {
    "startMs": 1071840,
    "text": "control the backend, to be able to"
  },
  {
    "startMs": 1073800,
    "text": "switch So they So they 5 was released a"
  },
  {
    "startMs": 1076120,
    "text": "few minutes ago. People want to switch"
  },
  {
    "startMs": 1078120,
    "text": "the backend and the intelligence and get"
  },
  {
    "startMs": 1079880,
    "text": "a lot of optionality on that, right?"
  },
  {
    "startMs": 1082240,
    "text": "When you're using a speech-to-speech"
  },
  {
    "startMs": 1083600,
    "text": "model, your your hands are a bit tied"
  },
  {
    "startMs": 1086280,
    "text": "with this model provider. And to give"
  },
  {
    "startMs": 1088440,
    "text": "you an idea of that,"
  },
  {
    "startMs": 1089880,
    "text": "until recently the AdSense voice mode"
  },
  {
    "startMs": 1091320,
    "text": "from OpenAI was powered by GPT-4o,"
  },
  {
    "startMs": 1093960,
    "text": "despite the fact that there have been"
  },
  {
    "startMs": 1095320,
    "text": "several generations of the text model"
  },
  {
    "startMs": 1097040,
    "text": "since then, because this process is so"
  },
  {
    "startMs": 1099080,
    "text": "expensive and so long."
  },
  {
    "startMs": 1101920,
    "text": "For this reason, we rather bet on the"
  },
  {
    "startMs": 1104000,
    "text": "hybrid approach because that will give"
  },
  {
    "startMs": 1106720,
    "text": "something that is not only very natural"
  },
  {
    "startMs": 1108920,
    "text": "and very nice for demos and impressive,"
  },
  {
    "startMs": 1111440,
    "text": "but also will be a viable alternative"
  },
  {
    "startMs": 1113960,
    "text": "from a economic point of view and"
  },
  {
    "startMs": 1115760,
    "text": "agentic capabilities point of view"
  },
  {
    "startMs": 1118360,
    "text": "to the best cascaded systems that are"
  },
  {
    "startMs": 1120640,
    "text": "still most of the market today in voice."
  },
  {
    "startMs": 1123080,
    "text": "So,"
  },
  {
    "startMs": 1124680,
    "text": "what now?"
  },
  {
    "startMs": 1125840,
    "text": "Uh you can use our models on gradium.ai."
  },
  {
    "startMs": 1128040,
    "text": "You can apply to gradium. We are"
  },
  {
    "startMs": 1130280,
    "text": "recruiting research scientists and"
  },
  {
    "startMs": 1131960,
    "text": "engineers. And thanks for your"
  },
  {
    "startMs": 1133880,
    "text": "attention."
  },
  {
    "startMs": 1136438,
    "text": ">> [applause]"
  },
  {
    "startMs": 1150848,
    "text": "[music]"
  }
]