[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12760,
    "text": ">> Hi everyone. Thanks for coming to our"
  },
  {
    "startMs": 14240,
    "text": "talk. We're going to talk a lot about"
  },
  {
    "startMs": 16480,
    "text": "how we are planning on voiceifying the"
  },
  {
    "startMs": 18720,
    "text": "agentic future with speech-to-speech"
  },
  {
    "startMs": 20320,
    "text": "research here at Google DeepMind. My"
  },
  {
    "startMs": 22440,
    "text": "name is Valeria. I'm the product lead"
  },
  {
    "startMs": 24080,
    "text": "for the speech-to-speech model in Gemini"
  },
  {
    "startMs": 25760,
    "text": "and Tom."
  },
  {
    "startMs": 27000,
    "text": ">> Yeah, my name is Tom. I work on"
  },
  {
    "startMs": 28800,
    "text": "speech-to-speech as an engineer and"
  },
  {
    "startMs": 30760,
    "text": "Gemini. So, yeah, great to be presenting"
  },
  {
    "startMs": 32880,
    "text": "our work."
  },
  {
    "startMs": 33960,
    "text": ">> Cool."
  },
  {
    "startMs": 35040,
    "text": "So, before we start diving into the"
  },
  {
    "startMs": 37040,
    "text": "history of Google research and audio and"
  },
  {
    "startMs": 39840,
    "text": "the latest research we've been doing, I"
  },
  {
    "startMs": 41480,
    "text": "just want to take a step back and really"
  },
  {
    "startMs": 43280,
    "text": "kind of like think about why this is"
  },
  {
    "startMs": 45000,
    "text": "important to us in the team. And the"
  },
  {
    "startMs": 47520,
    "text": "main reason is that voice is the most"
  },
  {
    "startMs": 49360,
    "text": "natural way for humans to interact with"
  },
  {
    "startMs": 51520,
    "text": "both the physical and the virtual world."
  },
  {
    "startMs": 53800,
    "text": "And we're already seeing so many"
  },
  {
    "startMs": 55080,
    "text": "applications that are starting even"
  },
  {
    "startMs": 56800,
    "text": "within the own Google products. We see"
  },
  {
    "startMs": 58920,
    "text": "voice being used to ask questions and"
  },
  {
    "startMs": 61480,
    "text": "you know, do homework or info seeking or"
  },
  {
    "startMs": 63200,
    "text": "EDU in search live and Gemini live. All"
  },
  {
    "startMs": 65840,
    "text": "the way to these same models being"
  },
  {
    "startMs": 67720,
    "text": "deployed in cloud and the API for"
  },
  {
    "startMs": 69680,
    "text": "enterprise or voice agent use cases."
  },
  {
    "startMs": 71680,
    "text": "Should I use this as a"
  },
  {
    "startMs": 73240,
    "text": "microphone?"
  },
  {
    "startMs": 75640,
    "text": "Hello. Okay."
  },
  {
    "startMs": 76880,
    "text": "Um"
  },
  {
    "startMs": 78360,
    "text": "So, because of that, we only think that"
  },
  {
    "startMs": 79800,
    "text": "the number of applications is going to,"
  },
  {
    "startMs": 81640,
    "text": "you know, exponentially increase over"
  },
  {
    "startMs": 83280,
    "text": "the next few years and we believe that"
  },
  {
    "startMs": 85280,
    "text": "speech-to-speech models are the way to"
  },
  {
    "startMs": 86800,
    "text": "go when we want to build robust"
  },
  {
    "startMs": 88360,
    "text": "universal voice agents. And with that,"
  },
  {
    "startMs": 90640,
    "text": "I'll hand it off to Tom."
  },
  {
    "startMs": 93560,
    "text": ">> All right. So, of course, like one way"
  },
  {
    "startMs": 94800,
    "text": "to create a voice agent is to just add"
  },
  {
    "startMs": 97280,
    "text": "an ASR like speech-to-text model in the"
  },
  {
    "startMs": 99600,
    "text": "pipeline and then a text-to-speech model"
  },
  {
    "startMs": 100960,
    "text": "on the other end, right? So, of course,"
  },
  {
    "startMs": 102400,
    "text": "that's where a lot of the history of"
  },
  {
    "startMs": 103760,
    "text": "speech come from. Of course, Google has"
  },
  {
    "startMs": 105000,
    "text": "been working on this for a long time."
  },
  {
    "startMs": 106840,
    "text": "I'm going to give you a little bit of a"
  },
  {
    "startMs": 107840,
    "text": "historical overview of what speech"
  },
  {
    "startMs": 109720,
    "text": "modeling, especially speech-to-text"
  },
  {
    "startMs": 111320,
    "text": "automatic"
  },
  {
    "startMs": 112760,
    "text": "speech recognition, typically looks"
  },
  {
    "startMs": 114120,
    "text": "like, right? So, up until around 2018,"
  },
  {
    "startMs": 116400,
    "text": "this usually involved a lot of different"
  },
  {
    "startMs": 117800,
    "text": "components. You have um you know of"
  },
  {
    "startMs": 119400,
    "text": "course feature extraction that's fairly"
  },
  {
    "startMs": 121080,
    "text": "general and then you have all these"
  },
  {
    "startMs": 122720,
    "text": "different pieces like acoustic modeling,"
  },
  {
    "startMs": 124479,
    "text": "pronunciation modeling, language"
  },
  {
    "startMs": 126160,
    "text": "modeling, a second pass rescoring that"
  },
  {
    "startMs": 128600,
    "text": "allows you to go from the audio input to"
  },
  {
    "startMs": 130600,
    "text": "a text transcription. Right? And of"
  },
  {
    "startMs": 132320,
    "text": "course like around 2018, these moved"
  },
  {
    "startMs": 134400,
    "text": "more and more towards end-to-end"
  },
  {
    "startMs": 135400,
    "text": "systems. You don't have to have so much"
  },
  {
    "startMs": 136560,
    "text": "domain knowledge. You can actually have"
  },
  {
    "startMs": 138400,
    "text": "mostly the the neural model learn this"
  },
  {
    "startMs": 141000,
    "text": "pattern and mapping between acoustic"
  },
  {
    "startMs": 143040,
    "text": "inputs and text. But these aren't really"
  },
  {
    "startMs": 145360,
    "text": "the end-to-end models that we think"
  },
  {
    "startMs": 146360,
    "text": "about when we think about LLMs. They're"
  },
  {
    "startMs": 147400,
    "text": "only doing kind of one thing, which is"
  },
  {
    "startMs": 149480,
    "text": "speech to a transcription of that"
  },
  {
    "startMs": 151160,
    "text": "speech. They're not responding, they're"
  },
  {
    "startMs": 152280,
    "text": "not translating. If you wanted to have"
  },
  {
    "startMs": 154600,
    "text": "the model tell you about the tone or the"
  },
  {
    "startMs": 156480,
    "text": "emotion or the speed of the speaker. If"
  },
  {
    "startMs": 158200,
    "text": "you wanted to bias it towards words or"
  },
  {
    "startMs": 160080,
    "text": "much less like images, those are all"
  },
  {
    "startMs": 161880,
    "text": "things you have to build yourself as"
  },
  {
    "startMs": 163360,
    "text": "part of the system and there's really"
  },
  {
    "startMs": 165120,
    "text": "really a barrier to how easily you can"
  },
  {
    "startMs": 166840,
    "text": "scale these systems. So,"
  },
  {
    "startMs": 169080,
    "text": "fast forward to now, which is kind of of"
  },
  {
    "startMs": 171320,
    "text": "course the LLM era, right? Like you"
  },
  {
    "startMs": 172959,
    "text": "know, the first LLMs were mostly text,"
  },
  {
    "startMs": 174480,
    "text": "but even there I think you could kind of"
  },
  {
    "startMs": 175800,
    "text": "hack audio embeddings into these text"
  },
  {
    "startMs": 177760,
    "text": "models and it kind of worked, but now of"
  },
  {
    "startMs": 179040,
    "text": "course for a long time now, Gemini"
  },
  {
    "startMs": 180720,
    "text": "models have been very natively"
  },
  {
    "startMs": 182440,
    "text": "multimodal. So, what does that mean? It"
  },
  {
    "startMs": 184320,
    "text": "means that when we train these models in"
  },
  {
    "startMs": 185560,
    "text": "pre-training, which is where the bulk of"
  },
  {
    "startMs": 186840,
    "text": "the data comes from, these are"
  },
  {
    "startMs": 188640,
    "text": "multimodal interleaved examples, right?"
  },
  {
    "startMs": 190800,
    "text": "So, the bottom uh diagram here gives you"
  },
  {
    "startMs": 193160,
    "text": "kind of one example of what that might"
  },
  {
    "startMs": 194680,
    "text": "look like. So, this is a task where"
  },
  {
    "startMs": 196239,
    "text": "you're asking the model to summarize a"
  },
  {
    "startMs": 198239,
    "text": "bedtime story and there's a text prompt"
  },
  {
    "startMs": 199920,
    "text": "in the beginning, but then there's this"
  },
  {
    "startMs": 201560,
    "text": "sequence of video and audio inputs that"
  },
  {
    "startMs": 204519,
    "text": "the model gets"
  },
  {
    "startMs": 205720,
    "text": "and then of course like what you expect"
  },
  {
    "startMs": 207280,
    "text": "the model to do here is produce um both"
  },
  {
    "startMs": 209720,
    "text": "the summary and also annotate timestamps"
  },
  {
    "startMs": 211720,
    "text": "for where interesting things happen and"
  },
  {
    "startMs": 213360,
    "text": "so forth. So, this example is teaching"
  },
  {
    "startMs": 215160,
    "text": "the model to translate its understanding"
  },
  {
    "startMs": 217560,
    "text": "of the audio and the video into text."
  },
  {
    "startMs": 219959,
    "text": "You might have other examples in"
  },
  {
    "startMs": 221040,
    "text": "pre-training that ask you to caption a"
  },
  {
    "startMs": 223120,
    "text": "video. So, you might have a video that"
  },
  {
    "startMs": 224480,
    "text": "has audio and the model is learning to"
  },
  {
    "startMs": 227239,
    "text": "bias towards both the video and the"
  },
  {
    "startMs": 228640,
    "text": "audio signal to caption this well. And"
  },
  {
    "startMs": 231080,
    "text": "of course like there's limitless like"
  },
  {
    "startMs": 233000,
    "text": "YouTube videos with captions that you"
  },
  {
    "startMs": 234040,
    "text": "can train these models on. Other models"
  },
  {
    "startMs": 235840,
    "text": "might actually try to generate audio"
  },
  {
    "startMs": 237280,
    "text": "from the video from the text, right? So,"
  },
  {
    "startMs": 239040,
    "text": "you can have ASR, TTS, or any"
  },
  {
    "startMs": 241240,
    "text": "combination of these plus all of these"
  },
  {
    "startMs": 242960,
    "text": "sort of agentic tasks all kind of"
  },
  {
    "startMs": 245680,
    "text": "learned under one unified token"
  },
  {
    "startMs": 247400,
    "text": "embedding space. So, this becomes a"
  },
  {
    "startMs": 249160,
    "text": "foundation for a lot of what we want to"
  },
  {
    "startMs": 250560,
    "text": "do in audio because we already have a"
  },
  {
    "startMs": 251920,
    "text": "model that understands audio, video,"
  },
  {
    "startMs": 254400,
    "text": "text, and how these things relate and"
  },
  {
    "startMs": 256760,
    "text": "transition from one to the next."
  },
  {
    "startMs": 259840,
    "text": "So, very quickly, like one of the"
  },
  {
    "startMs": 261400,
    "text": "applications that this enables that"
  },
  {
    "startMs": 262520,
    "text": "we've launched recently is live"
  },
  {
    "startMs": 264200,
    "text": "translation, right? And this is a kind"
  },
  {
    "startMs": 265600,
    "text": "of application that kind of only works"
  },
  {
    "startMs": 267560,
    "text": "when you have all of these capabilities"
  },
  {
    "startMs": 269000,
    "text": "working within the same model. You have"
  },
  {
    "startMs": 270840,
    "text": "basically state-of-the-art translation"
  },
  {
    "startMs": 272480,
    "text": "quality. Even though this model is"
  },
  {
    "startMs": 273720,
    "text": "translating basically as the user or"
  },
  {
    "startMs": 275800,
    "text": "speakers are speaking, you kind of ask,"
  },
  {
    "startMs": 277440,
    "text": "\"Hey, I I speak English. There's maybe"
  },
  {
    "startMs": 279560,
    "text": "friends who are talking in Spanish and"
  },
  {
    "startMs": 281240,
    "text": "Italian and Chinese.\" And it's"
  },
  {
    "startMs": 282320,
    "text": "translating all of them to your language"
  },
  {
    "startMs": 283600,
    "text": "as they talk. Um and then we're we're"
  },
  {
    "startMs": 285600,
    "text": "finding is the translation quality for"
  },
  {
    "startMs": 286960,
    "text": "this like streaming real-time"
  },
  {
    "startMs": 288200,
    "text": "translation is about as good as you"
  },
  {
    "startMs": 290280,
    "text": "would get with offline systems, right?"
  },
  {
    "startMs": 292320,
    "text": "Where you kind of know the full"
  },
  {
    "startMs": 293600,
    "text": "utterance"
  },
  {
    "startMs": 294760,
    "text": "um from the very beginning. So, that's"
  },
  {
    "startMs": 296200,
    "text": "something that has been classically very"
  },
  {
    "startMs": 298560,
    "text": "hard to do with these cascaded systems,"
  },
  {
    "startMs": 300880,
    "text": "but with LLMs, it actually just"
  },
  {
    "startMs": 303800,
    "text": "a lot of it comes out of the"
  },
  {
    "startMs": 304760,
    "text": "pre-training. So, of course, to do this"
  },
  {
    "startMs": 306680,
    "text": "task it needs to do multilingual"
  },
  {
    "startMs": 307800,
    "text": "switching because you could be"
  },
  {
    "startMs": 308600,
    "text": "translating across different languages."
  },
  {
    "startMs": 309919,
    "text": "You don't know what those languages are"
  },
  {
    "startMs": 310880,
    "text": "beforehand. It needs to preserve the"
  },
  {
    "startMs": 312400,
    "text": "speaker voice of the source speaker and"
  },
  {
    "startMs": 315840,
    "text": "be able to understand multiple speakers,"
  },
  {
    "startMs": 317919,
    "text": "be robust to noise, and of course, like"
  },
  {
    "startMs": 319919,
    "text": "do all this in real time, right? So,"
  },
  {
    "startMs": 321240,
    "text": "again, it would be very hard to try to"
  },
  {
    "startMs": 323200,
    "text": "engineer this, but then with the LLM and"
  },
  {
    "startMs": 325880,
    "text": "Gemini models, this almost becomes a"
  },
  {
    "startMs": 328280,
    "text": "prompting task."
  },
  {
    "startMs": 329720,
    "text": "And on that, like, I know, in this"
  },
  {
    "startMs": 331280,
    "text": "diagram we're saying, \"Hey, at the top"
  },
  {
    "startMs": 332720,
    "text": "with these models, if you prompt it to"
  },
  {
    "startMs": 334800,
    "text": "do the speaking like this streaming"
  },
  {
    "startMs": 336680,
    "text": "translation task,"
  },
  {
    "startMs": 338280,
    "text": "and you give it the audio, it will"
  },
  {
    "startMs": 339640,
    "text": "produce the streaming translation"
  },
  {
    "startMs": 340760,
    "text": "output, right? The same model, if you"
  },
  {
    "startMs": 342080,
    "text": "ask it to act like an agent and respond"
  },
  {
    "startMs": 344480,
    "text": "to maybe image and audio input, maybe"
  },
  {
    "startMs": 347480,
    "text": "asking questions about that image, it"
  },
  {
    "startMs": 348880,
    "text": "will give you an answer. And finally,"
  },
  {
    "startMs": 350400,
    "text": "like very well we'll show examples of"
  },
  {
    "startMs": 351560,
    "text": "this, you can also have it create this"
  },
  {
    "startMs": 353200,
    "text": "embodied, you know, virtual agent that"
  },
  {
    "startMs": 355320,
    "text": "has a face, that has, you know, things"
  },
  {
    "startMs": 357560,
    "text": "and tools that it can show you, and it"
  },
  {
    "startMs": 359680,
    "text": "will produce this sort of embodied agent"
  },
  {
    "startMs": 361760,
    "text": "experience. So, with that, I'm going to"
  },
  {
    "startMs": 363400,
    "text": "give it to Valeria to talk more about"
  },
  {
    "startMs": 364920,
    "text": "the North Star and some of the key demo"
  },
  {
    "startMs": 367720,
    "text": "products that we built."
  },
  {
    "startMs": 370200,
    "text": ">> Yeah, so to create this type of kind of"
  },
  {
    "startMs": 372320,
    "text": "universal, versatile, uh kind of"
  },
  {
    "startMs": 374760,
    "text": "model/product,"
  },
  {
    "startMs": 376320,
    "text": "there's like three vectors that we think"
  },
  {
    "startMs": 377760,
    "text": "about when we do research and product"
  },
  {
    "startMs": 379600,
    "text": "for these models. Um and they also come"
  },
  {
    "startMs": 381960,
    "text": "with some challenges, so I'll like walk"
  },
  {
    "startMs": 383560,
    "text": "you through some of them. So, at the"
  },
  {
    "startMs": 385240,
    "text": "core of a speech-to-speech model, the"
  },
  {
    "startMs": 387200,
    "text": "first thing that people usually think"
  },
  {
    "startMs": 388600,
    "text": "about is that it has very"
  },
  {
    "startMs": 389800,
    "text": "conversational, right? It's low latency,"
  },
  {
    "startMs": 391480,
    "text": "it's very conversational, very snappy,"
  },
  {
    "startMs": 393120,
    "text": "very natural. But, I think within our"
  },
  {
    "startMs": 395080,
    "text": "team, we really don't only want this"
  },
  {
    "startMs": 396919,
    "text": "model to sound nice. We also have two"
  },
  {
    "startMs": 399280,
    "text": "pillars at the top that we also really"
  },
  {
    "startMs": 400880,
    "text": "care about like pulling all together"
  },
  {
    "startMs": 402720,
    "text": "into one model, which is intelligence"
  },
  {
    "startMs": 404720,
    "text": "and it being multimodal. So, when we"
  },
  {
    "startMs": 406880,
    "text": "talk about intelligence, we talk about,"
  },
  {
    "startMs": 408360,
    "text": "you know, task completion, instruction"
  },
  {
    "startMs": 409840,
    "text": "following, reasoning, like capabilities"
  },
  {
    "startMs": 411919,
    "text": "that the model needs to have natively in"
  },
  {
    "startMs": 414000,
    "text": "order to complete tasks and to like do"
  },
  {
    "startMs": 416520,
    "text": "things uh that have high customer"
  },
  {
    "startMs": 418480,
    "text": "satisfaction, for example. And on the"
  },
  {
    "startMs": 420480,
    "text": "other side of the Venn diagram, we also"
  },
  {
    "startMs": 422200,
    "text": "have the idea that these models should"
  },
  {
    "startMs": 424040,
    "text": "be very multimodal, both in audio in and"
  },
  {
    "startMs": 426720,
    "text": "audio out, right? So, sometimes a user"
  },
  {
    "startMs": 428760,
    "text": "doesn't only want to input audio in and"
  },
  {
    "startMs": 431000,
    "text": "have that be the start of the"
  },
  {
    "startMs": 432040,
    "text": "conversation. We need video, your screen"
  },
  {
    "startMs": 434400,
    "text": "sharing, PDFs, whatever you would want"
  },
  {
    "startMs": 436480,
    "text": "the model to interpret and understand,"
  },
  {
    "startMs": 438240,
    "text": "we should be able to stream it in and"
  },
  {
    "startMs": 440160,
    "text": "also produce output out of it. So,"
  },
  {
    "startMs": 442000,
    "text": "that's kind of like the trifecta of"
  },
  {
    "startMs": 443840,
    "text": "which we think about speech-to-speech"
  },
  {
    "startMs": 445200,
    "text": "models. Um and I want to add a caveat"
  },
  {
    "startMs": 447440,
    "text": "about ITNN. I think actually the"
  },
  {
    "startMs": 449200,
    "text": "majority of our Gemini users are"
  },
  {
    "startMs": 450960,
    "text": "non-English speakers. Um so, we put a"
  },
  {
    "startMs": 452800,
    "text": "big focus on having and making sure that"
  },
  {
    "startMs": 454880,
    "text": "all these capabilities work within not"
  },
  {
    "startMs": 457440,
    "text": "only, you know, EN-US, but all the"
  },
  {
    "startMs": 459160,
    "text": "languages that our customers care about."
  },
  {
    "startMs": 461280,
    "text": "Um"
  },
  {
    "startMs": 461919,
    "text": "of course, this also our North Star also"
  },
  {
    "startMs": 464120,
    "text": "becomes one of the biggest challenges in"
  },
  {
    "startMs": 465760,
    "text": "our research because"
  },
  {
    "startMs": 467440,
    "text": "once you move one of the knobs, it's"
  },
  {
    "startMs": 469640,
    "text": "very easy for the other knobs to kind of"
  },
  {
    "startMs": 471840,
    "text": "like mess up, right? Like a very quick"
  },
  {
    "startMs": 473480,
    "text": "example, oh, how do we increase"
  },
  {
    "startMs": 475640,
    "text": "intelligence in the model? Well, you can"
  },
  {
    "startMs": 477800,
    "text": "turn thinking high or like the thinking"
  },
  {
    "startMs": 479600,
    "text": "is high as possible to have the model"
  },
  {
    "startMs": 481640,
    "text": "think a lot before calling a tool or"
  },
  {
    "startMs": 483520,
    "text": "answering a question, which in eval's it"
  },
  {
    "startMs": 485760,
    "text": "does show that it does improve the"
  },
  {
    "startMs": 487200,
    "text": "model's intelligence. But when what does"
  },
  {
    "startMs": 489400,
    "text": "that do to latency, right? And time to"
  },
  {
    "startMs": 491160,
    "text": "first audio and the naturalness of the"
  },
  {
    "startMs": 492800,
    "text": "conversation? So, within the Gemini"
  },
  {
    "startMs": 495040,
    "text": "team, we're really trying to push"
  },
  {
    "startMs": 496400,
    "text": "forward research initiatives that can"
  },
  {
    "startMs": 498240,
    "text": "kind of blend in the three of them"
  },
  {
    "startMs": 500560,
    "text": "without really sacrificing any of those"
  },
  {
    "startMs": 502880,
    "text": "by a lot. Um,"
  },
  {
    "startMs": 504400,
    "text": "but in the meantime, we're going to show"
  },
  {
    "startMs": 505480,
    "text": "you some of the demos that we think are"
  },
  {
    "startMs": 507280,
    "text": "hinting at how our speech-to-speech"
  },
  {
    "startMs": 509240,
    "text": "model can combine all these three into"
  },
  {
    "startMs": 512080,
    "text": "really cool application. So,"
  },
  {
    "startMs": 514440,
    "text": "the first one is you kind of saw this as"
  },
  {
    "startMs": 516159,
    "text": "a preview, but our live model, as you"
  },
  {
    "startMs": 518360,
    "text": "know, uh, powers streaming translation"
  },
  {
    "startMs": 520240,
    "text": "that supports 70+ languages. So, this is"
  },
  {
    "startMs": 522320,
    "text": "a little bit around the core of"
  },
  {
    "startMs": 524160,
    "text": "conversation quality and IT&N efforts"
  },
  {
    "startMs": 526080,
    "text": "that we have in the team. I'll play a"
  },
  {
    "startMs": 527920,
    "text": "quick video on how this works on Google"
  },
  {
    "startMs": 529720,
    "text": "Meets to help two people or maybe"
  },
  {
    "startMs": 531760,
    "text": "multiple people that are speaking"
  },
  {
    "startMs": 533040,
    "text": "different languages still have a live"
  },
  {
    "startMs": 534920,
    "text": "conversation."
  },
  {
    "startMs": 537000,
    "text": "Oh."
  },
  {
    "startMs": 538800,
    "text": "One sec. Okay."
  },
  {
    "startMs": 540560,
    "text": ">> Okay, let's turn on speech translation."
  },
  {
    "startMs": 544920,
    "text": ">> So, here the user can simply select the"
  },
  {
    "startMs": 546720,
    "text": "language that they want the translation"
  },
  {
    "startMs": 548160,
    "text": "to happen in and then the rest will be"
  },
  {
    "startMs": 550000,
    "text": "done in real time in multiple speakers"
  },
  {
    "startMs": 552320,
    "text": "kind of having a conversation back and"
  },
  {
    "startMs": 553760,
    "text": "forth. So, I'll show you a snippet of"
  },
  {
    "startMs": 555560,
    "text": "this video of what happens."
  },
  {
    "startMs": 560440,
    "text": ">> It's great to see you both. Cassie,"
  },
  {
    "startMs": 562680,
    "text": "how's the weather in Shanghai?"
  },
  {
    "startMs": 565320,
    "text": ">> It's nice to meet you. The weather here"
  },
  {
    "startMs": 567400,
    "text": "is really nice, sunny and bright. I"
  },
  {
    "startMs": 569880,
    "text": "spent all day Saturday in the park with"
  },
  {
    "startMs": 571720,
    "text": "my family."
  },
  {
    "startMs": 575360,
    "text": ">> That sounds wonderful."
  },
  {
    "startMs": 577320,
    "text": "Anna, you mentioned last week that you"
  },
  {
    "startMs": 579040,
    "text": "were celebrating your birthday. How was"
  },
  {
    "startMs": 580840,
    "text": "it?"
  },
  {
    "startMs": 581280,
    "text": ">> Oh, that was fantastic. I had dinner at"
  },
  {
    "startMs": 585040,
    "text": "my favorite restaurant with some"
  },
  {
    "startMs": 586720,
    "text": "friends. If you visit Sweden, you must"
  },
  {
    "startMs": 589680,
    "text": "try this restaurant. It was"
  },
  {
    "startMs": 592800,
    "text": ">> So, um a lot of things happening. Not"
  },
  {
    "startMs": 594720,
    "text": "only there's, you know, real-time"
  },
  {
    "startMs": 596040,
    "text": "translation. Whoops."
  },
  {
    "startMs": 598080,
    "text": "How do I"
  },
  {
    "startMs": 599640,
    "text": "back to Okay."
  },
  {
    "startMs": 600760,
    "text": "Um not only there's real-time"
  },
  {
    "startMs": 602200,
    "text": "translation that is happening, uh but we"
  },
  {
    "startMs": 604320,
    "text": "see that it's in a multi-speaker"
  },
  {
    "startMs": 605760,
    "text": "setting, you know, low latency, like"
  },
  {
    "startMs": 607520,
    "text": "right after the user starts speaking,"
  },
  {
    "startMs": 609080,
    "text": "the translation kicks off, so that it"
  },
  {
    "startMs": 610840,
    "text": "doesn't feel like it's really turn by"
  },
  {
    "startMs": 612720,
    "text": "turn and robotic and you're afraid to"
  },
  {
    "startMs": 614200,
    "text": "interrupt because, you know, once you"
  },
  {
    "startMs": 616120,
    "text": "start speaking, the model will start"
  },
  {
    "startMs": 617360,
    "text": "catching your translation directly. So,"
  },
  {
    "startMs": 619640,
    "text": "now I'll show some other applications"
  },
  {
    "startMs": 621400,
    "text": "where we see that we have one single"
  },
  {
    "startMs": 623400,
    "text": "speech-to-speech model for many"
  },
  {
    "startMs": 624800,
    "text": "conversational frontiers. So, two"
  },
  {
    "startMs": 626600,
    "text": "products that I want to highlight here"
  },
  {
    "startMs": 628160,
    "text": "are the same model that we power search"
  },
  {
    "startMs": 630440,
    "text": "live for everyday conversations in any"
  },
  {
    "startMs": 632800,
    "text": "language. It's the same model that we"
  },
  {
    "startMs": 634520,
    "text": "use to power developer experiences in"
  },
  {
    "startMs": 636840,
    "text": "our live API. As you'll see in the"
  },
  {
    "startMs": 638680,
    "text": "videos, both use cases are extremely"
  },
  {
    "startMs": 640760,
    "text": "different, but because we try and make"
  },
  {
    "startMs": 642880,
    "text": "this this model or this speech-to-speech"
  },
  {
    "startMs": 644839,
    "text": "model very promptable and universal, you"
  },
  {
    "startMs": 647120,
    "text": "can see how it adapts to the different"
  },
  {
    "startMs": 648680,
    "text": "use cases. So, let me just play the"
  },
  {
    "startMs": 650000,
    "text": "first video."
  },
  {
    "startMs": 672680,
    "text": "So, to highlight in this video, many"
  },
  {
    "startMs": 674120,
    "text": "things are happening. First, the"
  },
  {
    "startMs": 675320,
    "text": "multimodal, you know, it's real-time uh"
  },
  {
    "startMs": 677760,
    "text": "video and audio streaming in, so that"
  },
  {
    "startMs": 679280,
    "text": "you don't have to actually describe what"
  },
  {
    "startMs": 680839,
    "text": "the sofa looks like. But then the"
  },
  {
    "startMs": 682360,
    "text": "multimodal out, you see that there was a"
  },
  {
    "startMs": 684200,
    "text": "tool called made to pull up the right"
  },
  {
    "startMs": 685800,
    "text": "search cards if the user wanted to know"
  },
  {
    "startMs": 687520,
    "text": "more information. And then finally, the"
  },
  {
    "startMs": 689640,
    "text": "ITN component, you know, we're"
  },
  {
    "startMs": 691080,
    "text": "localizing the speaker's Spanish to"
  },
  {
    "startMs": 692800,
    "text": "Spain Spanish. And also, when we do the"
  },
  {
    "startMs": 696120,
    "text": "kind of responding in Spanish, we're"
  },
  {
    "startMs": 697760,
    "text": "also smart enough to know that some"
  },
  {
    "startMs": 699560,
    "text": "words are better to be left in English,"
  },
  {
    "startMs": 701360,
    "text": "like the mid-century type of word. It"
  },
  {
    "startMs": 703440,
    "text": "didn't translate it directly to Spanish"
  },
  {
    "startMs": 705360,
    "text": "cuz it knew that the term was commonly"
  },
  {
    "startMs": 706920,
    "text": "known in English. Sorry, in Spanish. So,"
  },
  {
    "startMs": 709720,
    "text": "this is one example of how users are"
  },
  {
    "startMs": 711600,
    "text": "using search live for like info-seeking"
  },
  {
    "startMs": 713600,
    "text": "questions. Now, let me show a very"
  },
  {
    "startMs": 715280,
    "text": "different demo on how we do this for"
  },
  {
    "startMs": 717360,
    "text": "conversational voice agents."
  },
  {
    "startMs": 722400,
    "text": "Oh."
  },
  {
    "startMs": 724680,
    "text": "Sorry."
  },
  {
    "startMs": 734320,
    "text": ">> Hi, I've blown a tire and had to pull"
  },
  {
    "startMs": 737120,
    "text": "over to the side of the road. Can you"
  },
  {
    "startMs": 738760,
    "text": "help?"
  },
  {
    "startMs": 740640,
    "text": ">> Hello, you've reached Auto Transform"
  },
  {
    "startMs": 743000,
    "text": "Insurance Roadside Assistance. My name"
  },
  {
    "startMs": 745280,
    "text": "is Ace. I understand this is stressful,"
  },
  {
    "startMs": 747440,
    "text": "but we'll take care of everything. To"
  },
  {
    "startMs": 749120,
    "text": "get started, could I please have your"
  },
  {
    "startMs": 750600,
    "text": "name and your policy number?"
  },
  {
    "startMs": 752120,
    "text": ">> Uh my name is Richard Jenkins. I I don't"
  },
  {
    "startMs": 754960,
    "text": "actually have my policy number to hand."
  },
  {
    "startMs": 759000,
    "text": ">> No worries, Richard. I can look you up"
  },
  {
    "startMs": 761080,
    "text": "with your registration plate and"
  },
  {
    "startMs": 762720,
    "text": "postcode instead. Could you provide"
  },
  {
    "startMs": 764640,
    "text": "those for me, please?"
  },
  {
    "startMs": 766440,
    "text": ">> Whoa, that was a bit close. Um yeah, my"
  },
  {
    "startMs": 769960,
    "text": "registration plate is BD21"
  },
  {
    "startMs": 774320,
    "text": "XYA"
  },
  {
    "startMs": 776240,
    "text": "and uh my postcode is SN48ZX."
  },
  {
    "startMs": 783520,
    "text": ">> Policy details BD21"
  },
  {
    "startMs": 786520,
    "text": "XYA. Thank you. For your safety, please"
  },
  {
    "startMs": 789920,
    "text": "stay clear of the road. I've found your"
  },
  {
    "startMs": 791880,
    "text": "details and I see you're in a blue Mini"
  },
  {
    "startMs": 794160,
    "text": "Cooper F-Series. Is that the vehicle"
  },
  {
    "startMs": 796480,
    "text": "you're in?"
  },
  {
    "startMs": 796880,
    "text": ">> Yeah, yeah, that that's the vehicle."
  },
  {
    "startMs": 800440,
    "text": ">> Got it."
  },
  {
    "startMs": 801600,
    "text": ">> Okay, so I'll pause it here, but as you"
  },
  {
    "startMs": 803000,
    "text": "can see, very different things are"
  },
  {
    "startMs": 804400,
    "text": "happening under the hood with the same"
  },
  {
    "startMs": 805920,
    "text": "model. You know, it's more about"
  },
  {
    "startMs": 807560,
    "text": "alphanumeric accuracy for complex kind"
  },
  {
    "startMs": 810120,
    "text": "of like postcode numbers or addresses."
  },
  {
    "startMs": 812920,
    "text": "There's like this feature that we have"
  },
  {
    "startMs": 814520,
    "text": "called proactive audio, which is the"
  },
  {
    "startMs": 816000,
    "text": "idea that the LLM knows when or when not"
  },
  {
    "startMs": 818600,
    "text": "to respond to an external input. So, for"
  },
  {
    "startMs": 820480,
    "text": "example, if you're in a conversation and"
  },
  {
    "startMs": 822440,
    "text": "someone else is talking or there's"
  },
  {
    "startMs": 823760,
    "text": "background noise happening in the back,"
  },
  {
    "startMs": 825920,
    "text": "the model knows to not stop or cut"
  },
  {
    "startMs": 827960,
    "text": "itself short because there's external"
  },
  {
    "startMs": 829600,
    "text": "noise happening. Because we realized"
  },
  {
    "startMs": 831520,
    "text": "that the majority of these conversations"
  },
  {
    "startMs": 833000,
    "text": "are not happening in an office room with"
  },
  {
    "startMs": 835320,
    "text": "like no noise, you know, in a sealed"
  },
  {
    "startMs": 836840,
    "text": "environment. They're happening on the"
  },
  {
    "startMs": 838440,
    "text": "go, on the train, while you're on a"
  },
  {
    "startMs": 839960,
    "text": "walk. Um and that's the type of"
  },
  {
    "startMs": 841440,
    "text": "experiences that we want to facilitate"
  },
  {
    "startMs": 843160,
    "text": "with the speech-to-speech model."
  },
  {
    "startMs": 845440,
    "text": "Um lastly, um another thing we're really"
  },
  {
    "startMs": 847560,
    "text": "excited about is not only voice out, but"
  },
  {
    "startMs": 849960,
    "text": "also multimodal out. We believe that,"
  },
  {
    "startMs": 852000,
    "text": "you know, the true AGI conversational"
  },
  {
    "startMs": 854000,
    "text": "frontier will also require visual"
  },
  {
    "startMs": 856080,
    "text": "presence. So, we were excited to launch"
  },
  {
    "startMs": 858240,
    "text": "our first kind of pilot demo with City"
  },
  {
    "startMs": 860720,
    "text": "um in Cloud Next, which supports"
  },
  {
    "startMs": 862520,
    "text": "customized real-time avatars."
  },
  {
    "startMs": 864680,
    "text": "Um and you can kind of like"
  },
  {
    "startMs": 866600,
    "text": "personalize anything from a"
  },
  {
    "startMs": 868080,
    "text": "hyperrealistic human to a cartoon"
  },
  {
    "startMs": 870360,
    "text": "appearance and everything in the middle."
  },
  {
    "startMs": 872280,
    "text": "And powered by the same speech-to-speech"
  },
  {
    "startMs": 873800,
    "text": "model we've been showing, it allows to"
  },
  {
    "startMs": 875640,
    "text": "have, you know, low-latency,"
  },
  {
    "startMs": 877120,
    "text": "multilingual lip-syncing, and a really"
  },
  {
    "startMs": 878800,
    "text": "kind of like uh ongoing fluid"
  },
  {
    "startMs": 880720,
    "text": "conversation that has visual presence."
  },
  {
    "startMs": 882880,
    "text": "So, let me show you a demo that also"
  },
  {
    "startMs": 884400,
    "text": "brings our Venn diagram together and"
  },
  {
    "startMs": 886120,
    "text": "what we're excited about."
  },
  {
    "startMs": 888760,
    "text": ">> All this is reminding me of my"
  },
  {
    "startMs": 889960,
    "text": "daughter's college fund. How are we"
  },
  {
    "startMs": 891320,
    "text": "tracking on that?"
  },
  {
    "startMs": 895000,
    "text": ">> You're tracking well, Jackson. I've also"
  },
  {
    "startMs": 897320,
    "text": "identified a new opportunity that may"
  },
  {
    "startMs": 899280,
    "text": "get you there even sooner."
  },
  {
    "startMs": 901120,
    "text": "Oh, and I can see Lisa just joined you."
  },
  {
    "startMs": 903640,
    "text": "Hi, Lisa. She must be so excited about"
  },
  {
    "startMs": 906160,
    "text": "her college acceptance. Congratulations."
  },
  {
    "startMs": 908600,
    "text": "It's wonderful to see your savings goals"
  },
  {
    "startMs": 910440,
    "text": "coming to life."
  },
  {
    "startMs": 911880,
    "text": ">> Excelente. Las cosas han estado tan"
  },
  {
    "startMs": 914040,
    "text": "inestables últimamente."
  },
  {
    "startMs": 918360,
    "text": ">> Sí. Los servicios de tecnología y"
  },
  {
    "startMs": 920800,
    "text": "comunicaciones han mostrado un desempeño"
  },
  {
    "startMs": 923080,
    "text": "sólido en lo que va del año."
  },
  {
    "startMs": 925280,
    "text": ">> That's good news."
  },
  {
    "startMs": 926560,
    "text": ">> I always joke that the user's audio in"
  },
  {
    "startMs": 928520,
    "text": "Spanish is worse than the audio model"
  },
  {
    "startMs": 930360,
    "text": "speaking back, but um this is kind of"
  },
  {
    "startMs": 932320,
    "text": "just to show how like our Venn diagram"
  },
  {
    "startMs": 934160,
    "text": "of combining, you know, multimodality in"
  },
  {
    "startMs": 936560,
    "text": "and out, you know, tool calling to like"
  },
  {
    "startMs": 938520,
    "text": "pull up the relevant examples from the"
  },
  {
    "startMs": 940120,
    "text": "user, and also conversational fluidity"
  },
  {
    "startMs": 942160,
    "text": "with ITNN are starting slowly to come"
  },
  {
    "startMs": 944520,
    "text": "together in these types of demos that"
  },
  {
    "startMs": 946080,
    "text": "we're excited to keep pushing the"
  },
  {
    "startMs": 947640,
    "text": "frontier of."
  },
  {
    "startMs": 949200,
    "text": "Um so with this parting thought, I guess"
  },
  {
    "startMs": 951280,
    "text": "last"
  },
  {
    "startMs": 952520,
    "text": "kind of thought that we have for you is"
  },
  {
    "startMs": 953720,
    "text": "that we believe that AGI will not be"
  },
  {
    "startMs": 955640,
    "text": "typed, that it will be spoken. Um and"
  },
  {
    "startMs": 958000,
    "text": "for it to be spoken, there's a lot of"
  },
  {
    "startMs": 959560,
    "text": "things that need to work together in a"
  },
  {
    "startMs": 961320,
    "text": "single promptable, versatile model that"
  },
  {
    "startMs": 964080,
    "text": "allows a user to switch between all the"
  },
  {
    "startMs": 966320,
    "text": "sorts of conversation modes that we're"
  },
  {
    "startMs": 967960,
    "text": "looking at, right? From translation to"
  },
  {
    "startMs": 969680,
    "text": "taking action to brainstorming to"
  },
  {
    "startMs": 971600,
    "text": "rambling, and we truly believe in the"
  },
  {
    "startMs": 973360,
    "text": "power of these speech-to-speech models"
  },
  {
    "startMs": 974840,
    "text": "to achieve that like seamless switching."
  },
  {
    "startMs": 977480,
    "text": "Um so we're excited to push the frontier"
  },
  {
    "startMs": 979000,
    "text": "on that. So if you're excited or want to"
  },
  {
    "startMs": 981240,
    "text": "learn more, please come talk to us, and"
  },
  {
    "startMs": 982960,
    "text": "thank you so much for coming."
  },
  {
    "startMs": 998231,
    "text": ">> [music]"
  }
]