[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 13599,
    "text": "So, my name is Charlie uh and I work on"
  },
  {
    "startMs": 16000,
    "text": "the developer experience team at OpenAI."
  },
  {
    "startMs": 18960,
    "text": "And part of my job is talking to"
  },
  {
    "startMs": 21359,
    "text": "developers to understand and see, you"
  },
  {
    "startMs": 24720,
    "text": "know, what and how they're building with"
  },
  {
    "startMs": 26640,
    "text": "our models. um whether that's text,"
  },
  {
    "startMs": 28880,
    "text": "image or audio. And lately I've been"
  },
  {
    "startMs": 32399,
    "text": "thinking about a misconception that I"
  },
  {
    "startMs": 35440,
    "text": "have seen or maybe it's just a"
  },
  {
    "startMs": 37280,
    "text": "misunderstanding"
  },
  {
    "startMs": 39040,
    "text": "and it's the idea that voice agents"
  },
  {
    "startMs": 43600,
    "text": "have to talk back."
  },
  {
    "startMs": 48480,
    "text": ">> And to some of you that might sound, you"
  },
  {
    "startMs": 50640,
    "text": "know, absurd. It's a voice agent. What"
  },
  {
    "startMs": 52239,
    "text": "do you mean it's not supposed to talk?"
  },
  {
    "startMs": 54079,
    "text": "Uh, but I think if there's one thing"
  },
  {
    "startMs": 56800,
    "text": "that you take away from this"
  },
  {
    "startMs": 59359,
    "text": "presentation, I would like it to be the"
  },
  {
    "startMs": 62079,
    "text": "idea that speech is not the only way"
  },
  {
    "startMs": 65760,
    "text": "that a voice model has to respond."
  },
  {
    "startMs": 72240,
    "text": "And I think models are getting"
  },
  {
    "startMs": 73920,
    "text": "intelligent enough and capable enough"
  },
  {
    "startMs": 76240,
    "text": "that they're starting to open up some uh"
  },
  {
    "startMs": 80400,
    "text": "new modes of design. I mean there's"
  },
  {
    "startMs": 82240,
    "text": "actually three kind of modes that that"
  },
  {
    "startMs": 84159,
    "text": "you know I kind of see emerging these"
  },
  {
    "startMs": 85840,
    "text": "days right uh speech to speech speech to"
  },
  {
    "startMs": 89680,
    "text": "action and event to speech and there's a"
  },
  {
    "startMs": 92880,
    "text": "couple of things I think worth pointing"
  },
  {
    "startMs": 94320,
    "text": "out about um these three categories. The"
  },
  {
    "startMs": 97680,
    "text": "first is that they're not new, right? I"
  },
  {
    "startMs": 101200,
    "text": "think as we've seen from from previous"
  },
  {
    "startMs": 103200,
    "text": "talks, even just today, um there's a"
  },
  {
    "startMs": 105360,
    "text": "long history of building these types of"
  },
  {
    "startMs": 108079,
    "text": "systems in and around voice. Um if you"
  },
  {
    "startMs": 110880,
    "text": "squint, you could make the argument that"
  },
  {
    "startMs": 112880,
    "text": "the movie phone hotline where you called"
  },
  {
    "startMs": 115040,
    "text": "in to get showtimes was an example of a"
  },
  {
    "startMs": 117680,
    "text": "speech-to-pech system. Uh and I think"
  },
  {
    "startMs": 119840,
    "text": "you could pretty reasonably make the"
  },
  {
    "startMs": 121119,
    "text": "argument that you know GPS navigation in"
  },
  {
    "startMs": 123360,
    "text": "your car which has existed since I was a"
  },
  {
    "startMs": 125920,
    "text": "kid is an example of an event to speech"
  },
  {
    "startMs": 128560,
    "text": "system. So, it's not that they are brand"
  },
  {
    "startMs": 131280,
    "text": "new, but I think it is that we are able"
  },
  {
    "startMs": 132720,
    "text": "to do some some much more interesting"
  },
  {
    "startMs": 134720,
    "text": "things with them uh now that uh we're in"
  },
  {
    "startMs": 137760,
    "text": "this era, right? And I think, you know,"
  },
  {
    "startMs": 139040,
    "text": "the other thing I would mention here is"
  },
  {
    "startMs": 140319,
    "text": "that um they're they're remixable,"
  },
  {
    "startMs": 144319,
    "text": "right? They're not meant to be mutually"
  },
  {
    "startMs": 145520,
    "text": "exclusive. Um and I think as we'll see"
  },
  {
    "startMs": 147440,
    "text": "in a little bit, the best products uh"
  },
  {
    "startMs": 149200,
    "text": "exist in a way that combines all of"
  },
  {
    "startMs": 151280,
    "text": "these modes. So, speechto um everybody"
  },
  {
    "startMs": 154959,
    "text": "knows it. Hopefully, everybody loves it."
  },
  {
    "startMs": 156480,
    "text": "the user talks uh and then the model"
  },
  {
    "startMs": 158720,
    "text": "talks back and I think there are a few"
  },
  {
    "startMs": 161120,
    "text": "examples that that I can give um for"
  },
  {
    "startMs": 163760,
    "text": "this type of use case right you've got"
  },
  {
    "startMs": 166080,
    "text": "things like live practice and coaching"
  },
  {
    "startMs": 168959,
    "text": "especially around language learning"
  },
  {
    "startMs": 170480,
    "text": "right I think the ability to um hear a"
  },
  {
    "startMs": 173040,
    "text": "lot of emphasis or emotion and give"
  },
  {
    "startMs": 175040,
    "text": "people that feedback is really powerful"
  },
  {
    "startMs": 177280,
    "text": "um I think you can have you know what I"
  },
  {
    "startMs": 178879,
    "text": "am sort of cheekily calling concierge"
  },
  {
    "startMs": 180879,
    "text": "experiences which I think is just"
  },
  {
    "startMs": 182480,
    "text": "another way to say customer support"
  },
  {
    "startMs": 184959,
    "text": "plus+ Um the first voice tutorial that"
  },
  {
    "startMs": 188319,
    "text": "most people you know try to build when"
  },
  {
    "startMs": 189680,
    "text": "they have access to this technology is"
  },
  {
    "startMs": 191280,
    "text": "some sort of customer support chatbot"
  },
  {
    "startMs": 193120,
    "text": "for very good reasons. But I think with"
  },
  {
    "startMs": 195760,
    "text": "you know when we add a richness and a"
  },
  {
    "startMs": 198159,
    "text": "depth to the voice models as has been"
  },
  {
    "startMs": 199920,
    "text": "happening in recent months in recent"
  },
  {
    "startMs": 201360,
    "text": "years um we can build something that is"
  },
  {
    "startMs": 203280,
    "text": "like much more enjoyable to use than"
  },
  {
    "startMs": 205360,
    "text": "like talking your way through a phone"
  },
  {
    "startMs": 207440,
    "text": "tree. Um, and so, you know, I have the"
  },
  {
    "startMs": 210480,
    "text": "hope that like soon, if not like, you"
  },
  {
    "startMs": 212159,
    "text": "know, now, we're capable of building"
  },
  {
    "startMs": 214319,
    "text": "support experiences with agents that"
  },
  {
    "startMs": 216239,
    "text": "actually feel much more enjoyable to"
  },
  {
    "startMs": 218080,
    "text": "talk to than like arguably like the"
  },
  {
    "startMs": 220159,
    "text": "median human support agent."
  },
  {
    "startMs": 222959,
    "text": "Um, and as we just saw if you hear the"
  },
  {
    "startMs": 225519,
    "text": "last talk, uh, live translation, right?"
  },
  {
    "startMs": 227360,
    "text": "The models have gotten good enough and"
  },
  {
    "startMs": 228959,
    "text": "fast enough that we can just dynamically"
  },
  {
    "startMs": 231280,
    "text": "translate content on the fly, um, with"
  },
  {
    "startMs": 233360,
    "text": "like little to no latency. It wouldn't"
  },
  {
    "startMs": 235440,
    "text": "shock me if at next year's keynote um"
  },
  {
    "startMs": 237519,
    "text": "you know they live streamed it from the"
  },
  {
    "startMs": 238879,
    "text": "main stage but also uh dubbed it in real"
  },
  {
    "startMs": 241120,
    "text": "time across multiple languages."
  },
  {
    "startMs": 245200,
    "text": "The second category is speech to action."
  },
  {
    "startMs": 247519,
    "text": "Uh these are talks and the model uses"
  },
  {
    "startMs": 250239,
    "text": "tools and I think this is one of the"
  },
  {
    "startMs": 252400,
    "text": "most underexplored areas that we have."
  },
  {
    "startMs": 255280,
    "text": "Um I actually almost titled this talk uh"
  },
  {
    "startMs": 257440,
    "text": "voice is the next capability overhang"
  },
  {
    "startMs": 259519,
    "text": "because I think there is just a vast"
  },
  {
    "startMs": 261600,
    "text": "vast amount of stuff um that we could be"
  },
  {
    "startMs": 264000,
    "text": "doing in this category that we are not"
  },
  {
    "startMs": 265759,
    "text": "currently doing. Uh for example um"
  },
  {
    "startMs": 269600,
    "text": "there's a broad spectrum I don't have"
  },
  {
    "startMs": 271120,
    "text": "you know there's way too many examples"
  },
  {
    "startMs": 272320,
    "text": "even fit on this slide but three"
  },
  {
    "startMs": 274400,
    "text": "categories that that I find particularly"
  },
  {
    "startMs": 276080,
    "text": "interesting. Uh first is form filling"
  },
  {
    "startMs": 278160,
    "text": "right um so much of the internet is just"
  },
  {
    "startMs": 281040,
    "text": "filling out forms. Um, and there is, you"
  },
  {
    "startMs": 284320,
    "text": "know, today no reason why you shouldn't"
  },
  {
    "startMs": 285919,
    "text": "be able to just talk. You know, I would"
  },
  {
    "startMs": 287199,
    "text": "love it if instead of spending an hour"
  },
  {
    "startMs": 288880,
    "text": "filling out a government document, I"
  },
  {
    "startMs": 290720,
    "text": "could just talk for five minutes and it"
  },
  {
    "startMs": 292720,
    "text": "would get 90% of it for me and I would"
  },
  {
    "startMs": 294880,
    "text": "do a quick check, you know, just to make"
  },
  {
    "startMs": 296320,
    "text": "sure that everything looked good, right?"
  },
  {
    "startMs": 297840,
    "text": "That is a vastly superior experience"
  },
  {
    "startMs": 299520,
    "text": "than like having to type in every single"
  },
  {
    "startMs": 301600,
    "text": "name and address that I've lived in the"
  },
  {
    "startMs": 303280,
    "text": "last 5 years and, you know, all of my"
  },
  {
    "startMs": 304960,
    "text": "previous identities. Um and so I think I"
  },
  {
    "startMs": 308000,
    "text": "think that one is though it may seem"
  },
  {
    "startMs": 309360,
    "text": "boring you know affects a a significant"
  },
  {
    "startMs": 311440,
    "text": "GDP of the internet right the next"
  },
  {
    "startMs": 313840,
    "text": "category is creative tools uh where I am"
  },
  {
    "startMs": 316400,
    "text": "privileged enough that I can speak the"
  },
  {
    "startMs": 318080,
    "text": "language of software and so I can tell"
  },
  {
    "startMs": 320080,
    "text": "codeex you know here's exactly what I"
  },
  {
    "startMs": 322560,
    "text": "want you to build and I can articulate"
  },
  {
    "startMs": 324000,
    "text": "it in a way that um I get much more"
  },
  {
    "startMs": 326080,
    "text": "leverage than sort of just like cludily"
  },
  {
    "startMs": 328080,
    "text": "trying to iterate one thing at a time"
  },
  {
    "startMs": 329919,
    "text": "but I can't do that when it comes to you"
  },
  {
    "startMs": 332080,
    "text": "know using making music or painting um"
  },
  {
    "startMs": 334720,
    "text": "and so if I don't have the ability to"
  },
  {
    "startMs": 337199,
    "text": "articulate um the exact aesthetic that"
  },
  {
    "startMs": 339520,
    "text": "I'm looking for. Um and if I don't know"
  },
  {
    "startMs": 341440,
    "text": "how to use Photoshop or Ableton, I'm"
  },
  {
    "startMs": 343680,
    "text": "left in this state where, you know, my"
  },
  {
    "startMs": 345199,
    "text": "my taste exceeds my capability. Um and"
  },
  {
    "startMs": 347680,
    "text": "so I'm really looking forward to"
  },
  {
    "startMs": 348880,
    "text": "integrating voice into creative tools so"
  },
  {
    "startMs": 350800,
    "text": "that I can just sort of cludgy go along"
  },
  {
    "startMs": 352960,
    "text": "and, you know, vibe create, vibe"
  },
  {
    "startMs": 355039,
    "text": "compose, vibe paint, um and make"
  },
  {
    "startMs": 356960,
    "text": "something that that's really beautiful"
  },
  {
    "startMs": 358400,
    "text": "to me. And I think the generalizable um"
  },
  {
    "startMs": 362560,
    "text": "category here, right, then just starts"
  },
  {
    "startMs": 364000,
    "text": "to become computer use. And we've"
  },
  {
    "startMs": 365600,
    "text": "already seen some companies start to do"
  },
  {
    "startMs": 367120,
    "text": "this. Um, you know, it raises the"
  },
  {
    "startMs": 368880,
    "text": "question of like, look, if the models"
  },
  {
    "startMs": 370479,
    "text": "are just getting good enough to do"
  },
  {
    "startMs": 371919,
    "text": "everything on a computer that a human"
  },
  {
    "startMs": 373680,
    "text": "can do, like why am I talking to an app?"
  },
  {
    "startMs": 377199,
    "text": "Why am I talking to a terminal? Why am I"
  },
  {
    "startMs": 378880,
    "text": "not just talking to the entire computer?"
  },
  {
    "startMs": 382000,
    "text": "Uh, and so I think that's sort of a"
  },
  {
    "startMs": 383520,
    "text": "really interesting uh way to start"
  },
  {
    "startMs": 385280,
    "text": "exploring. But if you're a developer"
  },
  {
    "startMs": 387199,
    "text": "today, right? Whoops. If you're a"
  },
  {
    "startMs": 388639,
    "text": "developer today, um, what does that mean"
  },
  {
    "startMs": 391039,
    "text": "for building your own software, right?"
  },
  {
    "startMs": 392240,
    "text": "And I think it is like much easier than"
  },
  {
    "startMs": 394319,
    "text": "you think to start adding audio as an"
  },
  {
    "startMs": 396479,
    "text": "intelligence layer to the intelligence"
  },
  {
    "startMs": 398479,
    "text": "layer to the apps that you already have."
  },
  {
    "startMs": 400319,
    "text": "Um, if you're building a modern web"
  },
  {
    "startMs": 401919,
    "text": "application, you already expose so much"
  },
  {
    "startMs": 403600,
    "text": "of it as like action as nouns and verbs,"
  },
  {
    "startMs": 406160,
    "text": "right? And if you think about all the"
  },
  {
    "startMs": 407600,
    "text": "verbs that you have, you have uh API"
  },
  {
    "startMs": 409919,
    "text": "endpoints, you have, you know, React"
  },
  {
    "startMs": 411680,
    "text": "hooks. Each of those things can like"
  },
  {
    "startMs": 413440,
    "text": "pretty relatively easily be converted"
  },
  {
    "startMs": 415199,
    "text": "into a tool that you expose to a model"
  },
  {
    "startMs": 417120,
    "text": "and then you can give the user the"
  },
  {
    "startMs": 418560,
    "text": "ability to just drive your existing"
  },
  {
    "startMs": 420080,
    "text": "software um with their voice, right? And"
  },
  {
    "startMs": 422319,
    "text": "yes, you still need guardrails, you"
  },
  {
    "startMs": 423599,
    "text": "still need safety checks. Like many of"
  },
  {
    "startMs": 425199,
    "text": "the talks today are going to talk about"
  },
  {
    "startMs": 426560,
    "text": "securing and you know productizing this,"
  },
  {
    "startMs": 428560,
    "text": "but um for this I just want you to think"
  },
  {
    "startMs": 430160,
    "text": "about you know what would it mean to"
  },
  {
    "startMs": 431520,
    "text": "take your existing software and just"
  },
  {
    "startMs": 433280,
    "text": "talk to it."
  },
  {
    "startMs": 436560,
    "text": "Um and to go back to that misconception,"
  },
  {
    "startMs": 438240,
    "text": "right? I think there are a lot of um you"
  },
  {
    "startMs": 440080,
    "text": "know like if you're talking to the"
  },
  {
    "startMs": 441039,
    "text": "software maybe it can talk back but"
  },
  {
    "startMs": 442800,
    "text": "we've been developing other ways of"
  },
  {
    "startMs": 444639,
    "text": "communicating with the user for decades"
  },
  {
    "startMs": 446479,
    "text": "right we know these things we know we"
  },
  {
    "startMs": 448000,
    "text": "can show notifications and popups we can"
  },
  {
    "startMs": 450400,
    "text": "change state like the color of a button"
  },
  {
    "startMs": 452080,
    "text": "or a drop shadow we can highlight text"
  },
  {
    "startMs": 454160,
    "text": "um if you've used computer use in the"
  },
  {
    "startMs": 455520,
    "text": "codeex app you know there's this amazing"
  },
  {
    "startMs": 457440,
    "text": "like little ghost cursor animation that"
  },
  {
    "startMs": 459360,
    "text": "goes around and clicks things for you so"
  },
  {
    "startMs": 461120,
    "text": "we don't have to use words to actually"
  },
  {
    "startMs": 462880,
    "text": "tell the user what is happening on"
  },
  {
    "startMs": 464319,
    "text": "screen with their software"
  },
  {
    "startMs": 468560,
    "text": "Uh and then the last bucket here is"
  },
  {
    "startMs": 469919,
    "text": "event to speech, right? Um the model"
  },
  {
    "startMs": 471360,
    "text": "receives an event and talks to the user."
  },
  {
    "startMs": 473440,
    "text": "Um and sort of the counterpoint from"
  },
  {
    "startMs": 475280,
    "text": "speech to action. I think this one is"
  },
  {
    "startMs": 476800,
    "text": "still very very exploratory, right? Um"
  },
  {
    "startMs": 479199,
    "text": "you know, if you saw Quinn's talk, I"
  },
  {
    "startMs": 480479,
    "text": "think there's a lot of uh space here of"
  },
  {
    "startMs": 482639,
    "text": "like things we can do. Um and to me, we"
  },
  {
    "startMs": 484879,
    "text": "haven't quite seen what AI native really"
  },
  {
    "startMs": 487360,
    "text": "looks like in this vein yet. But um of"
  },
  {
    "startMs": 489919,
    "text": "the things that I've seen, I think"
  },
  {
    "startMs": 491039,
    "text": "there's a couple of through lines that I"
  },
  {
    "startMs": 492479,
    "text": "tend to notice, right? The first is"
  },
  {
    "startMs": 495120,
    "text": "hands-free or screen-free experiences."
  },
  {
    "startMs": 497280,
    "text": "There might be times where uh I need to"
  },
  {
    "startMs": 499280,
    "text": "interact with software, interact with"
  },
  {
    "startMs": 500800,
    "text": "objects and I can't use my hands or more"
  },
  {
    "startMs": 502639,
    "text": "importantly my attention is diverted"
  },
  {
    "startMs": 504479,
    "text": "elsewhere. Um that might be something"
  },
  {
    "startMs": 506639,
    "text": "you know like uh a recipe app. Maybe I'm"
  },
  {
    "startMs": 509520,
    "text": "cooking and I need to just say like"
  },
  {
    "startMs": 510800,
    "text": "what's going on and and have something"
  },
  {
    "startMs": 512159,
    "text": "else have something happen. Um the other"
  },
  {
    "startMs": 515039,
    "text": "category is proactive outreach, right?"
  },
  {
    "startMs": 516800,
    "text": "Where you the model needs to be able to"
  },
  {
    "startMs": 518560,
    "text": "tell you something or get your attention"
  },
  {
    "startMs": 519919,
    "text": "in a way um that you might not be"
  },
  {
    "startMs": 521680,
    "text": "looking at, right? I think every"
  },
  {
    "startMs": 522880,
    "text": "developer um has uh an endless amount of"
  },
  {
    "startMs": 526240,
    "text": "notifications and events happening in"
  },
  {
    "startMs": 527920,
    "text": "their software. But um no developer in"
  },
  {
    "startMs": 530000,
    "text": "their right mind would sort of say I"
  },
  {
    "startMs": 531279,
    "text": "should show all of these logs. Nor you"
  },
  {
    "startMs": 532959,
    "text": "know would they say I should speak all"
  },
  {
    "startMs": 534320,
    "text": "of these logs. But we can start to"
  },
  {
    "startMs": 536000,
    "text": "conceive of voice as this like upper"
  },
  {
    "startMs": 538240,
    "text": "level in this escalatory path of like"
  },
  {
    "startMs": 540240,
    "text": "okay maybe you animate something and"
  },
  {
    "startMs": 541680,
    "text": "then maybe you pop something up and then"
  },
  {
    "startMs": 542959,
    "text": "if that doesn't work maybe you talk to"
  },
  {
    "startMs": 544320,
    "text": "the user to get their attention."
  },
  {
    "startMs": 547839,
    "text": "And underlying both of these categories"
  },
  {
    "startMs": 549279,
    "text": "and I think this this you know this"
  },
  {
    "startMs": 550560,
    "text": "whole presentation is this broader theme"
  },
  {
    "startMs": 552000,
    "text": "of accessibility. Um on a personal"
  },
  {
    "startMs": 554480,
    "text": "personal note, I know like multiple"
  },
  {
    "startMs": 556480,
    "text": "developers who um over the course of"
  },
  {
    "startMs": 558800,
    "text": "their careers lost mobility in their"
  },
  {
    "startMs": 561120,
    "text": "hands, lost dexterity in their fingers"
  },
  {
    "startMs": 563200,
    "text": "and for many of them, they thought their"
  },
  {
    "startMs": 565040,
    "text": "career as a programmer was more or less"
  },
  {
    "startMs": 566959,
    "text": "over. Um and then came large language"
  },
  {
    "startMs": 569680,
    "text": "models, right? Then came coding agents"
  },
  {
    "startMs": 571120,
    "text": "and voice agents and now they generate"
  },
  {
    "startMs": 573600,
    "text": "orders of magnitude more code than they"
  },
  {
    "startMs": 575600,
    "text": "like previously did um you know on a"
  },
  {
    "startMs": 577440,
    "text": "given given day or month. Um, and so I"
  },
  {
    "startMs": 579680,
    "text": "think there's there's a lot that we can"
  },
  {
    "startMs": 580800,
    "text": "unlock here uh for the broader world as"
  },
  {
    "startMs": 582959,
    "text": "well."
  },
  {
    "startMs": 585040,
    "text": "Um, to go back to like I said, you know,"
  },
  {
    "startMs": 586720,
    "text": "I think like when it comes to these"
  },
  {
    "startMs": 587920,
    "text": "three modalities, you can mix and match"
  },
  {
    "startMs": 589519,
    "text": "them and we already have some, you know,"
  },
  {
    "startMs": 591279,
    "text": "rudimentary ways that we're seeing this."
  },
  {
    "startMs": 593120,
    "text": "I think there's things like, you know,"
  },
  {
    "startMs": 594959,
    "text": "all of these pieces for in-car"
  },
  {
    "startMs": 596320,
    "text": "assistants exist, though nothing has"
  },
  {
    "startMs": 598000,
    "text": "quite like combined them into this"
  },
  {
    "startMs": 599680,
    "text": "seamless way. You can talk to like the"
  },
  {
    "startMs": 602240,
    "text": "CarPlay dashboard. Um, you can tell it,"
  },
  {
    "startMs": 604240,
    "text": "\"Hey, go play some Spotify music for"
  },
  {
    "startMs": 606080,
    "text": "me.\" Um, and then it can come back and"
  },
  {
    "startMs": 607680,
    "text": "tell you, Google Maps can come back and"
  },
  {
    "startMs": 609040,
    "text": "tell you, hey, like, you know, there's"
  },
  {
    "startMs": 610240,
    "text": "traffic on this route. We're going to"
  },
  {
    "startMs": 611360,
    "text": "reroute you. But like we can now start"
  },
  {
    "startMs": 613440,
    "text": "to think about what does it mean to"
  },
  {
    "startMs": 614480,
    "text": "combine that into like a single voice"
  },
  {
    "startMs": 616240,
    "text": "agent across multiple modes. [snorts]"
  },
  {
    "startMs": 618399,
    "text": "Um, similarly, you know, there's a lot"
  },
  {
    "startMs": 620000,
    "text": "of experimentation in the game space"
  },
  {
    "startMs": 621680,
    "text": "with multimodality. Um, you can think"
  },
  {
    "startMs": 623680,
    "text": "about a real life character where you're"
  },
  {
    "startMs": 625600,
    "text": "talking to it to, you know, mine"
  },
  {
    "startMs": 627279,
    "text": "information about the game, about the"
  },
  {
    "startMs": 628880,
    "text": "world. um you can talk to it to execute"
  },
  {
    "startMs": 630720,
    "text": "actions on your behalf and then it can"
  },
  {
    "startMs": 632880,
    "text": "react to like world events right that"
  },
  {
    "startMs": 634640,
    "text": "are happening and then give that"
  },
  {
    "startMs": 635920,
    "text": "information to you rather than just like"
  },
  {
    "startMs": 637440,
    "text": "a simple notification"
  },
  {
    "startMs": 641279,
    "text": "um and I think the question you know"
  },
  {
    "startMs": 642399,
    "text": "behind the question here right is like I"
  },
  {
    "startMs": 644000,
    "text": "mentioned we've had all these things for"
  },
  {
    "startMs": 645120,
    "text": "a while people have been prototyping"
  },
  {
    "startMs": 646320,
    "text": "them for a while why focus on them now"
  },
  {
    "startMs": 648720,
    "text": "why think about building with them now"
  },
  {
    "startMs": 650640,
    "text": "um and I think that brings me to uh a"
  },
  {
    "startMs": 653440,
    "text": "little bit of context here right as as"
  },
  {
    "startMs": 654959,
    "text": "I'm hopefully most of you know"
  },
  {
    "startMs": 656399,
    "text": "traditionally voice agents are built in"
  },
  {
    "startMs": 658399,
    "text": "this chain Ed model, right? You uh talk,"
  },
  {
    "startMs": 661360,
    "text": "you transcribe, you send that to a"
  },
  {
    "startMs": 662880,
    "text": "language model. It calls tools."
  },
  {
    "startMs": 664240,
    "text": "Hopefully, it doesn't take too long to"
  },
  {
    "startMs": 665519,
    "text": "respond. Um it then generates text"
  },
  {
    "startMs": 667920,
    "text": "output. You make that into audio and"
  },
  {
    "startMs": 669600,
    "text": "then you play that back to the user."
  },
  {
    "startMs": 672160,
    "text": "And some time ago, um OpenAI, you know,"
  },
  {
    "startMs": 674320,
    "text": "decided on a different approach, right?"
  },
  {
    "startMs": 676000,
    "text": "The real-time model family does not do"
  },
  {
    "startMs": 678560,
    "text": "any transcription behind the scenes. It"
  },
  {
    "startMs": 680800,
    "text": "is trained on native audio as tokens."
  },
  {
    "startMs": 683519,
    "text": "So, you send audio in and you get audio"
  },
  {
    "startMs": 686000,
    "text": "back out. And the industry I think in"
  },
  {
    "startMs": 688399,
    "text": "general has been, you know, trending"
  },
  {
    "startMs": 689600,
    "text": "more in this direction. Um, and not just"
  },
  {
    "startMs": 691519,
    "text": "making it native audio, but even just"
  },
  {
    "startMs": 693120,
    "text": "letting go of the turnbased abstraction"
  },
  {
    "startMs": 695279,
    "text": "that we've had, right? Um, and so making"
  },
  {
    "startMs": 697120,
    "text": "it that it's just continuous streaming"
  },
  {
    "startMs": 698880,
    "text": "audio in and out."
  },
  {
    "startMs": 702000,
    "text": "And the reason that opening I did this"
  },
  {
    "startMs": 703920,
    "text": "was, you know, turns out there's a lot"
  },
  {
    "startMs": 706000,
    "text": "of stuff that you lose when you"
  },
  {
    "startMs": 707440,
    "text": "transcribe speech and when you"
  },
  {
    "startMs": 709120,
    "text": "transcribe audio, right? Um there's the"
  },
  {
    "startMs": 710959,
    "text": "old saying that when humans communicate"
  },
  {
    "startMs": 712560,
    "text": "face to face, 55% of the information is"
  },
  {
    "startMs": 715040,
    "text": "in body language, another 38% is in your"
  },
  {
    "startMs": 717920,
    "text": "tone of voice and the last like 7% is"
  },
  {
    "startMs": 719839,
    "text": "the actual words you are saying. Um and"
  },
  {
    "startMs": 721680,
    "text": "so when you transcribe, you lose tone"
  },
  {
    "startMs": 723360,
    "text": "and cadence and emotional uh you know"
  },
  {
    "startMs": 725839,
    "text": "impact, you lose like whether they're"
  },
  {
    "startMs": 727279,
    "text": "trying to interrupt you, you lose"
  },
  {
    "startMs": 728240,
    "text": "background noise, all of this stuff"
  },
  {
    "startMs": 729519,
    "text": "which is really important context for"
  },
  {
    "startMs": 731279,
    "text": "the model to understand."
  },
  {
    "startMs": 733680,
    "text": "Um a much more quantitative reason to do"
  },
  {
    "startMs": 735440,
    "text": "it is that you know the first two uh"
  },
  {
    "startMs": 737680,
    "text": "voice modes in chat GBT were built with"
  },
  {
    "startMs": 740160,
    "text": "this chained approach. Um and as you can"
  },
  {
    "startMs": 741920,
    "text": "see had you know significantly higher"
  },
  {
    "startMs": 743839,
    "text": "latency than using uh the native"
  },
  {
    "startMs": 745760,
    "text": "approach with advanced voice mode."
  },
  {
    "startMs": 749040,
    "text": "And that brings me to GPT realtime 2. Um"
  },
  {
    "startMs": 752000,
    "text": "and this is going to be the one part of"
  },
  {
    "startMs": 753200,
    "text": "the talk where you know I make my"
  },
  {
    "startMs": 754880,
    "text": "shameless plug. Um real time 2 is the"
  },
  {
    "startMs": 757839,
    "text": "the latest model in the real time"
  },
  {
    "startMs": 759279,
    "text": "family. We released it a couple of"
  },
  {
    "startMs": 760720,
    "text": "months ago. Um, and the the really cool"
  },
  {
    "startMs": 763040,
    "text": "thing about this model is that it brings"
  },
  {
    "startMs": 765120,
    "text": "reasoning to the audio medium. Um, and"
  },
  {
    "startMs": 768240,
    "text": "so much like our text models, it can now"
  },
  {
    "startMs": 770160,
    "text": "think before it speaks. Um, I'm sure"
  },
  {
    "startMs": 772480,
    "text": "many of us have seen some demos of voice"
  },
  {
    "startMs": 774320,
    "text": "models saying things that are a little"
  },
  {
    "startMs": 776079,
    "text": "bit less than intelligent. Um, and so"
  },
  {
    "startMs": 778720,
    "text": "you can now, you know, try to ensure"
  },
  {
    "startMs": 780160,
    "text": "that you give it more reasoning budget"
  },
  {
    "startMs": 782399,
    "text": "uh to come up with a good answer. Part"
  },
  {
    "startMs": 784240,
    "text": "of why that's also useful is that we"
  },
  {
    "startMs": 785920,
    "text": "introduced tool calling a little while"
  },
  {
    "startMs": 787279,
    "text": "ago. And so the model in addition to"
  },
  {
    "startMs": 789120,
    "text": "thinking it can also delegate parallel"
  },
  {
    "startMs": 790959,
    "text": "tool calls. Um you can start to bring"
  },
  {
    "startMs": 792880,
    "text": "these together. Um though of course that"
  },
  {
    "startMs": 794800,
    "text": "adds latency, right? And you know that's"
  },
  {
    "startMs": 796959,
    "text": "why we also added preamles. Um preamles"
  },
  {
    "startMs": 799279,
    "text": "are a way that you can prompt the model"
  },
  {
    "startMs": 800800,
    "text": "to uh give the user a heads up if it's"
  },
  {
    "startMs": 803519,
    "text": "going to be thinking or if it's going to"
  },
  {
    "startMs": 804720,
    "text": "be calling tools. Um you know if you"
  },
  {
    "startMs": 807040,
    "text": "think about the scenario of a travel"
  },
  {
    "startMs": 809040,
    "text": "agent, right? If I called the travel"
  },
  {
    "startMs": 810560,
    "text": "agent on the phone, uh you would want"
  },
  {
    "startMs": 812720,
    "text": "the travel agent to say to say, \"Hey,"
  },
  {
    "startMs": 814480,
    "text": "like I'm going to go check flight"
  },
  {
    "startMs": 815680,
    "text": "prices, right? give me a couple seconds"
  },
  {
    "startMs": 817200,
    "text": "to do that. Um, and now with an AI"
  },
  {
    "startMs": 820240,
    "text": "travel agent, um, and preamles, you can"
  },
  {
    "startMs": 822240,
    "text": "actually have it communicate that to the"
  },
  {
    "startMs": 823839,
    "text": "user while it's performing actions in"
  },
  {
    "startMs": 825519,
    "text": "the background. Uh, there's a few other"
  },
  {
    "startMs": 827920,
    "text": "things here, right? It's got longer"
  },
  {
    "startMs": 829200,
    "text": "context, better domain understanding,"
  },
  {
    "startMs": 831120,
    "text": "um, more natural voices, and it's much"
  },
  {
    "startMs": 832880,
    "text": "more steerable. Um, there's some really"
  },
  {
    "startMs": 834639,
    "text": "cool features, uh, that it can do when"
  },
  {
    "startMs": 836880,
    "text": "it comes to like wake words and just"
  },
  {
    "startMs": 839120,
    "text": "waiting for you to to tell it. You you"
  },
  {
    "startMs": 841120,
    "text": "can give it a name. Uh, you can say"
  },
  {
    "startMs": 842880,
    "text": "like, you know, hey, Marin, do you want"
  },
  {
    "startMs": 844560,
    "text": "to say hi to the room? Um, and if you've"
  },
  {
    "startMs": 846320,
    "text": "like prompted that into the model, then"
  },
  {
    "startMs": 848160,
    "text": "it'll, you know, go ahead and and"
  },
  {
    "startMs": 849680,
    "text": "respond to you, right? Uh, and of"
  },
  {
    "startMs": 851839,
    "text": "course, you know, uh, obligatory"
  },
  {
    "startMs": 853279,
    "text": "benchmark slide, uh, it does pretty well"
  },
  {
    "startMs": 855040,
    "text": "on the the latest audio benchmarks, too."
  },
  {
    "startMs": 858959,
    "text": "So, TLDDR, uh, is a pretty good model."
  },
  {
    "startMs": 861519,
    "text": "Um,"
  },
  {
    "startMs": 863760,
    "text": "but I think the the kind of final thing"
  },
  {
    "startMs": 865519,
    "text": "that you know I want to leave you with"
  },
  {
    "startMs": 866880,
    "text": "here is um when building voice agents uh"
  },
  {
    "startMs": 871600,
    "text": "not to"
  },
  {
    "startMs": 873440,
    "text": "start with the question of like what"
  },
  {
    "startMs": 875279,
    "text": "kind of voice agent am I trying to"
  },
  {
    "startMs": 877199,
    "text": "build, right? I think the thing I want"
  },
  {
    "startMs": 878560,
    "text": "to leave you with is start with the"
  },
  {
    "startMs": 880399,
    "text": "question of like what is the role of"
  },
  {
    "startMs": 883440,
    "text": "voice and audio in this interaction? Um"
  },
  {
    "startMs": 886079,
    "text": "and then how do I move forward from"
  },
  {
    "startMs": 887839,
    "text": "there, right? Right? And often when I"
  },
  {
    "startMs": 889120,
    "text": "ask that question, it leads to a bunch"
  },
  {
    "startMs": 891199,
    "text": "more questions after that. Things like"
  },
  {
    "startMs": 893600,
    "text": "what can the model perceive? What"
  },
  {
    "startMs": 895120,
    "text": "context does it have? Right? Um what"
  },
  {
    "startMs": 897440,
    "text": "tools are available to it and which of"
  },
  {
    "startMs": 899680,
    "text": "those tools should it be, you know, uh"
  },
  {
    "startMs": 901760,
    "text": "executing safely and correctly? Um"
  },
  {
    "startMs": 904480,
    "text": "should it communicate now? Should it"
  },
  {
    "startMs": 906160,
    "text": "wait? Uh you know, how should it"
  },
  {
    "startMs": 908240,
    "text": "communicate? Should it be sending visual"
  },
  {
    "startMs": 910399,
    "text": "notifications or using audio? Um, and so"
  },
  {
    "startMs": 913839,
    "text": "taken together, yeah, I hope everybody"
  },
  {
    "startMs": 915360,
    "text": "in here can can start to build some much"
  },
  {
    "startMs": 917279,
    "text": "richer experiences with voice. Um,"
  },
  {
    "startMs": 919600,
    "text": "because like others have said, uh, I do"
  },
  {
    "startMs": 921519,
    "text": "believe that AGI will be spoken, not"
  },
  {
    "startMs": 923680,
    "text": "typed."
  },
  {
    "startMs": 926320,
    "text": "Thank you very much. I'll be at the"
  },
  {
    "startMs": 927600,
    "text": "OpenAI booth, uh, for any Q&A after. Um,"
  },
  {
    "startMs": 930240,
    "text": "yeah, have a good event."
  },
  {
    "startMs": 946485,
    "text": ">> [music]"
  }
]