[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12800,
    "text": "My name is Suman Yu and I'm the founder"
  },
  {
    "startMs": 15040,
    "text": "and CEO of Hamming. And before working"
  },
  {
    "startMs": 19039,
    "text": "on voice agent reliability and safety, I"
  },
  {
    "startMs": 22320,
    "text": "worked at a company called Citizen"
  },
  {
    "startMs": 25199,
    "text": "out of New York. Anybody here use"
  },
  {
    "startMs": 26960,
    "text": "Citizen app? Awesome. Thank you. Uh, and"
  },
  {
    "startMs": 31760,
    "text": "at Citizen,"
  },
  {
    "startMs": 33440,
    "text": "we listened to crime, thousands of hours"
  },
  {
    "startMs": 36960,
    "text": "of police radio station data and sent"
  },
  {
    "startMs": 40079,
    "text": "millions of alerts to users in San"
  },
  {
    "startMs": 42559,
    "text": "Francisco,"
  },
  {
    "startMs": 44559,
    "text": "New York, LA, uh, Chicago, Baltimore,"
  },
  {
    "startMs": 48320,
    "text": "and so on."
  },
  {
    "startMs": 50399,
    "text": "Some obviously gory and pretty sad. Uh,"
  },
  {
    "startMs": 54079,
    "text": "but others more funny like a person"
  },
  {
    "startMs": 56399,
    "text": "stealing bags of ice cream from Safeway"
  },
  {
    "startMs": 60079,
    "text": "or report of a man hanging off the side"
  },
  {
    "startMs": 62000,
    "text": "of the house after a woman stole his"
  },
  {
    "startMs": 64559,
    "text": "ladder."
  },
  {
    "startMs": 67680,
    "text": "If I actually take a look at the citizen"
  },
  {
    "startMs": 69040,
    "text": "app right now for those who are"
  },
  {
    "startMs": 70720,
    "text": "customers or users, I can see that there"
  },
  {
    "startMs": 73439,
    "text": "is a man yelling at person. There's"
  },
  {
    "startMs": 76400,
    "text": "indecent exposure. This is real. This is"
  },
  {
    "startMs": 78479,
    "text": "real time. This is, you know,"
  },
  {
    "startMs": 81280,
    "text": "couple hours ago. These are real-time"
  },
  {
    "startMs": 83040,
    "text": "alerts that we're sending."
  },
  {
    "startMs": 87439,
    "text": "Now, voice agents scare me more because"
  },
  {
    "startMs": 89840,
    "text": "they're finally graduating from demos"
  },
  {
    "startMs": 91759,
    "text": "and PC's to production. We should be"
  },
  {
    "startMs": 93520,
    "text": "super excited, but I'm nervous. I'm"
  },
  {
    "startMs": 95200,
    "text": "personally nervous. Uh, they're talking"
  },
  {
    "startMs": 97439,
    "text": "to users at a scale that would make Gary"
  },
  {
    "startMs": 99600,
    "text": "Tan and Polygram proud."
  },
  {
    "startMs": 103759,
    "text": "When I got started in voice agent"
  },
  {
    "startMs": 105200,
    "text": "reliability in early 2024, voice was"
  },
  {
    "startMs": 108720,
    "text": "just starting to work. It was not quite"
  },
  {
    "startMs": 110720,
    "text": "good yet, but it was just starting to"
  },
  {
    "startMs": 112079,
    "text": "work. You would have to pay me a lot of"
  },
  {
    "startMs": 114079,
    "text": "money for me to stop using, you know,"
  },
  {
    "startMs": 115920,
    "text": "Aqua voice, Super Whisper, uh, Whisper"
  },
  {
    "startMs": 118719,
    "text": "Flow, and so on. These products are just"
  },
  {
    "startMs": 120560,
    "text": "getting super, super good. And a big"
  },
  {
    "startMs": 122320,
    "text": "reason is because the underlying"
  },
  {
    "startMs": 123520,
    "text": "infrastructure is getting better, and"
  },
  {
    "startMs": 125040,
    "text": "the orchestration layer is getting"
  },
  {
    "startMs": 126399,
    "text": "meaningfully better. It's getting much"
  },
  {
    "startMs": 129039,
    "text": "faster to build products and voice"
  },
  {
    "startMs": 130959,
    "text": "experiences that maybe are 60% good in a"
  },
  {
    "startMs": 134560,
    "text": "pretty short period of time, but the"
  },
  {
    "startMs": 136000,
    "text": "long tail is still Hey, Gorov. the long"
  },
  {
    "startMs": 138560,
    "text": "tail is still uh wise away."
  },
  {
    "startMs": 141520,
    "text": "I think speech speech models are getting"
  },
  {
    "startMs": 143120,
    "text": "better. Um teams are experiment"
  },
  {
    "startMs": 145840,
    "text": "experimenting with hybrid architectures"
  },
  {
    "startMs": 148879,
    "text": "of combining more voicetovoice"
  },
  {
    "startMs": 152239,
    "text": "modalities and also cascading stacks to"
  },
  {
    "startMs": 154879,
    "text": "make the experience reliable but still"
  },
  {
    "startMs": 156959,
    "text": "pretty low latency."
  },
  {
    "startMs": 159599,
    "text": "Things are obviously getting better."
  },
  {
    "startMs": 161200,
    "text": "Agents are being connected to calendars,"
  },
  {
    "startMs": 162879,
    "text": "CRM, HRs, reservation systems, and so"
  },
  {
    "startMs": 165840,
    "text": "on. Voice agents can now take actions."
  },
  {
    "startMs": 170160,
    "text": "However, reliability is still the number"
  },
  {
    "startMs": 172160,
    "text": "one problem holding back most voice"
  },
  {
    "startMs": 174080,
    "text": "agent deployments at scale. This is"
  },
  {
    "startMs": 175760,
    "text": "still the number one problem. This is an"
  },
  {
    "startMs": 178000,
    "text": "example I found on Twitter pretty"
  },
  {
    "startMs": 179599,
    "text": "randomly, you know, two weeks ago and a"
  },
  {
    "startMs": 182879,
    "text": "person is trying to get information for"
  },
  {
    "startMs": 184319,
    "text": "a tradein and gets absolutely confused"
  },
  {
    "startMs": 186720,
    "text": "with the information that they're"
  },
  {
    "startMs": 188239,
    "text": "receiving. Alex now has to correct for"
  },
  {
    "startMs": 190800,
    "text": "this loss of trust but trying to, you"
  },
  {
    "startMs": 193599,
    "text": "know, call the person and see what see"
  },
  {
    "startMs": 195519,
    "text": "what happened and fix the situation. Let"
  },
  {
    "startMs": 197840,
    "text": "me see if audio works here."
  },
  {
    "startMs": 200640,
    "text": ">> Screwed up with another customer. We're"
  },
  {
    "startMs": 202800,
    "text": "getting it fixed, but I got to call him"
  },
  {
    "startMs": 204080,
    "text": "and see if I can work it out."
  },
  {
    "startMs": 205440,
    "text": ">> I'm like, dude, half the time I'm like,"
  },
  {
    "startMs": 207120,
    "text": "I don't know if I'm talking to AI. I"
  },
  {
    "startMs": 208800,
    "text": "don't know if I'm talking to a person."
  },
  {
    "startMs": 209920,
    "text": "It was just confusing, but we got there."
  },
  {
    "startMs": 212000,
    "text": ">> It probably is AI and human. And"
  },
  {
    "startMs": 217040,
    "text": ">> so I think voices sound very confident."
  },
  {
    "startMs": 219120,
    "text": "They sound very natural, but the"
  },
  {
    "startMs": 220640,
    "text": "information provided is often, you know,"
  },
  {
    "startMs": 222640,
    "text": "not correct. That's the biggest problem"
  },
  {
    "startMs": 224239,
    "text": "here."
  },
  {
    "startMs": 226000,
    "text": "This example is more personal. I had"
  },
  {
    "startMs": 228159,
    "text": "booked an appointment with a physician a"
  },
  {
    "startMs": 229920,
    "text": "couple weeks ago or I thought I did. I"
  },
  {
    "startMs": 232959,
    "text": "showed up to the appointment and turns"
  },
  {
    "startMs": 234400,
    "text": "out I was not actually on the schedule."
  },
  {
    "startMs": 237040,
    "text": "So the front desk, you me turned me"
  },
  {
    "startMs": 238640,
    "text": "away. I wasted 2 hours. For me, this was"
  },
  {
    "startMs": 241920,
    "text": "a waste of time. But what if this was"
  },
  {
    "startMs": 243760,
    "text": "actually your parent?"
  },
  {
    "startMs": 246159,
    "text": "What if this was your grandparent?"
  },
  {
    "startMs": 248799,
    "text": "What if this appointment was for a"
  },
  {
    "startMs": 250720,
    "text": "procedure instead of a regular checkup?"
  },
  {
    "startMs": 253120,
    "text": "The costs for these different"
  },
  {
    "startMs": 255200,
    "text": "permutations of the same failure mode"
  },
  {
    "startMs": 256959,
    "text": "can actually be super super high."
  },
  {
    "startMs": 260079,
    "text": "Now, let's compare crime to voice"
  },
  {
    "startMs": 262000,
    "text": "agents. Um, I think observation number"
  },
  {
    "startMs": 264400,
    "text": "one is crime is actually decreasing over"
  },
  {
    "startMs": 266960,
    "text": "time. This is a good thing and I hope it"
  },
  {
    "startMs": 270479,
    "text": "crosses the x- axis at some point you"
  },
  {
    "startMs": 272479,
    "text": "know in the future."
  },
  {
    "startMs": 275280,
    "text": "Voice on the other hand is generally"
  },
  {
    "startMs": 277280,
    "text": "taking off right we're seeing a pretty"
  },
  {
    "startMs": 278880,
    "text": "fast takeoff of voice agents being"
  },
  {
    "startMs": 280560,
    "text": "deployed in production. There's at least"
  },
  {
    "startMs": 282400,
    "text": "a trillion calls that are done every"
  },
  {
    "startMs": 284000,
    "text": "single year and majority of these will"
  },
  {
    "startMs": 286800,
    "text": "be done by conversational voice agents"
  },
  {
    "startMs": 288400,
    "text": "over the next you know five years. If"
  },
  {
    "startMs": 290400,
    "text": "you assume a 1% error rate that is still"
  },
  {
    "startMs": 293040,
    "text": "10 billion incidents per year. That's a"
  },
  {
    "startMs": 295440,
    "text": "lot."
  },
  {
    "startMs": 298080,
    "text": "In practice, we currently monitor 10,000"
  },
  {
    "startMs": 300479,
    "text": "agents and the error rate is closer to"
  },
  {
    "startMs": 303280,
    "text": "10% in practice. These range from agents"
  },
  {
    "startMs": 306880,
    "text": "saying they found the right policy when"
  },
  {
    "startMs": 308560,
    "text": "they actually skipped the eligibility or"
  },
  {
    "startMs": 310320,
    "text": "verification steps or applying discounts"
  },
  {
    "startMs": 312800,
    "text": "when they were not really supposed to,"
  },
  {
    "startMs": 314720,
    "text": "misharing what the person said,"
  },
  {
    "startMs": 316240,
    "text": "providing incorrect information, or"
  },
  {
    "startMs": 318240,
    "text": "claiming they booked an appointment when"
  },
  {
    "startMs": 319680,
    "text": "they actually did not, just like it"
  },
  {
    "startMs": 321039,
    "text": "happened for me."
  },
  {
    "startMs": 323440,
    "text": "Now, not every single call has an"
  },
  {
    "startMs": 325280,
    "text": "equally, you know, bad cost. Uh, some"
  },
  {
    "startMs": 329120,
    "text": "range, you know, in the crime land, some"
  },
  {
    "startMs": 330960,
    "text": "range from trash fires, which are kind"
  },
  {
    "startMs": 333440,
    "text": "of funny, annoying, not really hurting"
  },
  {
    "startMs": 335440,
    "text": "somebody. For a voice equivalent, that"
  },
  {
    "startMs": 338080,
    "text": "would be annoyances like repetition, um,"
  },
  {
    "startMs": 340639,
    "text": "or just sort of not quite understanding"
  },
  {
    "startMs": 342560,
    "text": "what the user is saying. all the way to"
  },
  {
    "startMs": 344639,
    "text": "safety risks like mass shootings or in"
  },
  {
    "startMs": 347680,
    "text": "the voice agent equivalent, it would be"
  },
  {
    "startMs": 349919,
    "text": "um a drive-thru that's deploying um"
  },
  {
    "startMs": 352400,
    "text": "voice agents at scale like a Taco Bell"
  },
  {
    "startMs": 354240,
    "text": "or McDonald's and a person orders a"
  },
  {
    "startMs": 357039,
    "text": "vegan burger with peanut allergies."
  },
  {
    "startMs": 359919,
    "text": "If one of those two situations are not"
  },
  {
    "startMs": 362160,
    "text": "handled correctly, that is definitely a"
  },
  {
    "startMs": 364000,
    "text": "safety concern at scale."
  },
  {
    "startMs": 367759,
    "text": "The other big difference between crime"
  },
  {
    "startMs": 370400,
    "text": "and and voice agent deployments is is"
  },
  {
    "startMs": 373039,
    "text": "crime generally tends to be pretty hyper"
  },
  {
    "startMs": 376319,
    "text": "local,"
  },
  {
    "startMs": 378000,
    "text": "tends to be very decentralized, right?"
  },
  {
    "startMs": 379759,
    "text": "Things like robbery or motor vehicle"
  },
  {
    "startMs": 382160,
    "text": "theft or lararseny. They're impacting a"
  },
  {
    "startMs": 385680,
    "text": "finite set of individuals that are"
  },
  {
    "startMs": 387280,
    "text": "involved in that um situation."
  },
  {
    "startMs": 392639,
    "text": "On the other hand, voice agents are much"
  },
  {
    "startMs": 394639,
    "text": "more centralized. a single prompt change"
  },
  {
    "startMs": 397039,
    "text": "or an architecture change can have"
  },
  {
    "startMs": 398880,
    "text": "pretty massive implications downstream"
  },
  {
    "startMs": 401520,
    "text": "for all of the millions of you know"
  },
  {
    "startMs": 403280,
    "text": "users that are um in the crossfire. So"
  },
  {
    "startMs": 406800,
    "text": "the blast radius is is quite quite"
  },
  {
    "startMs": 408960,
    "text": "massive. So the natural question is how"
  },
  {
    "startMs": 411039,
    "text": "do you make these incidents much more"
  },
  {
    "startMs": 412639,
    "text": "visible and obvious? That's the kind of"
  },
  {
    "startMs": 414720,
    "text": "obvious question here."
  },
  {
    "startMs": 417039,
    "text": "I'll borrow a framework from a couple of"
  },
  {
    "startMs": 418800,
    "text": "my friends who were OG growth folks at"
  },
  {
    "startMs": 421759,
    "text": "Facebook. So step one is to identify"
  },
  {
    "startMs": 424000,
    "text": "okay what are all the challenges and"
  },
  {
    "startMs": 425520,
    "text": "problems that um exist in your"
  },
  {
    "startMs": 428080,
    "text": "conversation experience. Step two is to"
  },
  {
    "startMs": 430560,
    "text": "prioritize an impact size. There's a"
  },
  {
    "startMs": 432160,
    "text": "frequency and severity analysis that's"
  },
  {
    "startMs": 434560,
    "text": "pretty important. Step three is to"
  },
  {
    "startMs": 436639,
    "text": "understand okay how do we actually fix"
  },
  {
    "startMs": 438080,
    "text": "this? Step four execute. Step five okay"
  },
  {
    "startMs": 441280,
    "text": "did my change actually work and did it"
  },
  {
    "startMs": 443520,
    "text": "cause any regressions somewhere else."
  },
  {
    "startMs": 445759,
    "text": "And lastly we continue to monitor in"
  },
  {
    "startMs": 447680,
    "text": "production."
  },
  {
    "startMs": 450080,
    "text": "On the y- axis, I think it's important"
  },
  {
    "startMs": 452160,
    "text": "to highlight there are known problems"
  },
  {
    "startMs": 454639,
    "text": "that already exist. Things like turnover"
  },
  {
    "startMs": 456960,
    "text": "latency, interruptions, um maybe some"
  },
  {
    "startMs": 459280,
    "text": "ASR problems you're, you know, aware of."
  },
  {
    "startMs": 461759,
    "text": "And these are known problems that exist"
  },
  {
    "startMs": 464639,
    "text": "that the team should track over time. On"
  },
  {
    "startMs": 467120,
    "text": "the other axis is actually emerging"
  },
  {
    "startMs": 469360,
    "text": "behavior or patterns that are only"
  },
  {
    "startMs": 471280,
    "text": "obvious across lots of conversations. Um"
  },
  {
    "startMs": 475520,
    "text": "on the x- axis, you have coverage just"
  },
  {
    "startMs": 477599,
    "text": "like insurance. Are you analyzing few"
  },
  {
    "startMs": 480400,
    "text": "conversations? Are you analyzing many,"
  },
  {
    "startMs": 482240,
    "text": "many conversations? Most teams will"
  },
  {
    "startMs": 484639,
    "text": "typically start by listening to calls"
  },
  {
    "startMs": 486319,
    "text": "manually."
  },
  {
    "startMs": 488400,
    "text": "And I think that's the best place to"
  },
  {
    "startMs": 489680,
    "text": "start. I don't think you should skip"
  },
  {
    "startMs": 491360,
    "text": "that step. There's a lot of depth and"
  },
  {
    "startMs": 493280,
    "text": "insights to get by actually listening to"
  },
  {
    "startMs": 495120,
    "text": "specific conversations and building that"
  },
  {
    "startMs": 497280,
    "text": "texture that that comes from that"
  },
  {
    "startMs": 499039,
    "text": "intuition. However, it's obviously not"
  },
  {
    "startMs": 501919,
    "text": "scalable. So most teams end up having a"
  },
  {
    "startMs": 504560,
    "text": "spreadsheet of I don't know five or 10"
  },
  {
    "startMs": 507599,
    "text": "different rubrics around greetings,"
  },
  {
    "startMs": 509599,
    "text": "closing, validation,"
  },
  {
    "startMs": 511680,
    "text": "um, core logic and so on. To scale that"
  },
  {
    "startMs": 515120,
    "text": "up even further, you then end up"
  },
  {
    "startMs": 516959,
    "text": "investing in some eval product, right?"
  },
  {
    "startMs": 519440,
    "text": "You might run some element as a judge"
  },
  {
    "startMs": 521120,
    "text": "and compute classic metrics and also"
  },
  {
    "startMs": 524480,
    "text": "more more deterministic and stoastic"
  },
  {
    "startMs": 527120,
    "text": "scoring logic. Um, but there you're"
  },
  {
    "startMs": 529200,
    "text": "still stuck with checking for"
  },
  {
    "startMs": 531200,
    "text": "consistency of known problems, but"
  },
  {
    "startMs": 533680,
    "text": "you're not really discovering novel"
  },
  {
    "startMs": 534959,
    "text": "insights that are actually happening"
  },
  {
    "startMs": 536240,
    "text": "across conversations. We're spending a"
  },
  {
    "startMs": 538720,
    "text": "ton of time on performing cross"
  },
  {
    "startMs": 541279,
    "text": "conversation analysis, not a pattern on"
  },
  {
    "startMs": 543120,
    "text": "a single call, but across conversations."
  },
  {
    "startMs": 545760,
    "text": "And some of the best teams that we work"
  },
  {
    "startMs": 547360,
    "text": "with are are doing the same."
  },
  {
    "startMs": 550800,
    "text": "Now, to prioritize an impact size, I"
  },
  {
    "startMs": 552800,
    "text": "think there's problems that are one-off"
  },
  {
    "startMs": 554959,
    "text": "that are low impact. I mean, who cares?"
  },
  {
    "startMs": 557440,
    "text": "uh even low impact and systematic"
  },
  {
    "startMs": 559440,
    "text": "problems in the crime world that would"
  },
  {
    "startMs": 561680,
    "text": "be a trash fire in a voice aation world"
  },
  {
    "startMs": 563920,
    "text": "it could be some repetitions the team is"
  },
  {
    "startMs": 565920,
    "text": "experiencing they're still annoying at"
  },
  {
    "startMs": 567839,
    "text": "scale and if you are doing a bake off"
  },
  {
    "startMs": 569680,
    "text": "it's still worth solving for them I"
  },
  {
    "startMs": 571600,
    "text": "would not ignore these class of problems"
  },
  {
    "startMs": 573920,
    "text": "oneoff and high impact well hope it"
  },
  {
    "startMs": 575600,
    "text": "doesn't chronic and I think systematic"
  },
  {
    "startMs": 578240,
    "text": "and high impact are obviously the P 0"
  },
  {
    "startMs": 580320,
    "text": "you know target areas um for the team to"
  },
  {
    "startMs": 582640,
    "text": "solve an example of that"
  },
  {
    "startMs": 585200,
    "text": "would be in a fins serve capacity"
  },
  {
    "startMs": 587040,
    "text": "There's a voice agent that um helps"
  },
  {
    "startMs": 589920,
    "text": "users freeze their credit cards. And if"
  },
  {
    "startMs": 591839,
    "text": "it doesn't do that, well, that's a"
  },
  {
    "startMs": 593120,
    "text": "massive fail."
  },
  {
    "startMs": 596560,
    "text": "All right. So, understand and execute."
  },
  {
    "startMs": 598399,
    "text": "I'm pretty sure everyone's doing this."
  },
  {
    "startMs": 600000,
    "text": "Please fix my agent. Uh I think fixing"
  },
  {
    "startMs": 602720,
    "text": "or rather attempting to make a fix is"
  },
  {
    "startMs": 605920,
    "text": "the simplest and the lowest effort"
  },
  {
    "startMs": 608640,
    "text": "component of this debugging pipeline and"
  },
  {
    "startMs": 611440,
    "text": "loop. Um the next step is all right, I"
  },
  {
    "startMs": 614880,
    "text": "made a change to my system. How do I"
  },
  {
    "startMs": 616399,
    "text": "actually know this thing works um for"
  },
  {
    "startMs": 619040,
    "text": "real? A great way that's naive is to"
  },
  {
    "startMs": 623200,
    "text": "take a real call, for example, in my"
  },
  {
    "startMs": 625360,
    "text": "case, I booked an appointment and it"
  },
  {
    "startMs": 627120,
    "text": "didn't get scheduled and replay that"
  },
  {
    "startMs": 629040,
    "text": "exact conversation and run that maybe 5,"
  },
  {
    "startMs": 632000,
    "text": "10, 20, 50 times and see, okay, what is"
  },
  {
    "startMs": 634480,
    "text": "my probability of passing this type of"
  },
  {
    "startMs": 637920,
    "text": "issue? A better way is to keep the same"
  },
  {
    "startMs": 640959,
    "text": "intent but change the wordings, change"
  },
  {
    "startMs": 644000,
    "text": "the patterns, change the accents, change"
  },
  {
    "startMs": 646480,
    "text": "the style, add one more intent to the"
  },
  {
    "startMs": 648160,
    "text": "mix. And that gives teams much more, you"
  },
  {
    "startMs": 651360,
    "text": "know, better coverage to feel confident"
  },
  {
    "startMs": 653839,
    "text": "that yes, I actually made a change and"
  },
  {
    "startMs": 656480,
    "text": "my changes are net positive instead of"
  },
  {
    "startMs": 658480,
    "text": "net negative."
  },
  {
    "startMs": 661600,
    "text": "There are certain fixes and I guess"
  },
  {
    "startMs": 665040,
    "text": "hypothesis that are very difficult to"
  },
  {
    "startMs": 667519,
    "text": "test in a pre-eployment synthetic"
  },
  {
    "startMs": 669600,
    "text": "setting. And so AB testing ends up"
  },
  {
    "startMs": 671680,
    "text": "being, you know, pretty pretty critical"
  },
  {
    "startMs": 673040,
    "text": "for those circumstances. For example, if"
  },
  {
    "startMs": 675360,
    "text": "you have an outbound agent, the first 5"
  },
  {
    "startMs": 677519,
    "text": "seconds of a conversation tends to be"
  },
  {
    "startMs": 679760,
    "text": "the most important. And so the vocal"
  },
  {
    "startMs": 682399,
    "text": "quality um and the specific words you"
  },
  {
    "startMs": 685600,
    "text": "end up using, they matter the most. And"
  },
  {
    "startMs": 687680,
    "text": "so AB testing that is the only way in in"
  },
  {
    "startMs": 689920,
    "text": "kind of real life setting to to get"
  },
  {
    "startMs": 692480,
    "text": "results. You can't really do it through"
  },
  {
    "startMs": 694160,
    "text": "simulations alone."
  },
  {
    "startMs": 697200,
    "text": "And so there we have the loop. Identify,"
  },
  {
    "startMs": 699760,
    "text": "prioritize, impact size, understand the"
  },
  {
    "startMs": 702560,
    "text": "fix, execute, check, make sure it didn't"
  },
  {
    "startMs": 705920,
    "text": "break anything, and then continue"
  },
  {
    "startMs": 707360,
    "text": "monitoring."
  },
  {
    "startMs": 711920,
    "text": "So I think making voice agents useful is"
  },
  {
    "startMs": 714640,
    "text": "already hard as it is. even when dealing"
  },
  {
    "startMs": 718160,
    "text": "with earnest users on the other line,"
  },
  {
    "startMs": 721040,
    "text": "right? These are people who who just"
  },
  {
    "startMs": 722399,
    "text": "want their problem solved. They're not"
  },
  {
    "startMs": 723600,
    "text": "trying to mess with you. These are like"
  },
  {
    "startMs": 724880,
    "text": "legit normal people."
  },
  {
    "startMs": 728639,
    "text": "Now, what happens when mythos learns how"
  },
  {
    "startMs": 730399,
    "text": "to dial?"
  },
  {
    "startMs": 733600,
    "text": "So, if it can extract trade secrets and"
  },
  {
    "startMs": 736959,
    "text": "uh you know, from the NSA, it can"
  },
  {
    "startMs": 738959,
    "text": "certainly, you know, seduce you into"
  },
  {
    "startMs": 742000,
    "text": "revealing PHI and PII data as well."
  },
  {
    "startMs": 746959,
    "text": "And I think both voice agents and humans"
  },
  {
    "startMs": 749200,
    "text": "will [clears throat] be targeted here."
  },
  {
    "startMs": 751120,
    "text": "Voice agents because there's a pressure"
  },
  {
    "startMs": 753760,
    "text": "to make these more capable. Give them"
  },
  {
    "startMs": 757200,
    "text": "access to more data. Give them access to"
  },
  {
    "startMs": 759120,
    "text": "more tools."
  },
  {
    "startMs": 761440,
    "text": "Deploy them quickly."
  },
  {
    "startMs": 765279,
    "text": "The more the capability, the bigger the"
  },
  {
    "startMs": 766720,
    "text": "surface area. This is this is pretty"
  },
  {
    "startMs": 768639,
    "text": "pretty common sense. And the more the"
  },
  {
    "startMs": 770880,
    "text": "voice agents become natural and human"
  },
  {
    "startMs": 774720,
    "text": "sounding, the more humans will be"
  },
  {
    "startMs": 776959,
    "text": "tricked along the way as well for those"
  },
  {
    "startMs": 778720,
    "text": "who are weaponizing."
  },
  {
    "startMs": 781680,
    "text": "Uh we ship a we shipped a red tipping"
  },
  {
    "startMs": 783519,
    "text": "product um back in April just to test"
  },
  {
    "startMs": 785760,
    "text": "out this hypothesis for how many agents"
  },
  {
    "startMs": 787760,
    "text": "can we actually break from a adversarial"
  },
  {
    "startMs": 790720,
    "text": "capacity and we can probably break one"
  },
  {
    "startMs": 792880,
    "text": "in five agents at this point. We've"
  },
  {
    "startMs": 794399,
    "text": "tested this across financial services,"
  },
  {
    "startMs": 796480,
    "text": "healthcare, um consumer and so on. We've"
  },
  {
    "startMs": 799839,
    "text": "bypassed verification. Uh we've"
  },
  {
    "startMs": 802240,
    "text": "definitely had agents, you know, we've"
  },
  {
    "startMs": 804560,
    "text": "been able to promject uh several agents"
  },
  {
    "startMs": 807279,
    "text": "and and gotten data we should not have."
  },
  {
    "startMs": 809279,
    "text": "So this is not theoretical. This is"
  },
  {
    "startMs": 811120,
    "text": "actually a real a real concern."
  },
  {
    "startMs": 817519,
    "text": "I think the only real defense against"
  },
  {
    "startMs": 819200,
    "text": "the dark arts is"
  },
  {
    "startMs": 821760,
    "text": "step one to invest deeply in"
  },
  {
    "startMs": 823760,
    "text": "pre-eployment testing. This could be"
  },
  {
    "startMs": 825920,
    "text": "textto text. This could be voice to"
  },
  {
    "startMs": 828000,
    "text": "voice. There's pros and cons to both."
  },
  {
    "startMs": 830079,
    "text": "Happy to chat offline if folks are"
  },
  {
    "startMs": 831680,
    "text": "interested. And this is just making sure"
  },
  {
    "startMs": 834240,
    "text": "you're not self-owning, you know, when"
  },
  {
    "startMs": 836399,
    "text": "you're talking to real people who just"
  },
  {
    "startMs": 837839,
    "text": "want to get their problem solved. Step"
  },
  {
    "startMs": 839920,
    "text": "two is to have a great monitoring system"
  },
  {
    "startMs": 842720,
    "text": "of all kinds. And I've highlighted, you"
  },
  {
    "startMs": 844880,
    "text": "know, different flavors of monitoring"
  },
  {
    "startMs": 846880,
    "text": "per call scoring, manual kind of evals,"
  },
  {
    "startMs": 850240,
    "text": "you know, listening to conversations and"
  },
  {
    "startMs": 851760,
    "text": "cross call analysis. And this is helpful"
  },
  {
    "startMs": 854320,
    "text": "both for monitoring what the agent is"
  },
  {
    "startMs": 856399,
    "text": "saying and behaving and how it's"
  },
  {
    "startMs": 857760,
    "text": "actually doing, but also the users. Are"
  },
  {
    "startMs": 859839,
    "text": "the users being adversarial? Are they"
  },
  {
    "startMs": 861360,
    "text": "being annoying? Are are they trying to"
  },
  {
    "startMs": 862800,
    "text": "trick the agent into doing things it's"
  },
  {
    "startMs": 864560,
    "text": "not supposed to be doing?"
  },
  {
    "startMs": 866639,
    "text": "And I think our new recommendation now"
  },
  {
    "startMs": 868720,
    "text": "is to run 24/7 red teaming um for your"
  },
  {
    "startMs": 871839,
    "text": "agents, especially if you believe the"
  },
  {
    "startMs": 874160,
    "text": "cost of bad interactions can be can be"
  },
  {
    "startMs": 876160,
    "text": "rather large."
  },
  {
    "startMs": 878800,
    "text": "So, I think voice agents um have this"
  },
  {
    "startMs": 880959,
    "text": "awesome potential of of making the world"
  },
  {
    "startMs": 883680,
    "text": "feel much more human compared to"
  },
  {
    "startMs": 888000,
    "text": "interacting with clunky IVR trees or"
  },
  {
    "startMs": 890320,
    "text": "chat bots or worse um being stuck on a"
  },
  {
    "startMs": 893040,
    "text": "on a hold."
  },
  {
    "startMs": 895040,
    "text": "And when we think about crime, we often"
  },
  {
    "startMs": 897040,
    "text": "think of crime happening to somebody"
  },
  {
    "startMs": 899199,
    "text": "else. You know, crime does not happen to"
  },
  {
    "startMs": 901839,
    "text": "you typically with voice agents,"
  },
  {
    "startMs": 904959,
    "text": "especially bad actors. As these agents"
  },
  {
    "startMs": 907440,
    "text": "are deployed and as bad actors start to"
  },
  {
    "startMs": 909279,
    "text": "exploit a lot of the vulnerabilities,"
  },
  {
    "startMs": 911360,
    "text": "the number of incidents is about to kind"
  },
  {
    "startMs": 913519,
    "text": "of go way way up. And so the reason I"
  },
  {
    "startMs": 916079,
    "text": "fear voice agents more than crime is"
  },
  {
    "startMs": 918480,
    "text": "that one of these incidents is going to"
  },
  {
    "startMs": 920160,
    "text": "impact you. It already did for me."
  },
  {
    "startMs": 924240,
    "text": "Awesome. So it's time for me to shill."
  },
  {
    "startMs": 925920,
    "text": "Well, we burn a lot of tokens. If you"
  },
  {
    "startMs": 928000,
    "text": "are interested in working in this space,"
  },
  {
    "startMs": 931360,
    "text": "please come and talk to us. And if you"
  },
  {
    "startMs": 933680,
    "text": "are deploying voice agents and want to"
  },
  {
    "startMs": 935839,
    "text": "validate whether your architectured or"
  },
  {
    "startMs": 939120,
    "text": "your eval are set up correctly, please"
  },
  {
    "startMs": 940880,
    "text": "come and talk to us. We'll be outside."
  },
  {
    "startMs": 942639,
    "text": "And here's here's my number. Here's my"
  },
  {
    "startMs": 944000,
    "text": "WhatsApp. Thanks everyone."
  },
  {
    "startMs": 962206,
    "text": ">> [music]"
  }
]