Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Hultman Kramer (Daily)

Talk transcript

1. 0:00

   [upbeat music] All right, Squabbert, you ready to get packed up?

2. 0:17

   I don't know. I'm pretty nervous.

3. 0:20

   Oh, relax. You got nothing to worry about.

4. 0:22

   But this is, like, the worst idea ever for a live demo. A live, unscripted conversation with a non-deterministic LLM on conference Wi-Fi?

5. 0:32

   Come on, your prompt is great, and I mean, it's gonna be to-

6. 0:34

   With onstage audio in a room full of echoes? Sometimes my text-to-speech even mispronounces my own name. Why did you even name me Squabbert? What if I say Squibbly or something again?

7. 0:46

   Okay. [sighs] Take a breath. Well, you don't really do that. Let's just take it one step at a time. Just start with the intro.

8. 0:53

   I guess you're right. Okay, here I go. Hi, everybody. I'm Squabbert. Here to take us on a whirlwind tour of the wonderful world of WebRTC, please welcome Sean and Kwind.

9. 1:07

   Hey, I'm Sean. I work on WebRTC at OpenAI. Um, some of the things you might be familiar with are the Realtime API or 1-800-CHATGPT. You can call it right from your phone.

10. 1:17

    Um, before I worked at OpenAI, I worked on the Go implementation of WebRTC called Pion.

11. 1:23

    And I'm Kwind. I work at Daily on real-time audio and video infrastructure, and on an open source voice agent framework called Pipecat. Uh, today we're gonna talk about how to build natural, fast, human-like voice experiences.

12. 1:35

    Uh, we're gonna give you a crash coise-- course on low latency audio and video, and I hope we'll show you a couple of things you might not have thought of around voice AI before.

13. 1:47

    Um, if you wanna build a conversational voice experience that people really love, you re-- you're gonna stress a lot about latency. Nothing else matters if your AI responds too slowly.

14. 1:56

    Building voice AI experiences is similar to other kinds of AI engineering in most ways. If you've built multi-turn agents, a lot of that will port over to building voice agents.

15. 2:07

    But the big difference is latency.

16. 2:11

    Everything in a voice AI app needs to be groun-- built from the ground up for fast response times. If you're talking to a person, around 500 milliseconds sounds natural.

17. 2:20

    When talking to an AI system, people bring those same expectations. Response latencies much above a second, in general, doom your voice agent to very low completion rates and n-- low NPS scores and hang-ups.

18. 2:32

    And we're talking here about voice-to-voice latency, so this is the time, uh, between when I, the human, stop talking and the time I hear the first audio byte come back from the LLM.

19. 2:46

    Let's take a look at how latency adds up in a typical voice-to-voice AI application. So this is a breakdown from a real voice AI app running in a web browser on macOS, talking over the internet to a voice agent running in the cloud, running on Pipecat.

20. 3:02

    A couple of things to note. Our voice latency is just under a second. That's good, but not great. We can make things a little faster, but that comes with trade-offs, lower quality or cost.

21. 3:13

    Second, it's frustratingly easy to do even worse than this. Your LLM might be slower, other things get in the way, or worst of all, Bluetooth.

22. 3:21

    Don't get me started on Bluetooth.

23. 3:25

    But the single biggest mistake we see people making is using the wrong approach to sending and receiving audio over the network. It's time to talk about WebRTC and WebSockets.

24. 3:35

    If you're new to building voice applications, you probably think, "Hey, I need a, like, a long-lived, uh, connection. I'm gonna send audio and video over this long-lived connection. I've used WebSockets for long-lived data connections before.

25. 3:48

    I'm just gonna write some WebSocket code." That's great if you're doing long-lived, short, small amounts of data. It doesn't work for real-time audio. In fact, WebSockets are almost the opposite of what you want from a network engineering perspective for real-time audio and video.

26. 4:05

    So let's do a compare and contrast on this. WebSockets are great if you're trying to deliver audio and you want something really easy that can target all platforms. If you're trying to build a prototype, lots of different platforms, WebSocket's the way to go.

27. 4:19

    On the other hand, WebRTC solves a bunch of things around handling, giving you high quality audio, high bandwidth, low latency. But the, the catch is it can be a lot more complicated to implement, and, um, it specializes, but it can be frustrating.

28. 4:36

    Lots of applications use both, but for different things.

29. 4:40

    So here's the TLDR if you only remember one thing from this talk. Use WebSockets for those server-to-server use cases and small amounts of structured data in places that you want to prototype.

30. 4:50

    Use WebRTC if you're sending audio and video streams over the internet from your web app, your native, um, that's where it excels.

31. 4:57

    So why is it so important to use WebRTC for real-time edge-to-cloud audio? A WebSocket is a TCP connection. TCP guarantees in order delivery of network packets. If you send some data, that data's gonna arrive exactly as you sent it, or it's not gonna arrive at all.

32. 5:13

    You put packets in your operating system's send queue. That O-OS queue is gonna keep trying to send them until they either get ACKed by the other side or your connection completely times out.

33. 5:23

    And this is in general what you want if you're doing most network programming. If you're making a web request, for example, this is perfect. It's not what you want if you're aiming for conversational latency.

34. 5:34

    Remember that we're trying to hit a voice-to-voice latency of under one second, and ideally even better. What we want to ignore is things like the occasional packet loss. So imagine if a packet is dropped, I don't really care about what happened a second ago.

35. 5:48

    So WebRTC does clever math and buffer management that we're gonna talk about more to hide where that happens.

36. 5:53

    So this is the first and most important thing WebRTC does for you. It's all that machinery that sends packets as fast as possible, ignores packets that don't arrive inside that very tight latency budget we're operating within.

37. 6:05

    You can think of it as, like, super fast, best effort networking If this were all WebRTC could do for you compared to WebSockets, it would still be worth using WebRTC just for this, because you literally can't implement this on top of a TCP stack or on top of WebSockets.

38. 6:20

    Again, the operating system is just gonna try to keep sending whatever you tell it to send. It's gonna block everything if you have any packet loss or significant sort of jitter or delay in the network.

39. 6:29

    And in real world, we have lots and lots of real world data on this. In the real world, this means you will get audio glitchiness or high latency or unexpected socket disconnections in 10 to 15% of your network connections.

40. 6:42

    But WebRTC does a lot more than that. Um, if you go and you try to build the same application in WebSockets, you have to handle resampling, you have to handle packetization and doing all that, bandwidth estimation.

41. 6:54

    Networks are constantly changing and fluctuating, so you can't just send one bit rate. Um, and you also get standard APIs for getting the stats and observability. This is all just built into WebRTC, but if you decide to do WebSockets, you have to build it yourself.

42. 7:08

    So if you look at this code up on the screen, on the right side is an example of WebRTC sending one bidirectional stream of audio. On the left side is WebSockets.

43. 7:17

    And you wanna spend more time building your application and less time worrying about things like sample rates. That's why you pick WebRTC.

44. 7:25

    And this is real code using your OpenAI Realtime API.

45. 7:27

    Yep.

46. 7:27

    And which you, you offer both, both options for developers.

47. 7:31

    Yes.

48. 7:31

    So I hope we've convinced you that you should use WebRTC if you're doing edge-to-cloud audio, oh, and especially audio and video. Um, we love talking about this stuff. If you come find us later, we will talk your ear off about jitter bu- buffers and packet management and bandwidth shaping and all that stuff.

49. 7:45

    But, uh, what we, we wanna do now is move on and talk about a whole nother category of fun stuff, which is what you can actually do with WebRTC.

50. 7:52

    Um, I'll start by saying you can embed real-time audio in any app you write, any website, any iOS app, any Android app, lots of fun embedded stuff, and the network connec- connections will just work.

51. 8:02

    You will get good audio on any device, any platform, almost any real-world network connection.

52. 8:08

    And I bet you use WebRC- WebRTC today already if you used Facebook Messenger, WhatsApp, Zoom, Discord, you know, any of these applications, they're using WebRTC. Um, but you didn't know that there's even more cool things happening with WebRTC.

53. 8:22

    Um, I worked with a company that was doing surgery over the internet. Um, people will teleop, um, con- vehicles in the field. It's super cool. Um, WebRTC is kind of the standard language of the real-time world.

54. 8:34

    Um, and that's why it makes so easy that we can go build conversational intelligence on top of it. Um, all the stuff's already been solved.

55. 8:40

    I mean, in the new LLM era, we know lots of people who spend hours talking to computers, uh, driving their development environments with voice, doing brainstorming, treating the computer as a personal assistant, a coach, a therapist, a researcher.

56. 8:53

    Uh, I'm convinced voice is going to be the core building block of the next generation of UIs, of the UIs for the generative AI era. We have to do a little iteration before we figure out really what those UIs look and sound like, uh, but that future seems very clear to me.

57. 9:09

    One of the things I say to people to kinda try to communicate how excited I am about building all this stuff is, we all lived through the last platform shift.

58. 9:17

    It was recent enough that the move for most computing from desktop to mobile happened within our memories. And so by analogy, we are in late 2007 now. We have the first iPhones, but we haven't yet invented pull to refresh.

59. 9:35

    What keeps me so excited and motivated is I feel like w- uh, voice is like the next, um, bicycle for the mind. You know, today we've only been able to use computers with our eyes and our hands, but now I can talk to it as well.

60. 9:46

    Um, think of all those situations where, like, you c- you have your voice available but you don't have your hands, um. And the other great thing is you can keep all this computing power remotely and access it and have small devices near you.

61. 9:59

    So on the small devices front, we brought a friend with us today. Uh, he's a little shy sometimes, so give him just a second.

62. 10:15

    So, uh, Squabert, can you tell people a little bit about your tech stack?

63. 10:21

    Hang on one second.

64. 10:23

    This is Squabert, who Chad built and Chad's daughter, Ella, named Squabert.

65. 10:38

    Squabert, can you tell the people a little bit about your tech stack?

66. 10:44

    I'm a friendly stuffed animal running a web interface locally on a Raspberry Pi. I'm connected directly to a Python process running on a laptop using MLX Whisper, Gemma 3, and a mildly buggy custom Logit sampler that Quinn wrote.

67. 10:57

    I mean, come on now. Like, I had a good reason that I wanted to write a Logit sampler, and I, I, I wanted you to be able to do something that even the really amazing big LLMs running in the cloud can't do.

68. 11:09

    Do you know what I'm talking about?

69. 11:13

    You're talking about my syllable counting talent. It's pretty neat, even if I do say so myself.

70. 11:19

    I mean, counting syllables is like the 2025 version of how many Rs are in strawberry. And, you know, Squ- Squabert, why don't we, why don't we demonstrate that a little bit?

71. 11:29

    Can you make up a poem, like a four-line poem about computer programming, and use only two-syllable words?

72. 11:40

    Two. Okay, listen. Logic, coding, people, knowing. Systems, working, future, growing.

73. 11:51

    One more maybe? Uh, that was good, but I've heard you do better.

74. 11:57

    Okay, here's another one. Pixels dancing, colors bright. Program running, feels so right. Coding faster, day and night. Making magic, pure delight.

75. 12:06

    You forgot about the two-syllable words thing, but we'll, we'll, we'll let it go. Yeah, uh, definitely the first one had two-syllable words. I think, I think we should say goodbye just in the interest of time, Squabert, but thank you so much[audience applauding]

76. 12:23

    I think Squabert did really, really well. Um, so that, that is a Raspberry Pi connected with a peer-to-peer WebRTC connection directly to my laptop over the same local area network.

77. 12:36

    So that was the serverless WebRTC connection. Squabert's talking directly to the laptop. But what's super cool about WebRTC is you have all these different choices to how you want to connect to things.

78. 12:45

    So you could do this local connection, or something like Squabert could connect to a server, um, up running on another... in the cloud and do all the AI stuff.

79. 12:55

    And then the third option is you can go and connect up to something like Pipecat and make it multi-party, so bring LLMs into meetings or other places like that.

80. 13:03

    Um, super awesome how, like, you get all this flexibility to build these things the way that, like, matches your application.

81. 13:09

    So we want to close the video with actually a builder who's in the audience. Um, super excited. And, um, what's the best part about this is that, um, when she built this, she had never written any code before this.

82. 13:21

    And so what makes me so excited about the future of voice is if we can make this easy enough, people that have really innovative, inspirational ideas can go and do stuff themselves.

83. 13:32

    Hi, I'm Yaxin. I'm a mom of two bilingual kids.

84. 13:37

    I'm raising my kids bilingual because I want them to connect with my cultural roots, and that's a wish shared by many parents of bilingual children. But raising bilingual kids is hard.

85. 13:48

    It's expensive, it's time-consuming, and it often feels like a big chore on both the kids and parents.

86. 13:56

    I believe we can do a lot better. With today's technology, I really believe we can make language education feel more natural and even fun for the kids. Here's a quick clip of what I've been working on.

87. 14:11

    Hi there, buddy. Ready to have some fun today? [speaking Mandarin] [speaking Mandarin]

88. 14:17

    How about we start with something fun? Let's say hello. In Mandarin, we say [speaking Mandarin]

89. 14:31

    . Can you say [speaking Mandarin]? [speaking Mandarin] Okay, it's still early, but I'm excited about what's possible. I'm not technical, but with just a little bit of guidance from a kind member of this community, I was able to bring this first version to life. [audience laughing]

90. 14:51

    I already have a group of eager testers, mostly parents like me, who are excited to try this.

91. 14:58

    If this also sounds exciting to you, I would love to connect. Thanks so much for watching, and let's build this future together. [audience cheering]

92. 15:13

    So we put the QR code up here for Yaxin's project, which I absolutely love. She's here. Uh, if you're interested... If you're here in the audience and, uh, you're interested in multilingual stuff or building these kind of things, please find her.

93. 15:26

    Such a great, great, great thing. Um, if you're watching on YouTube, here's the QR code. Sean and I are super excited about these kind of projects. And, I mean, the, the person Yaxin shouted out in the video is, of course, Sean, who has done more than anybody I know to make WebRTC accessible to everyone.

94. 15:43

    Um, the idea that we want to leave you with is if you have an idea in voice AI and WebRTC, whether you've been a programmer for years or you're just getting started, like, we are here to support you.

95. 15:53

    We're so excited, and we believe that things like Pipecat and libpeer, like, if we can make this easier, we're gonna see the next generation of really exciting, innovative projects.

96. 16:02

    So come find us in the hallway or online. We hang out on Discord and Twitter and LinkedIn.

97. 16:07

    And, uh, here are the resources that you can scan and, uh, hopefully they find helpful. So Quinn wrote an amazing book that I believe is in your bag. Um, and then, uh, yeah.

98. 16:17

    So thank you so much.

99. 16:18

    We can't wait to see what you build. [audience cheering] [upbeat music]