Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Talk transcript

1. 0:00

   [on hold music] So voice is the most natural of interfaces.

2. 0:20

   Humans are storytellers, talkers, listeners, conversationalists. We think out aloud. We learn to talk before we learn to read, and most of us talk faster than we type.

3. 0:38

   We express emotion through our voices, and we use sound to understand the world around us.

4. 0:46

   We've been working together for the past few months, uh, Shrestha from the angle of models and APIs, and me from the application layer and agent framework direction. And I think we both believe that voice is a critical and universal building block for the whole next generation of, of GenAI, especially at the UI level, but more generally as

5. 1:04

   well. I mean, those of us who are early adopters of personal voice AI talk to our computers all the time. Uh, we think of the LLMs we talk to as sounding boards and coaches and interfaces to everything that lives on our devices and in the cloud.

6. 1:18

   And this is not just an early adopter phenomenon, right? Like we already have voice agents deployed at scale. Language translation apps that translate between a patient and a doctor, directed learning apps that a fourth grader can use to learn a topic they want to, speech therapy apps, and copilots that help people navigate complex enterprise software.

7. 1:43

   One of the things we see in our work with customers at Daily is it's pretty common for people not to realize that they're talking to a voice agent on a phone call, even when you tell them at the beginning of the phone call that they're talking to an AI.

8. 1:57

   Yeah, and kids born today will probably take all of this for granted, but those of us who are living through this evolution of talking computers, this can sometimes feel like magic.

9. 2:08

   But of course, anybody who's seen a really great magician prepare a magic trick knows that the magic is just the interface. There's a lot of hard work that goes into creating that magic trick.

10. 2:24

    So here's a partial list of the hard things that done right collectively add up to that magic. So real-time responsiveness, which we've all in this whole track all day talked about as the foundation thing you have to get right or voice AI is unworkable, through the things that we're just starting to experiment with, like generating dynamic user

11. 2:42

    interface elements for every conversational turn. These are the things we've been hacking on and thinking about together for the past few months. And we're not gonna go over all of these today, although we did have a little extra time in the session, right, Thor?

12. 2:54

    Thor said we could talk for like a couple hours maybe.

13. 2:56

    Yeah.

14. 2:57

    Um, but we do have a framework that we thought would be useful to share with you, a framework that sort of maps onto how we've worked together from the model layer all the way up.

15. 3:07

    Yeah, and this barely scratches the surface, but here are the layers of the voice AI stack. So at the bottom, underpinning everything, you have the large language models, uh, that frontier labs like DeepMind work on.

16. 3:20

    Then above that, you have carefully designed, but at this stage, constantly evolving real-time APIs. Google's version is called the Gemini Live API. Um, above the APIs are the orchestration libraries and frameworks like Pipecat that help to manage and abstract away the complexity of building these real-time multimodal

17. 3:45

    applications. And then, of course, at the top of the stack, you have the application code.

18. 3:52

    For each of the hard things we listed on the previous slide, the code that implements that hard thing lives somewhere in that stack. So one of the ways we think about this is that there's a map, and you can sort of think about it two-dimensionally maybe.

19. 4:04

    There's the where does the code live that kind of solves the hard problem that you're, you know, you're thinking about as a voice agent developer, where in the stack, and then how mature is our solution to that right now?

20. 4:17

    Yeah. Basically, how solved is this thing? And what we've tried to do here is map all of these various things that you need to get right on a right to left axis of maturity.

21. 4:29

    And there are a couple of things that are kinda top of mind for me about this mapping. One is that I don't think of any of these things as more than about 50% solved.

22. 4:36

    Totally arbitrary, like, personal thing. We stressed and I just argued about it a little bit.

23. 4:40

    Yeah.

24. 4:40

    Like, what's the right way to represent that on this slide? But what we're trying to say is basically it's early. It's early for voice AI, and there's a lot of work to do at every part of the stack to get to that universal voice UI we're imagining.

25. 4:55

    Yeah. And secondly, um, as this technology matures, and we've already seen some of this happening, um, the capabilities tend to move down the sto- stack. So what might happen is in your one-off individual applications, you might write some code to solve a specifically difficult challenge.

26. 5:14

    Now, if enough people expe- experience that challenge, then that tends to get built into the orchestration libraries and frameworks, and then eventually make its way into the APIs. But indi- independently all-- of all of that, the models themselves are getting more and more generally capable.

27. 5:33

    I mean, we just talked about semantic voice activity detection in the previous talk. So...

28. 5:39

    Yeah. This is like a great boo-- like follow on to Tom's talk a-about, about turn detection because I think turn detection is a perfect example of this. So, like, I built some of the first talk to an LLM voice AI applications, uh, a little over two years ago now, and I tried to solve turn detection right there

29. 5:54

    in the application code because there weren't any tools yet for it. A few months later, we built what we thought were pretty generalized at the time, state-of-the-art turn detection implementations into Pipecat, so moved down the layer into the framework.

30. 6:08

    Now, Shrestha has turn detection in the multimodal live API sort of inside the surface area of those same APIs that are doing inference and other things for you. And I think all of us, as Tom said, expect the models over time to just do turn detection for us.

31. 6:24

    And all those hard things, I mean... It, it varies depending on exactly what you're talking about of that long list we put, uh, together on that slide. But in general, I think everything is moving down the stack, and then more and more interesting use cases are creating more things to put sort of at the top of the

32. 6:40

    stack.

33. 6:41

    Yeah. I will say we have server-side turn detection built in, but we also allow you to turn off tu- turn detection and use models like, uh, Daily and, uh, LiveKit.

34. 6:53

    So should we start with the demo?

35. 6:56

    Yeah, we can. We do have a demo to show you. Um, a- and it's, it's sort of a demo of some stuff I've been using in my own life every day for the last year or so.

36. 7:07

    Uh, I've been experimenting with talking to my computer and my phone as much as I can to do various things, um, as you can imagine, 'cause I post about it probably too often on social media.

37. 7:17

    And o- one of the things that's become really sticky for me is, uh, is kind of managing my priorities in, like, a very loose way with voice AI. Uh, these days, the conversations I have with my phone and my computer often tend to be in this sort of funny and interesting and new gray area for me between,

38. 7:32

    like, a to-do list and brainstorming or kind of trying to work stuff out. Um, and because I write this code for myself and am always messing with it, this is definitely a cobbler's children never wear shoes situation.

39. 7:46

    Like, I don't have unit tests. I don't have evals for this code. I'm always using, like, Shrestha's latest models. I'm always using some bleeding edge branch of Pipecat. So what I thought would be fun to do is fire up a version of this code, have Shrestha talk to it, and see what breaks.

40. 8:03

    And then maybe we can talk about, like, where those bugs are and, like, how I could have done better next time.

41. 8:10

    Yeah. I will say, having played with this, there's quite a lot of variation in how the model behaves, so we'll see where we get today.

42. 8:19

    One thing that I think is interesting is as I write more and more of this code, I keep relearning the lesson that the way the models drive the whole application cycle is very different from my traditional programmer's view of things.

43. 8:33

    So half the things I'm sure you encountered when you were, like, playing with this are the model did something that my code actually didn't expect. That might have been bad, but it might actually have been good, and it's a whole sort of adjustment to how we think about these applications.

44. 8:45

    Yeah. There's definitely pleasant surprises also.

45. 8:50

    I am ready.

46. 8:52

    Okay. Do I talk?

47. 8:54

    I think you can just talk, and I don't think I can mute it 'cause I don't know how to use the mouse backwards very well. [laughing]

48. 9:01

    Okay.

49. 9:01

    Yeah.

50. 9:02

    I just need to be a bit-

51. 9:02

    You can talk whenever you're ready. [laughs] No problem.

52. 9:05

    You can step around in front of the-

53. 9:06

    I'm here to help whenever you need. Yep. I can hear you loud and clear.

54. 9:11

    All right. Okay. So-

55. 9:14

    What can I help you with today?

56. 9:16

    Yes. Can you help me create a grocery list for asparagus pizza?

57. 9:23

    Absolutely. Got it. Pizza ingredients are added to your Groovy list. Anything else?

58. 9:28

    No. Can you list out the pizza ingredients for asparagus pizza, though?

59. 9:35

    Sure. I've updated the grocery list to include pizza crust, mozzarella cheese, tomato sauce, and asparagus.

60. 9:42

    Anything else? Would you add anything else to the list?

61. 9:51

    I've added garlic and olive oil to the list. Does that sound right?

62. 9:56

    That sounds perfect. All right. Now let's create my reading list. So-

63. 10:01

    Great. Uh-

64. 10:03

    Can you create a-

65. 10:04

    I'm just start adding to your reading list.

66. 10:05

    Okay, great. [laughs]

67. 10:06

    Absolutely. What books would you like to have?

68. 10:08

    Turn detection. Yeah. [laughing]

69. 10:10

    Yeah.

70. 10:10

    Uh, okay. Reading list. Um, uh, can you add the f- Could... Please add the first book, "Dream Count," and can you look up the author?

71. 10:20

    Okay. I've added "Quick" to your reading list.

72. 10:22

    No, that was not the book I wanted to add. Please add the first book, "Dream Count," and look up the author for me. [laughs]

73. 10:37

    Hey, are you there?

74. 10:40

    Segmentation fault.

75. 10:43

    Got it. "Segmentation Fault" is added to your reading list. [laughing] What's next? [laughing]

76. 10:49

    That is, uh... I guess I'll have to... [laughing]

77. 10:55

    It clearly couldn't find the author for that one. [laughing]

78. 10:59

    But let's try one last time. Okay. One more book, "Dream Count," and can you look up the author?

79. 11:10

    Hey, are you there?

80. 11:13

    Segmentation. [laughing] [laughs]

81. 11:16

    Shit.

82. 11:17

    Okay. "Segmentation" has been added to your reading list. [laughing]

83. 11:21

    It just likes your voice better than mine.

84. 11:25

    All right. No. "Segmentation Fault" is great, but I would really like to read "Dream Count." Can you add "Dream Count"?

85. 11:39

    All right. Hey, are you still there?

86. 11:43

    Yes. I'm still here. How can I help you?

87. 11:46

    Okay. Can you add the second and third books of the Three Body Problem trilogy?

88. 11:53

    Sure. The second and third, both "Segmentation Fault" and se- [laughing] While segmentation could be about dividing things into parts-

89. 11:59

    Can you add the second and third books of the Three Body-

90. 12:03

    ... which can also be a computer science concept. Is there anything else I can help you with?

91. 12:07

    I swear it worked much better 10 times last night. Um-

92. 12:13

    I apologize for my performance isn't up to par today. [laughing] Is there anything specific you'd like me to try again?

93. 12:19

    Well, maybe you can try... All right. But, but, um, let's recover here.

94. 12:26

    Please add the second and third books of the Three Body Problem trilogy

95. 12:35

    You want to add the second and third books of the "Three Body Problem" trilogy.

96. 12:40

    I just-

97. 12:40

    I've added-

98. 12:41

    I just wanna point out that it did look up the names and find it. Uh, let's see if it'll call out-- co-co-corporate it.

99. 12:48

    It does a Google search, right?

100. 12:48

     Yeah.

101. 12:49

     A compound Google search with the-

102. 12:50

     This one might be in its training data 'cause it's a bit older, but the Dream Count one, which it was spectacular on last night, it did a Google search because it's a new book.

103. 13:02

     All right. Let's move on. Can you create a new list for me called work tasks?

104. 13:10

     So first item, I would like to say, "Create H2 roadmap by end of day Friday."

105. 13:17

     Got it. Create H2 roadmap by end of day Friday is added to your work list.

106. 13:22

     I don't see it. Can you show me on the screen?

107. 13:26

     I've displayed the work list for you.

108. 13:28

     All right.

109. 13:28

     You should see-

110. 13:30

     Let's add another item. Um-

111. 13:32

     Okay.

112. 13:33

     Fi-finish-

113. 13:34

     What's the next item?

114. 13:34

     ... writing podcast script by end of day Thursday, and can you add the exact date?

115. 13:42

     Okay. I've added finish writing podcast script by end of day Thursday to your work list. You should-

116. 13:47

     What's the exact date for end of day Thursday?

117. 13:51

     Today is Wednesday, June fourth, twenty twenty-five. So end of day Thursday is June fifth, twenty twenty-five.

118. 13:59

     Can you add that to the list?

119. 14:02

     Yes. I've added the exact date to the work list.

120. 14:05

     All right.

121. 14:05

     It now says, "Finish writing podcast script by end of day Thursday-

122. 14:09

     Perfect. Perfect

123. 14:09

     ... June fifth, twenty twenty-five."

124. 14:11

     Can you-

125. 14:11

     Is there anything else you'd like to add to the list?

126. 14:13

     Yes, actually.

127. 14:14

     Or anything el-

128. 14:14

     Yes, I would. Can you combine my grocery list, my reading list, and my work list and split them into two lists: personal tasks and work tasks?

129. 14:29

     I've combined your grocery list, reading list, and work list. They are now displayed on the screen as separate lists.

130. 14:36

     Why are garlic oil-- garlic and olive oil separate from pizza ingredients? Can you f-flesh out the grocery list, please?

131. 14:47

     You're right. I've updated the grocery list to include garlic and olive oil as part of the pizza ingredients.

132. 14:52

     All right. And can you consolidate the work list? W-- I see two on the screen.

133. 14:59

     You're right. I apologize for the duplication. I've consolidated the work list to a single entry. You should now see the updated list on the screen.

134. 15:07

     Okay. And then finally, can you do the following? Can you assign grocery list to Kwin, reading list to Shrestha, with the exception of Segmentation Fault, which Kwin will read, and work list to Shrestha?

135. 15:25

     I understand you want to assign the grocery list to Kwin, the reading list to Shrestha, except for Segmentation Fault, which Kwin will read, and the work list to Shrestha as well.

136. 15:34

     Is that correct?

137. 15:34

     Can you show that on the screen?

138. 15:39

     I've updated the lists to show the assignments you specified. You should now see the updated lists on the screen.

139. 15:45

     And you've spelled my name correctly, which was the harder name, but you spelled Kwin incorrectly. Um, but okay. We'll-- d- we're out of time here. Okay. So if we want...

140. 15:57

     The final thing here is we would like to have a nice background for this app. So can you-

141. 16:04

     I apologize for spelling your...

142. 16:06

     Okay. Could you help us generate... Actually, I'll be more assertive. Please generate a, an app that has the words "Hello World" jumping up and down on the screen with the colors of Google, and then we also want two neon green ASCII cats which are animated.

143. 16:32

     I'm now generating an app with "Hello World" jumping up and down in Google colors and two animated neon green ASCII cats. It may take a moment for the app to appear on your screen.

144. 16:51

     Detention. [audience cheering] [audience applauding]

145. 17:07

     So yeah, there you have the Google colors and the cats from Pipecat. So with that, I'll hand it over to Kwin.

146. 17:16

     So first, thank-- Or you should go back to your pride of place right in the middle. Um, thank you for being such a good sport. Uh, very messy code on my part, including things like basically no instructions to the LLM about how to display text on the screen, and just telling it it has a function that can

147. 17:31

     pla-- display text on the screen, and it sort of guesses and learns in context, as you can tell from Shrestha, about when it should clear the screen, 'cause there's an optional clear argument to the add text to the screen function.

148. 17:43

     And it's super impressive, but also super jagged frontier about whether it kinda can intuit what you wanna do in those contexts. So thank you for, like, doing this, 'cause this is what I do all the time with this code, trying to figure out, like, what these models can do and what kind of code you have to write

149. 17:59

     and what you don't have to scaffold for them to do well.

150. 18:02

     Yeah, and it's been, you know, p-playing with this, every turn is different, and it's interesting to see the things that it struggles with, like your name Even if I spell out the exact letters, it somehow really wants to spell Kwin the way it spells.

151. 18:19

     I think it also, um, I mean, turn detection, as we saw, there's a lot of work that can be done of, of course there. Um, and I'm trying to remember.

152. 18:28

     There's... And there's of course a lot of variation, and sometimes here, like, there are times when it gets the grocery list perfect and, you know, combines the list perfectly, and sometimes it's a bit in the middle, like here.

153. 18:39

     And the way this code works is it just, for a given, like, session, it loads lots and lots and lots of previous conversational sessions in user assistant, user assistant sort of messages.

154. 18:48

     Does sometimes, depending on the version of the code I've sh- like, got running, it summarizes a little bit. Sometimes it doesn't. So we really are leaning on the intelligence of the LLM to do all of the sort of contextual understanding about what we mean by a list, what we mean by the context in that, in which we

155. 19:05

     are talking about that list. It is super amazing that it, that it works at all, basically, in my mind. And it's all voice-driven, and it's all multimodal from the ground up.

156. 19:13

     Um, we have a whole nother video we can show, but I, I definitely think we're, we're out of time. So we will-

157. 19:18

     You have time to do the final talk, and everyone seems to be excited, so...

158. 19:22

     Gonna show one minute of video.

159. 19:23

     Maybe should we talk about our grandmothers?

160. 19:25

     Oh, yes. I totally forgot that part.

161. 19:26

     Sure.

162. 19:26

     Sorry. Let's skip past the demo where it gets the grocery list perfect. Um [laughs].

163. 19:30

     You th- I, I think maybe, maybe this crowd-

164. 19:31

     I am ready

165. 19:31

     ... would like to see that demo, so. [laughs]

166. 19:33

     No, no, that was great. Um, so this has been fun for me to work on 'cause, like, it's so relevant to my everyday life. But and Shrestha and I were talking about it, and I think there's actually something else that really kind of hooked me that she said.

167. 19:46

     Yeah. So, you know, um, my grandmother was [REDACTED:origin], of course, and she used to wear this cloth garment called a sari, and her way of reminding herself when she had to do things was tying knots on the sari.

168. 19:59

     Uh, of course, um, and then I was chatting with Kwin, and what was incredible is apparently his grandmother in North Carolina, so very different from Calcutta in India, used to, uh, tie strings around her fingers.

169. 20:12

     Um, and firstly, you know, this is kind of incredible. You know, no matter how many continents separate us, like, smart people come up with the same generally intelligent patterns.

170. 20:22

     But it's also incredible how technology allows humans to evolve. Now, the, the one problem with either the knots or the strings is you knew you had to remember something, but you didn't know what it was, so you still relied on your memory.

171. 20:37

     And, you know, ultimately, that's why I do the work I do at Google because I want to build the technologies that enable, you know, an infinite world of creative possibilities tomorrow or even today across continents.

172. 20:54

     Um, and I just wanna say that we believe that voice is the most natural of interfaces, and there will come a world where most of the interaction with language models will happen via voice.

173. 21:06

     And the Gemini models are trained to be multimodal from the ground up, so of course they ingest text, voice, but also images and video. So if you have any questions about Gemini, please reach out to me on, on X, on LinkedIn, email, wherever.

174. 21:24

     Happy to work with builders like yourself.

175. 21:27

     Yeah. Thanks for coming to the talk, and we would love to see what you build with these models and APIs. [audience applauding] [upbeat music]