Full Workshop: Realtime Voice AI — Mark Backman, Daily

Talk transcript

1. 0:00

   [on-hold music] I'm Mark with Daily.

2. 0:16

   This is Aleix.

3. 0:17

   Aleix.

4. 0:18

   And then we have a few other Daily folks, uh, Quinn, Nina, Varun, and then I'm not sure where he went, but Philip from... Right there. From the Google team, Google DeepMind team.

5. 0:30

   So this session, we're going to spend just a few minutes getting everyone started. The idea here is it's going to be a hands-on workshop where all the folks I just called out are gonna be available to help out.

6. 0:40

   We'll walk you through a quick start to get you up and running, and then the idea is to build something. So build a voice bot in the next 78 minutes and 12 seconds or whatever time we have left.

7. 0:51

   Um, the one... I guess there's one, uh, consideration is the Wi-Fi. If you don't have good Wi-Fi, you might wanna try to tether. I was able to tether, and it worked fairly well, but the conference Wi-Fi was a little shaky.

8. 1:03

   This is real-time, so you will be streaming data. It does require a viable connection and not just sending, you know, a few bits over. So just a heads-up if you hit that as a, as a snag.

9. 1:15

   So I guess before I get started, who here knows about Pipecat or has built anything with voice AI? Okay.

10. 1:25

    A smaller audience. Has anyone built any real-time applications with LLMs or AI?

11. 1:32

    Maybe slightly bigger. Okay. Great. So Pipecat, um, is a, uh... It's an open source repo. It's a Python framework for building voice and AI multimodal agents. And it's built by the team at Daily, um, but we're an open source.

12. 1:49

    It's an open source project that, uh, anyone can contribute to. It's been around for, I don't know, just over a year now, and-

13. 1:57

    Yeah. Like, I would say officially Pipecat was March 2024.

14. 2:02

    Okay.

15. 2:03

    Something like that.

16. 2:03

    So 13 months.

17. 2:04

    So over a year, yeah.

18. 2:05

    There we go. So just a quick, uh, walkthrough, maybe just to kind of ground everyone in the thinking around voice AI. Uh, these slides weren't built for this talk, but I'm gonna use them.

19. 2:15

    So the, you know, voice AI or v- real-time applications are tough because there's just... You know, we as humans communicate all the time with each other, thousands, ten thou- tens of thousands of years of evolution baked into our brains.

20. 2:26

    So it, it's pretty tough to make a machine, you know, work on, on the same level. So we have great expectations being the user in it. So, you know, you need a good listener, somebody that's smart and conversational.

21. 2:37

    You need to be connected to data stores. Uh, it has to sound normal or natural. Think back to even just maybe two, three years ago what your, you know, voice bots sound like.

22. 2:46

    And many of them on, if you call them on the phone, still sound like... It needs to sound natural. And actually, kudos to the Google team. The latest, uh, Gemini Live, uh, native audio dialogue is quite good in that regard.

23. 3:00

    It has to be fast. So the whole end-to-end communication needs to happen in roughly, you know, kind of the, the benchmark is around 800 milliseconds. Um,

24. 3:09

    y- you could strive for more. I think we see, uh, maybe on the human level, it might be 500 milliseconds or somewhere on that, uh, order. So it is pretty fast.

25. 3:17

    So there's a lot to kind of, to get all the way there. And this is something that we at Daily, everyone building Pipecat has been working very, very hard on, uh, getting all the way to meeting all these expectations.

26. 3:29

    So just to kind of ground you in some of this since we're going to be working in Pipecat, Pipecat has a pipeline. I don't know if maybe, Aleix, you wanna talk a little bit about the origin of that quickly.

27. 3:38

    Sure. Sure. Uh, you can think about it as a multimedia pipeline. And you would think, "What is a multimedia pipeline?" It's basically just think about, like, boxes that receive input, and input could be audio or video.

28. 3:51

    And then those boxes just will stream those, uh, same data or modified data or new data to the following, uh, to the following elements or processors. In, in Pipecat, well, we call them processors.

29. 4:05

    So in, in Pipecat, you would have a pipeline where you have a transport, which is the,

30. 4:13

    the inc- the transport of your data or the input of your data. For example, when you're talking, [clears throat]

31. 4:19

    you could be talking in a, in a meeting, so that would be the audio of the user. Then you would have, uh, another box following that, which is the speech-to-text service.

32. 4:28

    So y- the speech-to-text service would, would grab, uh, audio from the user. It would tran- uh, transcribe it. Then you would get text. That would be the following, uh, data that goes through the pipeline.

33. 4:40

    And then the next one would be the LLM. So now the LLM has what the user has said. [clears throat]

34. 4:46

    And then it generates, uh, output, whatever, uh, whatever the LLM. That would be tokens. Then those tokens are co- are, are converted into, um, text-to-speech. And then the text-to-speech outputs audio, and then the audio goes back to the transport, so you could hear what the LLM has said.

35. 5:05

    Today, uh, what we're gonna do today with Gemini Live, a lot of those boxes go away because, uh, um, the LLM will do a bunch of the things. It will do transcription.

36. 5:16

    It will do the LLM. It will do the text-to-speech in, in one of these boxes. But you might still require, for example, if you wanna save the audio or record the audio into a file, you need a bunch of utilities to do that, and Pipecat has all that built for you.

37. 5:31

    Um, so basically-

38. 5:33

    Yeah

39. 5:33

    ... that's, that's it.

40. 5:34

    I mean, well, a lot, a lot of this is really just or- it's orchestration. So if you think about what Pipecat offers, it's orchestration. It also offers a lot of abstractions for a lot of common utilities like Aleix had said.

41. 5:44

    So recording, uh, transcript outputs, artifacts you might wanna produce, or even ways that you might manipulate the information in the pipeline itself. So this, uh, image here, which is what Aleix actually just talked through, is what you would call, I guess, a cascaded model, where you have This flow through of information.

42. 6:02

    So we're going to be-- You can build with Google and many different services in this way. Uh, in the last year, there's been an emergence of speech-to-speech models that now take audio in natively and audio out natively, and those models also allow for audio in and then optionally text and/or audio out.

43. 6:19

    So you can actually, for example, take a raw, you know, microphone input or, uh, you know, audio input, and then

44. 6:28

    the model would run all of its logic, and you can actually opt to have it output text if you wanna, say, parse the text output before speaking. So there are a few different demos we'll look at that offer that.

45. 6:39

    Uh, and in Pipecat, we show kind of all the ways to do things 'cause that's, that's what we offer, or at least what Pipecat offers as a, as a value proposition.

46. 6:47

    So the, um-

47. 6:48

    Yeah, just one thing. I don't think-

48. 6:49

    Yeah

49. 6:50

    ... you mentioned it in the slides, but all these boxes you can plug and play the service you want in Pipecat. So the speech-to-spe-- uh, speech-to-text, it could be, I don't know, Deepgram, for example.

50. 7:01

    The LLM could be, uh, Google or OpenAI or whatever. You ju- can just plug and play any service you want.

51. 7:08

    Right. Yeah. The modularity, I guess, is the other big strength. So there's no... You know, you can change out a service without changing out your underlying application code, which makes it easy.

52. 7:16

    And we see with this a lot of companies that are building for voice AI might have, uh, maybe even a more complex thing. A pipeline here runs straight down, but you can actually have split branches where you might have one leg that's running some logic and the other running different.

53. 7:30

    We call that a parallel pipeline. So if you wanted to have, say, a failover, if vendor A goes down, you can move to vendor B dynamically, w- even within the same conversation.

54. 7:41

    That's something that Pipecat affords as well. Um, and that can allow you to transfer contexts over. So a lot of really cool stuff. Uh, the goal again today being to get you familiar with just building a voice agent and building one to get started.

55. 7:56

    So one of the cool things, um, like Aleix had pointed out, that with the cascaded models, there's a lot of complexity, but with your speech-to-speech model, things get, you know, dramatically simplified.

56. 8:08

    You know, your code may have looked something like this. This is, like, an old example of some, like, a ton of, uh, orchestration in the pipeline. But with a speech-to-speech model, you may be able to simplify it down to this, but then you have to remember you actually need orchestration around it.

57. 8:24

    So it does get simpler to some regard. Um, so it's, it's more about the services you interface with. Uh, I think with that, why don't we transition now, 'cause I'm realizing we have only about maybe 70, 70, 75 minutes left, to looking at, um, the actual activity for today.

58. 8:42

    All right. So there's a public repo. I don't know how big or small this is, but it's under Daily Co. So on GitHub, Daily Co, daily-[REDACTED:url]-workshop. I'll give everyone a chance to make sure your internet's working and-

59. 8:59

    Can everyone see the, the repo URL?

60. 9:00

    Yeah, it's really tiny. Yeah. Okay. I should have it in big text somewhere. Oh, no. Not really.

61. 9:10

    No. Here we go. All right.

62. 9:30

    Let's take a look at this. So what, what I want to do... So there are-- I spent a little bit of time, Aleix and I spent time writing up this repo.

63. 9:37

    This is meant to be just a jumping-off point. I'm gonna get you oriented, and then I want to look through one of the bot files, which is kind of the main Pipecat code with you.

64. 9:46

    And then we'll break and make this, uh, an interactive session where we can answer a bunch of questions. So in the repo, um, you could either start doing it now or maybe take a pause, but this will give you the steps to walk through getting the quick start running.

65. 10:01

    Before we do that, I wanna take a moment here. Let's see how the Wi-Fi's doing. That's not a good sign.

66. 10:11

    Uh, Wi-Fi is down. It's down? Ah. I take comfort in no Wi-Fi. Well, hey, you know, it is tough to do Wi-Fi for this many people.

67. 10:19

    Yeah.

68. 10:19

    Very, very tough.

69. 10:19

    Instead of real-time, it's gonna be real slow.

70. 10:22

    Real slow. Yeah. [chuckles] Real slow voice communication. All right. Well, we're using WebRTC.

71. 10:27

    Yeah. [chuckles]

72. 10:28

    Yes. Yeah. There will be some... Okay.

73. 10:31

    Yeah.

74. 10:31

    So in, uh... Here is this gemini, uh, bot.py files. This is all Python again, so we... Everything will be in Python. There will be some client code options which, which we'll look at in a second.

75. 10:43

    Um, just to orient you, I'll just jump right into the meat of the pipeline. We have this main, uh, function that runs your bot. Everything's going to run kind of encapsulated within an AIOHTTP session.

76. 10:57

    We're gonna pass that session around. That's more of just kind of the mechanics of things. In our pipeline, let's just jump to the, the simple part here. We'll have Daily as the transport.

77. 11:06

    Daily is-- We are a WebRTC provider as well as we build Pipecat. Uh, there will be context aggregation. So one important note, when you speak, uh, every turn of the bot is a, like a, a discrete point in time.

78. 11:22

    And this is maybe less so the case for a speech-to-speech model, but for basic LLMs, they get discrete inputs. Everything is like a REST API call, so you're going to get a snapshot of the conversation.

79. 11:33

    The context aggregator is going to collect the, all the bits of the conversation, both from the user and the assistant, uh, and will put that into the form of what the LLMs can handle.

80. 11:44

    So this, in this case, is more for function calling and kind of logistics and management. Gemini is amazing because it offers a lot of this for you. But if you're to build with, say, the, uh, just build with Gemini not live, the actual kind of just the text-based LLM, you'd have to have this context aggregation Uh, that

81. 12:05

    will then go to your LLM, which is going to be Gemini Live, Gemini Multimodal Live, and then it's going to be outputted through, uh, Daily again on, on that side of the transport.

82. 12:16

    So you have Daily. We'd configure our service, which takes a number of arguments. Like, you set up a room with a token, give it a name, and then have some properties.

83. 12:25

    There are docs which I'll link to, uh,

84. 12:29

    in the-- It's linked in the quick start. There's also a Gemini Multimodal Live LLM service, which is a Pipecat class that is a wrapper around the Gemini Live API.

85. 12:39

    So this, again, you just initialize and run. With the LLM, you see we do a few special things. We're going to define tools. This one has just two really basic kind of canned functions.

86. 12:51

    Fortunately, we're not calling out to the internet 'cause it's not working very well. Um, our connection's not working well. So this one just has the dummy, like fetch weather, and we'll give you a restaurant recommendation.

87. 13:01

    So these are two handlers that when your function is called, we'll just return this result information. Uh, so we have the actual functions themselves that are defined in this function schema, which is you can use just native function, uh, definitions using whatever LLM format.

88. 13:20

    We also have created this function schema, which is a universal schema that lets you define, uh, and move between any LLM without having to kind of transform your LLM calls from OpenAI to Anthropic to Gemini to Bedrock, 'cause they're all a little bit different, or Groq.

89. 13:36

    You know, they all have slightly different formats. So this is more of kind of a universal transform for that.

90. 13:42

    And then they're collected and trans-translated into the native format in this tool schema. So we'll pass the tools then to the Gemini service, and that's how it gets access to use and run those tools.

91. 13:55

    There's also a prompt above which I think in this simple example, we just say, "Hey, you're a chatbot. You have these tools available." Uh, and that's that.

92. 14:04

    We're also setting up our context aggregation, which

93. 14:08

    for better or worse, we use a-- OpenAI as kind of the default, like, uh, the lingua franca for context, so everything gets kind of folded back into OpenAI, uh, at a certain level.

94. 14:18

    Um, and then we define the pipeline. So the pipeline is, again, like Aleix had said, just a list of all of your different, uh, or is I guess a tuple of all of your different services, um, that are running in the pipeline.

95. 14:31

    And you can write your own. So if you want to instead make the LLM output text and you want to either extract information from them, maybe you in-- have it in code, some XML or some type of information, you can actually extract it, store it.

96. 14:44

    Maybe your application does something with it. Um, or maybe you want to inject a text-to-speech type of, uh, frame. You can actually do that by separating the LLM, the, that audio output from audio output to, uh, just be text, and then we-- you would add like a text-to-speech service here.

97. 15:02

    Or you could write your own processor, which may be not within the next hour. And then lastly, all of these are, not all, but many of them have, uh, events they emit.

98. 15:11

    The transport emits handlers for when the client connects and disconnects. So in this case, uh, we use this line here to actually inject a context frame into Gemini to k-kick off the conversation.

99. 15:25

    So when your client application connects, it's going to queue a frame. So again, frames being kind of the base format for, uh, information. Think of it as like an object for your pipeline.

100. 15:36

     You're going to queue one of those frames, and what this function does is it just grabs the latest context. So when we set this one above, I think that just says hello, that's going to pass that into, uh, or basically push that into the pipeline, which then it will make its way to, uh, to Gemini to initialize

101. 15:53

     the conversation. And the rest of this you could think of as boilerplate to run it. You create a runner, which then is the thing that actually runs your task, and the task is what runs the pipeline.

102. 16:03

     So maybe beyond today, but just know that that's something that's required to run your code. Okay, I'm gonna pause here just for any questions, 'cause I do wanna get to developing soon.

103. 16:14

     Um, so you said it was a WebRTC. So you can provide a WebRTC server. You provide WebSockets as well?

104. 16:20

     Mm-hmm. We provide both. That's a whole topic that is a talk in itself. Um, the short answer is if you're building a client-server app, you should, with like strong emphasis, use WebRTC.

105. 16:32

     It has a whole bunch of properties that are relevant, like error correction, better audio quality, et cetera, et cetera. If you're building server to server, so one of the options today is to build a phone chatbot.

106. 16:43

     You can use, and it's probably the best option to use WebSockets. So you can bring your own, uh, transport. We actually, in Pipecat, there's a, a FastAPI version of that that's a server that you can use to exchange messages with a WebSocket.

107. 16:57

     So yeah, it's really up to you. So I guess maybe the, the takeaway is if you're building client server, you really want WebRTC. You could technically use WebSockets, but you'll hit like a long tail of err- of errors when you get to production, and then server to server, totally fine.

108. 17:10

     You're gonna be fine with WebSockets.

109. 17:15

     Could anyone download the repo by any chance?

110. 17:19

     Did anyone not-

111. 17:20

     Just one person

112. 17:20

     ... download the repo?

113. 17:21

     Two persons. Three, four.

114. 17:23

     Wi-Fi's dead. Man.

115. 17:24

     Wi-Fi's dead.

116. 17:25

     Is it-- Can anyone-- Can you tether? Is that an option? Do folks have tether hotspots on your phone or... That's even shaky? Oh, geez. Okay.

117. 17:32

     You have a hotspot. Yeah.

118. 17:34

     I can dance. Could do some... I don't know. [laughing] You know, I, I'm not really, you know, it's not my thing, but I could try.

119. 17:41

     True. Yeah. We-- I guess I could just make this a code walkthrough. Um-

120. 17:46

     Or we could write it from-

121. 17:48

     What about VADs?

122. 17:50

     What-

123. 17:50

     If you use some other VADs-

124. 17:51

     Yeah.

125. 17:51

     ... Gemini have its own VAD or...?

126. 17:52

     Gemini does have its own VAD. Yeah.

127. 17:54

     What's your favorite choice in voice activity detection?

128. 17:57

     Gemini does have its own VAD. In fact, if you're using a speech-to-text service, that likely also brings its own VAD to the, to the equation. What we've found is that, so Maybe to use some of our extra time because we're having internet issues.

129. 18:10

     Um, the VAD serves a really important purpose of detecting when a user starts speaking. So in the whole kind of life cycle of a turn, that user speaking kind of ushers in the user's turn for the conversation.

130. 18:22

     So Pipecat will emit a user started speaking frame, and that will also push through an interruption. So the user will interrupt like anything that's talking. So if the bot was speaking or whatever, uh, it basically clears the way because the user has expressed that they want to speak.

131. 18:37

     The idea with the VAD is that we want it to be extremely accurate and extremely fast. So running something on device, we recommend Solero, which is an open source option.

132. 18:46

     It works incredibly well. I don't know what the inference time is like. I don't know.

133. 18:51

     Millisecond.

134. 18:52

     Millisecond. Extremely fast. So you're going to get an event back, uh, fast. In fact, you have the ability to tune how long to hear human speech before that event gets emitted.

135. 19:02

     And there-- the defaults are pretty good in Pipecat, and there are maybe scenarios where you wanna change that. But the VAD is a really important, uh, consideration. It's extremely low CPU consumption.

136. 19:12

     Quinn has a great, uh, spreadsheet of breaking down the full cost analysis of an, an agent, and really the CPU is going to be extremely low. Interestingly, the TTS tokens or characters are, are the most expensive by far.

137. 19:25

     Um, so when you think about it, running that local VAD gives you superior performance, and it allows you-- And it-- There's not much of a cost hit. I mean, it's maybe like a fraction of one percent to run a local VAD.

138. 19:36

     Um, but yeah, you have all sorts of choices. But we find, we find that to work really well. Mm-hmm.

139. 19:43

     Is Pipecat integrated with phone carrier and like that or just web?

140. 19:48

     Uh, it's integrated with phone carriers too. So there are a ton-- There are-- I found out-- I didn't work much with phone. Varun's been our phone expert, though I don't-- Varun is like a figure in the WebRTC community.

141. 20:00

     He's like an author of many things. But he's also a phone expert, which I don't know if that crosses over too much. [chuckles] It was-- Maybe that happened before. Um,

142. 20:11

     phones are super complicated, how you actually make calls. Um, Pipecat supports all of them. So maybe a very quick list. You can make a WebSocket connection with a phone provider like a Twilio or Telnyx or Plivo or Ex-Exotel and exchange, uh, media streams.

143. 20:27

     So that's a way to have just a, a native WebSocket connection from Pipecat to Twilio. You'd call Twilio, it's going to emit the WebSocket, and there'll be a handshake to get connected.

144. 20:38

     You can also use, uh, PSTN, which is a public switched telephone network, uh, which is lets you dial in, and that's gonna be kind of a different mechanism. There's also SIP, which is its own separate thing again, and all of the telephony providers would also support this as well.

145. 20:55

     With SIP, you would call, um, something say like Twilio, and you would call into like say a server like, uh, or a SIP provider like Daily, which is offers SIP provided rooms.

146. 21:07

     And you then have the ability to kind of, uh, bring the two together via that SIP connection. The nice thing about SIP is that you have the ability to have like superior call control.

147. 21:16

     It does-- It is slightly more complicated, whereas that WebSocket connection is instantaneous. Your bot needs to be up and running. So there's a whole... We're not gonna talk about it today, but like cold starts for agents, they need to start immediately.

148. 21:28

     So if you don't have resources provisioned, you don't want your users waiting like twenty seconds while the bot comes online. So a long-ish, medium-ish answer for a very complicated question, if you didn't know that.

149. 21:42

     Okay. You can-

150. 21:42

     Yeah, yeah, yeah.

151. 21:44

     Could you compare with something like Cartesia, Pipecat plus, uh, Pipecat, Pipecat giving birth to something like Cartesia?

152. 21:55

     Can take that. Yeah.

153. 21:56

     Yeah. I'll, I'll let you answer that.

154. 21:57

     Yeah, yeah. Cart-Cartesia of-

155. 22:00

     What's the question?

156. 22:01

     Yeah. Can you-- I didn't hear it. Can you repeat?

157. 22:03

     Well, I think you were, you were asking if Pipecat compares to or how it compares to, uh, Cartesia. Is that-

158. 22:11

     Yeah, yeah. We heard first.

159. 22:11

     Yeah. Cartesia is, uh... Well, I think they're gonna do more stuff, but, uh, as of today, it, it's a text-to-speech service. And you can clone your voice or you can, you know...

160. 22:22

     And then they provide a real-time, uh, API. You can just via WebSockets, you pass the text, and they reply with, with audio, basically, with audio frames. And then Pipecat integrates with Cartesia as any other text-to-speech service like ElevenLabs or, or anything.

161. 22:39

     Like think about Pipecat as a j- it's just a framework for developers where they can plug and play the service they want. So they can plug and-- They can take Cartesia, they can put Cartesia, they can take it out, put ElevenLabs.

162. 22:52

     Oops. Whoop. Even closer. Okay. Now I can hear myself. Um, yeah, you can plug and play any service you want. Like you can change LLM, you can use Llama, you can use Anthropic, you can use, uh, Cartesia or ElevenLabs.

163. 23:09

     Then for, uh, speech-to-text, you can use Deepgram, or you can use one box like Gemini Live, which has all that, uh, built for you. So

164. 23:20

     is that, that clear? Yeah. Yeah. Okay. Yeah.

165. 23:24

     Um, what about what behaviors? Okay. Uh-

166. 23:27

     About what, sir?

167. 23:28

     What behaviors. What behaviors. So, uh, uh, uh, I, uh, I mean, so we are talking about, uh, uh, voice product, right? The right to production. For instance, if you are trying to use, right?

168. 23:43

     The voice value. So what about the responsibility AI? How, how, how are you, uh, like focusing on create-- Uh, sorry about my English. These, what behaviors, right? Is that, uh, for guarantee that the models were responsible, something that is for right for the, for the professional and the author and the user, uh-

169. 24:12

     Mm-hmm.

170. 24:13

     I don't know if I can... If I get myself muted.

171. 24:16

     You-- Um, uh, I, m- I'm not sure [chuckles] if I understood. Are you talking like how to ensure that the LLM says what the right thing or not the right thing?

172. 24:30

     Yeah. Well, Pipecat doesn't have control about that. It's up to you to define the prompt or define how the LLM will, um, uh, will reply. You can put-- If you want, you can put, uh...

173. 24:44

     You'll be able to write your own processors that we call them, those little boxes, and check what the LLM has said, for example, before it's, um... Like put some kind of real-time eval, like to, to make sure, uh, that the LLM has re-- You could, you could do that, yeah.

174. 25:02

     You-- The, the-- All that, that pipeline is very flexible, so you can, you can, you can put whatever you want there, like in parallel, not in parallel. For example, Mark was saying about parallel pipelines, like if you have video and audio at the same time.

175. 25:19

     Uh, with Gemini Live, you can do everything in one box. But let's say you don't have Gemini Live, you wanna use other, uh, other services that one does video and the, the other one does audio.

176. 25:29

     You can have a parallel pipeline which, you know, it's like a, a tree, right? You, you s- have your transfer input, and then if it's audio, it goes this way; if it's video, it goes that, that way, and you can, um, you can do things dynamically like, like that, yeah.

177. 25:44

     Yes, next question. Yep.

178. 25:46

     Yeah, I have a related question. Um, is that common that people put some kind of check, uh, in place and you see like what is the extra latency of sort, uh, that that produces?

179. 25:57

     And then for models like Gemini Live that produce audio and video, it seems like... Do they also produce the text with it, or do you have to kind of do speech-to-text again and then back to-

180. 26:10

     S-sure

181. 26:10

     ... speech and you're losing all that extra [audio static]

182. 26:14

     Okay. So the question was, um, are guar- are guardrails requirements, and do-- how do people use them? And how does that apply for speech-to-speech models? So the answer is they're not required.

183. 26:27

     In fa-- Uh, there is a challenge here, and actually be-- Uh, I talked about this on like one of the first slides. Latency is absolutely critical, so what you want to avoid are unnecessary turns.

184. 26:37

     You know, obviously, LLMs are amazing language processors, so if you had all the time in the world, you could do hallucination checking against the LLM. There are other strategies to handle this.

185. 26:48

     One of the big things is that we see because of the aggregate nature of the context, it grows over the course of the conversation. You actually... You can find better, um, better accuracy with the responses if you have more control over how you prompt the LLM.

186. 27:04

     So this is, uh, a whole topic and talk in itself. Uh, what we found is there are two ways to handle this. Well, at least two. Um, one, if you...

187. 27:14

     For a lot of conversations, they're going to be task-oriented. So let's say, uh, something simple like a restaurant reservation bot. It may have to take your name, get your time, log the time to a database.

188. 27:24

     You can chunk that out, even that small conversation, into just discrete tasks, and LLMs are really good at, uh, following the most recent input. So if you kind of feed it task by task, that helps.

189. 27:37

     Also, if you control the context window, like the size, it can really be beneficial to kind of manage that really judiciously. So you could either reset. Um, one example might be, let's say you're building a patient intake bot at a doctor's office.

190. 27:50

     They may... The very first thing it may do is verify the date of birth, which serves no utility beyond just the very first, you know, checkpoint. So you may actually remove that from the context, like completely get it out of there, 'cause otherwise it's just cruft that hangs on, and instead you kind of reset and then maybe

191. 28:07

     roll through the tasks. You could also, for really, really long conversations, summarize the c- the context. So you, you may wanna do an out-of-band LLM call. And this is something actually Quinn just ta-- We talked internally about this, that we're going to see more and more of this mixture of LLMs where even in the context of real-time,

192. 28:23

     you may have an out-of-band like REST call to the, to the text-based LLM just to do a summary and then return it back so that you can kind of compress that context window.

193. 28:33

     And just to give a call-out to Google, the, the live APIs, so maybe transitioning there, they offer context, uh, management through a bunch of different str- different strategies, like a rolling...

194. 28:42

     They have a rolling window or sliding window. I think they offer like token caps for that, so you can have some control. Or if you want, you can output text and then kind of do whatever you want with it.

195. 28:52

     They also take text input, so there's a lot of, a lot of flexibility with speech-to-speech models. They do offer or they do pose some other maybe development challenges, but offer like tremendous benefits in terms of the features they offer.

196. 29:05

     All right. I'm gonna-

197. 29:05

     I think there's a question-

198. 29:06

     I'm gonna hold questions just for a sec 'cause I, I've heard the Wi-Fi is back. Can folks try downloading the repo again? 'Cause I-

199. 29:14

     Slack channel.

200. 29:15

     Slack channel. Yes. Quinn, can you maybe come to the mic and... I can't remember what you told me.

201. 29:21

     Uh, uh, workshop-voice-gemini-pipecat with dashes in between on AI Engineer Slack.

202. 29:28

     Okay.

203. 29:28

     So-

204. 29:28

     Does everyone know where that is? I know this is like day one, hour three.

205. 29:32

     Workshop-voice-gemini-pipecat is the channel name.

206. 29:36

     So if you go to the AI Engineer Slack and search for workshop Gemini, it should come up. And Quinn and Aleix will be posting links as we go along if the Wi-Fi stays up.

207. 29:47

     Um, if anyone can get on that channel, can you raise your hand so we-

208. 29:51

     Can people jo-- Like do they have a-- Do you guys have access to the Slack, AI Engineer Slack?

209. 29:57

     Get Slack. There's two channels. There's two channels. Well, workshop-gemini-pipecat and workshop-

210. 30:05

     Oh, yeah. There's like... Yeah.

211. 30:09

     So we'll explain what that happened when the Wi-Fi was down and we were trying to

212. 30:14

     Yeah. I'll take a question over here

213. 30:15

     On your bot, there should be the Join a Slack group

214. 30:18

     All right. I'd say, uh, I'm gonna t- I'll take that one, then I'll walk forward. If I could, let's just try maybe listen and also try to get the repo.

215. 30:25

     I'd like to walk through the quick start, and then we can look at some examples.

216. 30:29

     You mentioned moving the VAD to local processing. Do you find that there's, like, any advantage in moving more of the pipeline locally, or is that kind of offset by the lack of processing power on client-side devices?

217. 30:41

     I... You know, it's something actually, uh, Quinn has done a ton of work with this. I have not personally. I mean, there's obviously massive latency benefits because you cut out network round trips all over the place, so you save a ton.

218. 30:51

     And depending on where you are in the world, that can save a lot. If you're US-based, you know, your network latency is gonna be relatively low. But a lot of the developers in the community are in Europe, and a lot of these AI services are relatively new with, you know, uh, data centers only in the US.

219. 31:05

     So there are different challenges when doing that, though, when running it. You... It... Actually, there are great, um, hosting providers like Modal that offer really good options for running, like, your own local LLM, which then you're not...

220. 31:17

     You know, you're buying or I guess leasing, like, the GPU time instead of running everything on a GPU, which would probably be cost-prohibitive 'cause because of the way processes run.

221. 31:25

     But that's, uh, that is also a talk in itself. Um, next question.

222. 31:32

     Just one on kind of state management with Pipecat.

223. 31:34

     Sure.

224. 31:36

     Um, have, have you seen specific use cases where like making tools more stateless or instead of just pushing everything into this, um, large context window-

225. 31:46

     Oh, yeah. Definitely

226. 31:47

     ... makes a significant difference? But that can also, from your experience, that that causes more latency for the local device.

227. 31:56

     Well, the large context windows definitely cause LLMs to process slower. Um-

228. 32:02

     Can you repeat the question?

229. 32:02

     Yeah, I'm sorry. The question was around state management with, um, with LLMs and whether it's better to, I guess, chunk or have more kind of deterministic input versus just a large context where you just dump everything in.

230. 32:14

     This is actually an extension of what the other gentleman was asking. Um, the idea being that, uh... And actually, w- so Daily, we built the chat widget that's on the homepage of the Voice AI World's Fair, uh, page.

231. 32:30

     I, I, I personally built that. What was interesting there is that, um, and Aleix and I were just talking about this, function calls in the context of real-time are still slow, unfortunately.

232. 32:40

     Like, too slow. Um, actually, Gemini, [chuckles] I'll give more props to Gemini being, like, maybe one of the fastest. If you run, like, a basic local, like, one of these demos and you ask it what the weather is, it will return back with time to first byte in, I don't know, less than 500 milliseconds.

233. 32:56

     Whereas other vendors, not trying to throw OpenAI under the bus, but it has, has gotten slower. Like, the... You might see upwards of 1.5 to two seconds of waiting just to get that first token back.

234. 33:07

     And we dug into it. Actually, this is just something recently this morning. The issue is when you get the normal streamed response for the conversation, you can start playing that audio out once you get the first sentence.

235. 33:15

     The issue being when you get a tool, you need the entire JSON response before you can actually do anything with it. So that's slow. Uh, that's one part of it.

236. 33:24

     Se- separately, chunking the prompts is absolutely the way to go. Uh, in building that World's Fair bot, um, I was kind of balancing between the two worlds because, uh, you can talk to it and ask it about speakers for the session.

237. 33:37

     Route one would've said, "Let's use, like, a mag approach and put all of the, the speaker JSON in something that could be a tool that could be accessed." And I tried that, and unfortunately, it's just a little too slow.

238. 33:49

     It's a giant context. It takes a while to come back. What's interesting is if you instead move that all just directly into the context with Gemini Live, it's a little bit variable, but under good conditions, you'll get a response back on that, like, 800-millisecond latency.

239. 34:03

     So it actually has access to, like, the full context. The one trade-off, though, is what this gentleman over here was asking is accuracy. It's gonna get confused 'cause especially when you get a JSON with a lot of speaker...

240. 34:14

     And this isn't-- This is all LLMs. It's not specifically Gemini. With that type of even structured data, it becomes very hard to kind of discern what's what when a lot of it looks the same.

241. 34:23

     So it's all... I mean, a lot of this is emerging like other things in, in AI. We're trying to just do it as fast as possible with voice. Before I take one more question, I wanna check in on have folks have, uh, any luck with the repo?

242. 34:36

     Got a thumbs up. All right. I'm gonna pause on questions for the time being. We can take them at the tail end. I do wanna try to go through the quick start if we could.

243. 34:45

     That would be great. And, and again, to know, um, we consolidate channels. There's one channel in the Slack. It's workshop-voice-gemini-pipecat. Quinn is in there answering questions. Roen will join, and we're gonna share links there.

244. 35:00

     So Mark, that's great.

245. 35:01

     Okay, great. So I would recommend, um, maybe we'll just take, like, a few minutes of independent getting set up if you all go to the README on in the Gemini Pipecat workshop.

246. 35:14

     And if you'd roll through the first few steps, maybe get a, I don't know, a hand up. You could just flash it up real quick so I could see when people start to get through it.

247. 35:23

     Um, you don't have to hold it up. If you don't mind, if you brought a device. Okay, we've got one,

248. 35:29

     two.

249. 35:32

     So do we need a key?

250. 35:33

     Ah. So for this workshop, I have leaked my key, a key from one of my accounts, which I'll cycle after this, so you don't need to sign up for a Daily account.

251. 35:42

     You do need to sign up for a Gemini account. So I don't have a key I can just give out. So in the environment.example, you'll see there's already a Daily key which you can use.

252. 35:54

     Do people know how to sign up for a Gemini-

253. 35:56

     I have it in the README

254. 35:57

     ... account? Do you have a direct link?

255. 35:57

     Uh, it's through AI Studio. Is that a good spot to do it? Okay. So...

256. 36:10

     You can do it. Can you do that? You can do that. Get your computer.

257. 36:14

     There we go.

258. 36:16

     All right, I'll take a question while we're waiting

259. 36:24

     In the Pipe- in Pipecat, or are you still needing to do-

260. 36:26

     We-

261. 36:27

     ... most of it yourself?

262. 36:28

     So in terms of, um, the pipe... That's a good question. In terms of the Pipecat interface, you interface with the Pipecat class, so there's a service class. In- within Pipecat, so how you write your application code, is going to be uniform across all of the services.

263. 36:43

     The individual, um, providers haven't really... Unlike the text-based LLMs, they haven't really settled on kind of a, a standard. So Pipecat handles all that translation on your behalf. And I think there are other frameworks that do similar things.

264. 36:56

     The idea is to provide, like, a uniform, simple interface, uh, s- so that you could take... And this is part of the modularity. If you wanted to, you could swap this bot out for OpenAI real-time or, you know, a text-based model with a TTS, an STT paired with it.

265. 37:12

     That's kind of the whole, whole idea. There is maybe a little bit in terms of, uh... The one thing LLM providers maybe that... I don't know if anybody here can [chuckles] nudge anyone you know.

266. 37:22

     The system instruction or system prompt is not unifor- uniformly dealt with. I don't know if that's well understood. But OpenAI has this, like, user that is system that you can inject anywhere at any time, which is really fantastic.

267. 37:35

     But, um, Anthropic and Google require, like, a named system instruction that's a special one-time, like, at constructor time, um, instruction. So there's, uh, there's some differences. As much as best we can, we u- we unify.

268. 37:52

     All right. Any... How are folks doing on getting quick start going? I see a hand up. Is that a question or... Okay. I'll take... Maybe it's related to the quick start.

269. 38:00

     So one of the problems that I've seen with VAD is that it kind of struggles in a really noisy environment.

270. 38:07

     Oh, yeah. Great topic.

271. 38:10

     I was wondering if, um, there are any tricks you know to improve the, the VAD.

272. 38:15

     Yeah. So there's a question about noisy environments, which is, like, voice AI's kryptonite. Um, [chuckles] so the- with the VAD, no, not at the moment. Uh, but there are... Again, this is where, like, Pipecat is the assembler of all things.

273. 38:28

     What do you wanna plug in? Um, that you can run separately from the VAD. Uh, we found, like Krisp is one that's a partner of ours. They're have- they have fantastic noise cancellation.

274. 38:41

     You can run something outside of the loop that, uh, would actually clean up, like in the t- in Pipecat, it would be in the transport itself. So in that audio input, it would take, uh, the audio input and remove any ambient noise, so, like, chip bags opening or dogs barking.

275. 38:55

     But maybe more impressively, uh, human background voice. So it'll remove that from the feed. So you could be in this conference, and it picks up the primary speaker for the device, like, incredibly well.

276. 39:06

     At the moment, they're the only ones that I'm aware that know how to do that and do it that well. But it's, I mean, it's phenomenal. But you're right, the VAD is...

277. 39:12

     I mean, it was one of the biggest problems that we saw until we found Krisp, and they're fantastic.

278. 39:18

     Is that C-R-I-S-P?

279. 39:20

     K-R-I-S-P.

280. 39:23

     Okay.

281. 39:23

     So big props to the Krisp team.

282. 39:26

     Which speech, speech model are you guys using right now?

283. 39:30

     Well, we use, uh, we use all of them. Pipecat's open source, so we bring... We ha- I think options are Gemini, Multimodal Live, OpenAI real-time, and then, uh, AWS just launched a new one called Novasonic.

284. 39:41

     So we have those three, um, within Pipecat.

285. 39:47

     How, how, how... I mean, what's the... Is it the latest speakers

286. 39:51

     They're all pretty... They're all very much on par. Actually, well, they're all, they're all very similar. Um, I'm not gonna... I don't wanna, like, nitpick on all of the vendors here, but they're, I mean, they're...

287. 40:01

     They have strengths and weaknesses each 'cause it's still an emerging field. Um, but they're all... Latency-wise, that is not an issue. Latency is fantastic for all the providers. Yeah.

288. 40:14

     Thank you.

289. 40:15

     All right. Okay. Maybe Aleix will walk through a quick start. And why don't we... He's gonna just do some live coding here, and maybe this will help to understand what's, how this all works.

290. 40:24

     And then perhaps we, I stop chatting, and maybe you could grab me if you have questions and just do some heads down working time.

291. 40:32

     I don't have Vims.

292. 40:34

     Okay.

293. 40:35

     Yeah, I cannot type and [chuckles]

294. 40:39

     Teamwork.

295. 40:40

     Teamwork, yeah. Uh, I'll start from the very, very, from nothing, from scratch. Um, so this is a Python project. Um, actually, let me, let me try again. Yeah. This is a Python project.

296. 40:53

     The first thing we'll do is create an environment,

297. 40:57

     like a virtual environment that is called. I like to call it .env. That, that's how you create a virtual environment in, in, in Python. Um, a virtual environment will have-

298. 41:09

     Can we turn the lights up a little? It's hard to see the screen.

299. 41:11

     Oh, yeah.

300. 41:12

     What's that?

301. 41:13

     Your screen is hard to see.

302. 41:14

     Oh, it's hard to see. Um, how do I do that?

303. 41:18

     Or make it a bit bigger. Make the font bigger maybe. Try that.

304. 41:22

     Yeah, I guess. Could you use just, like, VS Code?

305. 41:24

     Better?

306. 41:25

     That's right. You're talking.

307. 41:27

     No.

308. 41:28

     Expert. Probably too.

309. 41:29

     It's like Emacs.

310. 41:30

     What's that?

311. 41:31

     Okay.

312. 41:31

     White.

313. 41:31

     Okay.

314. 41:33

     Can you s- do you-- can you change your theme to, like, white for the time being?

315. 41:37

     I guess. I only have one theme. Oh.

316. 41:53

     Maybe.

317. 41:56

     Emacs, yeah.

318. 42:03

     The only issue is I only have one theme. [chuckles] But, uh-

319. 42:06

     Is it better? Can you see it better?

320. 42:08

     Yeah.

321. 42:09

     Yeah.

322. 42:09

     Better?

323. 42:10

     Nice.

324. 42:10

     Okay. Yay. Okay. So we'll start with, um, uh... Yes, it's Emacs. You can judge me. Uh- [chuckles] ... I'm, I'm, I'm not gonna change. I have a few years only coding, so that's what it is.

325. 42:25

     Um, so I'm gonna create a requirements file. Sorry, I loaded... First thing, I did the virtual environment We did this, then we're gonna load it. Now you, we can start installing, uh, packages in, in this environment, like Python packages.

326. 42:41

     So the ones we have to do is, uh, Pipecat, of course. So I'm just creating a new file which has pipecat-ai, and Pipecat has a few options. So for this example, we're gonna use Daily, we're gonna use, uh, Google, and we're gonna use Silero.

327. 43:01

     Right? Silero is the VAD that we, we've been talking about. Um, the other thing we will do is we will load the envi- there's a package called python.env just to load the environment variables that we'll have, uh, in this, in here.

328. 43:17

     Okay? Um, that's it. One file. Now we're gonna... I'm gonna create, uh, an environment variable. I'm gonna just copy it from the,

329. 43:29

     from the kick start. It just, uh, oops. It's just a .env, and we can take a look at it. It... I will...

330. 43:39

     These are the, the API keys. We'll get rid of them after this workshop, so [laughs] But yeah, these are the... You just need two keys, Google API key, and just, uh, go to AI Studio and create your own one, and a Daily API key.

331. 43:54

     This one we already... This one you can copy. It's a bit long, but, uh, I don't know. It doesn't-

332. 44:00

     It's in the project.

333. 44:01

     Yeah, it's in the project. Yeah. Yeah, if you, if you are able to go to the, to the repo on GitHub, you will, you can just copy it from there.

334. 44:09

     All right. Next, we're gonna create a Python file. Let's call it, uh, bot.py. Again, empty file. Uh, first thing we'll do is typical, I think it's called name, uh, equals main.

335. 44:27

     No help [laughs] from an LLM. And then we do asyncio.run. Ah, a little bit of help. Asyncio.run, and we're gonna write a main, a main function. Function,

336. 44:43

     and then let's write the main function. All right. So now we start. So the first thing we, uh, we s- we, we said is we're gonna need a transport.

337. 44:55

     The transport input is like we're gonna, um, like the Dai- uh, Daily transport, we're gonna say. That's... I'm gonna speak to the, to the bot through the Daily transport, and that's what I'm gonna create.

338. 45:06

     And then we want... It's a transport for incoming audio from me talking, and then a transport, uh, for outputting what the, what the LLM will say. So I'm just gonna create a transport, um, and this is a Daily transport,

339. 45:22

     and it has a few, uh, arguments. It has a room URL. I'm just gonna create mine for now. Um, this is, uh, again, a public, public room that I will have to delete.

340. 45:37

     Uh, I don't need a token, and params.

341. 45:43

     Okay. And then the Daily params is like, what do we want this app-, uh, this transport to do? So we're gonna say that we want, uh, input enabled, which means get audio from the transport, and we also want to, [clears throat]

342. 46:00

     to send audio to the transport. Okay? Um,

343. 46:06

     all right. What else? Uh, it's complaining for something.

344. 46:11

     VAD.

345. 46:11

     Oh, yeah, and we also need a VAD analyzer, which is gonna be a Silero VAD analyzer. Um, so the transport is gonna be able to, uh, use this VAD analyzer to detect if the user has spoken or not.

346. 46:27

     You missed a name.

347. 46:28

     What's that?

348. 46:29

     You missed a name.

349. 46:30

     Oh, yeah. And this is gonna be AI engineer. There we go.

350. 46:37

     Okay, now we have the transport. Now we're gonna create the LLM. Uh, in this ca- it's Google, um, uh, Gemini Live, so I don't need to create an, uh, speech-to-text or a text-to-speech.

351. 46:50

     We just create, uh, the Gemini Live. And for that, I'll need to copy it for... 'cause I don't know that from memory. But, um...

352. 47:00

     Oops. That's in this file. Gemini bot. I just wanna copy these lines here. There we go.

353. 47:11

     All right. So this is my LLM, and again, it uses this Gemini multimodal live LLM service.

354. 47:22

     Um, Gemini multimodal live LLM service. That's gonna add my import. Okay. And now it needs a couple of things, a system ins- instruction, which is like what the, what the agent is gonna, is gonna do, and some tools.

355. 47:36

     The tools, I'm gonna skip them for now. So let's, let's do the system instruction. Again, I'm gonna copy it from somewhere. There it is.

356. 47:46

     This is like the prompt, like the main prompt of the... Well, let... Yeah.

357. 47:52

     All right. So the system instruction is, "You're a helpful assistant who can answer questions and use tools." For now, we're not gonna use any tools. Um, you know what?

358. 48:02

     Let me get rid of the tools. Just copy here for later. I'm gonna comment this out, and you are just the helpful assistant. Okay? So that's...

359. 48:18

     All right. Um, so no complaints here. All right. And now we just create the, [clears throat]

360. 48:26

     a pipeline. I'm gonna avoid storing the context 'cause I don't think we need it for now

361. 48:34

     And this is the pipeline. The pipeline just receives a list of, um, of processors or elements, and the first one is a transport.input. That's the input transport, so how we get audio from the, uh, from the Daily room in this case, the LLM and the transport.output.

362. 48:59

     All right. Now we need some... This just defines the pi- A pipeline also is another processor, so you could build a pipeline of pipelines of pipelines of pipelines. So you can build, uh, or you can plug and play, uh, as, as, um, the way you liked it.

363. 49:15

     So how do you run a pipeline? You need a task, what we call a pipeline task,

364. 49:21

     that receives a pipeline, and the pipeline task also has some params, which are called pipeline params.

365. 49:30

     Um, and we're gonna say... Oops. That we allow interruptions,

366. 49:40

     and I think that's enough.

367. 49:44

     Yeah.

368. 49:44

     And how do you run a task? Uh, you can create more than one pipeline task if you wanted. In this case, we just have one. Usually you just, you just have one.

369. 49:53

     Uh, you're gonna create a runner, and guess what? It's called pipeline runner. Um, it's pipeline runner, and then we just do await

370. 50:05

     runner run task, and some completions please. Pipeline

371. 50:15

     runner. All right. And I think that's it. Uh, we'll try it. Oh, OS, import OS, I think.

372. 50:26

     I think there's no more warnings. Oh, and I need to load the environment variables, which is this line here, load.env. I'm just copying it from another file.

373. 50:40

     Okay. And where do we get load.env? This is just a function that, um,

374. 50:50

     um, that imports, that imports the environment variable. All right. And yeah, let's, let's try. I'm just gonna open the... Oops. [clears throat]

375. 51:05

     I'm just gonna open the terminal here, and I'm just gonna run it. I think we call it bot.py.

376. 51:14

     Uh, no model. Oh, maybe I need to install the requirements. I forgot this step.

377. 51:26

     There it is. Okay. So initia- at the beginning I, uh, wrote that file requirements.text, which has, uh, had a bunch of, uh... Well, again, just a few requirements, but I forgot to, to install them.

378. 51:49

     In the meantime, I'm just gonna go to the Daily room that I just pointed the bot to.

379. 52:07

     Uh, no video. Okay. All right. So that's, uh, right now it's just me in that, in that room, so... And now we just have to wait for this to, to finish, and hopefully the bot will join the room and we'll be able to talk to it.

380. 52:27

     Hopefully.

381. 52:32

     Yeah. How important is, um, like Daily as part of this flow? Like that room component, like is it interchangeable?

382. 52:41

     It is inter- Yeah, it's... This is because, uh, we're using the Daily transport, and the Daily transport just connects to a Daily room.

383. 52:49

     Okay.

384. 52:49

     So, but you could have a, a WebSocket transport and then use Twilio with a phone number and Twilio being connected to that.

385. 52:57

     Gotcha.

386. 52:57

     If we have time, we can even try that. Um, so I think-

387. 53:01

     What's the problem with SmallWebRTC transport? Can you talk about SmallWebRTC transport and why you didn't use it?

388. 53:06

     You wanna, you wanna talk about that?

389. 53:09

     Yeah, sure.

390. 53:09

     Oh.

391. 53:09

     Yeah.

392. 53:09

     Yeah, I'll wait. [chuckles] I'll just...

393. 53:11

     So we als- uh, in Pipecat we also have added, um, based off of the AIORTC Python package, which is how, uh, WebRTC package in Python. We've added a, a new transport called SmallWebRTC transport.

394. 53:25

     It is a peer-to-peer WebRTC communication that's free, so it's separate from any vendor. Uh, though the one downside is that it requires a turn server, which we... You bring your own, so we, [chuckles] we didn't...

395. 53:37

     You know, we weren't prepared for that for the conference, and also just the conference Wi-Fi makes that a little challenging. But normally, if you're running any of the, any of the, um...

396. 53:45

     We call them foundational examples in Pipecat. Think of them as the, like, essential, um, examples that show how to do very specific functions. There's probably about 100 of them in Pipecat.

397. 53:55

     But one by one it shows you how to, like, record or add an SDT or push frames or show images or sync images and, and sound. Those all use the peer-to-peer WebRTC transport.

398. 54:06

     So we, we would have loved to have used that. You wouldn't need a key. But unfortunately, firewall rules have trumped.

399. 54:13

     All right. Uh, so I'm just running the bot and see how it fails, because it has to fail the first time.

400. 54:26

     Yeah, this is the mass th-th-there 'cause there's a bunch of the, um, Python packages, and Python just decides to take, uh, a time to load them. But you see how easy it was to write, um-

401. 54:43

     Like an agent, like a voice agent with Gemini Live, uh, if it work, it, it just a few lines of code, uh, that we wrote in... I don't know how long it took me, but, uh, maybe like five, 10 minutes.

402. 54:58

     Um, yeah. Are there any questions [chuckles] on the example or-

403. 55:04

     It worked here.

404. 55:04

     What's that?

405. 55:05

     It worked.

406. 55:06

     It worked?

407. 55:06

     Working, yeah. It worked.

408. 55:07

     What worked?

409. 55:09

     The bot.

410. 55:09

     The bot worked?

411. 55:10

     Yes.

412. 55:10

     Okay. All righty. [chuckles]

413. 55:14

     Great.

414. 55:15

     All right. Nice.

415. 55:17

     You need friends with good Wi-Fi.

416. 55:19

     Yeah. [laughs]

417. 55:23

     There was a couple of things that popped up there with the words later in it. Like where are you at in terms of production ready and stuff like that for commercial use?

418. 55:31

     Copy that.

419. 55:33

     Uh, we have-- I, I don't know the actual number of customers, but, I mean, Pipecat probably serves hundreds of thousands of calls a day. I don't know. A lot.

420. 55:41

     Quinn, you probably have a better idea.

421. 55:44

     Oh, yeah. I mean, Pipecat is used by some very large companies in production. Uh, and people are contributing to it from NVIDIA, AWS, uh, OpenAI, Google, uh, lots of, lots of big companies.

422. 55:57

     Yeah. There's one thing we didn't mention about Pipecat is that what you see now in the screen, this runs on the server side, but we do have client SDKs for Android, iOS, JavaScript, and React, and I think that's about it.

423. 56:14

     But, uh, Ragnar, even a, a C++ client, um, uh, if you want. Um, so yeah. So that's the server side, but you can plug your, your client and connect to the, to the agent on your, on your phone.

424. 56:30

     And they're all WebRTC connections or like-

425. 56:32

     That would be-- It-- That depends on the transport you use. But yeah, you could, you could have your client connect to a Daily or we support LiveKit as well, but to a Daily room.

426. 56:44

     We like Daily because we are [chuckles] working Daily. But, uh, you connect, you can connect to a Daily room, and then the bot would connect or the agent will connect to the Daily room as well, and then that's the transport, the WebRTC transport.

427. 56:56

     Yeah. I think there were questions there. I can't see. [background chattering]

428. 57:14

     Uh, say it again.

429. 57:15

     So are there other ways to test the bot, like other than talking to the-

430. 57:21

     Yes. Yes, there is. Um, actually, uh, for the previous version, um, I just hacked together a thing ca- that wa- that I call release evals, which is a bot talking to a bot.

431. 57:34

     And what it does is I put this bot, uh, up and then it joins a Daily room. And then I have a, an eval bot, and the eval bot, um, what it's gonna do, it has a prompt, which is ask a simple addition.

432. 57:50

     Okay? And then that eval bot is gonna connect to the room. It's gonna add, "What is two plus two?" And then the other bot is gonna reply, "Two plus two is four."

433. 58:00

     And the eval bot, the LLM, uh, [clears throat] it checks if the answer of the user is correct, and the user in this case is another LLM. So it verifies. It's like an end-to-end.

434. 58:12

     The good thing is we, we run the-- We used to run like more than a hundred examples every release just to make sure they work. So I just got tired of it 'cause it's very painful and very slow.

435. 58:24

     So we have this, uh, eval bot or eva-- uh, release evals that are gonna test each service. Like we test Gemini Live, we test Cartesia, Deep Gram, like all the services like end-to-end, and then the bots basically talk to each other with voice.

436. 58:41

     That's the, that's the, the nice thing. So yeah.

437. 58:45

     Okay, may- Oh, is this on? Maybe real quick just to show for those that didn't see it, it is working. Hey, can you tell me a joke?

438. 58:54

     Why don't scientists trust atoms? Because they make up everything. [audience applauding]

439. 59:01

     Which if you build Pipecat, I've heard that joke probably, I mean, not like five thousand times. Like for-- You can even try to seed it with something different, but it still tells the same joke.

440. 59:10

     Okay. I will try to come up with a new joke next time.

441. 59:11

     What-- Yeah. Why do, why do you always say the same joke?

442. 59:17

     That's an interesting question. Why do you think people keep telling the same jokes?

443. 59:22

     I don't know. You tell me.

444. 59:28

     From my search, there appear to be several reasons why people repeat jokes. [audience laughing] Enjoyment and reinforcement. People repeat things they find funny because they want to re-experience the good feeling associated with laughter.

445. 59:39

     It feels good, and repeating it is a way to try and recreate that sensation. Memory aid.

446. 59:45

     Okay. Okay, that's enough.

447. 59:45

     Repeating something can help you remember.

448. 59:47

     Yeah. Um-

449. 59:49

     Is there anything else I can help you with today?

450. 59:51

     No, thank you. That, that was it.

451. 59:53

     Thanks.

452. 59:53

     Thank you so much.

453. 59:54

     Well, just to show, there was a question about interruptions. We could just have it... Like my favorite is to ask it to tell you like a really long story and then interrupt it.

454. 1:00:01

     So can you tell me a really long and really long story?

455. 1:00:06

     Okay, I can do that. Here's a story, and feel free to interrupt whenever you like. Once upon a time-

456. 1:00:12

     Okay. Actually, tell me that new joke.

457. 1:00:14

     Following the clues, she discovered-

458. 1:00:16

     Nope. Hey, tell me that new joke.

459. 1:00:17

     Oh, you'd like to... Why don't scientists trust atoms? Because they make up everything. Okay, here's one-

460. 1:00:24

     Again? What-- That was the same joke. [laughs] [laughs]

461. 1:00:28

     That was the same joke. All right. All right.

462. 1:00:31

     That was it.

463. 1:00:34

     No. Well, I find like a lot of conversational agents, like they choke based on when you pause talking, right? Like that's the turn-

464. 1:00:44

     Oh

465. 1:00:52

     So I think the questions asked during this workshop could map out like years of work. So this is like the-- another one of those fantastic cutting-edge things. So again, back to like human evolution.

466. 1:01:05

     We all know and when we talk... Actually, it's even hard for humans to talk to not speak over each other. So the way that it works mechanically is when the user stops speaking, the VAD has a timeout.

467. 1:01:14

     You tell it and program it, wait, let's say one second, point eight seconds, half second, whatever feels natural. And you're trying to balance low latency response with giving the user enough time to speak.

468. 1:01:25

     It's a really hard thing, and it's one of the biggest complaints is that agents will speak over the human. So if you're-- Let's say you're building an interview bot, like you're using, um, like Tavis, one of their digital twins.

469. 1:01:36

     You wanna have like a, a real, like, likeness, and you wanna speak to it.

470. 1:01:41

     You may take time to think 'cause sometimes you have to take time to think, and that's a really difficult thing for bots to do 'cause again, it's driven by like a simple stop speaking algorithm.

471. 1:01:51

     So this is a new, uh, I guess it's like an emerging, uh, field of models, which is looking at semantic, uh, end of turn. So driven off of things like, um,

472. 1:02:03

     like speech filler words, uh, pauses, uh, intonation. So things in the audio realm and also things in the text-based realm. So just looking at context. So we've actually started, uh...

473. 1:02:15

     We're one of many that are doing this, I think. We c- we launched a model. Uh, if you look at it on GitHub, it's under smart-turn. It's a native audio in, uh, classifier that runs an inference on the in- input audio, and it simply outputs either complete or incomplete.

474. 1:02:31

     And the way Pipecat uses this is that if you get an incomplete response, we can dynamically adjust the VAD timeout. So we can tell the Pipecat bot, "Okay, he's, you know, he or she is not done speaking.

475. 1:02:42

     Let's actually move the-- Let's give three seconds to complete the thought," and if it's not done, then the bot will actually respond. So you can create a little bit of like dynamic interaction there.

476. 1:02:52

     And that's one of the first things. Uh, I'm sure Goo- the Google team is working on th- similar things. I know OpenAI is, and all the STT vendors are, are also looking at their own thing.

477. 1:03:01

     So I'd say right now it is very much an unsolved problem. But I would imagine, given how fast things are going in the next 12 months, we'll have great solutions that will make it even more natural to talk to a bot.

478. 1:03:13

     It's a good question.

479. 1:03:20

     Any more questions?

480. 1:03:21

     Any questions?

481. 1:03:23

     Well, uh, actually, I'm kind of wondering like, uh,

482. 1:03:27

     like is there a way to see the transcription of the text as it's happening versus only-

483. 1:03:32

     Oh, yeah. Yeah, yeah. One of-- Uh, this is actually back to the... Well, for-- This is specific to Pipecat but also, um, like Gemini Live will output audio and text and other speech-to-speech LLMs do this.

484. 1:03:46

     Uh, Pipecat offers in terms of its, again, orchestration role when you get a, uh... Actually, it's going to be specific to TTS provider. Um, many TT- There are great TTS providers that do word and timestamp synchronization.

485. 1:04:00

     So they'll give pairs. They call them like alignment pairs. So if you're using a Cartesia or an ElevenLabs or Rime, they all output these pairs. One of the really cool things with Pipecat is that the TTS services output not only the audio stream but also the text stream.

486. 1:04:16

     So they'll output text frames, TTS text frames we call them in Pipecat. And if you place, uh, we have-- In terms of how the client software works, there is, uh, like an observer role where you can actually watch.

487. 1:04:27

     There's a process that can watch things that happen in the pipeline and emit events. So we've instrumented that for the clients so that whenever you see those text frames move through the transport, you can get synchronized word and audio output.

488. 1:04:40

     So in your client, if you wanted to have word by word output synchronized to the audio, you can do that with Pipecat. And it's as simple as just adding an event.

489. 1:04:48

     I think you listen to like bot TTS text output or on bot TTS text, and it will give you the synchronized output.

490. 1:04:56

     If I wanted to build a fully offline voice agent, what box in that pipeline would be the hardest or toughest to do well?

491. 1:05:06

     Fully offline. Um, well, they're all doable. There are great models. I think it really depends on what your bot needs to accomplish. Um, a lot of the state-of-the-art models to do all the best and smartest things need to have some...

492. 1:05:17

     Like, they're going to be run like on-prem or in, in the cloud. Um, but if you have-- And a lot of j- bots do jobs. Like, if you wanted to build a, like a restaurant reservation one like I referenced earlier, it's a very simple job.

493. 1:05:29

     You could probably run it with some version of Llama running locally. Um, there are great local... And something, again, Quinn has been experimenting with, uh, a lot of great local models.

494. 1:05:40

     Like, um, Whisper has challenges, you know. It, I mean, it's, it has a lot, uh, it has some challenges, uh, as an open source model for, uh, STT, but there are good and emerging TTS services.

495. 1:05:53

     So, you know, I, I-- Things are only as good as the input, and we've actually seen this with some of the speech-to-speech models that sometimes they mistranscribe. So you really need...

496. 1:06:01

     I mean, it, it's, you know, there-- Every part is critical, but if you can't transcribe the speech really well, nothing really matters. Like, it has to understand you. And having like disfluencies or like hallucinated responses or even just inaccurate responses kind of breaks everything down.

497. 1:06:15

     So things mostly start at, at the STT. So maybe that's the hardest. I don't know if there are a lot of good open source options for that right now.

498. 1:06:21

     You said there's someone on your team looking at that?

499. 1:06:23

     I don't know. We don't, we're not doing anything in the STT world. No, no.

500. 1:06:28

     No. It-- That's a whole different ballgame. Good question, though.

501. 1:06:32

     Just made me realize we could have used local models [chuckles] and avoid this.

502. 1:06:37

     We could have. Yeah.

503. 1:06:37

     This mess. [laughs]

504. 1:06:38

     Well, we're, we're partnering with the, the Google team.

505. 1:06:41

     Yeah, yeah. As a... Yeah, exactly. Yeah.

506. 1:06:45

     Has anyone looked at any of the sample projects and had questions? There's a lot of interesting things there. If any of this has, like, interested you, we do have a Discord.

507. 1:06:54

     You're welcome to get on it. Um, you can find us at pipecat.ai And find our Discord there. You can ask questions. Um, there's some really cool stuff with Gemini that can be done.

508. 1:07:05

     There, uh, in particular in the Pipecat repo, we built, uh, like, I don't know if you know the game Catch Phrase, where you describe a word and something, you know, guesses it.

509. 1:07:13

     We built a version of that. We had to brand it something else called Word Wrangler, and you as the human, you describe a word, and then you have the AI agent try to answer it.

510. 1:07:22

     So we built a client-server version of that, which I linked to in the repo, and then we have one that's a phone-based one that's, I think, particularly sophisticated and interesting 'cause you might think like, "How the hell would I build this with a speech-to-speech model?"

511. 1:07:34

     We actually use two Gemini agents in the same call, and we use a parallel pipeline where one agent is the host giving out the, the questions to the human user.

512. 1:07:45

     The other is the guesser, and we ha-- you know, kind of limit the audio flow so that the guesser, the AI player, can only hear the user. So there's a bunch of really interesting things getting into the, majorly into the weeds of some of the powers of, um, Pipecat.

513. 1:07:58

     But it also speaks to the strength of having just native audio input being really, really help- helpful. So I'd recommend checking those out. Um, really cool, easy demos to run.

514. 1:08:07

     One's Twilio. The other is, again, a client server. I think it's like a React Next.js project.

515. 1:08:16

     What's that? Word Wrangler. Word Wrangler? Yeah. Yeah. I mean, we could, we could run the Word Wrangler client app. It's actually just on the web.

516. 1:08:26

     Test.

517. 1:08:28

     Welcome to Word Wrangler. I'll, I'll try to guess the words you describe. Remember, don't say any part of the word itself. Ready? Let's go.

518. 1:08:39

     Uh, I'm gonna skip to something easier. Okay, this is something you take pictures with. It's on your phone.

519. 1:08:48

     Is it camera?

520. 1:08:50

     All right. This is a field, uh, related to the study of languages, I think.

521. 1:08:56

     Is it linguistics?

522. 1:08:57

     All right. This is a game with a yellow ball you play with rackets. You hit the ball over the net.

523. 1:09:03

     Is it tennis?

524. 1:09:05

     All right. This is a, a round dessert with chocolate chips sometimes and other fun goodies.

525. 1:09:14

     Is it cookie?

526. 1:09:14

     It's really good even when I'm bad at giving answers, so pretty cool. This is with the-- built with Gemini Live. But again, just an example of things you can build with, uh, with voice AI, so cool, unique interactions.

527. 1:09:31

     All right. I think that's about it. Thanks, everybody. [upbeat music]