Pipecat Cloud: Enterprise Voice Agents Built On Open Source

Talk transcript

1. 0:00

   [upbeat music] Hi, everybody.

2. 0:15

   My name's Quinn. I am a co-founder of a company called Daily. Daily's other founder is in the back there, Nina. Uh, I'm stepping in for my colleague, Mark, who couldn't make it today, so we're gonna do this fast and very informally, but I think that's a good way to do it at an engineering conference.

3. 0:28

   I don't have as much code to show as the last awesome presentation, but I'll try to show a little bit. We're gonna talk about building voice agents today. Uh, w- I work on a open source, vendor-neutral project called Pipecat, um, and a lot of other people at Daily do, too, because voice AI is growing fast and is

4. 0:44

   super interesting and is a good fit for what we do as a company. We started in 2016. We are global infrastructure for real-time audio, video, and now AI for developers.

5. 0:55

   Pipecat sits somewhere higher up in the stack than our traditional infrastructure business. So we'll talk a little bit about how you can build reliable, performant voice AI agents completely using open source software.

6. 1:08

   We also recently launched a layer just on top of our traditional infrastructure designed for hosting voice AI agents. We'll talk just a little bit about that. Um, so we've been doing this a long time.

7. 1:18

   We care a lot about the developer experience for very fast, very responsive real-time audio and video. We have a long list of engineering firsts we're proud of, but that's not why you're here today.

8. 1:28

   Uh, happy to talk about that later, though. Um, if we step back and orient a little bit, what are you doing when you build a voice agent? I, I tend to sort of orient people with three things they have to think about.

9. 1:40

   You've gotta write the code, you have to deploy that code somewhere, and then you have to connect users over the network or over a telephony connection to that agent.

10. 1:52

    A few things here. User expectations are high. Voice AI is new, but it's growing fast, I think, because we're able to with sort of the best technologies that are just now becoming available to meet user expectations.

11. 2:04

    But users expect the AI to understand what they're saying, to feel smart and conversational and human, uh, to be connected to knowledge bases, to have actual access to useful information for whatever they are doing for that user, uh, to sound natural.

12. 2:18

    There's definitely an uncanny valley problem that in generative AI we fell into for a very long time. Now we're on the other side of that for voice AI, which is really exciting.

13. 2:28

    Um, the agents have to respond fast. Humans expect... It varies by language and by culture and by individual, but roughly speaking, humans expect a 500-millisecond response time in natural human conversation.

14. 2:40

    If you don't do that in your voice AI interface, you are probably gonna lose most of your normal users. So we tell people target 800-millisecond voice-to-voice response times. That's not easy to do with today's technology, but it is definitely possible, and build UIs very thoughtfully to understand that humans expect fast responses.

15. 2:59

    The other thing that's hard, uh, a little bit like fast response times, is knowing when to respond. Uh, humans are good but not perfect at knowing when somebody we're talking to is done talking and when we should start talking.

16. 3:12

    Uh, voice AI agents are not as good at that yet, but they're getting better. So we'll talk a tiny bit about that. So why do developers use a framework like Pipecat instead of writing all the code themselves?

17. 3:22

    Well, a little bit of it is all those hard things on the previous slide that you probably don't wanna write the code for yourself if you're mostly thinking about your business logic and your user experience and connecting to all of your systems.

18. 3:33

    You wanna use battle-tested implementations of things like turn detection, interruption handling, context management, calling out to other tools, function, function calling in an asynchronous environment, all that stuff. Um, so developers tend to use frameworks these days for lots of agentic things they do, uh, and voice AI, I think, is even more important to s- sit, sit on

19. 3:55

    top of, uh, really well-tested infrastructure and code components than even in other domains. Um, Pipecat appeals to developers because it's 100% open source and completely vendor neutral. You can use it with lots of different providers at every single layer of the stack that Pipecat enables.

20. 4:12

    Um, for example, there's native telephony support in Pipecat, so you can use Pipecat with lots of different telephony providers in a plug-and-play manner. You can use Twilio, uh, for example, which a lot of developers know.

21. 4:23

    If you're in a geography like India where Twilio doesn't have phone numbers, you can use Plivo. A bunch of other telephony providers are supported. Um, there's a native audio smart turn model that's completely open source in Pipecat.

22. 4:35

    So the community has gotten large enough that there's kinda cutting-edge ML research, at least in the small model domain, coming out of this open source community, which is really fun.

23. 4:43

    Um, Pipecat Cloud, I think, is a really nice advantage for the Pipecat ecosystem. It's the first open source voice AI cloud sort of built from the ground up to host code that you write but that is designed for the problems of voice AI.

24. 4:57

    Uh, and Pipecat supports a lot of models and services. It count to something like 60 plus. All the things you would wanna use in a voice AI agent, uh, are probably in Pipecat main branch.

25. 5:09

    Um, so you probably don't have to write code to get started, though the appeal is that you can write lots and lots of code if you want to. So there's no ceiling.

26. 5:17

    Um, I'll talk a little bit about what the architecture looks like, and we probably won't have time to talk about client SDKs because most of you in this room are probably building for telephony use cases.

27. 5:27

    But there's a really rich and growing set of JavaScript, React, iOS, Android client-side components and SDKs that people in the Pipecat community are using to build multimodal applications that run in the web and on native mobile platforms.

28. 5:42

    Um, we talked about this, so I will actually just skip this slide. Uh, I hope we'll have time for Q&A. That's the most fun part. Um, here's the other piece that, uh, often helps orient people.

29. 5:52

    This is what a Pipecat agent looks like. So you're building a pipeline of programmable media handling elements. Uh, these are all written in Python, although lots of the performant, uh, sensitive ones bottom out in some kind of C code.

30. 6:08

    Uh- Is, uh, it's pretty common in real-time media handling. Um, you probably don't have to worry about that level though. You're probably just thinking in pipe- pipe- pipe- Python.

31. 6:18

    Um, pipe cat- pipelines can be really simple. Uh, they can have just a couple, maybe just three elements, something for the network, something that's doing some processing, and something that's sending stuff back out the network, or they can be quite complicated.

32. 6:29

    And we see enterprise voice agents often become quite complicated because they're doing complicated things and connecting out to a large variety of existing legacy systems. Um, so an example of a little bit of that span, the left two screenshots here are from the Pipecat docs about how you work with the OpenAI audio-centric models in Pipecat.

33. 6:51

    Uh, OpenAI gives you a couple of different shapes of models and APIs that you can use. One is chaining together transcription, large language model operating in text mode, and voice output.

34. 7:02

    The other is using their new and, uh, in some ways experimental speech-to-speech models, which are also really awesome and promising. Um, you can do either of those approaches in Pipecat just by changing probably three or four lines of code.

35. 7:17

    Uh, on the right is the Python, sort of the chunk, core chunk of a few hundred lines of Python code and a flow diagram for a more complicated, uh, pipeline.

36. 7:27

    This is one of my favorite starter kit examples for Pipecat. It uses two, uh, instances of the Gemini multimodal live API in audio native mode. Uh, and one is the conversational flow, and the other is another participant in the conversation that plays a game with the user.

37. 7:45

    So, sort of an LLM as a judge pattern here, but in the context of a game. Uh, and you're moving the audio frames around through both pipelines selectively depending on the results of the real-time inference, uh, which is a pattern we also see in enterprise use cases, but it's fun to clone this and run it and play

38. 8:02

    the game. Um, we listed some of the services here. Uh, we can talk a lot more if, uh, you want to in the Q&A about sort of what we see people actually using in production most often in terms of models and services, uh, in enterprise voice AI.

39. 8:18

    So, that's a very quick rundown of the Pipecat framework, which is how you write the code. Now, how do you deploy it, and why am I talking to you about Pipecat Cloud today?

40. 8:29

    Um, there are a bunch of hard things about voice AI that are unique to these use cases. These are long-running sessions. Uh, they have to use network protocols that are designed for low latency.

41. 8:40

    Um, things like auto-scaling are not available out of the box for these workloads the way they are for something like HTTP workloads. So, I was actually quite resistant for a long time to building anything commercial around Pipecat at Daily because we do the low-level infrastructure.

42. 8:57

    We already have things that we do that serve the Pipecat community. But it got to the point where there... very large percentage of the questions in the Pipecat Discord were about how to deploy and scale.

43. 9:08

    Um, and I- I- I initially sort of felt like that was a solved enough problem because what we do in the infrastructure level helps you in one way. What a lot of our friends and, and customers do much higher up in the stack with

44. 9:22

    platforms that sort of wrap all of the voice AI problem set in very easy-to-use dashboards and tools and GUIs are also really good solutions. But what we came to realize is that there was sort of a middle of the stack that people were asking about a lot in the open source community that boiled down to, "How do

45. 9:39

    I do my Kubernetes?" Um, so people would ask questions in the Pipecat Discord about deployment and scaling, and we would say, "Oh, well, if you really wanna run this stuff yourself on your own infrastructure, here are the five things you do in Kubernetes."

46. 9:51

    And people would say some version of, "Kuber-what?" Um, and we don't have a good answer to that. So we thought we'd come up with a good answer to that, which is a very thin layer on top of our existing global media-oriented real-time infrastructure designed as what I think of as not a very good marketing tagline, but I

47. 10:10

    think of this as a very thin wrapper around Docker and Kubernetes optimized for voice AI. Um, so what are the things we're trying to solve for? Fast start times are very important.

48. 10:21

    If somebody calls your voice agent, uh, and they hear ringing, they want to hear that voice agent pick up the phone and say hello pretty fast. Um, no- almost no matter what you do in AI, you care about cold start times, but it's even more important when the user is initiating some action and expects you to hear

49. 10:39

    audio back. Um, cold starts are hard. If you've built gen AI infrastructure, you know that. Uh, we try to solve the cold start, uh, problem for voice AI. Happy to talk about cold starts a- a- at great detail 'cause it's something I've been thinking a lot about over the last few months.

50. 10:56

    Um, auto-scaling is a little bit related to cold starts. You want your resources to expand as your traffic pattern expands. The alternative is you know exactly what your traffic pattern is, and you just deploy a bunch of resources.

51. 11:09

    Uh, that doesn't work for most workloads. Most people have time-dependent or completely unpredictable workloads, uh, so you need to scale up and scale down. Um, real-time is different from non-real time, and by non-real time, I mean everything that's not conversational latency of a few hundred milliseconds or less.

52. 11:28

    If you are making an HTTP request, you want it to be fast, but you don't really care if your P95 is fifteen hundred milliseconds or two thousand milliseconds in most cases.

53. 11:38

    In a voice AI conversation, you care a lot if your P95 goes up above eight hundred, nine hundred, a thousand milliseconds for the entire voice-to-voice, uh, response chain. Uh, all the little inference calls you make as part of that have to be much faster than that by definition.

54. 11:55

    Um, so the whole networking stack from client to wherever your Pipecat Cloud is running and inside that Kubernetes cluster has to be optimized for real-time.

55. 12:07

    Uh, you probably need global deployment. Uh, you probably have, uh, GDPR or data Residency or other kinds of data privacy requirements, or you just need global deployment because you want these servers close to users because that helps with latency.

56. 12:23

    And all these things have to be, like, delivered at reasonable cost. So we try to take these things off of your plate and help you build quickly and get to market, uh, with your voice agents.

57. 12:36

    Um, couple other things that are just worth flagging here. We've done a lot of work on turn detection, which is sort of one of the twenty twenty-five top three problems most people in voice AI are thinking about how to make better.

58. 12:46

    Um, check out the open source smart turn model that's part of the Pipecat ecosystem if you're interested in, in that. Uh, the open source smart turn model is built into Pipecat Cloud and runs for free.

59. 12:57

    Our friends at Fal host it. Um, you've probably heard of Fal if you're doing gen AI stuff. Very fast, very good GPU-optimized inference. Um, and ambient noise and background voices.

60. 13:08

    So one problem with voice AI is that even though transcription models today are very res- like resilient to all kinds of noisy environments, the LLMs themselves are not. So if you are trying to do transcription and figure out when people are talking and figure out when to fire inference down the chain and ask your LLMs to do

61. 13:29

    something, having background noise that sounds a little bit like speech will trigger lots of interruptions that you don't mean to happen and will inject lots of spurious pseudo-speech into your transcripts.

62. 13:42

    So... And that's true even for speech-to-speech models today. Uh, they're not very resilient to background noise. Um, the best, uh, uh, the, the, the best solution to background noise today is a commercial model from a really great small company called Krisp.

63. 13:55

    The Krisp, uh, model is only available with sort of big chunk of commercial licensing. Uh, you can use Krisp for free inside Pipecat Cloud if you run on Pipecat Cloud.

64. 14:04

    You can also use Krisp in your own Pipecat pipelines with your own license if you run Krisp somewhere else. Uh, finally, agents are non-deterministic. As we all know, there's a whole evals and PM track here, and in every other track we talk about this problem.

65. 14:17

    Um, we've got some nice, uh, low-level building blocks for logging and observability natively in Pipecat and exposed through Pipecat Cloud and a bunch of partners we work with on that.

66. 14:26

    I'm happy to introduce you to the great teams we work with at various companies that are building observability stuff. That is my speed run. I came in twenty seconds under the fifteen minutes.

67. 14:39

    But because we are the last talk in this block, if people want to do Q&A, totally happy to. [audience applauding]

68. 14:50

    Thanks, Grant. Uh, wonderful bit. One, um, actually I have two questions. Two very quick questions.

69. 14:55

    Yeah.

70. 14:55

    One is, we're based out of Sydney, Australia.

71. 14:58

    Yeah.

72. 14:58

    One of the problems we've run into is that eight hundred millisecond thing is the time to, like, go into our OpenAI-

73. 15:03

    Yeah

74. 15:03

    ... and come back. I'm all the way in Australia.

75. 15:04

    Yeah.

76. 15:05

    So OpenAI processing is kind of-

77. 15:06

    Yeah

78. 15:07

    ... extremely. Do you have any alternatives for that? Have you looked at other alternatives for people outside the States?

79. 15:11

    Yes, that's a great question. So the question-- I will repeat the question. The question is, if you're in a geography that is a long way from your inference servers, so in the case of this particular question, you're, uh, serving users in Australia, you're using OpenAI.

80. 15:24

    OpenAI only has inference servers in the US. You don't wanna make extra round trips to the US. So there's a couple answers to that. One is if you make one long haul to the US for all the audio that at the beginning of the chain and at the end of the chain, that is much better than making

81. 15:41

    three inference round trips for transcription, OpenAI, and, uh, voice generation. So that's one tool. We often say to people, "Just deploy close to the inference servers rather than close to the users, and optimize for having one long trip and then a bunch of very, very fast short trips."

82. 15:58

    That's good, but not great. The other option is to run stuff in, uh, on, uh, using open-weights models locally in Australia, which you can definitely do. Uh, it- it's a longer conversation about what use cases you can use, say, the best open-weights models versus the, you know, GPT-4o and Gemini 2 Flash, uh, level models.

83. 16:19

    But there are definitely some voice AI workloads now that you can reliably run on, like the Gemma or the Quinn 3 or the Llama 4 models.

84. 16:26

    Okay. Second question maybe just related to that is, let's say, can we-- if we basically host models in Australia itself-

85. 16:33

    Yeah

86. 16:34

    ... um, what's the interconnectivity with the network from your cloud, Pipecat? Is something like-- do you go through, like, the internet exchange locally out there?

87. 16:41

    Yes. So we have endpoints all over the world that are-- We-- In our world we call them points of presence. So we have the, the sort of the edge server close to the user, and we'll terminate the WebRTC or the telephony connection there.

88. 16:54

    And then we'll route over our own private AWS or OCI backbones to wherever you need to route to. If you're hosting in Australia, uh, you should be able to just hit our endpoints and then, uh, your-

89. 17:07

    Store

90. 17:07

    ... your hosting in Australia. We also-- We have s- some regional availability at Pipecat Cloud now. We will launch a bunch more regional avail- availability at Pipecat Cloud over the next quarter.

91. 17:17

    So I hope we actually have Pipecat Cloud in Australia soon. Although you, you can also obviously self-host in Australia and still use either Pipecat itself or Pipecat Plus Daily in other ways.

92. 17:28

    All right. Cool. Thanks.

93. 17:28

    Thank you. Yeah. Oh, sorry. [laughs]

94. 17:31

    Yeah. So thanks. Uh, thanks for the talk.

95. 17:33

    Thank you.

96. 17:33

    So, uh, there are models like Moshi, I don't know if you've heard-

97. 17:35

    Yeah, yeah, yeah. I love Moshi.

98. 17:37

    So they basically claim that, uh, turn detection is no longer needed because they inherently encode both the speaker and the language model. Do you have experience running those? Do they actually scale?

99. 17:47

    Yes.

100. 17:48

     Can you share-

101. 17:48

     The question is about a, a, a really cool open-weights model called Moshi by a, a French lab called Kyoti. Um, Moshi is a, is, is a sort of next generation research model where the architecture is constant bi-directional streaming.

102. 18:01

     So you're always streaming tokens in, and the model is always streaming tokens out. In a conversational voice situation, which Moshi was designed for, most of the tokens streaming out are silence tokens of some kind.

103. 18:13

     And when they're not silence tokens, it's because the model decided it was gonna do whatever the model's trained to do. Which is really cool because that can mean not just that the model does natural turn-taking, but also that the model can do things like backchanneling.

104. 18:27

     So the model can do the things the humans do that its dataset has audio for. Like when you're talking, I can say, "Mm, ah, yeah. Mm-hmm, yeah. Uh-huh." And it's not actually- A new inference call, it's just streaming.

105. 18:41

     Um, that paper-- the, the Kyotai Labs Moshi architecture paper was my very favorite ML research paper from last year. Now [chuckles] that model itself is not usable in production for a bunch of reasons, including that it is too small a language model to be useful for, uh, basically any real world use case.

106. 19:03

     Um, I, I have more to say about that, but I'm super, super excited about that architecture, but I don't think-- I mean, we're a couple years away from that architecture being actually usable a-and trained as a production model.

107. 19:16

     There are speech-to-speech models from the, from the large labs that are closer to being able to be used in production. Uh, now they are not streaming architecture models, but they are native audio speech-to-speech models, which have a bunch of advantages, including really great multilingual support.

108. 19:31

     So like mixed language stuff is great from those models. Um, in theory, latency reductions. Um, so OpenAI has a, a real-time model called GPT-4o Audio Preview, uh, that sits behind their real-time API.

109. 19:45

     It's a, it's a good model. Uh, Gemini 2.0 Flash, uh, is avail-- is usable in an audio to audio mode, and they're training to, or they're-- they have preview releases of two five Flash.

110. 19:56

     These models are now good enough that you can use them for use cases where you are more concerned about naturalness of the human conversation than you are about reliable instruction following and function calling.

111. 20:07

     They are less reliable in audio mode than the text mode, the, the Soda models operating in text mode. So what we generally see is that for a, a small subset of voice AI use cases today that are really about like conversational dynamics, narrative, storytelling, those models are starting to get adopted.

112. 20:26

     For the majority of sort of enterprise voice AI use cases where you really need best possible instruction following and function calling, those models are not yet the right choice, but they are getting better every release, and all of us expect the world to move to speech-to-speech models being the default for like ninety-five percent of voice AI sometime

113. 20:44

     in the next two years. The question is when in your use case will a particular model architecture sort of cross that threshold in your evals.

114. 20:51

     Sorry, what about Sesame? Would you put Sesame in that same bucket as Gemini and OpenAI, or?

115. 20:56

     Sesame is closer to Moshi. In fact, Sesame-- So there's another open, uh, weights or partly open weights and really interesting model called Sesame. Uh, it's a little like Moshi.

116. 21:08

     It in fact uses the Moshi neural encoder.

117. 21:11

     Yeah.

118. 21:11

     Yeah. It uses Mimi. Um, Ses-so Sesame's not yet been fully released. There isn't a full Sesame release. Uh, also I think Sesame is smaller than probably you would need to use for most enterprise use cases today.

119. 21:27

     Although the lab training Sesame, I think has bigger versions coming. Uh, there's also a speech-to-speech model called Ultravox, which is really good, which is trained on the Llama 3 seventy B backbone, and that team supports that model and has a production voice AI API.

120. 21:42

     That model is worth trying if you are really interested in speech-to-speech models. If Llama three seventy B can do what you want, I think Ultravox is a good choice.

121. 21:50

     If Llama three seventy B isn't quite there for your use case, probably not, but you know, the next release of Ultravox. So speech-to-speech is definitely the future. I, I generally tell people, experiment with it.

122. 22:02

     Don't necessarily start assuming you're gonna use it for your enterprise use case though today.

123. 22:07

     Um.

124. 22:07

     Hey, uh, given your ven-vendor neutrality, can you speak to the strengths and weaknesses of using like the leading edge, uh, multimodal input models like OpenAI and Gemini? Uh, when, when should I use-- choose OpenAI, or when should I choose Gemini?

125. 22:24

     So my opinion is that GPT-4o in text mode and Gemini two oh Flash in text mode are roughly equivalent models for the use cases that I test every day.

126. 22:36

     Um, so I would make the decision-- If you can, I would build a Pipecat pipeline and then just swap the two models and run your evals, um, because they're both really good models.

127. 22:48

     Uh, one of the advantages of Gemini is that it's extremely aggressively priced. So

128. 22:54

     y-you know, a, a, a thirty-minute conversation on Gemini is probably ten times cheaper than a thirty-minute conversation on GPT-4o. Um, you know, that may or may not stay true as they both change their prices, but that's definitely something we hear a lot from customers today, is that they like the pricing of Gemini.

129. 23:11

     The other interesting thing about Gemini is that it operates in native audio input m-mode very well. So you can use Gemini in native audio input mode and then text output mode in a pipeline, and that has advantages for some use cases and some languages, and you can again, test that on your evals.

130. 23:28

     And OpenAI also has native audio support in some of their newer models, but I, I think they're a l-- just a little bit behind the Gemini models in that, uh, in that regard.

131. 23:37

     Um, time for one more or are we done? O-one more, and then we're done. Yeah.

132. 23:42

     What are the general advantages of speech-to-speech versus going speech-to-text, doing something, and then going back to text-to-speech?

133. 23:50

     So what are the general advantages of speech-to-speech instead of text, text inference, uh, to speech and out? So it's super interesting question, and I have a, like a, a practical answer and a philosophical answer.

134. 24:02

     I'll keep them both short. The, the, the practical answer is that you lose information when you transcribe. And so if there's information that's useful in the, um, in the transcription step that i-if there's information in the audio that you would lose that's useful for your s-use case, then a speech-to-speech model is great.

135. 24:21

     Um, so for example, things like mixed language are very hard for small transcription models. Um, you're almost always sort of losing a bunch more information in a mixed language transcription than you are in like a, an optimized model, monolingual transcription.

136. 24:33

     So why not go to the big LLM that just has all this like language knowledge and can do a better job on the multilingual input? Um, the other advantage is potentially you have lower latency.

137. 24:43

     Like if you're-- if you've trained an end-to-end model for speech-to-speech and it's all one model and you're not like chaining together inference calls, you, you can probably get lower latency.

138. 24:51

     Uh, in practice, whether that's true today depends more on the sort of APIs and inference stack than it does on the model architecture. But I think we're all going towards assuming that we just wanna do one inference call for like the bulk of things, and then we might use other little models on the side for, for like

139. 25:05

     subsets. The philosophical answer, though, is that those advantages are probably outweighed by the challenges to today's LLMs architecture. LLM architecture is when you have big context, and big context and audio tokens take up a lot of context tokens.

140. 25:20

     So when you're operating in audio mode, you're just sort of expanding the context massively relative to operating in text mode, and that tends to degrade the performance of the model.

141. 25:29

     I think a little bit relatedly, nobody has as much audio data as they have text data for training. So even though a big model is doing a bunch of transfer learning when you give it a bunch of audio, and it is in theory sort of mapping all that audio to the same latent space as its text reasoning,

142. 25:45

     in practice, it's definitely not doing that exactly. It's doing something like that, but not that. And so because we don't have as much audio data, you see a lot of issues with audio-to-audio models, like the model will sometimes just respond in a totally different language.

143. 26:01

     And that's cool, but it's never what you want in the enterprise, you know, voice AI use case. And the best guess for why that's happening is it's in some right part of the latent space from some projection, but then from some other projection, it's totally in a different part of the latent space when you gave it audio

144. 26:19

     instead of text, even though if you transcribed that text, it would be exactly the same as the audio. Um, so you know, latent spaces are big, and to like actually find our way through them in post-training, you really have to have a lot of data, and nobody has enough audio data yet.

145. 26:36

     But the big labs are gonna fix that because audio matters and multi-turn conversations matter. [upbeat music]