AI Engineer World's Fair 2025
Pipecat Cloud: Enterprise Voice Agents Built On Open Source
Read the talk
Building Voice Agents with Pipecat: Pipelines, Latency, and Model Choice
A responsive voice agent needs more than a good model: it needs programmable media handling, deployment suited to live conversations, and careful choices about where inference runs.
From a talk by Kwindla Hultman Kramer
Before you start: Basic familiarity with Python, language-model APIs, and network latency will help with the pipeline and deployment examples.
What does it take to build a voice agent?
How do you build a voice agent that understands a person, responds naturally, and remains reliable over a real network? Pipecat addresses the application layer of that problem: an open-source, vendor-neutral framework above the infrastructure that carries audio and video. Kwindla Hultman Kramer, Daily co-founder, presents it here in place of his colleague Mark. Daily began in 2016 as infrastructure for real-time media; Pipecat adds agent programming, while Pipecat Cloud adds a hosting layer on that infrastructure.
The work divides into three responsibilities:
- Write the agent: implement its behavior and connect its models and business systems.
- Deploy the agent: give that code somewhere to run.
- Connect the user: carry the conversation over a network or telephony connection.
These are separate engineering problems, but users experience their combined result as a single conversation.
Users expect accurate understanding, useful access to knowledge, conversational intelligence, and a natural voice. Kramer’s assessment is that voice AI has finally crossed the uncanny valley, making those expectations possible to satisfy. Speed is part of that experience: Kramer approximates the expected response time in human conversation at 500 milliseconds, varying with language, culture, and individual. He recommends an 800-millisecond voice-to-voice response target for agents. That is an engineering target, not a measured Pipecat result, and the interface needs to account for the user’s expectation of a prompt response.
Fast generation alone does not tell an agent when to speak. It must also decide whether the person has finished their turn. Humans sometimes get this wrong; an agent needs explicit machinery to avoid answering during a pause or waiting long after the user is done.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse the conversational machinery
The reason to use a framework is to keep business logic from becoming an implementation of every difficult real-time behavior. Pipecat supplies building blocks for turn detection, interruption handling, context management, tool calls, and asynchronous function calling. Those components need to work together while audio continues arriving, not just between isolated requests.
Provider choice extends through the stack. Kramer describes Pipecat as entirely open source and vendor neutral, including native telephony integrations. He gives Twilio and Plivo as alternatives, citing Indian phone-number availability as a reason developers might choose Plivo. The ecosystem also includes the open-source native-audio Smart Turn model, an example of small-model research emerging from the community.
Pipecat Cloud occupies the hosting layer for developers who want to run their own agent code. The distinction matters: Pipecat is the open-source framework; Cloud is managed hosting for it. Kramer positions Cloud as a first-of-its-kind offering for this ecosystem, rather than another constraint on which models developers can use. He reports approximately 60-plus supported models and services in Pipecat’s main branch at the time of the talk. Existing integrations reduce the code needed to start, while programmable pipelines leave room for more elaborate applications.
Telephony is only one client. JavaScript, React, iOS, and Android components and SDKs support multimodal applications on the web and native mobile platforms. The same framework can therefore sit behind a phone call or an application with a richer audio and video interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose the agent as a media pipeline
A Pipecat agent is a pipeline of programmable media-handling elements. Application code is Python, with performance-sensitive components reaching into C underneath. A minimal pipeline can have just three elements: network input, processing, and network output. Enterprise pipelines grow because their tasks involve more processing and connections to existing systems.
The basic composition is explicit in Python. With configured transport, service objects, and a context aggregator supplied by the application, a transcription-based pipeline can be assembled like this:
python
from pipecat.pipeline.pipeline import Pipeline
def build_voice_pipeline(transport, stt, llm, tts, context_aggregator):
return Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
])
The ordering expresses the media path: incoming audio becomes a transcript, the context aggregator’s user processor turns transcripts into conversational context for the language model, and text-to-speech produces outgoing audio. The assistant processor records the assistant’s response in that context for subsequent turns. The OpenAI examples shown in the talk contrast that chain with a native speech-to-speech service. Kramer estimates that switching between the two approaches in those examples requires changing three or four lines of code.
A more distinctive example uses two instances of the Gemini Multimodal Live API in native-audio mode. One handles the conversational flow; the other participates in a game, applying an LLM-as-judge pattern. The starter kit’s core is a few hundred lines of Python, accompanied by a flow diagram. Its important mechanism is selective audio-frame routing: real-time inference results determine which audio frames move through which pipeline. This is more than placing two models next to each other. The outputs of inference govern the media flow, a pattern Kramer also sees in enterprise applications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy for live sessions, not isolated requests
Once the pipeline works, deployment introduces another set of constraints. Voice agents run long-lived sessions over protocols chosen for low latency. The autoscaling patterns available out of the box for HTTP workloads do not automatically solve that problem. Deployment and scaling became such a large share of Pipecat community questions that Daily reconsidered whether its existing low-level infrastructure was enough.
There were already useful solutions above and below this gap: media infrastructure at the bottom, and platforms with dashboards and graphical tools at the top. Developers writing their own Pipecat code still needed help operating it. Advice about the things to configure in Kubernetes often met with a simpler question: what is Kubernetes? Pipecat Cloud was designed for that middle layer. Kramer describes it as a thin wrapper around Docker and Kubernetes, optimized for voice AI and built on Daily’s global real-time infrastructure.
The cold-start problem becomes concrete when someone calls an agent. While the caller hears ringing, the system must get ready to pick up and produce a greeting. A delayed first response is already part of the user experience, before any substantive conversation begins. Fast starts are therefore a central hosting requirement, separate from the latency of later conversational turns.
Capacity must also follow traffic. Preprovisioning a fixed fleet works only when the workload is sufficiently predictable; time-dependent or unpredictable demand requires resources to expand and contract. The latency tolerance differs from ordinary request handling:
| Workload | Kramer’s latency comparison |
|---|---|
| Many HTTP applications | P95 of 1,500–2,000 ms may be tolerable |
| Voice conversation | P95 around 800–1,000 ms for the entire voice-to-voice chain is already concerning |
These are his workload comparisons, not hosting benchmark results. Because the conversational budget covers the entire chain, each inference call must consume only part of it. Network behavior matters both between the client and the deployment and inside the Kubernetes cluster.
Global deployment serves more than speed. It can place servers near users, but it may also be necessary for GDPR, data residency, or other privacy requirements. The hosting goal is to absorb these operational concerns at a reasonable cost so developers can get an agent into use sooner.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep noise from becoming a conversational event
Turn detection remains one of the central voice-agent problems. Kramer describes Smart Turn as built into Pipecat Cloud and free to run there, with fal providing GPU-optimized hosting. Its role is to improve the decision about when the user’s turn is complete.
Background noise creates a related but distinct failure. A transcription model may be resilient enough to recover words in a noisy room, while the surrounding agent still reacts incorrectly to sound that resembles speech. That sound can trigger an unintended interruption, insert spurious pseudo-speech into the transcript, and cause downstream inference to run when it should not. Native speech-to-speech models are also susceptible, in Kramer’s assessment. Accurate transcription does not by itself imply reliable conversational control.
Kramer recommends Krisp as the strongest noise-suppression option available at the time. He describes it as included without an additional charge inside Pipecat Cloud, while independently hosted Pipecat pipelines can use it with their own commercial license. Those are the talk’s hosting and licensing terms.
Even with clean audio and better turn timing, agents remain nondeterministic. Pipecat provides low-level logging and observability building blocks, exposed through Cloud, with partners offering additional observability tooling. These give developers a way to inspect the behavior of a live agent rather than assuming that a successfully connected pipeline is behaving correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Place the agent near the inference it needs
The first audience question exposes a limit to the advice to deploy near users. A developer in Sydney reports that the OpenAI trip and return alone takes roughly 800 milliseconds. If that consumes the conversational budget, where should the agent run?
Kramer answers from the premise of inference in the United States. His blanket statement that OpenAI inference was US-only needs qualification: OpenAI had already announced European processing for eligible API customers and endpoints, although that does not establish an alternative for this developer’s account or audio endpoint. The placement principle remains useful: avoid repeatedly crossing the ocean inside a sequential inference chain.
If transcription, language-model inference, and voice generation are remote, an agent running near the user may make a long-distance round trip for each stage. Instead, place the agent near those inference servers. Audio travels the long-haul path into and out of the system, while the internal inference calls take short paths.
| Placement | Long-distance work |
|---|---|
| Agent near user, inference overseas | Separate remote calls for transcription, LLM, and voice generation |
| Agent near inference | Audio ingress and egress; short internal inference calls |
Kramer describes this as an improvement, not an ideal solution: the long-haul audio path still exists.
The alternative is to host open-weight models locally in Australia. Whether that works depends on the task’s language-model requirements. Kramer names Gemma, Qwen 3, and Llama 4 as viable for some voice workloads, while distinguishing their suitability from that of GPT-4o and Gemini 2.0 Flash. Geography can be improved by changing the model deployment, but the replacement still has to satisfy the application’s evaluations.
The follow-up asks how locally hosted models connect to Daily’s network. Kramer describes worldwide points of presence that terminate WebRTC or telephony near the user, then route onward over private AWS or OCI backbones. Australian hosting can connect through those local endpoints. Pipecat Cloud had some regional availability at the time, with more planned for the following quarter; Australia was a hoped-for addition, not a confirmed region. Self-hosting in Australia remained an option with Pipecat alone or Pipecat plus Daily.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Moshi makes turn-taking part of generation
Could a model make separate turn detection unnecessary? An audience member raises Moshi, Kyutai’s speech-text foundation model, and asks whether its approach scales. The architectural difference is continuous bidirectional streaming: tokens keep arriving at the model, and tokens keep leaving it.
Most outgoing tokens during listening represent silence. When the model produces something else, it is choosing to speak within the same ongoing stream. That supports both turn-taking and backchanneling: a brief acknowledgment while the other person continues talking. Such an acknowledgment does not require starting a new inference call. It emerges from streaming generation and the conversational behavior represented in the training audio.
Kramer strongly endorses the architecture paper while separating that enthusiasm from production readiness. He judges the model’s language capability too small for real-world enterprise tasks and forecasts that a production-trained version of this architecture is still a couple of years away. His answer supplies an architectural explanation and a readiness judgment, rather than a scaling measurement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Natural conversation and dependable task execution
Large-lab native speech-to-speech models offer a different route to more natural conversations. Kramer distinguishes them from Moshi’s continuously bidirectional architecture. Their attractions include multilingual and mixed-language handling, with the possibility of lower latency. He refers to GPT-4o Audio Preview when discussing OpenAI’s Realtime API; the historical Realtime API launch names its model gpt-4o-realtime-preview, distinct from the audio model naming used in Chat Completions. He also points to Gemini 2.0 Flash in audio-to-audio mode and preview releases of Gemini 2.5 Flash.
The adoption decision turns on what the application cannot afford to get wrong:
- Conversational dynamics: narrative, storytelling, and other experiences centered on natural interaction are beginning to use native audio.
- Enterprise task execution: applications that require the best available instruction following and function calling still favor leading text-mode models in Kramer’s assessment.
Audio mode can make a conversation feel better while making its actions less dependable. The useful threshold is therefore the one established by the application’s own evaluations. Kramer forecasts speech-to-speech becoming the default for about 95% of voice AI within two years; that is a prediction, not measured adoption.
An audience follow-up brings in Sesame. Kramer places it closer to Moshi and notes its use of Mimi, the neural audio codec. The released CSM component generates speech using Mimi codes; it should not be equated with the complete conversational demo. At the time of the talk, Kramer describes the broader release as incomplete and worries that the available model is too small for most enterprise tasks, while expecting larger versions.
He also recommends trying Ultravox, describing the version under discussion as a speech-to-speech offering built on a Llama 3 70B backbone, supported by its team and a production voice-AI API. His recommendation is conditional: if that backbone can do the required task, Ultravox is worth evaluating; if it cannot, the audio interface does not remove the capability limit. Experiment with speech-to-speech, but do not assume it is already the right foundation for an enterprise workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare providers inside the same pipeline
Asked when to choose OpenAI or Gemini, Kramer starts with task performance. For the use cases he tests daily, he finds GPT-4o and Gemini 2.0 Flash roughly equivalent in text mode. His practical procedure is straightforward: build the Pipecat pipeline, swap the models, and run the same evaluations. Keeping the surrounding pipeline stable makes the provider choice something the application can test.
Cost can then separate otherwise suitable options. Kramer estimates that a 30-minute Gemini conversation was probably about ten times cheaper than a 30-minute GPT-4o conversation at talk-time pricing. The estimate does not specify token volumes, modalities, caching, or model snapshots, so it cannot serve as a reproducible cost calculation; Kramer also explicitly expects prices to change.
Input and output modalities provide another axis of comparison. Gemini can take native audio input and produce text output, preserving audio as model input without requiring an audio response from that same model. Kramer sees advantages for some languages and use cases and recommends testing those in the application’s evaluations. He judges OpenAI’s newer native-audio input support somewhat behind Gemini’s in this respect at the time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What direct audio preserves—and what it costs
The final question asks why speech-to-speech should be preferable to transcription, text inference, and speech synthesis at all. The practical answer starts with information loss: transcription discards information in the audio. If the discarded information matters to the task, feeding audio directly to the model can help. Mixed-language speech is a concrete example. Small transcription models can struggle more with it than with optimized monolingual transcription, while a large model has broader language knowledge to apply to the original audio.
A single end-to-end model can also avoid chaining multiple inference calls, potentially reducing latency. Whether that advantage appears in practice depends on the APIs and inference infrastructure, not just the model architecture. Kramer expects a direction in which one main inference call handles most of the interaction, with smaller models alongside it for particular subtasks.
The costs begin with context. Audio consumes many more context tokens than an equivalent text representation, greatly expanding the context the model must handle. Kramer connects that expansion with degraded performance. Training data adds another limitation: there is much less audio data than text data, so learning from audio does not reproduce text-mode reasoning behavior exactly, even when transfer learning brings the representations closer together.
A concrete symptom is an audio-to-audio model unexpectedly replying in a different language. That may be an interesting generation, but it is the wrong behavior for an enterprise conversation. Kramer offers a hypothesis: audio and its equivalent transcript may reach related regions of the model’s latent space without supporting the same behavior across all relevant dimensions. His explanation is speculative, but the operational failure is clear—the modality can change the response even when the linguistic content is equivalent.
More audio data and post-training are needed to make that behavior reliable. Kramer closes by expecting the major labs to address the gap because audio and multi-turn conversation matter. The reason to keep experimenting is the information direct audio preserves; the reason to keep evaluating is that preserving more input does not yet guarantee better task execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python framework for composing voice and multimodal agents from media pipelines, transports and model integrations.
Pipecat's turn-detection project, with instructions for running the model locally.
Historical introduction to the Realtime API and its distinction from audio support in Chat Completions.
Kyutai's full-duplex dialogue architecture models user and assistant speech as parallel streams.
Speech-generation model using a Llama backbone and Mimi audio codes.
Official Ultravox project repository with model-weight links and training resources.
Further reading
Original cross-language study measuring the gaps between questions and responses in natural conversation.
Updates since the talk
The January 2026 announcement covering managed deployment, scaling, regions and bundled services.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everybody.
- 0:15
My name's Quinn. I am a co-founder of a company called Daily. Daily's other founder is in the back there, Nina. Uh, I'm stepping in for my colleague, Mark, who couldn't make it today, so we're gonna do this fast and very informally, but I think that's a good way to do it at an engineering conference.
- 0:28
I don't have as much code to show as the last awesome presentation, but I'll try to show a little bit. We're gonna talk about building voice agents today. Uh, w- I work on a open source, vendor-neutral project called Pipecat, um, and a lot of other people at Daily do, too, because voice AI is growing fast and is
- 0:44
super interesting and is a good fit for what we do as a company. We started in 2016. We are global infrastructure for real-time audio, video, and now AI for developers.
- 0:55
Pipecat sits somewhere higher up in the stack than our traditional infrastructure business. So we'll talk a little bit about how you can build reliable, performant voice AI agents completely using open source software.
- 1:08
We also recently launched a layer just on top of our traditional infrastructure designed for hosting voice AI agents. We'll talk just a little bit about that. Um, so we've been doing this a long time.
- 1:18
We care a lot about the developer experience for very fast, very responsive real-time audio and video. We have a long list of engineering firsts we're proud of, but that's not why you're here today.
- 1:28
Uh, happy to talk about that later, though. Um, if we step back and orient a little bit, what are you doing when you build a voice agent? I, I tend to sort of orient people with three things they have to think about.
- 1:40
You've gotta write the code, you have to deploy that code somewhere, and then you have to connect users over the network or over a telephony connection to that agent.
- 1:52
A few things here. User expectations are high. Voice AI is new, but it's growing fast, I think, because we're able to with sort of the best technologies that are just now becoming available to meet user expectations.
- 2:04
But users expect the AI to understand what they're saying, to feel smart and conversational and human, uh, to be connected to knowledge bases, to have actual access to useful information for whatever they are doing for that user, uh, to sound natural.
- 2:18
There's definitely an uncanny valley problem that in generative AI we fell into for a very long time. Now we're on the other side of that for voice AI, which is really exciting.
- 2:28
Um, the agents have to respond fast. Humans expect... It varies by language and by culture and by individual, but roughly speaking, humans expect a 500-millisecond response time in natural human conversation.
- 2:40
If you don't do that in your voice AI interface, you are probably gonna lose most of your normal users. So we tell people target 800-millisecond voice-to-voice response times. That's not easy to do with today's technology, but it is definitely possible, and build UIs very thoughtfully to understand that humans expect fast responses.
- 2:59
The other thing that's hard, uh, a little bit like fast response times, is knowing when to respond. Uh, humans are good but not perfect at knowing when somebody we're talking to is done talking and when we should start talking.
- 3:12
Uh, voice AI agents are not as good at that yet, but they're getting better. So we'll talk a tiny bit about that. So why do developers use a framework like Pipecat instead of writing all the code themselves?
- 3:22
Well, a little bit of it is all those hard things on the previous slide that you probably don't wanna write the code for yourself if you're mostly thinking about your business logic and your user experience and connecting to all of your systems.
- 3:33
You wanna use battle-tested implementations of things like turn detection, interruption handling, context management, calling out to other tools, function, function calling in an asynchronous environment, all that stuff. Um, so developers tend to use frameworks these days for lots of agentic things they do, uh, and voice AI, I think, is even more important to s- sit, sit on
- 3:55
top of, uh, really well-tested infrastructure and code components than even in other domains. Um, Pipecat appeals to developers because it's 100% open source and completely vendor neutral. You can use it with lots of different providers at every single layer of the stack that Pipecat enables.
- 4:12
Um, for example, there's native telephony support in Pipecat, so you can use Pipecat with lots of different telephony providers in a plug-and-play manner. You can use Twilio, uh, for example, which a lot of developers know.
- 4:23
If you're in a geography like India where Twilio doesn't have phone numbers, you can use Plivo. A bunch of other telephony providers are supported. Um, there's a native audio smart turn model that's completely open source in Pipecat.
- 4:35
So the community has gotten large enough that there's kinda cutting-edge ML research, at least in the small model domain, coming out of this open source community, which is really fun.
- 4:43
Um, Pipecat Cloud, I think, is a really nice advantage for the Pipecat ecosystem. It's the first open source voice AI cloud sort of built from the ground up to host code that you write but that is designed for the problems of voice AI.
- 4:57
Uh, and Pipecat supports a lot of models and services. It count to something like 60 plus. All the things you would wanna use in a voice AI agent, uh, are probably in Pipecat main branch.
- 5:09
Um, so you probably don't have to write code to get started, though the appeal is that you can write lots and lots of code if you want to. So there's no ceiling.
- 5:17
Um, I'll talk a little bit about what the architecture looks like, and we probably won't have time to talk about client SDKs because most of you in this room are probably building for telephony use cases.
- 5:27
But there's a really rich and growing set of JavaScript, React, iOS, Android client-side components and SDKs that people in the Pipecat community are using to build multimodal applications that run in the web and on native mobile platforms.
- 5:42
Um, we talked about this, so I will actually just skip this slide. Uh, I hope we'll have time for Q&A. That's the most fun part. Um, here's the other piece that, uh, often helps orient people.
- 5:52
This is what a Pipecat agent looks like. So you're building a pipeline of programmable media handling elements. Uh, these are all written in Python, although lots of the performant, uh, sensitive ones bottom out in some kind of C code.
- 6:08
Uh- Is, uh, it's pretty common in real-time media handling. Um, you probably don't have to worry about that level though. You're probably just thinking in pipe- pipe- pipe- Python.
- 6:18
Um, pipe cat- pipelines can be really simple. Uh, they can have just a couple, maybe just three elements, something for the network, something that's doing some processing, and something that's sending stuff back out the network, or they can be quite complicated.
- 6:29
And we see enterprise voice agents often become quite complicated because they're doing complicated things and connecting out to a large variety of existing legacy systems. Um, so an example of a little bit of that span, the left two screenshots here are from the Pipecat docs about how you work with the OpenAI audio-centric models in Pipecat.
- 6:51
Uh, OpenAI gives you a couple of different shapes of models and APIs that you can use. One is chaining together transcription, large language model operating in text mode, and voice output.
- 7:02
The other is using their new and, uh, in some ways experimental speech-to-speech models, which are also really awesome and promising. Um, you can do either of those approaches in Pipecat just by changing probably three or four lines of code.
- 7:17
Uh, on the right is the Python, sort of the chunk, core chunk of a few hundred lines of Python code and a flow diagram for a more complicated, uh, pipeline.
- 7:27
This is one of my favorite starter kit examples for Pipecat. It uses two, uh, instances of the Gemini multimodal live API in audio native mode. Uh, and one is the conversational flow, and the other is another participant in the conversation that plays a game with the user.
- 7:45
So, sort of an LLM as a judge pattern here, but in the context of a game. Uh, and you're moving the audio frames around through both pipelines selectively depending on the results of the real-time inference, uh, which is a pattern we also see in enterprise use cases, but it's fun to clone this and run it and play
- 8:02
the game. Um, we listed some of the services here. Uh, we can talk a lot more if, uh, you want to in the Q&A about sort of what we see people actually using in production most often in terms of models and services, uh, in enterprise voice AI.
- 8:18
So, that's a very quick rundown of the Pipecat framework, which is how you write the code. Now, how do you deploy it, and why am I talking to you about Pipecat Cloud today?
- 8:29
Um, there are a bunch of hard things about voice AI that are unique to these use cases. These are long-running sessions. Uh, they have to use network protocols that are designed for low latency.
- 8:40
Um, things like auto-scaling are not available out of the box for these workloads the way they are for something like HTTP workloads. So, I was actually quite resistant for a long time to building anything commercial around Pipecat at Daily because we do the low-level infrastructure.
- 8:57
We already have things that we do that serve the Pipecat community. But it got to the point where there... very large percentage of the questions in the Pipecat Discord were about how to deploy and scale.
- 9:08
Um, and I- I- I initially sort of felt like that was a solved enough problem because what we do in the infrastructure level helps you in one way. What a lot of our friends and, and customers do much higher up in the stack with
- 9:22
platforms that sort of wrap all of the voice AI problem set in very easy-to-use dashboards and tools and GUIs are also really good solutions. But what we came to realize is that there was sort of a middle of the stack that people were asking about a lot in the open source community that boiled down to, "How do
- 9:39
I do my Kubernetes?" Um, so people would ask questions in the Pipecat Discord about deployment and scaling, and we would say, "Oh, well, if you really wanna run this stuff yourself on your own infrastructure, here are the five things you do in Kubernetes."
- 9:51
And people would say some version of, "Kuber-what?" Um, and we don't have a good answer to that. So we thought we'd come up with a good answer to that, which is a very thin layer on top of our existing global media-oriented real-time infrastructure designed as what I think of as not a very good marketing tagline, but I
- 10:10
think of this as a very thin wrapper around Docker and Kubernetes optimized for voice AI. Um, so what are the things we're trying to solve for? Fast start times are very important.
- 10:21
If somebody calls your voice agent, uh, and they hear ringing, they want to hear that voice agent pick up the phone and say hello pretty fast. Um, no- almost no matter what you do in AI, you care about cold start times, but it's even more important when the user is initiating some action and expects you to hear
- 10:39
audio back. Um, cold starts are hard. If you've built gen AI infrastructure, you know that. Uh, we try to solve the cold start, uh, problem for voice AI. Happy to talk about cold starts a- a- at great detail 'cause it's something I've been thinking a lot about over the last few months.
- 10:56
Um, auto-scaling is a little bit related to cold starts. You want your resources to expand as your traffic pattern expands. The alternative is you know exactly what your traffic pattern is, and you just deploy a bunch of resources.
- 11:09
Uh, that doesn't work for most workloads. Most people have time-dependent or completely unpredictable workloads, uh, so you need to scale up and scale down. Um, real-time is different from non-real time, and by non-real time, I mean everything that's not conversational latency of a few hundred milliseconds or less.
- 11:28
If you are making an HTTP request, you want it to be fast, but you don't really care if your P95 is fifteen hundred milliseconds or two thousand milliseconds in most cases.
- 11:38
In a voice AI conversation, you care a lot if your P95 goes up above eight hundred, nine hundred, a thousand milliseconds for the entire voice-to-voice, uh, response chain. Uh, all the little inference calls you make as part of that have to be much faster than that by definition.
- 11:55
Um, so the whole networking stack from client to wherever your Pipecat Cloud is running and inside that Kubernetes cluster has to be optimized for real-time.
- 12:07
Uh, you probably need global deployment. Uh, you probably have, uh, GDPR or data Residency or other kinds of data privacy requirements, or you just need global deployment because you want these servers close to users because that helps with latency.
- 12:23
And all these things have to be, like, delivered at reasonable cost. So we try to take these things off of your plate and help you build quickly and get to market, uh, with your voice agents.
- 12:36
Um, couple other things that are just worth flagging here. We've done a lot of work on turn detection, which is sort of one of the twenty twenty-five top three problems most people in voice AI are thinking about how to make better.
- 12:46
Um, check out the open source smart turn model that's part of the Pipecat ecosystem if you're interested in, in that. Uh, the open source smart turn model is built into Pipecat Cloud and runs for free.
- 12:57
Our friends at Fal host it. Um, you've probably heard of Fal if you're doing gen AI stuff. Very fast, very good GPU-optimized inference. Um, and ambient noise and background voices.
- 13:08
So one problem with voice AI is that even though transcription models today are very res- like resilient to all kinds of noisy environments, the LLMs themselves are not. So if you are trying to do transcription and figure out when people are talking and figure out when to fire inference down the chain and ask your LLMs to do
- 13:29
something, having background noise that sounds a little bit like speech will trigger lots of interruptions that you don't mean to happen and will inject lots of spurious pseudo-speech into your transcripts.
- 13:42
So... And that's true even for speech-to-speech models today. Uh, they're not very resilient to background noise. Um, the best, uh, uh, the, the, the best solution to background noise today is a commercial model from a really great small company called Krisp.
- 13:55
The Krisp, uh, model is only available with sort of big chunk of commercial licensing. Uh, you can use Krisp for free inside Pipecat Cloud if you run on Pipecat Cloud.
- 14:04
You can also use Krisp in your own Pipecat pipelines with your own license if you run Krisp somewhere else. Uh, finally, agents are non-deterministic. As we all know, there's a whole evals and PM track here, and in every other track we talk about this problem.
- 14:17
Um, we've got some nice, uh, low-level building blocks for logging and observability natively in Pipecat and exposed through Pipecat Cloud and a bunch of partners we work with on that.
- 14:26
I'm happy to introduce you to the great teams we work with at various companies that are building observability stuff. That is my speed run. I came in twenty seconds under the fifteen minutes.
- 14:39
But because we are the last talk in this block, if people want to do Q&A, totally happy to. [audience applauding]
- 14:50
Thanks, Grant. Uh, wonderful bit. One, um, actually I have two questions. Two very quick questions.
- 14:55
Yeah.
- 14:55
One is, we're based out of Sydney, Australia.
- 14:58
Yeah.
- 14:58
One of the problems we've run into is that eight hundred millisecond thing is the time to, like, go into our OpenAI-
- 15:03
Yeah
- 15:03
... and come back. I'm all the way in Australia.
- 15:04
Yeah.
- 15:05
So OpenAI processing is kind of-
- 15:06
Yeah
- 15:07
... extremely. Do you have any alternatives for that? Have you looked at other alternatives for people outside the States?
- 15:11
Yes, that's a great question. So the question-- I will repeat the question. The question is, if you're in a geography that is a long way from your inference servers, so in the case of this particular question, you're, uh, serving users in Australia, you're using OpenAI.
- 15:24
OpenAI only has inference servers in the US. You don't wanna make extra round trips to the US. So there's a couple answers to that. One is if you make one long haul to the US for all the audio that at the beginning of the chain and at the end of the chain, that is much better than making
- 15:41
three inference round trips for transcription, OpenAI, and, uh, voice generation. So that's one tool. We often say to people, "Just deploy close to the inference servers rather than close to the users, and optimize for having one long trip and then a bunch of very, very fast short trips."
- 15:58
That's good, but not great. The other option is to run stuff in, uh, on, uh, using open-weights models locally in Australia, which you can definitely do. Uh, it- it's a longer conversation about what use cases you can use, say, the best open-weights models versus the, you know, GPT-4o and Gemini 2 Flash, uh, level models.
- 16:19
But there are definitely some voice AI workloads now that you can reliably run on, like the Gemma or the Quinn 3 or the Llama 4 models.
- 16:26
Okay. Second question maybe just related to that is, let's say, can we-- if we basically host models in Australia itself-
- 16:33
Yeah
- 16:34
... um, what's the interconnectivity with the network from your cloud, Pipecat? Is something like-- do you go through, like, the internet exchange locally out there?
- 16:41
Yes. So we have endpoints all over the world that are-- We-- In our world we call them points of presence. So we have the, the sort of the edge server close to the user, and we'll terminate the WebRTC or the telephony connection there.
- 16:54
And then we'll route over our own private AWS or OCI backbones to wherever you need to route to. If you're hosting in Australia, uh, you should be able to just hit our endpoints and then, uh, your-
- 17:07
Store
- 17:07
... your hosting in Australia. We also-- We have s- some regional availability at Pipecat Cloud now. We will launch a bunch more regional avail- availability at Pipecat Cloud over the next quarter.
- 17:17
So I hope we actually have Pipecat Cloud in Australia soon. Although you, you can also obviously self-host in Australia and still use either Pipecat itself or Pipecat Plus Daily in other ways.
- 17:28
All right. Cool. Thanks.
- 17:28
Thank you. Yeah. Oh, sorry. [laughs]
- 17:31
Yeah. So thanks. Uh, thanks for the talk.
- 17:33
Thank you.
- 17:33
So, uh, there are models like Moshi, I don't know if you've heard-
- 17:35
Yeah, yeah, yeah. I love Moshi.
- 17:37
So they basically claim that, uh, turn detection is no longer needed because they inherently encode both the speaker and the language model. Do you have experience running those? Do they actually scale?
- 17:47
Yes.
- 17:48
Can you share-
- 17:48
The question is about a, a, a really cool open-weights model called Moshi by a, a French lab called Kyoti. Um, Moshi is a, is, is a sort of next generation research model where the architecture is constant bi-directional streaming.
- 18:01
So you're always streaming tokens in, and the model is always streaming tokens out. In a conversational voice situation, which Moshi was designed for, most of the tokens streaming out are silence tokens of some kind.
- 18:13
And when they're not silence tokens, it's because the model decided it was gonna do whatever the model's trained to do. Which is really cool because that can mean not just that the model does natural turn-taking, but also that the model can do things like backchanneling.
- 18:27
So the model can do the things the humans do that its dataset has audio for. Like when you're talking, I can say, "Mm, ah, yeah. Mm-hmm, yeah. Uh-huh." And it's not actually- A new inference call, it's just streaming.
- 18:41
Um, that paper-- the, the Kyotai Labs Moshi architecture paper was my very favorite ML research paper from last year. Now [chuckles] that model itself is not usable in production for a bunch of reasons, including that it is too small a language model to be useful for, uh, basically any real world use case.
- 19:03
Um, I, I have more to say about that, but I'm super, super excited about that architecture, but I don't think-- I mean, we're a couple years away from that architecture being actually usable a-and trained as a production model.
- 19:16
There are speech-to-speech models from the, from the large labs that are closer to being able to be used in production. Uh, now they are not streaming architecture models, but they are native audio speech-to-speech models, which have a bunch of advantages, including really great multilingual support.
- 19:31
So like mixed language stuff is great from those models. Um, in theory, latency reductions. Um, so OpenAI has a, a real-time model called GPT-4o Audio Preview, uh, that sits behind their real-time API.
- 19:45
It's a, it's a good model. Uh, Gemini 2.0 Flash, uh, is avail-- is usable in an audio to audio mode, and they're training to, or they're-- they have preview releases of two five Flash.
- 19:56
These models are now good enough that you can use them for use cases where you are more concerned about naturalness of the human conversation than you are about reliable instruction following and function calling.
- 20:07
They are less reliable in audio mode than the text mode, the, the Soda models operating in text mode. So what we generally see is that for a, a small subset of voice AI use cases today that are really about like conversational dynamics, narrative, storytelling, those models are starting to get adopted.
- 20:26
For the majority of sort of enterprise voice AI use cases where you really need best possible instruction following and function calling, those models are not yet the right choice, but they are getting better every release, and all of us expect the world to move to speech-to-speech models being the default for like ninety-five percent of voice AI sometime
- 20:44
in the next two years. The question is when in your use case will a particular model architecture sort of cross that threshold in your evals.
- 20:51
Sorry, what about Sesame? Would you put Sesame in that same bucket as Gemini and OpenAI, or?
- 20:56
Sesame is closer to Moshi. In fact, Sesame-- So there's another open, uh, weights or partly open weights and really interesting model called Sesame. Uh, it's a little like Moshi.
- 21:08
It in fact uses the Moshi neural encoder.
- 21:11
Yeah.
- 21:11
Yeah. It uses Mimi. Um, Ses-so Sesame's not yet been fully released. There isn't a full Sesame release. Uh, also I think Sesame is smaller than probably you would need to use for most enterprise use cases today.
- 21:27
Although the lab training Sesame, I think has bigger versions coming. Uh, there's also a speech-to-speech model called Ultravox, which is really good, which is trained on the Llama 3 seventy B backbone, and that team supports that model and has a production voice AI API.
- 21:42
That model is worth trying if you are really interested in speech-to-speech models. If Llama three seventy B can do what you want, I think Ultravox is a good choice.
- 21:50
If Llama three seventy B isn't quite there for your use case, probably not, but you know, the next release of Ultravox. So speech-to-speech is definitely the future. I, I generally tell people, experiment with it.
- 22:02
Don't necessarily start assuming you're gonna use it for your enterprise use case though today.
- 22:07
Um.
- 22:07
Hey, uh, given your ven-vendor neutrality, can you speak to the strengths and weaknesses of using like the leading edge, uh, multimodal input models like OpenAI and Gemini? Uh, when, when should I use-- choose OpenAI, or when should I choose Gemini?
- 22:24
So my opinion is that GPT-4o in text mode and Gemini two oh Flash in text mode are roughly equivalent models for the use cases that I test every day.
- 22:36
Um, so I would make the decision-- If you can, I would build a Pipecat pipeline and then just swap the two models and run your evals, um, because they're both really good models.
- 22:48
Uh, one of the advantages of Gemini is that it's extremely aggressively priced. So
- 22:54
y-you know, a, a, a thirty-minute conversation on Gemini is probably ten times cheaper than a thirty-minute conversation on GPT-4o. Um, you know, that may or may not stay true as they both change their prices, but that's definitely something we hear a lot from customers today, is that they like the pricing of Gemini.
- 23:11
The other interesting thing about Gemini is that it operates in native audio input m-mode very well. So you can use Gemini in native audio input mode and then text output mode in a pipeline, and that has advantages for some use cases and some languages, and you can again, test that on your evals.
- 23:28
And OpenAI also has native audio support in some of their newer models, but I, I think they're a l-- just a little bit behind the Gemini models in that, uh, in that regard.
- 23:37
Um, time for one more or are we done? O-one more, and then we're done. Yeah.
- 23:42
What are the general advantages of speech-to-speech versus going speech-to-text, doing something, and then going back to text-to-speech?
- 23:50
So what are the general advantages of speech-to-speech instead of text, text inference, uh, to speech and out? So it's super interesting question, and I have a, like a, a practical answer and a philosophical answer.
- 24:02
I'll keep them both short. The, the, the practical answer is that you lose information when you transcribe. And so if there's information that's useful in the, um, in the transcription step that i-if there's information in the audio that you would lose that's useful for your s-use case, then a speech-to-speech model is great.
- 24:21
Um, so for example, things like mixed language are very hard for small transcription models. Um, you're almost always sort of losing a bunch more information in a mixed language transcription than you are in like a, an optimized model, monolingual transcription.
- 24:33
So why not go to the big LLM that just has all this like language knowledge and can do a better job on the multilingual input? Um, the other advantage is potentially you have lower latency.
- 24:43
Like if you're-- if you've trained an end-to-end model for speech-to-speech and it's all one model and you're not like chaining together inference calls, you, you can probably get lower latency.
- 24:51
Uh, in practice, whether that's true today depends more on the sort of APIs and inference stack than it does on the model architecture. But I think we're all going towards assuming that we just wanna do one inference call for like the bulk of things, and then we might use other little models on the side for, for like
- 25:05
subsets. The philosophical answer, though, is that those advantages are probably outweighed by the challenges to today's LLMs architecture. LLM architecture is when you have big context, and big context and audio tokens take up a lot of context tokens.
- 25:20
So when you're operating in audio mode, you're just sort of expanding the context massively relative to operating in text mode, and that tends to degrade the performance of the model.
- 25:29
I think a little bit relatedly, nobody has as much audio data as they have text data for training. So even though a big model is doing a bunch of transfer learning when you give it a bunch of audio, and it is in theory sort of mapping all that audio to the same latent space as its text reasoning,
- 25:45
in practice, it's definitely not doing that exactly. It's doing something like that, but not that. And so because we don't have as much audio data, you see a lot of issues with audio-to-audio models, like the model will sometimes just respond in a totally different language.
- 26:01
And that's cool, but it's never what you want in the enterprise, you know, voice AI use case. And the best guess for why that's happening is it's in some right part of the latent space from some projection, but then from some other projection, it's totally in a different part of the latent space when you gave it audio
- 26:19
instead of text, even though if you transcribed that text, it would be exactly the same as the audio. Um, so you know, latent spaces are big, and to like actually find our way through them in post-training, you really have to have a lot of data, and nobody has enough audio data yet.
- 26:36
But the big labs are gonna fix that because audio matters and multi-turn conversations matter. [upbeat music]