Pipecat Cloud: Enterprise Voice Agents Built On Open Source

How do you build a voice agent that understands a person, responds naturally, and remains reliable over a real network? Pipecat addresses the application layer of that problem: an open-source, vendor-neutral framework above the infrastructure that carries audio and video. Kwindla Hultman Kramer, Daily co-founder, presents it here in place of his colleague Mark. Daily began in 2016 as infrastructure for real-time media; Pipecat adds agent programming, while Pipecat Cloud adds a hosting layer on that infrastructure.

The work divides into three responsibilities:

1. Write the agent: implement its behavior and connect its models and business systems.
2. Deploy the agent: give that code somewhere to run.
3. Connect the user: carry the conversation over a network or telephony connection.

These are separate engineering problems, but users experience their combined result as a single conversation.

Users expect accurate understanding, useful access to knowledge, conversational intelligence, and a natural voice. Kramer’s assessment is that voice AI has finally crossed the uncanny valley, making those expectations possible to satisfy. Speed is part of that experience: Kramer approximates the expected response time in human conversation at 500 milliseconds, varying with language, culture, and individual. He recommends an 800-millisecond voice-to-voice response target for agents. That is an engineering target, not a measured Pipecat result, and the interface needs to account for the user’s expectation of a prompt response.

Fast generation alone does not tell an agent when to speak. It must also decide whether the person has finished their turn. Humans sometimes get this wrong; an agent needs explicit machinery to avoid answering during a pause or waiting long after the user is done.

The reason to use a framework is to keep business logic from becoming an implementation of every difficult real-time behavior. Pipecat supplies building blocks for turn detection, interruption handling, context management, tool calls, and asynchronous function calling. Those components need to work together while audio continues arriving, not just between isolated requests.

Provider choice extends through the stack. Kramer describes Pipecat as entirely open source and vendor neutral, including native telephony integrations. He gives Twilio and Plivo as alternatives, citing Indian phone-number availability as a reason developers might choose Plivo. The ecosystem also includes the open-source native-audio Smart Turn model, an example of small-model research emerging from the community.

Pipecat Cloud occupies the hosting layer for developers who want to run their own agent code. The distinction matters: Pipecat is the open-source framework; Cloud is managed hosting for it. Kramer positions Cloud as a first-of-its-kind offering for this ecosystem, rather than another constraint on which models developers can use. He reports approximately 60-plus supported models and services in Pipecat’s main branch at the time of the talk. Existing integrations reduce the code needed to start, while programmable pipelines leave room for more elaborate applications.

Telephony is only one client. JavaScript, React, iOS, and Android components and SDKs support multimodal applications on the web and native mobile platforms. The same framework can therefore sit behind a phone call or an application with a richer audio and video interface.

A Pipecat agent is a pipeline of programmable media-handling elements. Application code is Python, with performance-sensitive components reaching into C underneath. A minimal pipeline can have just three elements: network input, processing, and network output. Enterprise pipelines grow because their tasks involve more processing and connections to existing systems.

The basic composition is explicit in Python. With configured transport, service objects, and a context aggregator supplied by the application, a transcription-based pipeline can be assembled like this:

from pipecat.pipeline.pipeline import Pipeline


def build_voice_pipeline(transport, stt, llm, tts, context_aggregator):
    return Pipeline([
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
    ])

The ordering expresses the media path: incoming audio becomes a transcript, the context aggregator’s user processor turns transcripts into conversational context for the language model, and text-to-speech produces outgoing audio. The assistant processor records the assistant’s response in that context for subsequent turns. The OpenAI examples shown in the talk contrast that chain with a native speech-to-speech service. Kramer estimates that switching between the two approaches in those examples requires changing three or four lines of code.

A more distinctive example uses two instances of the Gemini Multimodal Live API in native-audio mode. One handles the conversational flow; the other participates in a game, applying an LLM-as-judge pattern. The starter kit’s core is a few hundred lines of Python, accompanied by a flow diagram. Its important mechanism is selective audio-frame routing: real-time inference results determine which audio frames move through which pipeline. This is more than placing two models next to each other. The outputs of inference govern the media flow, a pattern Kramer also sees in enterprise applications.

Once the pipeline works, deployment introduces another set of constraints. Voice agents run long-lived sessions over protocols chosen for low latency. The autoscaling patterns available out of the box for HTTP workloads do not automatically solve that problem. Deployment and scaling became such a large share of Pipecat community questions that Daily reconsidered whether its existing low-level infrastructure was enough.

There were already useful solutions above and below this gap: media infrastructure at the bottom, and platforms with dashboards and graphical tools at the top. Developers writing their own Pipecat code still needed help operating it. Advice about the things to configure in Kubernetes often met with a simpler question: what is Kubernetes? Pipecat Cloud was designed for that middle layer. Kramer describes it as a thin wrapper around Docker and Kubernetes, optimized for voice AI and built on Daily’s global real-time infrastructure.

The cold-start problem becomes concrete when someone calls an agent. While the caller hears ringing, the system must get ready to pick up and produce a greeting. A delayed first response is already part of the user experience, before any substantive conversation begins. Fast starts are therefore a central hosting requirement, separate from the latency of later conversational turns.

Capacity must also follow traffic. Preprovisioning a fixed fleet works only when the workload is sufficiently predictable; time-dependent or unpredictable demand requires resources to expand and contract. The latency tolerance differs from ordinary request handling:

These are his workload comparisons, not hosting benchmark results. Because the conversational budget covers the entire chain, each inference call must consume only part of it. Network behavior matters both between the client and the deployment and inside the Kubernetes cluster.

Global deployment serves more than speed. It can place servers near users, but it may also be necessary for GDPR, data residency, or other privacy requirements. The hosting goal is to absorb these operational concerns at a reasonable cost so developers can get an agent into use sooner.

Turn detection remains one of the central voice-agent problems. Kramer describes Smart Turn as built into Pipecat Cloud and free to run there, with fal providing GPU-optimized hosting. Its role is to improve the decision about when the user’s turn is complete.

Background noise creates a related but distinct failure. A transcription model may be resilient enough to recover words in a noisy room, while the surrounding agent still reacts incorrectly to sound that resembles speech. That sound can trigger an unintended interruption, insert spurious pseudo-speech into the transcript, and cause downstream inference to run when it should not. Native speech-to-speech models are also susceptible, in Kramer’s assessment. Accurate transcription does not by itself imply reliable conversational control.

Kramer recommends Krisp as the strongest noise-suppression option available at the time. He describes it as included without an additional charge inside Pipecat Cloud, while independently hosted Pipecat pipelines can use it with their own commercial license. Those are the talk’s hosting and licensing terms.

Even with clean audio and better turn timing, agents remain nondeterministic. Pipecat provides low-level logging and observability building blocks, exposed through Cloud, with partners offering additional observability tooling. These give developers a way to inspect the behavior of a live agent rather than assuming that a successfully connected pipeline is behaving correctly.

The first audience question exposes a limit to the advice to deploy near users. A developer in Sydney reports that the OpenAI trip and return alone takes roughly 800 milliseconds. If that consumes the conversational budget, where should the agent run?

Kramer answers from the premise of inference in the United States. His blanket statement that OpenAI inference was US-only needs qualification: OpenAI had already announced European processing for eligible API customers and endpoints, although that does not establish an alternative for this developer’s account or audio endpoint. The placement principle remains useful: avoid repeatedly crossing the ocean inside a sequential inference chain.

If transcription, language-model inference, and voice generation are remote, an agent running near the user may make a long-distance round trip for each stage. Instead, place the agent near those inference servers. Audio travels the long-haul path into and out of the system, while the internal inference calls take short paths.

Kramer describes this as an improvement, not an ideal solution: the long-haul audio path still exists.

The alternative is to host open-weight models locally in Australia. Whether that works depends on the task’s language-model requirements. Kramer names Gemma, Qwen 3, and Llama 4 as viable for some voice workloads, while distinguishing their suitability from that of GPT-4o and Gemini 2.0 Flash. Geography can be improved by changing the model deployment, but the replacement still has to satisfy the application’s evaluations.

The follow-up asks how locally hosted models connect to Daily’s network. Kramer describes worldwide points of presence that terminate WebRTC or telephony near the user, then route onward over private AWS or OCI backbones. Australian hosting can connect through those local endpoints. Pipecat Cloud had some regional availability at the time, with more planned for the following quarter; Australia was a hoped-for addition, not a confirmed region. Self-hosting in Australia remained an option with Pipecat alone or Pipecat plus Daily.

Could a model make separate turn detection unnecessary? An audience member raises Moshi, Kyutai’s speech-text foundation model, and asks whether its approach scales. The architectural difference is continuous bidirectional streaming: tokens keep arriving at the model, and tokens keep leaving it.

Most outgoing tokens during listening represent silence. When the model produces something else, it is choosing to speak within the same ongoing stream. That supports both turn-taking and backchanneling: a brief acknowledgment while the other person continues talking. Such an acknowledgment does not require starting a new inference call. It emerges from streaming generation and the conversational behavior represented in the training audio.

Kramer strongly endorses the architecture paper while separating that enthusiasm from production readiness. He judges the model’s language capability too small for real-world enterprise tasks and forecasts that a production-trained version of this architecture is still a couple of years away. His answer supplies an architectural explanation and a readiness judgment, rather than a scaling measurement.

Large-lab native speech-to-speech models offer a different route to more natural conversations. Kramer distinguishes them from Moshi’s continuously bidirectional architecture. Their attractions include multilingual and mixed-language handling, with the possibility of lower latency. He refers to GPT-4o Audio Preview when discussing OpenAI’s Realtime API; the historical Realtime API launch names its model gpt-4o-realtime-preview, distinct from the audio model naming used in Chat Completions. He also points to Gemini 2.0 Flash in audio-to-audio mode and preview releases of Gemini 2.5 Flash.

The adoption decision turns on what the application cannot afford to get wrong:

• Conversational dynamics: narrative, storytelling, and other experiences centered on natural interaction are beginning to use native audio.
• Enterprise task execution: applications that require the best available instruction following and function calling still favor leading text-mode models in Kramer’s assessment.

Audio mode can make a conversation feel better while making its actions less dependable. The useful threshold is therefore the one established by the application’s own evaluations. Kramer forecasts speech-to-speech becoming the default for about 95% of voice AI within two years; that is a prediction, not measured adoption.

An audience follow-up brings in Sesame. Kramer places it closer to Moshi and notes its use of Mimi, the neural audio codec. The released CSM component generates speech using Mimi codes; it should not be equated with the complete conversational demo. At the time of the talk, Kramer describes the broader release as incomplete and worries that the available model is too small for most enterprise tasks, while expecting larger versions.

He also recommends trying Ultravox, describing the version under discussion as a speech-to-speech offering built on a Llama 3 70B backbone, supported by its team and a production voice-AI API. His recommendation is conditional: if that backbone can do the required task, Ultravox is worth evaluating; if it cannot, the audio interface does not remove the capability limit. Experiment with speech-to-speech, but do not assume it is already the right foundation for an enterprise workload.

Asked when to choose OpenAI or Gemini, Kramer starts with task performance. For the use cases he tests daily, he finds GPT-4o and Gemini 2.0 Flash roughly equivalent in text mode. His practical procedure is straightforward: build the Pipecat pipeline, swap the models, and run the same evaluations. Keeping the surrounding pipeline stable makes the provider choice something the application can test.

Cost can then separate otherwise suitable options. Kramer estimates that a 30-minute Gemini conversation was probably about ten times cheaper than a 30-minute GPT-4o conversation at talk-time pricing. The estimate does not specify token volumes, modalities, caching, or model snapshots, so it cannot serve as a reproducible cost calculation; Kramer also explicitly expects prices to change.

Input and output modalities provide another axis of comparison. Gemini can take native audio input and produce text output, preserving audio as model input without requiring an audio response from that same model. Kramer sees advantages for some languages and use cases and recommends testing those in the application’s evaluations. He judges OpenAI’s newer native-audio input support somewhat behind Gemini’s in this respect at the time.

The final question asks why speech-to-speech should be preferable to transcription, text inference, and speech synthesis at all. The practical answer starts with information loss: transcription discards information in the audio. If the discarded information matters to the task, feeding audio directly to the model can help. Mixed-language speech is a concrete example. Small transcription models can struggle more with it than with optimized monolingual transcription, while a large model has broader language knowledge to apply to the original audio.

A single end-to-end model can also avoid chaining multiple inference calls, potentially reducing latency. Whether that advantage appears in practice depends on the APIs and inference infrastructure, not just the model architecture. Kramer expects a direction in which one main inference call handles most of the interaction, with smaller models alongside it for particular subtasks.

The costs begin with context. Audio consumes many more context tokens than an equivalent text representation, greatly expanding the context the model must handle. Kramer connects that expansion with degraded performance. Training data adds another limitation: there is much less audio data than text data, so learning from audio does not reproduce text-mode reasoning behavior exactly, even when transfer learning brings the representations closer together.

A concrete symptom is an audio-to-audio model unexpectedly replying in a different language. That may be an interesting generation, but it is the wrong behavior for an enterprise conversation. Kramer offers a hypothesis: audio and its equivalent transcript may reach related regions of the model’s latent space without supporting the same behavior across all relevant dimensions. His explanation is speculative, but the operational failure is clear—the modality can change the response even when the linguistic content is equivalent.

More audio data and post-training are needed to make that behavior reliable. Kramer closes by expecting the major labs to address the gap because audio and multi-turn conversation matter. The reason to keep experimenting is the information direct audio preserves; the reason to keep evaluating is that preserving more input does not yet guarantee better task execution.