Full Workshop: Realtime Voice AI — Mark Backman, Daily

The assignment is straightforward: build a voice bot during the workshop. The first obstacle is the conference Wi-Fi. Mark Backman suggests tethering because realtime audio requires a sustained streaming connection, not just enough connectivity to send a short request. That dependency will soon interrupt the planned hands-on session.

Pipecat is the open-source Python framework used to assemble the bot. Daily builds it with community contributors; Aleix tentatively dates its official beginning to March 2024. Its scope includes voice and multimodal agents, where listening, reasoning, and output must operate as a continuous interaction.

A useful voice agent must listen accurately, converse intelligently, access databases or internet services, and sound natural. Mark singles out Gemini Live native audio dialogue for its voice quality. Speed is another requirement: Mark gives roughly 800 ms as a voice-response target, with a tentative human comparison around 500 ms. The completed slide specifies a median target below 800 ms; these are conversational design targets, not timings measured in this workshop.

Pipecat organizes work as a multimedia pipeline. Each processor receives data and forwards the original data, modified data, or newly generated data to the next processor. In a conventional cascade, the transport receives microphone audio; speech-to-text produces a transcript; the LLM generates tokens; text-to-speech turns those tokens into audio; and the output transport delivers that audio to the listener.

Gemini Live combines several of those responsibilities in one service. Separate transcription, language generation, and speech-synthesis services are no longer necessary for the basic conversation. Recording audio, producing transcript artifacts, and manipulating data still require application orchestration. Fewer model services do not eliminate the pipeline around them.

The workshop also discusses choosing text output from a native-audio service so an application can parse it before speech synthesis. Preserve that historical configuration when following the recording: current Gemini Live API documentation describes native-audio models using AUDIO responses with optional output transcription, which is distinct from selecting text-only responses for a separate TTS stage.

The processors also create substitution points. Deepgram can supply STT, while Google or OpenAI supplies the language model. Changing a service need not mean rewriting the surrounding application. A parallel pipeline adds branches for different processing paths; Mark describes using that structure to move from one vendor to another during a conversation and transfer context. Even a compact speech-to-speech implementation still needs these surrounding decisions about routing, artifacts, and service behavior.

The Pipecat + Gemini workshop repository provides the README quick start and sample bot files. Mark starts with repository orientation before opening the Python implementation. As the conference connection fails, the session shifts toward a code walkthrough; WebRTC is the transport used for the workshop bot.

The sample's main function runs inside an aiohttp session that can be shared by services. Its processing path uses Daily for transport, context aggregation around the model, Gemini Live for inference, and Daily again for output. A text LLM normally receives a discrete snapshot of the conversation on each request. The context aggregator collects user and assistant contributions and constructs that input. Gemini Live manages more conversational state internally, so aggregation here is particularly useful for tool calls and application management.

The Daily service takes a room, token, participant name, and transport properties. GeminiMultimodalLiveLLMService is the Pipecat wrapper shown for the Gemini Live API. The sample gives it two deliberately simple tools: a weather handler and a restaurant-recommendation handler. Both return canned local results, so their execution does not depend on another internet lookup.

Tool definitions can use a provider's native format, or Pipecat's common function schema. The latter describes the function once and lets the tool schema translate it into the provider-specific representation used by OpenAI, Anthropic, Gemini, Bedrock, or Groq. The service receives those tools together with a prompt explaining their availability. The sample also uses an OpenAI-style context representation as a common format; that representation does not mean the conversation is being sent to OpenAI.

A pipeline is an ordered collection of processors, including processors you write yourself. For example, text output can contain structured information that a processor extracts and stores before downstream speech synthesis. Another processor could inject speech content into the stream. This is where the placement of a separate TTS service becomes useful: the application can act on model text before it becomes audible.

Transport events connect application lifecycle to conversation lifecycle. When the client connects, the sample queues the latest context as a frame, Pipecat's object for carrying information through the pipeline. That frame reaches Gemini and initiates the conversation. Disconnect events are available too. Execution has three layers: the pipeline defines the processing path, a task manages that pipeline, and a runner executes the task.

The transport question has two different answers depending on where the connection terminates.

Pipecat supports both, including a FastAPI WebSocket option. Mark strongly recommends WebRTC for client-facing applications: WebSockets can carry the data, but leave a long tail of realtime media problems for the application to solve in production. Phone-provider integrations are a common server-to-server use case for WebSockets.

A separate concern is voice activity detection, or VAD. Gemini and many STT services have their own detectors, but a local detector can let the application react promptly when the user begins speaking. Pipecat emits a user-started-speaking frame and propagates an interruption, clearing the way for the user's turn while the bot is talking.

Mark recommends Silero VAD for local detection. Aleix informally estimates inference at about a millisecond, without specifying hardware or a benchmark configuration. That processing time is separate from the amount of speech the detector must hear before emitting an event, which Pipecat lets developers tune.

Mark also estimates local VAD at a fraction of one percent of agent cost, citing Quinn's cost analysis in which TTS tokens or characters dominate. The spreadsheet and its configuration are not provided, so the practical point is the small local processing burden described for this setup, rather than a general pricing ratio.

Phone integration adds connection and call-control choices:

• Provider media streams: Twilio, Telnyx, Plivo, and Exotel can exchange audio with Pipecat over WebSockets. An incoming call triggers the connection handshake.
• PSTN dial-in: The public switched telephone network provides another route into the conversation.
• SIP: A telephony provider can connect to a SIP-enabled Daily room. Mark describes richer call control at the cost of more setup complexity.

The media connection expects an agent ready to receive it. Resources need to be provisioned so callers do not sit waiting for a cold start; Mark's example of a twenty-second wait illustrates the experience to avoid.

An audience comparison with Cartesia clarifies the layers. In the workshop, Cartesia is described as a TTS service with voice cloning and a realtime WebSocket API: send text, receive audio frames. Pipecat integrates that service. Cartesia or ElevenLabs can fill the speech-output slot; Llama or an Anthropic model can supply language generation; Deepgram can supply transcription. Gemini Live combines those model functions, while Pipecat remains the framework assembling the application.

Aleix interprets a question about responsible behavior as asking how to ensure the model says appropriate things. Pipecat does not make that judgment automatically. The application supplies prompts and can insert processors that inspect responses or perform realtime evaluation before downstream use. Branching is also available: one transport can route audio into one service and video into another, while Gemini Live can handle both within one service.

Checks compete with the response budget. An additional LLM call to inspect every answer introduces another synchronous dependency. Guardrail processors are optional in Pipecat; whether an application needs them is a separate product decision. Mark instead develops a complementary approach: constrain the task and manage the context the model sees.

A restaurant-reservation conversation can proceed through discrete tasks:

1. Collect the customer's name.
2. Collect the requested time.
3. Record the booking in a database.

Feeding the current task to the model keeps the next action explicit. Context can also be retired after it has served its purpose. In Mark's patient-intake example, date-of-birth verification is an initial checkpoint; once complete, its conversation details may no longer belong in subsequent model input.

For longer conversations, an out-of-band request to a text LLM can summarize accumulated context and return a compressed version. This work need not be another serial step before every spoken response. Mark also points to rolling or sliding context windows in Google's Live APIs, tentatively mentioning token caps, and notes that speech-to-speech services can accept text input as well as audio. These mechanisms give the application ways to manage a conversation beyond continually appending its entire history.

As connectivity returns, participants retry the repository download and use the AI Engineer Slack for help. The intended channel is workshop-voice-gemini-pipecat; a second similarly named channel briefly creates confusion, and badge instructions provide the route into Slack.

Moving inference closer can remove network round trips. Mark points to European users reaching services hosted only in US regions as an example where distance matters, while qualifying that Quinn has done more direct experimentation with local models. He also mentions Modal as a way to rent GPU time for self-hosted inference. Hosting your own model and running it on a client device are different choices; the discussion does not establish a device-performance comparison.

State management introduces another latency tradeoff. A large context gives the model more material to process, but fetching information through a tool can introduce its own delay. Mark reports sub-500 ms time to first byte for a basic local Gemini weather demo, versus waits of 1.5–2 seconds for the first token with some other providers, including OpenAI. These are informal observations with unspecified model versions, sample sizes, and measurement boundaries.

The mechanism matters more than the provider ranking. In the conversational path Mark describes, playback can begin after the first sentence arrives. In the tool path, the application waits for the complete JSON invocation before acting on it. Streaming some bytes therefore does not necessarily mean useful work can begin.

The World's Fair homepage voice widget makes the tradeoff concrete. Mark first tried exposing speaker JSON through a retrieval tool, but found the response too slow. He then put the speaker data directly into Gemini Live's context.

Mark reports roughly 800 ms response latency under good conditions when the speaker data was placed directly in Gemini Live's context, with variable results. The larger context made all records available immediately, but many similarly structured speaker records could confuse the model—a problem he describes across LLMs.

With the Slack channels consolidated, participants return to the README setup. The workshop supplies a temporary Daily key in .env.example, which Mark says he intends to rotate afterward. Participants obtain their own Gemini key through Google AI Studio. For a new setup, use personal credentials rather than the workshop's shared key.

Pipecat's service classes normalize much of the application-facing interface while adapters translate provider protocols. The bot could use OpenAI Realtime, or a text model with separate STT and TTS, instead of Gemini Live. The abstraction has limits: Mark contrasts the workshop-era handling of OpenAI system-role messages with the named system instructions configured when constructing Anthropic and Google services. Similar interfaces do not make every provider's prompt lifecycle interchangeable.

Noisy environments expose a different boundary. VAD detects speech; it does not inherently know which nearby person should control the conversation. Mark describes using Daily partner Krisp as an input filter in the transport, separately from VAD. It cleans incoming audio before detection, targeting noises such as chip bags and barking dogs as well as background voices. He particularly praises its ability to isolate the primary speaker in conference conditions.

The speech-to-speech options named at this point are Gemini Multimodal Live, OpenAI Realtime, and the then-new AWS Nova Sonic. Mark describes their general conversational latency as similarly good, with different strengths and weaknesses; this is separate from his earlier observations about tool-call delays. Aleix then takes over to build the smallest bot from scratch.

Aleix begins with an isolated Python environment named .venv:

python -m venv .venv
source .venv/bin/activate

The environment keeps the bot's packages separate from other Python projects.

Next comes requirements.txt. The Pipecat extras enable the three integrations used here: Daily transport, Google inference, and Silero VAD. python-dotenv loads configuration from .env.

pipecat-ai[daily,google,silero]
python-dotenv

The environment file holds Google and Daily API keys. Aleix copies the workshop configuration, reiterating that its shared credentials are temporary.

In bot.py, an asynchronous main function is launched through asyncio.run(main()) under the standard __name__ == "__main__" guard. The first configured component is DailyTransport: a room URL, a token argument, a participant name, and DailyParams. The demonstration uses a temporary public room without a token. Both incoming and outgoing audio are enabled, and SileroVADAnalyzer lets the transport detect user speech. Aleix corrects a missing display name to AI engineer.

The model configuration comes from the sample's GeminiMultimodalLiveLLMService. Because Gemini Live receives and produces audio, this version needs neither an STT service nor a TTS service. Aleix removes the tools and reduces the system instruction to a helpful assistant. He also omits explicit context storage for this minimal conversation. The processing path is now just transport input, model, transport output.

The execution portion can be expressed as a small function receiving the configured transport and model:

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask


async def run_bot(transport, llm):
    pipeline = Pipeline([
        transport.input(),
        llm,
        transport.output(),
    ])
    task = PipelineTask(
        pipeline,
        params=PipelineParams(allow_interruptions=True),
    )
    runner = PipelineRunner()
    await runner.run(task)

This preserves the workshop's task configuration, including allow_interruptions=True. A pipeline is itself a processor, so larger applications can compose pipelines inside other pipelines. The example needs one task, though the runner architecture allows more. Finally, os supplies access to environment values and load_dotenv() loads the configuration before those values are read.

The first launch fails with a missing-module error: writing requirements.txt did not install its contents. The missing step is:

python -m pip install -r requirements.txt
python bot.py

While packages install, Aleix opens the configured Daily room with video disabled. The browser participant and the bot must join the same room before they can exchange audio.

The Daily room belongs to the selected transport, not to Pipecat's core model. Replacing that transport with a WebSocket connection to Twilio would make the agent reachable by phone instead. Mark also introduces SmallWebRTC, a vendor-independent peer-to-peer transport built on Python's aiortc.

The workshop uses Daily because the team has not prepared TURN infrastructure for the conference network. TURN relays media when direct peer connectivity fails; it is network-dependent, not universally required for peer-to-peer WebRTC. The Small WebRTC transport documentation describes the signaling and STUN/TURN configuration. Mark points to roughly 100 foundational examples covering recording, STT, frame injection, images, and synchronized image-and-audio output, described as using peer-to-peer transport without a Daily key.

Package loading delays the presenter's launch, but an attendee reports a working bot. Aleix estimates the small agent took about five to ten minutes of coding. Asked whether beta labels imply it is unsuitable for commercial use, Mark tentatively estimates hundreds of thousands of Pipecat calls per day while explicitly saying he does not know the actual count. Quinn instead points to large-company production use and contributions from NVIDIA, AWS, OpenAI, and Google.

The Python program is the server-side agent. Client SDKs cover Android, iOS, JavaScript, React, and C++. Connection behavior depends on the chosen transport; Aleix mentions both Daily and LiveKit. For this example, the client and agent become participants in the same Daily room, and WebRTC carries their audio.

Testing does not have to mean a developer manually talking to every bot. Aleix describes a recently assembled release-evaluation setup:

1. Start the agent under test in a Daily room.
2. Join an evaluator bot to the same room.
3. Have the evaluator ask a simple spoken question, such as what two plus two equals.
4. Let the agent answer through its normal audio path.
5. Have the evaluator judge the spoken answer.

This exercises the transport and model services together. Aleix says releases previously involved manually running more than 100 examples; bot-to-bot evaluations are intended to exercise services such as Gemini Live, Cartesia, and Deepgram through actual voice exchanges.

The live bot then answers a request for a joke with the familiar scientists-and-atoms punchline. When challenged about repeating it, the bot asks a question back and eventually gives an extended explanation of why people repeat jokes. Its introduction, “From my search,” does not establish that a search tool ran; the minimal build had deliberately omitted tools.

After stopping that explanation, Mark requests a long story so he can demonstrate interruption. As the story begins, he redirects the bot to a new joke. It briefly continues the story, then changes course after a second redirection—and repeats the same atoms joke. The exchange shows working voice interaction and an imperfect handoff, rather than an instantaneous stop on the first attempted interruption.

An audience member raises the opposite problem: agents often start talking when a person merely pauses. A basic VAD timeout waits after speech stops—Mark offers one second, 0.8 seconds, or half a second as possible settings—then gives the bot its turn. A shorter timeout improves responsiveness but increases the chance of speaking over someone. In an interview with a Tavus-style digital twin, a person may need time to think before continuing.

Semantic end-of-turn detection adds evidence beyond silence duration. Filler words, pauses, intonation, and conversational context can indicate whether the speaker has finished. Mark introduces Smart Turn as a native-audio classifier that returns complete or incomplete. In the integration he describes, an incomplete result can extend the VAD timeout, giving the speaker three seconds to finish the thought before a response.

This is a dynamic decision about how long to wait, rather than one fixed silence threshold for every utterance. Mark describes natural turn-taking as unresolved at the time of the workshop, with model providers and STT vendors exploring similar approaches, and expects substantial progress over the following year.

To show text while the bot speaks, the application needs more than the completed response string. Mark notes audio-and-text output from speech-to-speech services, then explains the more specific TTS alignment path. Providers such as Cartesia, ElevenLabs, and Rime supply word-and-timestamp pairs. Pipecat TTS services emit audio and text frames, and an observer watches those frames pass through the transport to emit client events. The client can then display words in sync with playback. Mark recalls the callback name tentatively, so the useful contract here is the synchronized bot-TTS-text event rather than a particular spelling.

A fully offline agent is possible in principle, but its task determines the model and hardware requirements. A narrow restaurant-reservation bot might work with a local Llama model; more demanding reasoning may require larger on-premises or cloud resources. Mark mentions challenges with Whisper for STT and emerging local TTS options, without prescribing a complete offline stack.

Speech understanding is an upstream dependency for everything else. Misrecognition can make even a capable model answer the wrong question, and native speech-to-speech models can misunderstand input too. Mark tentatively identifies STT as the hardest offline component to get right. Quinn has been experimenting with local models, but Mark clarifies that Daily is not developing an STT model. Aleix observes that local inference could also have avoided the workshop's network dependency.

The closing example is Word Wrangler, a Catch Phrase-style game: the human describes a word without saying any part of it, and an AI player guesses. Mark directs participants to Pipecat's community and sample projects, including both a browser version and a phone version of the game.

The phone version puts two Gemini agents in one call using parallel pipelines. One is the host, which gives the human the prompts. The other is the guesser. Selective audio routing keeps the guesser from hearing the host: it receives the human's clues instead. This makes routing part of the game rules, because the guesser must infer the answer from the player's description rather than hear it announced. Mark identifies Twilio for the phone version and a React/Next.js-style client for the web version.

The live web game opens by explaining the rule against saying the target word. Mark describes something on a phone used to take pictures; the agent guesses camera. A field concerned with language becomes linguistics, a racket sport with a yellow ball becomes tennis, and a round dessert with chocolate chips becomes cookie. The displayed final score is 4. The workshop ends with Gemini Live participating in a concrete game whose behavior depends on both native audio understanding and the application's control over who hears what.