AI Engineer World's Fair 2025
Realtime Conversational Video with Pipecat and Tavus — Chad Bailey and Brian Johnson, Daily & Tavus
Read the talk
Building realtime conversational video with Pipecat and Tavus
A responsive video replica needs more than fast rendering: it needs streaming orchestration, sensible turn timing, synchronized media, and infrastructure that connects each user to a bot.
From a talk by Chad Bailey and Brian Johnson
Before you start: Familiarity with LLM message history, streaming output, and basic Python will help with the pipeline walkthrough.
What would make the robot concierge work?
A visitor approaches a robot concierge at a counter. The promise is obvious: ask a question and have a useful conversation. The familiar experience is less convincing. Chad Bailey opens with that gap—these interfaces ought to work, but too often they are frustrating to use. Building a good version is now possible; making the interaction work requires more than choosing a model.
There are three responsibilities to solve in sequence: models, which interpret input and generate responses; orchestration, which coordinates the conversation; and deployment, which makes a bot available when someone wants to talk. A compelling demo starts with the first. A usable service needs all three.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From speech pipelines to video replicas
The conventional voice pipeline separates transcription, reasoning, and speech synthesis. Voice-to-voice models offer another route, but neither approach removes the additional work involved in generating realtime video.
| Approach | Processing path |
|---|---|
| Cascading voice pipeline | Speech-to-text → LLM → text-to-speech |
| Voice-to-voice model | Spoken input → model → spoken output |
The choice depends on the application. Video adds another output modality that must participate in the same live interaction.
Tavus began with a rendering model. Brian Johnson describes the next requirement as speed: rendering had to work in a realtime context to become conversational. Once it did, the missing pieces became apparent—turn detection, response timing, interpreting signals, and orchestration. Tavus initially built its own system without knowing about Pipecat, then began partnering with the project.
Brian points viewers to the Tavus website instead of running a live demo. The product he describes is an end-to-end conversational video interface: create a replica of yourself, put it online, and converse with it. Brian reports a response time of around 600 milliseconds for the Tavus interface described in the talk. That is a reported figure, not a benchmark with specified measurement conditions—and he immediately adds that it can be too fast. Sometimes the system needs to delay its response.
Alongside the basic conversational stack, Tavus includes proprietary models Sparrow-0 and Raven-0. At this point in the recording, they are offered inside Tavus's stack, with access through systems such as Pipecat described as a future direction. The developer overview brings those pieces together as perception, thinking, and responding around the user's input and output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understanding why the bot behaves that way
A first conversation can feel remarkable. Production raises less glamorous questions: why did the bot pause, which part of the system was slow, and what caused it to respond that way? Observability and control over conversation flow mean being able to investigate those questions and change the behavior, rather than merely observe the final reply. Capturing metrics becomes part of building the interaction.
Pipecat is the open-source, vendor-neutral orchestration framework Bailey introduces for this work. Its job is to receive a user's audio or video, coordinate processing, and deliver audio or video back with low latency. The conference website's Talk to AIE bot provides a concrete example: Bailey says it uses Pipecat with Gemini Live, a voice-to-voice model.
An integrated model still leaves application behavior to coordinate. Bailey invokes Gemini's documentation to distinguish experimentation from production orchestration. That recommendation should not be read as a Google requirement to use Pipecat: the current Gemini Live API overview supports multiple integration approaches. The engineering issue is the work surrounding the model, even when the model handles the voice exchange itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Frames, processors, and pipelines
Pipecat's work divides into three stages:
- Input: Receive microphone audio and, where needed, camera video. A replica can respond to what it sees as well as what it hears.
- Processing: Run multiple models in sequence, or delegate much of the interaction to an integrated service such as Gemini Live or Tavus.
- Output: Deliver synthesized speech, generated video, or application UI updates. For a video replica, the audio and video must remain synchronized.
That last requirement is separate from generating plausible speech or imagery: the user has to receive them as one coherent interaction.
The implementation rests on frames, processors, and pipelines. A frame is a typed container for data. Bailey's examples include small audio snippets, such as 10 or 20 milliseconds of sound, and camera video frames. Events also travel as frames: voice activity detection, or VAD, can produce a user-started-speaking frame. Media and information about the conversation therefore move through the same processing structure.
A processor receives frames and emits frames. An LLM processor, for example, receives conversation context and produces a stream of text frames. A pipeline composes these processors in the order needed for the bot's behavior. Pipecat runs the work asynchronously to minimize the delay each piece of information experiences as it moves through the system. Streaming output can keep progressing instead of waiting for an entire response to become one finished object.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From incoming audio to a completed turn
Rather than live coding, Bailey walks through a Tavus example from the repository, with QR links to the example and the pipeline documentation. The first processor is transport input. It turns media arriving over WebRTC, WebSockets, or Twilio WebSockets into frames that enter the pipeline.
The speech-to-text processor collects incoming audio frames. The example uses 20-millisecond chunks to explain the flow, but the transcription service decides how much audio it needs before emitting text. Deepgram and Whisper are examples of transcription choices. The important boundary is that incoming audio frames and outgoing transcription fragments do not have a one-to-one relationship.
A transcription fragment is not yet a conversational turn. The STT processor emits text according to its own processing behavior. A context aggregator combines those fragments with other signals moving through the pipeline: the user started speaking, microphone activity dropped, and the user may now have stopped. A drop in activity is a cue to consider ending the turn, rather than proof that the person has finished their thought.
Once the aggregator decides to release the turn, the rest of the sequence follows:
- Group the accumulated transcription fragments and emit a context aggregation frame.
- Supply the LLM with the conversation context, including message history and tools.
- Stream the LLM's output as text frames, accompanied by start and end frames.
- Let text-to-speech accumulate text and generate speech.
Aggregation is therefore a control point: it determines when a stream of partial observations becomes a request for the model to answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turning synthesized speech into synchronized video
The bot file shown uses an explicitly older Tavus integration. Pipecat generates speech first, then sends the audio to Tavus to generate video. Tavus returns the audio synchronized with that video, and those media frames leave through transport output. The transport is the delivery boundary in both directions: input brings the user's media into the bot; output sends the replica's media back.
In Python, the composition can be expressed as an ordered pipeline of configured processors:
python
from pipecat.pipeline.pipeline import Pipeline
def build_replica_pipeline(
transport, stt, context_aggregator, llm, tts, tavus
):
return Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
tavus,
transport.output(),
])
This captures the audio-to-video arrangement being explained: Tavus sits after speech synthesis because generated audio is its input. Service construction and transport setup depend on the integration version; the current Tavus integration guide is the setup reference, rather than the historical example's configuration. The useful architectural property is that additional processors can be inserted around the renderer without replacing the whole conversation system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Running analysis beside the conversation
A pipeline need not be a single straight path. Pipecat supports parallel pipelines, so another LLM can perform sentiment analysis while the main conversation continues. That makes analysis a concurrent activity instead of an obligatory serial step before every reply.
Bailey describes a user applying Gemini Live Multimodal to an outbound-call problem: did a person answer, or is the bot hearing a voicemail greeting? One model makes that determination and sends a signal back to the pipeline. The signal selects a human branch or a voicemail branch. The classifier decides what was heard; orchestration decides which behavior follows.
Even when Tavus handles most of what makes the interaction feel convincing, peripheral tasks still need coordination. The runnable example makes that surrounding machinery visible. Bailey describes signing up for Tavus, supplying a key, and running the example unchanged at the time of the talk. Its UI supports conversation with the replica while a debug panel exposes activity inside Pipecat. The result is both an interaction surface and a way to inspect how the interaction is being produced.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Moving model capabilities into the orchestration layer
Tavus's own experience illustrates the cost of rebuilding orchestration. Brian says the team spent the preceding year learning lessons Pipecat had already addressed in aggregation, communication, and coordinating the conversation. He estimates that reusing those capabilities could save months of work. This is an engineering experience claim, not a measured development-time comparison.
At the time of the talk, Tavus was not using Pipecat internally. Enterprise customers were already using it, however, and wanted Tavus's models available inside their existing pipelines. Phoenix, the rendering model, had already been integrated. Turn-taking, response timing, and perception were planned next; adopting Pipecat inside Tavus was also a future intention. Brian grounds that intention in a concrete frustration: he had just spent days debugging a problem Pipecat already solved. These plans are distinct from the rendering integration already available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finishing a turn and choosing when to answer
Tavus's multilingual turn-detection model addresses a deceptively difficult question: has the person finished speaking? A fast pipeline that answers too early talks over the user. A slow one leaves the user wondering whether it is broken. Better turn detection aims to avoid both outcomes by making the decision about completion more accurate.
Response timing is a separate decision, made after the person has finished. Brian's example is someone describing a grandmother moving into a home and feeling sad. An immediate answer can feel inappropriate; a considered pause fits the exchange better. Casual chitchat calls for a quicker response. This explains why the earlier fast-response capability sometimes needs deliberate slowing.
| Decision | Question |
|---|---|
| Turn detection | Has the user finished speaking? |
| Response timing | How soon should the bot answer afterward? |
Minimum processing latency and appropriate conversational timing are different goals. The response-timing model described here was still being developed, with Pipecat integration planned.
Multimodal perception adds information beyond the words themselves: emotions, surroundings, and what the person is wearing. The planned next step is to feed those observations into turn-taking and response timing, making the conversational flow sensitive to more of the situation. Perception becomes useful not just because the replica can describe what it sees, but because it can influence how the replica participates.
The closing integration example returns to control over the pipeline boundary. Some functions can remain inside Tavus; others can be exposed as components the application coordinates. As models change, a differently composed Pipecat pipeline lets the application retain control where it needs it while delegating other work to Tavus.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Starting a bot and connecting its media
Deployment completes the system. Pipecat leaves substantial freedom in how bots are hosted, but the application still needs a concrete startup path:
- A client calls a REST API to request a conversation.
- Infrastructure starts a bot instance promptly and connects it to the requesting user.
- A media transport carries audio and video in both directions.
Bailey favors WebRTC for that transport. Starting the process alone is insufficient: the infrastructure must also configure the connection that lets the user and bot exchange media.
Pipecat Cloud is presented as the paid, managed option for teams that do not want to operate infrastructure such as Kubernetes themselves. Operating the infrastructure independently remains an option. Bailey closes by pointing to his colleague Mark's follow-up talk on deploying bots at scale, covering both the managed service and doing it yourself. Either route has the same responsibility: turn a request to talk into a running bot with a working media connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source framework and examples for building real-time voice and multimodal agents.
Managed infrastructure for deploying agents, scaling sessions, and connecting users through Daily WebRTC.
Further reading
Explains frames, processors, context aggregation, processing order, and parallel pipeline branches.
The March 2025 introduction of Phoenix-3, Raven-0, and Sparrow-0 explains their rendering, perception, and conversational timing roles.
Updates since the talk
Current setup instructions for adding Tavus rendering as a room participant or a video service within a Pipecat pipeline.
Current integration approaches for Gemini's real-time audio and vision API, including Pipecat.
Read the complete timestamped transcript
- 0:00
[upbeat music] We're here to talk about real-time conversational video, uh, with Pipecat, that's me, and with Tavus, that's Brian.
- 0:19
We'll introduce ourselves a little bit more, but in the interest of keeping it moving, let's talk about what we're here for. Have anybody, have any of you ever seen one of these robot concierge things?
- 0:28
Do they work? No, they don't. They're terrible, right? Um, it's actually possible nowadays to build this kind of thing, but actually good. Um, it's a little bit tricky, but that's what we're here to show you how to do.
- 0:40
Um, there are three things you need to think about when you wanna build real-time AI. Um, the first is your models. Obviously, we all know what models are, that's why we're here at this conference.
- 0:50
Um, the thing that you don't necessarily know you need to think about is your orchestration layer. We're gonna talk a little bit about that. And then, of course, you need to deploy these bots somewhere.
- 1:00
That's the third step, deployment. Talk about that as well.
- 1:04
So step one, models. I come from a little bit more of a traditional, it's funny to say that, voice AI world, where the traditional pipeline people talk about is speech-to-text, so transcription, right?
- 1:17
And then LLMs for your inference, and then text-to-speech, right? Like, that's the typical kind of cascading pipeline you hear. Sure enough, people are nowadays are using some voice-to-voice models.
- 1:25
That is a use case for this kind of thing. But there are reasons sometimes you might use one or the other. Um, real-time video is a lot more complicated.
- 1:35
It doesn't have to be, but it can be, and I think Brian will tell you that it should be. Um, there's a lot more stuff you need to think about to do video generation in real time.
- 1:45
So Brian, you wanna tell us a little bit about Tavus and how y'all are thinking about this?
- 1:48
Thanks, Chad. So Tavus started out as a, as a AI research company, and we started off with a, a single model that was like a rendering model. What we quickly realized is that we need to be able to put this into a real-time context for it to be useful, so it needs to be fast.
- 2:05
And once we did that, we started realizing there's a lot of missing, there are a lot of missing pieces, things like turn detection, response timing, uh, picking up signals and orchestration.
- 2:15
And uh, we started off in, in the beginning, we didn't know about Pipecat when we first built it, uh, but we've been partnering with Pipecat over the last year, and it's come to our realization that, like, a lot of the stuff Pipecat does is gonna be very important for conversational AI and making it real.
- 2:31
Um, I think we can go to the next one.
- 2:34
Yeah. Yep.
- 2:34
Uh, we have a demo. You can go to our site, tavus.io. I was gonna do it live, but for the sake of time, just check it out. You can go check it out on our website, tavus.io.
- 2:44
And I'll hand it back... Well, no, there's one more thing.
- 2:46
Yep.
- 2:47
So, uh, what we do at Tavus now is we offer a conversational video interface. It is a end-to-end pipeline that allows you to have a conversation with a replica of anyone.
- 2:58
You can create your own replica of yourself, you can put it online, and you can have a conversation. The response time is around six hundred milliseconds, but that's not ideal because a lot of times that's too fast.
- 3:09
Um, so we have to slow that down sometimes based on some of these models that we're using. Um, and there are a lot of steps that go into this.
- 3:17
You can see there's like, uh, where like Ch- like Chad talked about, the, the basic layers of a conversational stack. But we've also ha- we also have these proprietary models, Sparrow-0 and Raven-0, that we've created, which is kind of like our IP or what we're offering.
- 3:32
And we're going to... W- Right now, the r- we offer those in our stack, but we're moving towards a world where we're gonna offer those in things like Pipecat.
- 3:41
So models, and we'll come back to the Tavus models in a little bit and how they are getting better and some of the cool new things that are coming from Tavus that you will want to use.
- 3:50
Um, orchestration is where my, my world steps in. So that's, that's Pipecat. That's the thing on my water bottle and my shirt and my jacket and all that kind of stuff.
- 3:59
Um, let's talk a little bit about what Pipecat is. Um, there's a really interesting phrase on Brian's slide, real-time observability and control into the flow of a conversation. Um, that's a, th- A lot of those are words that you, that, that don't really mean anything until you actually go build one of these things.
- 4:18
And when you build it, the first time you use it, you go, "Wow, this is amazing. This is great." And then as you start to actually think about what it's gonna mean to have that in production, you realize, oh, wait, there are a lot of like boring infrastructure kinds of things that we need to solve.
- 4:32
The ability to understand, like, to, to have observability into how the bot is behaving and why it's behaving that way. The ability to get capture metrics on things and understand things like, sometimes the bot takes a long time to respond.
- 4:43
I wonder why that is. Well, it turns out there's a whole lot of these kinds of things that you need for a real live production app, for a real live production bot, that you need something like Pipecat.
- 4:53
Pipecat is an open source framework. Uh, it's built by my company, but is open source and actually fully vendor neutral, and it's designed to be this orchestration layer for real-time AI.
- 5:03
And by that I mean, you are, you, you have a user that is going to be producing either video and/or audio, and you wanna also be delivering video and/or audio to that user, and you wanna do that with a low latency as possible.
- 5:15
That's the real time part of this whole conversation. Um, if you went to the AI Engineer website, and you saw the little button on the bottom right that says "Talk to AIE," uh, that's powered by Pipecat.
- 5:25
It's actually using the Gemini Live model, so it's using a voice-to-voice model. But there's still so much other stuff you have to do to go from voice-to-voice demo bot on the web to, like, like, in your browser or on the web, to an actual, like, shipping production app, that even Google themselves, even the Gemini documentation says, you
- 5:46
can go use our own, like, tools and our, you know, like, our, our browser tools and things to experiment with Gemini multimodal live. But when you want to take it to production, you do need something like Pipecat to actually orchestrate what's happening in your entire app.
- 6:03
Um, I'm gonna try to do this slide very quickly. And there, there are a few QR codes coming up, so now would be a good time to, uh, get those buttons ready.
- 6:13
Um, Pipecat itself, two lists of three that you need to think about to understand what Pipecat does. The first one is something I just kind of already talked about.
- 6:23
Um, the three things that Pipecat is doing for you is handling input, it's handling the processing, and the output. Input is receiving media from your user. So in the case of a traditional voice bot, that's just voice.
- 6:36
Um, in the case of a, of a Tavus replica, that's sending voice, and they even... They're doing some interesting things that we'll talk about with inputting your user's video and allowing a Tavus replica to respond to not only what it's hearing in the voice, but what it's seeing in the video coming from the user.
- 6:53
Um, getting into that, that's the processing part. That's step two. That's where, that's where essentially you're going to run through a bunch of different models. Uh, in some cases, you can do almost all of what you need with a single model.
- 7:04
Um, in the case of, like, Gemini multimodal live for voice or a Tavus replica, they-- Th- there, there is a way that you use Tavus inside a Pipecat bot where you can basically let Tavus kind of do everything for you, um, run kind of as just one integrated piece.
- 7:18
Um, and then, of course, all of those models, hopefully, this is supposed to be real-time and interactive video. Hopefully, those models are producing some kind of output that you wanna show to your user.
- 7:26
That's the video and the audio being produced by your tools. In a typical voice bot, that is, you know, that is, uh, text-to-speech that is being played out as audio.
- 7:35
It might also be things like UI updates, if you're in a web app, that you're pushing UI updates, that kind of thing. And of course, in the Tavus case, it's video and audio that are hopefully presented in a way where the video stays synchronized to the audio, for example.
- 7:48
That's a really, really hard thing to do well, depending on exactly how you build this whole thing. The three fundamental pieces of the-- of Pipecat that enable those things to work are frames, processors, and pipelines.
- 7:59
Um, Pipecat's name comes from the fact that it is about building a pipeline. Um, and a pipeline is comprised or is composed of processors. Processors are things that handle frames.
- 8:11
Frames are essentially any... It, it is basically a type container for a kind of data. So in a Pipecat pipeline, you will see a whole bunch of frames with things like little snippets of user audio, like ten or twenty milliseconds of audio comes across as an audio frame, or video frames from the user's camera device you can
- 8:30
capture. But even things like, uh, voice activity detection, VAD, comes across as a user-started speaking frame in Pipecat. All of those frames progress through a series of processors, and a processor just takes in some frames and outputs other frames.
- 8:44
So a good example would be, like, the LLM processor, for example, is taking in frames that are essentially context fra- like, completed context turns from the user and the bot, and it is outputting a stream of text frames.
- 8:59
So, so if you're capturing streaming output from your LLM, in Pipecat, that looks like a bunch of text frames coming out of that processor. And all those are put together in a pipeline, and the pipeline is how you describe what you want your bot to do.
- 9:13
And the idea behind how Pipecat runs your pipeline is that it's doing all of that stuff asynchronously and doing its best to minimize the latency of every piece of information as it goes through the pipeline.
- 9:25
So there is a, a, a much better and longer explanation. I know that that was a lot. Um, there's a much better and longer explanation in the Pipecat docs.
- 9:33
That is that QA file. In terms of what it actually looks like, it was gonna be a little tight to try to get in and do some live coding during fifteen minutes.
- 9:42
Um, but this is a QR code that links to this example file. There is so much stuff in the Pipecat repo that shows you this. But just to step through these pieces real quick, at the top, there's the transport input.
- 9:53
Th- this is the core pipeline inside this bot file, and this is actually the, one of the Tavus examples that we have in the repo. First thing is transport input.
- 10:01
That's where the frames come in from your media transport. So whether it's WebRTC or WebSockets or Twilio WebSockets or anything like that. Frames start pouring in from the transport input.
- 10:12
They go to a speech-to-text processor. Um, that's where transcription is happening. So for example, one thing that frame processor is doing is it's collecting snippets of audio at a, you know, a frame at a time, twenty milliseconds at a time.
- 10:24
But it is sort of up to your transcription processor, whatever that is, Deepgram or Whisper running on something or whatever, to exactly collect a bunch of frames, collect however many frames it needs to then output a snippet of transcription information, right?
- 10:37
So that happens in speech-to-text. From there, we go into something called the context aggregator. That's because the transcription or the, the STT processor is emitting transcriptions whenever it feels like it.
- 10:49
So we use other frames in the pipeline that have made their way through to understand, okay, the user has started talking. The user's microphone has, you know, microphone level has dropped, so it looks like the user has stopped talking.
- 10:59
Maybe now is a good time to group all of the various pieces of transcription we've gotten over the past few seconds together and emit a single context aggregation frame.
- 11:08
That's what triggers the LLM to run. And so we, we, we grab the context, and if you've, you know, of course, if you've programmed with LLMs, you know you get the context with all the array of messages and the tools and everything.
- 11:18
You shove that to the LLM, and then it starts streaming tokens back. Those tokens come out of LLM as text frames as well as there's, like, a start and end frame.
- 11:26
And if you're, if you're familiar with this approach, you can probably see all these other frames as they start to exist in here. But then TTS essentially accumulates those and generates speech.
- 11:37
This bot file is actually an older example that uses an older Tavus model where we were actually generating audio, and then we were sending the audio over, I believe, a WebSocket.
- 11:48
It's not important. We were sending audio to a Tavus model that was generating the video based on the audio and then sending back to us, back to Pipecat, audio and video.
- 11:59
So essentially the same audio but synchronized with the video. Those come as a different series of frames that then go out through transport output. And that is, again, essentially the same transport that we're using on the import side, input side, but this is the output side, and so that's where all that media goes back to the other
- 12:13
user. So you can, you can start to see how with this structure, um, it looks very simple right here, but it is incredibly powerful when you realize that you can kind of put anything you want in this pipeline.
- 12:25
Uh, we have people, for example, that, like there's a construct in Pipecat called parallel pipelines, and so we have people that have this exact same workflow. But at the same time, in real time, they're running another LLM that is doing things like, um, you know, sentiment analysis.
- 12:40
Or we have-- there, there's, there's one, uh, Pipecat user I talked to that is using Gemini Live Multimodal to detect if the person answering the phone is a person or if it's a voicemail greeting, but they have a separate...
- 12:54
they have separate pipelines running for whether it's a voicemail or whether it's a human, and all that happens in Pipecat through the use of a parallel pipeline. Run one model to determine, and then it sends a signal to the p- back to the pipeline to say, "Do the voicemail branch or do the human branch."
- 13:07
So you can start to get a, get an idea of what you can build. Even if you have a model like Tavus that is doing ninety percent of the hard work of making the actual interaction feel good, there's just enough other stuff that's gonna happen around the periphery that it just makes a lot of sense to wrap
- 13:24
what you're doing inside something like Pipecat. This is what-- So Brian showed a picture. This is that same Tavus avatar. Um, if you go to the QR code on the last slide, um, which is gonna come up again in a second, um, you can basically run that example.
- 13:42
Like you just need to dr- sign up for Tavus. You get a key. You drop a key in there. You run that example code unmodified, and it will pop up this UI where you can both talk to that avatar in real time, talk to the replica in real time, but also you can see some of like the
- 13:55
interesting guts of what's happening inside Pipecat in that debug panel over there. Do you wanna tell us a little bit about why this architecture is interesting and what we can do in the near future with it?
- 14:06
Yeah. So as I mentioned, when we first built Tavus' conversational video, video interface, we built it ourselves 'cause we didn't know about Pipecat. So we've spent the last year learning a lot of the lessons that Pipecat has already solved.
- 14:18
There are a ton of orchestration, aggregation, communication
- 14:27
functionalities that are in Pipecat already that are going to basically save you months of time. I mean, it's gonna save you a lot of time. So, um, when we first talked about having this talk, I was like, "Well, we're not using Pipecat internally."
- 14:41
I was like, "I, I can't really say we're using it internally." But the thing is, our customers that have come to us that are enterprise customers, they're using Pipecat, and they want to be able to use our stuff in Pipecat.
- 14:51
So now we're, we're getting ready to move our best models into Pipecat. We've already moved, uh, Phoenix, which is our rendering model, but we're also gonna be moving turn-taking, response timing, perception models, things like that.
- 15:04
And eventually, we're gonna mate up and actually bring Pipecat internally as well because it... I spent like the last couple days actually de- debugging a problem that Pipecat's already solved really well, and I don't wanna have to do that anymore.
- 15:20
Uh, yeah, so I talked about these models that are coming. So we, we, we have a couple different, uh, unique models. Our turn detection model is a multilingual model that determines when a person is done speaking.
- 15:32
You wouldn't believe how important that is in a conversational AI. It's going to make your AI faster, and it's gonna make it so it doesn't interrupt people simultaneously. If you, if you have a very fast, uh, a, uh, conversational pipeline, oftentimes it will actually talk over the, the user.
- 15:53
But-- and if you have a slow one, it will take so long to respond that people will be like, "Is it broken?" You wanna get the best of both worlds, and that's what turn detection does.
- 16:01
We're also working on a response timing, uh, model right now, and that response timing-- We're, we're bringing all these to Pipecat soon. That response timing model will determine how quickly it should respond even though the person's done.
- 16:12
Because if I'm telling you about my, my, my grandmother who's like going into a, into a, into a home and she's sad, you're not gonna wanna like quickly respond to that.
- 16:21
You, you wanna think and take your time, right? But if we're having a chitchat, you wanna be fast. So that's what that's all about. And then finally, our multimodal perception is able to look at emotions, look at the surroundings, what the person's wearing, and also we're, we'll be feeding that into the turn-taking and the response timing so
- 16:38
that we're, we're able to provide much more nuanced conversational flow. So those things are coming to Pipecat.
- 16:47
And so, and so this is another example. Um, I will tear through the last of these 'cause we're, we are already out of time, and that's my fault. This is another example showing essentially a different way that you can integrate Tavus into Pipecat, and this is part of the flexibility.
- 16:59
As they develop new models, there are gonna be things that will run directly inside Tavus. There are things that you wanna have a little bit of control, and so you just drop them into, into a slightly differently shaped pipeline, and you can get your bot to actually do what you wanna do.
- 17:11
Um, I will talk about step three, which is deployment extremely quickly. Um, there are a lot of different ways that you can ship these bots. Pipecat is-- I-- Sometimes I call it open source to a fault.
- 17:21
I wish it had a little-- a, a, a few more opinions on some things. Um, really what you need is kind of two pieces. You need some kind of REST API to essentially, to allow your app, whatever your client app is, you need some kind of basic REST API to tell your app that, that a user wants
- 17:38
to talk to a bot. And when that happens, you need something to relatively quickly spin up a new instance of your bot and connect it to that user. And this is what essentially that's showing here.
- 17:47
Um, and then you also need a thing we haven't talked about, again, go read the docs, is the transport layer. That's the, that's the hopefully WebRTC part that actually moves the media back and forth.
- 17:57
That's part of what your infrastructure is configuring. You have a user that wants to use a bot. You need a, you need an API that can start a bot and g- and connect that bot to your user.
- 18:06
The very short version of how if you wanna just solve this problem with a little bit of money, come talk to us at our booth because Pipecat Cloud is...
- 18:13
Like, if you don't wanna mess with Kubernetes and all that kind of stuff, if Kubernetes makes you... We used to have this thing in Heroku where it would replace Kubernetes with scare quotes around it in the Heroku Slack, which was fun.
- 18:22
Um, come to this talk. This is Mark, uh, one of my colleagues, talking a lot more about Pipecat Cloud and how we solve the problems of deploying bots at scale and how you can either use Pipecat Cloud.
- 18:32
But if you want to actually just do it yourself, this is where you can learn how to do that. And that's our time. Thank y'all very much. [outro music]