Realtime Conversational Video with Pipecat and Tavus — Chad Bailey and Brian Johnson, Daily & Tavus

Talk transcript

1. 0:00

   [upbeat music] We're here to talk about real-time conversational video, uh, with Pipecat, that's me, and with Tavus, that's Brian.

2. 0:19

   We'll introduce ourselves a little bit more, but in the interest of keeping it moving, let's talk about what we're here for. Have anybody, have any of you ever seen one of these robot concierge things?

3. 0:28

   Do they work? No, they don't. They're terrible, right? Um, it's actually possible nowadays to build this kind of thing, but actually good. Um, it's a little bit tricky, but that's what we're here to show you how to do.

4. 0:40

   Um, there are three things you need to think about when you wanna build real-time AI. Um, the first is your models. Obviously, we all know what models are, that's why we're here at this conference.

5. 0:50

   Um, the thing that you don't necessarily know you need to think about is your orchestration layer. We're gonna talk a little bit about that. And then, of course, you need to deploy these bots somewhere.

6. 1:00

   That's the third step, deployment. Talk about that as well.

7. 1:04

   So step one, models. I come from a little bit more of a traditional, it's funny to say that, voice AI world, where the traditional pipeline people talk about is speech-to-text, so transcription, right?

8. 1:17

   And then LLMs for your inference, and then text-to-speech, right? Like, that's the typical kind of cascading pipeline you hear. Sure enough, people are nowadays are using some voice-to-voice models.

9. 1:25

   That is a use case for this kind of thing. But there are reasons sometimes you might use one or the other. Um, real-time video is a lot more complicated.

10. 1:35

    It doesn't have to be, but it can be, and I think Brian will tell you that it should be. Um, there's a lot more stuff you need to think about to do video generation in real time.

11. 1:45

    So Brian, you wanna tell us a little bit about Tavus and how y'all are thinking about this?

12. 1:48

    Thanks, Chad. So Tavus started out as a, as a AI research company, and we started off with a, a single model that was like a rendering model. What we quickly realized is that we need to be able to put this into a real-time context for it to be useful, so it needs to be fast.

13. 2:05

    And once we did that, we started realizing there's a lot of missing, there are a lot of missing pieces, things like turn detection, response timing, uh, picking up signals and orchestration.

14. 2:15

    And uh, we started off in, in the beginning, we didn't know about Pipecat when we first built it, uh, but we've been partnering with Pipecat over the last year, and it's come to our realization that, like, a lot of the stuff Pipecat does is gonna be very important for conversational AI and making it real.

15. 2:31

    Um, I think we can go to the next one.

16. 2:34

    Yeah. Yep.

17. 2:34

    Uh, we have a demo. You can go to our site, tavus.io. I was gonna do it live, but for the sake of time, just check it out. You can go check it out on our website, tavus.io.

18. 2:44

    And I'll hand it back... Well, no, there's one more thing.

19. 2:46

    Yep.

20. 2:47

    So, uh, what we do at Tavus now is we offer a conversational video interface. It is a end-to-end pipeline that allows you to have a conversation with a replica of anyone.

21. 2:58

    You can create your own replica of yourself, you can put it online, and you can have a conversation. The response time is around six hundred milliseconds, but that's not ideal because a lot of times that's too fast.

22. 3:09

    Um, so we have to slow that down sometimes based on some of these models that we're using. Um, and there are a lot of steps that go into this.

23. 3:17

    You can see there's like, uh, where like Ch- like Chad talked about, the, the basic layers of a conversational stack. But we've also ha- we also have these proprietary models, Sparrow-0 and Raven-0, that we've created, which is kind of like our IP or what we're offering.

24. 3:32

    And we're going to... W- Right now, the r- we offer those in our stack, but we're moving towards a world where we're gonna offer those in things like Pipecat.

25. 3:41

    So models, and we'll come back to the Tavus models in a little bit and how they are getting better and some of the cool new things that are coming from Tavus that you will want to use.

26. 3:50

    Um, orchestration is where my, my world steps in. So that's, that's Pipecat. That's the thing on my water bottle and my shirt and my jacket and all that kind of stuff.

27. 3:59

    Um, let's talk a little bit about what Pipecat is. Um, there's a really interesting phrase on Brian's slide, real-time observability and control into the flow of a conversation. Um, that's a, th- A lot of those are words that you, that, that don't really mean anything until you actually go build one of these things.

28. 4:18

    And when you build it, the first time you use it, you go, "Wow, this is amazing. This is great." And then as you start to actually think about what it's gonna mean to have that in production, you realize, oh, wait, there are a lot of like boring infrastructure kinds of things that we need to solve.

29. 4:32

    The ability to understand, like, to, to have observability into how the bot is behaving and why it's behaving that way. The ability to get capture metrics on things and understand things like, sometimes the bot takes a long time to respond.

30. 4:43

    I wonder why that is. Well, it turns out there's a whole lot of these kinds of things that you need for a real live production app, for a real live production bot, that you need something like Pipecat.

31. 4:53

    Pipecat is an open source framework. Uh, it's built by my company, but is open source and actually fully vendor neutral, and it's designed to be this orchestration layer for real-time AI.

32. 5:03

    And by that I mean, you are, you, you have a user that is going to be producing either video and/or audio, and you wanna also be delivering video and/or audio to that user, and you wanna do that with a low latency as possible.

33. 5:15

    That's the real time part of this whole conversation. Um, if you went to the AI Engineer website, and you saw the little button on the bottom right that says "Talk to AIE," uh, that's powered by Pipecat.

34. 5:25

    It's actually using the Gemini Live model, so it's using a voice-to-voice model. But there's still so much other stuff you have to do to go from voice-to-voice demo bot on the web to, like, like, in your browser or on the web, to an actual, like, shipping production app, that even Google themselves, even the Gemini documentation says, you

35. 5:46

    can go use our own, like, tools and our, you know, like, our, our browser tools and things to experiment with Gemini multimodal live. But when you want to take it to production, you do need something like Pipecat to actually orchestrate what's happening in your entire app.

36. 6:03

    Um, I'm gonna try to do this slide very quickly. And there, there are a few QR codes coming up, so now would be a good time to, uh, get those buttons ready.

37. 6:13

    Um, Pipecat itself, two lists of three that you need to think about to understand what Pipecat does. The first one is something I just kind of already talked about.

38. 6:23

    Um, the three things that Pipecat is doing for you is handling input, it's handling the processing, and the output. Input is receiving media from your user. So in the case of a traditional voice bot, that's just voice.

39. 6:36

    Um, in the case of a, of a Tavus replica, that's sending voice, and they even... They're doing some interesting things that we'll talk about with inputting your user's video and allowing a Tavus replica to respond to not only what it's hearing in the voice, but what it's seeing in the video coming from the user.

40. 6:53

    Um, getting into that, that's the processing part. That's step two. That's where, that's where essentially you're going to run through a bunch of different models. Uh, in some cases, you can do almost all of what you need with a single model.

41. 7:04

    Um, in the case of, like, Gemini multimodal live for voice or a Tavus replica, they-- Th- there, there is a way that you use Tavus inside a Pipecat bot where you can basically let Tavus kind of do everything for you, um, run kind of as just one integrated piece.

42. 7:18

    Um, and then, of course, all of those models, hopefully, this is supposed to be real-time and interactive video. Hopefully, those models are producing some kind of output that you wanna show to your user.

43. 7:26

    That's the video and the audio being produced by your tools. In a typical voice bot, that is, you know, that is, uh, text-to-speech that is being played out as audio.

44. 7:35

    It might also be things like UI updates, if you're in a web app, that you're pushing UI updates, that kind of thing. And of course, in the Tavus case, it's video and audio that are hopefully presented in a way where the video stays synchronized to the audio, for example.

45. 7:48

    That's a really, really hard thing to do well, depending on exactly how you build this whole thing. The three fundamental pieces of the-- of Pipecat that enable those things to work are frames, processors, and pipelines.

46. 7:59

    Um, Pipecat's name comes from the fact that it is about building a pipeline. Um, and a pipeline is comprised or is composed of processors. Processors are things that handle frames.

47. 8:11

    Frames are essentially any... It, it is basically a type container for a kind of data. So in a Pipecat pipeline, you will see a whole bunch of frames with things like little snippets of user audio, like ten or twenty milliseconds of audio comes across as an audio frame, or video frames from the user's camera device you can

48. 8:30

    capture. But even things like, uh, voice activity detection, VAD, comes across as a user-started speaking frame in Pipecat. All of those frames progress through a series of processors, and a processor just takes in some frames and outputs other frames.

49. 8:44

    So a good example would be, like, the LLM processor, for example, is taking in frames that are essentially context fra- like, completed context turns from the user and the bot, and it is outputting a stream of text frames.

50. 8:59

    So, so if you're capturing streaming output from your LLM, in Pipecat, that looks like a bunch of text frames coming out of that processor. And all those are put together in a pipeline, and the pipeline is how you describe what you want your bot to do.

51. 9:13

    And the idea behind how Pipecat runs your pipeline is that it's doing all of that stuff asynchronously and doing its best to minimize the latency of every piece of information as it goes through the pipeline.

52. 9:25

    So there is a, a, a much better and longer explanation. I know that that was a lot. Um, there's a much better and longer explanation in the Pipecat docs.

53. 9:33

    That is that QA file. In terms of what it actually looks like, it was gonna be a little tight to try to get in and do some live coding during fifteen minutes.

54. 9:42

    Um, but this is a QR code that links to this example file. There is so much stuff in the Pipecat repo that shows you this. But just to step through these pieces real quick, at the top, there's the transport input.

55. 9:53

    Th- this is the core pipeline inside this bot file, and this is actually the, one of the Tavus examples that we have in the repo. First thing is transport input.

56. 10:01

    That's where the frames come in from your media transport. So whether it's WebRTC or WebSockets or Twilio WebSockets or anything like that. Frames start pouring in from the transport input.

57. 10:12

    They go to a speech-to-text processor. Um, that's where transcription is happening. So for example, one thing that frame processor is doing is it's collecting snippets of audio at a, you know, a frame at a time, twenty milliseconds at a time.

58. 10:24

    But it is sort of up to your transcription processor, whatever that is, Deepgram or Whisper running on something or whatever, to exactly collect a bunch of frames, collect however many frames it needs to then output a snippet of transcription information, right?

59. 10:37

    So that happens in speech-to-text. From there, we go into something called the context aggregator. That's because the transcription or the, the STT processor is emitting transcriptions whenever it feels like it.

60. 10:49

    So we use other frames in the pipeline that have made their way through to understand, okay, the user has started talking. The user's microphone has, you know, microphone level has dropped, so it looks like the user has stopped talking.

61. 10:59

    Maybe now is a good time to group all of the various pieces of transcription we've gotten over the past few seconds together and emit a single context aggregation frame.

62. 11:08

    That's what triggers the LLM to run. And so we, we, we grab the context, and if you've, you know, of course, if you've programmed with LLMs, you know you get the context with all the array of messages and the tools and everything.

63. 11:18

    You shove that to the LLM, and then it starts streaming tokens back. Those tokens come out of LLM as text frames as well as there's, like, a start and end frame.

64. 11:26

    And if you're, if you're familiar with this approach, you can probably see all these other frames as they start to exist in here. But then TTS essentially accumulates those and generates speech.

65. 11:37

    This bot file is actually an older example that uses an older Tavus model where we were actually generating audio, and then we were sending the audio over, I believe, a WebSocket.

66. 11:48

    It's not important. We were sending audio to a Tavus model that was generating the video based on the audio and then sending back to us, back to Pipecat, audio and video.

67. 11:59

    So essentially the same audio but synchronized with the video. Those come as a different series of frames that then go out through transport output. And that is, again, essentially the same transport that we're using on the import side, input side, but this is the output side, and so that's where all that media goes back to the other

68. 12:13

    user. So you can, you can start to see how with this structure, um, it looks very simple right here, but it is incredibly powerful when you realize that you can kind of put anything you want in this pipeline.

69. 12:25

    Uh, we have people, for example, that, like there's a construct in Pipecat called parallel pipelines, and so we have people that have this exact same workflow. But at the same time, in real time, they're running another LLM that is doing things like, um, you know, sentiment analysis.

70. 12:40

    Or we have-- there, there's, there's one, uh, Pipecat user I talked to that is using Gemini Live Multimodal to detect if the person answering the phone is a person or if it's a voicemail greeting, but they have a separate...

71. 12:54

    they have separate pipelines running for whether it's a voicemail or whether it's a human, and all that happens in Pipecat through the use of a parallel pipeline. Run one model to determine, and then it sends a signal to the p- back to the pipeline to say, "Do the voicemail branch or do the human branch."

72. 13:07

    So you can start to get a, get an idea of what you can build. Even if you have a model like Tavus that is doing ninety percent of the hard work of making the actual interaction feel good, there's just enough other stuff that's gonna happen around the periphery that it just makes a lot of sense to wrap

73. 13:24

    what you're doing inside something like Pipecat. This is what-- So Brian showed a picture. This is that same Tavus avatar. Um, if you go to the QR code on the last slide, um, which is gonna come up again in a second, um, you can basically run that example.

74. 13:42

    Like you just need to dr- sign up for Tavus. You get a key. You drop a key in there. You run that example code unmodified, and it will pop up this UI where you can both talk to that avatar in real time, talk to the replica in real time, but also you can see some of like the

75. 13:55

    interesting guts of what's happening inside Pipecat in that debug panel over there. Do you wanna tell us a little bit about why this architecture is interesting and what we can do in the near future with it?

76. 14:06

    Yeah. So as I mentioned, when we first built Tavus' conversational video, video interface, we built it ourselves 'cause we didn't know about Pipecat. So we've spent the last year learning a lot of the lessons that Pipecat has already solved.

77. 14:18

    There are a ton of orchestration, aggregation, communication

78. 14:27

    functionalities that are in Pipecat already that are going to basically save you months of time. I mean, it's gonna save you a lot of time. So, um, when we first talked about having this talk, I was like, "Well, we're not using Pipecat internally."

79. 14:41

    I was like, "I, I can't really say we're using it internally." But the thing is, our customers that have come to us that are enterprise customers, they're using Pipecat, and they want to be able to use our stuff in Pipecat.

80. 14:51

    So now we're, we're getting ready to move our best models into Pipecat. We've already moved, uh, Phoenix, which is our rendering model, but we're also gonna be moving turn-taking, response timing, perception models, things like that.

81. 15:04

    And eventually, we're gonna mate up and actually bring Pipecat internally as well because it... I spent like the last couple days actually de- debugging a problem that Pipecat's already solved really well, and I don't wanna have to do that anymore.

82. 15:20

    Uh, yeah, so I talked about these models that are coming. So we, we, we have a couple different, uh, unique models. Our turn detection model is a multilingual model that determines when a person is done speaking.

83. 15:32

    You wouldn't believe how important that is in a conversational AI. It's going to make your AI faster, and it's gonna make it so it doesn't interrupt people simultaneously. If you, if you have a very fast, uh, a, uh, conversational pipeline, oftentimes it will actually talk over the, the user.

84. 15:53

    But-- and if you have a slow one, it will take so long to respond that people will be like, "Is it broken?" You wanna get the best of both worlds, and that's what turn detection does.

85. 16:01

    We're also working on a response timing, uh, model right now, and that response timing-- We're, we're bringing all these to Pipecat soon. That response timing model will determine how quickly it should respond even though the person's done.

86. 16:12

    Because if I'm telling you about my, my, my grandmother who's like going into a, into a, into a home and she's sad, you're not gonna wanna like quickly respond to that.

87. 16:21

    You, you wanna think and take your time, right? But if we're having a chitchat, you wanna be fast. So that's what that's all about. And then finally, our multimodal perception is able to look at emotions, look at the surroundings, what the person's wearing, and also we're, we'll be feeding that into the turn-taking and the response timing so

88. 16:38

    that we're, we're able to provide much more nuanced conversational flow. So those things are coming to Pipecat.

89. 16:47

    And so, and so this is another example. Um, I will tear through the last of these 'cause we're, we are already out of time, and that's my fault. This is another example showing essentially a different way that you can integrate Tavus into Pipecat, and this is part of the flexibility.

90. 16:59

    As they develop new models, there are gonna be things that will run directly inside Tavus. There are things that you wanna have a little bit of control, and so you just drop them into, into a slightly differently shaped pipeline, and you can get your bot to actually do what you wanna do.

91. 17:11

    Um, I will talk about step three, which is deployment extremely quickly. Um, there are a lot of different ways that you can ship these bots. Pipecat is-- I-- Sometimes I call it open source to a fault.

92. 17:21

    I wish it had a little-- a, a, a few more opinions on some things. Um, really what you need is kind of two pieces. You need some kind of REST API to essentially, to allow your app, whatever your client app is, you need some kind of basic REST API to tell your app that, that a user wants

93. 17:38

    to talk to a bot. And when that happens, you need something to relatively quickly spin up a new instance of your bot and connect it to that user. And this is what essentially that's showing here.

94. 17:47

    Um, and then you also need a thing we haven't talked about, again, go read the docs, is the transport layer. That's the, that's the hopefully WebRTC part that actually moves the media back and forth.

95. 17:57

    That's part of what your infrastructure is configuring. You have a user that wants to use a bot. You need a, you need an API that can start a bot and g- and connect that bot to your user.

96. 18:06

    The very short version of how if you wanna just solve this problem with a little bit of money, come talk to us at our booth because Pipecat Cloud is...

97. 18:13

    Like, if you don't wanna mess with Kubernetes and all that kind of stuff, if Kubernetes makes you... We used to have this thing in Heroku where it would replace Kubernetes with scare quotes around it in the Heroku Slack, which was fun.

98. 18:22

    Um, come to this talk. This is Mark, uh, one of my colleagues, talking a lot more about Pipecat Cloud and how we solve the problems of deploying bots at scale and how you can either use Pipecat Cloud.

99. 18:32

    But if you want to actually just do it yourself, this is where you can learn how to do that. And that's our time. Thank y'all very much. [outro music]