AI Engineer World's Fair 2025
Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat
Read the talk
Milliseconds to Magic: Building Voice Workflows with Gemini Live and Pipecat
A voice-driven personal assistant turns grocery lists, book requests, and animated cats into a practical exploration of turn detection, tool behavior, and conversational state.
From a talk by Kwindla Kramer and Shrestha Basu Mallick
Before you start: Familiarity with LLM tool calling and the distinction between application code, frameworks, and APIs will help with the implementation discussion.
What makes talking to a computer feel natural?
People think aloud. We learn to speak before we learn to read, usually talk faster than we type, and convey emotion through our voices. A conversational interface can work with those habits: the computer becomes something to think with, rather than another form to fill out.
Making that interface work requires cooperation across layers. Shrestha Basu Mallick approaches the problem through models and APIs; Kwindla Kramer through applications and agent frameworks. For early adopters, voice agents already serve as sounding boards, coaches, and interfaces to information on devices and in the cloud. The use cases extend further: translation between patients and doctors, directed learning for a fourth grader, speech therapy, and help navigating enterprise software.
Kramer reports that Daily customers sometimes encounter callers who do not realize they are speaking to an agent, even after an initial AI disclosure. That apparent ease conceals substantial engineering. The magic is the interface: underneath it are problems ranging from real-time responsiveness—the foundation of a workable voice conversation—to generating a different visual interface for each conversational turn.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Locate the responsibility, then assess its maturity
The voice stack has four broad layers. Read from the bottom upward, each supplies capabilities that the next layer can compose:
| Layer | Responsibility |
|---|---|
| Model weights | Language and multimodal capabilities |
| Real-time APIs | Live access to inference and session behavior |
| Orchestration | Coordination of real-time application components |
| Application code | The particular workflow and interface |
The Gemini Live API occupies the API layer. Pipecat occupies the orchestration layer, helping manage the complexity of real-time multimodal applications.
For any difficult behavior, ask two questions: Where does the code responsible for it live, and how solved is the problem? The slide combines stack position with a right-to-left maturity axis. Kramer puts none of the listed challenges beyond about 50% solved, explicitly describing that as an arbitrary personal estimate. The map is a way to discuss unfinished work, not a measured performance scale.
As a problem becomes common, its solution tends to move down the stack. An application developer first writes a specialized implementation. Once enough applications need it, a framework can generalize it; later, an API may absorb it. Meanwhile, models become more capable independently of that software migration.
Turn detection makes this progression concrete. Following the discussion of semantic voice activity detection in the preceding talk, Kramer traces his own work: a little over two years earlier, he implemented turn detection inside an application because suitable tools did not yet exist. A few months later, generalized implementations moved into Pipecat. By this talk, detection was available inside the same live API surface that handled inference. Having models handle it themselves remained the expected next step. New use cases, meanwhile, keep creating new responsibilities at the application layer.
Built-in detection is also a choice, not a requirement. Basu Mallick explains that developers can disable the server-side detector and use alternatives from ecosystems such as Daily or LiveKit. In that arrangement, the client supplies speech-boundary signals; Daily and LiveKit are not themselves model names. This option already existed in the historical Live API, with activityStart and activityEnd events, rather than being a later capability to project backward onto the demo.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with an asparagus-pizza grocery list
Kramer’s personal assistant lives somewhere between a to-do list and a brainstorming partner. After roughly a year of using voice to manage priorities, he finds that loose, conversational workflow particularly useful. The version onstage is deliberately experimental: no unit tests or evals, the latest models, and a bleeding-edge Pipecat branch. Basu Mallick will use it live to expose where it breaks.
Both presenters have already seen substantial variation between runs. The deeper adjustment for Kramer is that the model drives the application cycle in ways his conventional programming expectations do not anticipate. An unexpected action may be a failure, but it may also be a useful behavior he never explicitly implemented.
The first task is straightforward: create a grocery list for asparagus pizza. The agent initially announces that it has added pizza ingredients without naming them. Basu Mallick asks it to spell out the ingredients, and it reports pizza crust, mozzarella cheese, tomato sauce, and asparagus. Asked whether it would add anything else, it adds garlic and olive oil. The workflow progresses from a broad intention to an explicit list, then to an open-ended suggestion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When incidental speech becomes application state
Switching to a reading list exposes the conversational boundary problem. The agent begins responding while Basu Mallick is still forming her request, prompting her to remark on turn detection. She then asks to add Dream Count and look up its author. The agent instead announces that it has added Quick.
She repeats the request, then asks whether the agent is still there. Kramer says “Segmentation fault,” and the agent treats the aside as a book title, announcing that it has added Segmentation Fault. Another attempt to request Dream Count stalls; Kramer says “Segmentation,” and that becomes another claimed entry. These are spoken jokes absorbed into the workflow, not evidence that the application process crashed. Basu Mallick jokes that the agent prefers his voice, then tries once more to add the intended book.
The next request is for the second and third books in the Three Body Problem trilogy. Initially, the agent continues discussing segmentation faults and talks over a repeated request. Basu Mallick says it worked much better in ten trials the previous night. That is her account of rehearsal behavior, rather than an evaluation with recorded success criteria.
After another clear request, the agent repeats the intended task, and Basu Mallick reports that it did find the book names. The conversation turns to Google search, including a compound search, but she distinguishes two possible sources of knowledge: the older trilogy might be in the model’s training data, whereas the newer Dream Count had triggered Google search during the previous night’s successful run. The onstage result therefore does not establish that every book lookup used retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adding a task is not the same as displaying it
Basu Mallick creates a work list and asks for an item: create the H2 roadmap by end of day Friday. The agent says it has added the task, but she cannot see it. Only after an explicit request to show the list does it announce a display update. A spoken acknowledgment and a visible interface update are separate parts of the workflow.
The next item is to finish writing a podcast script by end of day Thursday, including the exact date. The agent initially acknowledges the task without the date, so the repair takes two more requests:
- Ask for the exact date corresponding to Thursday. The agent states that today is Wednesday, June 4, 2025, making Thursday June 5, 2025.
- Ask it to add that date to the list. The agent then reads back the task with June 5 included.
The interaction separates interpreting a relative date from putting the resolved value into the task text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Grouping, duplication, and ownership
The requests now become transformations of existing information. Basu Mallick asks the agent to combine groceries, reading, and work into two lists: personal tasks and work tasks. Its response says the lists have been combined and displayed, but the result needs repair. Garlic and olive oil are separate from the pizza ingredients, so she asks for the grocery list to be fleshed out. She then points out two work lists and asks for consolidation. Although the agent claims it has consolidated them, the accompanying frame still shows two Work List groups alongside Tasks & Errands and Reading List.
Next comes an ownership rule with an exception:
| Items | Requested owner |
|---|---|
| Groceries | Kwin |
| Reading, except Segmentation Fault | Shrestha |
| Segmentation Fault | Kwin |
| Work | Shrestha |
The agent first restates the assignments. Basu Mallick has to ask again for them to appear on the screen. After the announced update, she observes that it has spelled Shrestha correctly but Kwin incorrectly. The workflow now requires the model to preserve items, regroup them, apply an exception, and present the result with the right names.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask for an interface, not just another list
A request for a nicer background becomes something more ambitious. Basu Mallick asks for an app with Hello World jumping up and down in Google colors, plus two animated neon-green ASCII cats. The agent announces that generation has started and that the app may take a moment to appear. After a wait, the audience applauds, and she confirms the Google colors and the cats, connecting them to Pipecat.
The voice request initiates the workflow; it does not imply that the live audio model directly produces the rendered interface. The companion demo repository describes an asynchronous tool that calls the standard, non-Live Gemini API to generate embedded UI. It identifies the demonstrated model as Gemini 2.5 Flash Preview. That is historical preview context, not a claim about current model availability or the exact deployed configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the application leaves for the model to infer
The display problems become easier to understand when Kramer explains the tool contract. His code gives the model a function that can add text to the screen, with an optional clear argument, but almost no instructions about how to use it. The model has to infer from the conversation when it should append content and when it should replace what is already there.
A small Python example makes that distinction explicit. Here, two existing work groups remain visible when a consolidated group is merely appended; clearing first produces a replacement view:
python
screen = [
"Work List: Create H2 roadmap by end of day Friday",
"Work List: Finish writing podcast script by June 5, 2025",
]
def add_text(text: str, clear: bool = False) -> None:
if clear:
screen.clear()
screen.append(text)
consolidated = (
"Work List\n"
"- Create H2 roadmap by end of day Friday\n"
"- Finish writing podcast script by June 5, 2025"
)
add_text(consolidated) # Appends; previous groups remain.
add_text(consolidated, clear=True) # Replaces the displayed content.
This illustrates the display operation Kramer describes, rather than reproducing his implementation. Correctly composing the consolidated text does not by itself choose the right operation on the existing screen.
Kramer describes a jagged frontier between what the model can infer and what application code must scaffold. Basu Mallick adds that Kwin’s name can remain misspelled even when she supplies the letters explicitly. Turn detection still needs work, and grocery completeness and list consolidation vary from one run to another. The successful animated app and the awkward list repairs coexist in the same system.
The application also delegates much of its contextual understanding to conversation history. Each session loads many previous conversations as alternating user and assistant messages. Some versions summarize that history; others do not. The model must work out what a list means and which conversational context applies to it. This explains the assistant’s flexibility: the workflow is voice-driven and multimodal, with much of its interpretation carried by the LLM rather than explicit application rules.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remembering what the reminder was for
The purpose of this loose personal assistant comes into focus through two family stories. Basu Mallick’s grandmother in India tied knots in her sari to remind herself to do things. Kramer’s grandmother in North Carolina tied strings around her fingers. Across Calcutta and North Carolina, they arrived at a similar reminder mechanism.
Both mechanisms preserve the fact that something needs remembering, but not what that something is. The wearer still has to supply the missing context from memory. A conversational assistant offers the possibility of preserving that context too: the task, the surrounding discussion, and the meaning that makes the reminder useful. Basu Mallick connects her work at Google to expanding those creative possibilities across continents.
Her outlook is that most interaction with language models will eventually happen through voice. That is a prediction about the interface, while the underlying models remain broader: Gemini is trained multimodally to ingest text, voice, images, and video. She closes by inviting builders to bring their Gemini questions through X, LinkedIn, or email—the next applications will need both the conversational interface and the engineering underneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Companion code for the experimental personal assistant, with a Pipecat bot, React frontend, conversation storage, and generated interfaces. Includes historical setup instructions.
Open-source framework for building real-time voice and multimodal agents.
Further reading
A contemporary introduction to session management, configurable voice activity detection, interruptions, and Pipecat and LiveKit integrations.
- Gemini API I/O updatesArticle
May 2025 release context for native audio preview models and asynchronous function calling, coauthored by Shrestha Basu Mallick.
Updates since the talk
Current implementation guidance for live sessions, audio, tools, and automatic or client-controlled speech boundaries.
Current guidance on voice activity detection, turn completion, realtime model integration, and interruption handling.
Read the complete timestamped transcript
- 0:00
[on hold music] So voice is the most natural of interfaces.
- 0:20
Humans are storytellers, talkers, listeners, conversationalists. We think out aloud. We learn to talk before we learn to read, and most of us talk faster than we type.
- 0:38
We express emotion through our voices, and we use sound to understand the world around us.
- 0:46
We've been working together for the past few months, uh, Shrestha from the angle of models and APIs, and me from the application layer and agent framework direction. And I think we both believe that voice is a critical and universal building block for the whole next generation of, of GenAI, especially at the UI level, but more generally as
- 1:04
well. I mean, those of us who are early adopters of personal voice AI talk to our computers all the time. Uh, we think of the LLMs we talk to as sounding boards and coaches and interfaces to everything that lives on our devices and in the cloud.
- 1:18
And this is not just an early adopter phenomenon, right? Like we already have voice agents deployed at scale. Language translation apps that translate between a patient and a doctor, directed learning apps that a fourth grader can use to learn a topic they want to, speech therapy apps, and copilots that help people navigate complex enterprise software.
- 1:43
One of the things we see in our work with customers at Daily is it's pretty common for people not to realize that they're talking to a voice agent on a phone call, even when you tell them at the beginning of the phone call that they're talking to an AI.
- 1:57
Yeah, and kids born today will probably take all of this for granted, but those of us who are living through this evolution of talking computers, this can sometimes feel like magic.
- 2:08
But of course, anybody who's seen a really great magician prepare a magic trick knows that the magic is just the interface. There's a lot of hard work that goes into creating that magic trick.
- 2:24
So here's a partial list of the hard things that done right collectively add up to that magic. So real-time responsiveness, which we've all in this whole track all day talked about as the foundation thing you have to get right or voice AI is unworkable, through the things that we're just starting to experiment with, like generating dynamic user
- 2:42
interface elements for every conversational turn. These are the things we've been hacking on and thinking about together for the past few months. And we're not gonna go over all of these today, although we did have a little extra time in the session, right, Thor?
- 2:54
Thor said we could talk for like a couple hours maybe.
- 2:56
Yeah.
- 2:57
Um, but we do have a framework that we thought would be useful to share with you, a framework that sort of maps onto how we've worked together from the model layer all the way up.
- 3:07
Yeah, and this barely scratches the surface, but here are the layers of the voice AI stack. So at the bottom, underpinning everything, you have the large language models, uh, that frontier labs like DeepMind work on.
- 3:20
Then above that, you have carefully designed, but at this stage, constantly evolving real-time APIs. Google's version is called the Gemini Live API. Um, above the APIs are the orchestration libraries and frameworks like Pipecat that help to manage and abstract away the complexity of building these real-time multimodal
- 3:45
applications. And then, of course, at the top of the stack, you have the application code.
- 3:52
For each of the hard things we listed on the previous slide, the code that implements that hard thing lives somewhere in that stack. So one of the ways we think about this is that there's a map, and you can sort of think about it two-dimensionally maybe.
- 4:04
There's the where does the code live that kind of solves the hard problem that you're, you know, you're thinking about as a voice agent developer, where in the stack, and then how mature is our solution to that right now?
- 4:17
Yeah. Basically, how solved is this thing? And what we've tried to do here is map all of these various things that you need to get right on a right to left axis of maturity.
- 4:29
And there are a couple of things that are kinda top of mind for me about this mapping. One is that I don't think of any of these things as more than about 50% solved.
- 4:36
Totally arbitrary, like, personal thing. We stressed and I just argued about it a little bit.
- 4:40
Yeah.
- 4:40
Like, what's the right way to represent that on this slide? But what we're trying to say is basically it's early. It's early for voice AI, and there's a lot of work to do at every part of the stack to get to that universal voice UI we're imagining.
- 4:55
Yeah. And secondly, um, as this technology matures, and we've already seen some of this happening, um, the capabilities tend to move down the sto- stack. So what might happen is in your one-off individual applications, you might write some code to solve a specifically difficult challenge.
- 5:14
Now, if enough people expe- experience that challenge, then that tends to get built into the orchestration libraries and frameworks, and then eventually make its way into the APIs. But indi- independently all-- of all of that, the models themselves are getting more and more generally capable.
- 5:33
I mean, we just talked about semantic voice activity detection in the previous talk. So...
- 5:39
Yeah. This is like a great boo-- like follow on to Tom's talk a-about, about turn detection because I think turn detection is a perfect example of this. So, like, I built some of the first talk to an LLM voice AI applications, uh, a little over two years ago now, and I tried to solve turn detection right there
- 5:54
in the application code because there weren't any tools yet for it. A few months later, we built what we thought were pretty generalized at the time, state-of-the-art turn detection implementations into Pipecat, so moved down the layer into the framework.
- 6:08
Now, Shrestha has turn detection in the multimodal live API sort of inside the surface area of those same APIs that are doing inference and other things for you. And I think all of us, as Tom said, expect the models over time to just do turn detection for us.
- 6:24
And all those hard things, I mean... It, it varies depending on exactly what you're talking about of that long list we put, uh, together on that slide. But in general, I think everything is moving down the stack, and then more and more interesting use cases are creating more things to put sort of at the top of the
- 6:40
stack.
- 6:41
Yeah. I will say we have server-side turn detection built in, but we also allow you to turn off tu- turn detection and use models like, uh, Daily and, uh, LiveKit.
- 6:53
So should we start with the demo?
- 6:56
Yeah, we can. We do have a demo to show you. Um, a- and it's, it's sort of a demo of some stuff I've been using in my own life every day for the last year or so.
- 7:07
Uh, I've been experimenting with talking to my computer and my phone as much as I can to do various things, um, as you can imagine, 'cause I post about it probably too often on social media.
- 7:17
And o- one of the things that's become really sticky for me is, uh, is kind of managing my priorities in, like, a very loose way with voice AI. Uh, these days, the conversations I have with my phone and my computer often tend to be in this sort of funny and interesting and new gray area for me between,
- 7:32
like, a to-do list and brainstorming or kind of trying to work stuff out. Um, and because I write this code for myself and am always messing with it, this is definitely a cobbler's children never wear shoes situation.
- 7:46
Like, I don't have unit tests. I don't have evals for this code. I'm always using, like, Shrestha's latest models. I'm always using some bleeding edge branch of Pipecat. So what I thought would be fun to do is fire up a version of this code, have Shrestha talk to it, and see what breaks.
- 8:03
And then maybe we can talk about, like, where those bugs are and, like, how I could have done better next time.
- 8:10
Yeah. I will say, having played with this, there's quite a lot of variation in how the model behaves, so we'll see where we get today.
- 8:19
One thing that I think is interesting is as I write more and more of this code, I keep relearning the lesson that the way the models drive the whole application cycle is very different from my traditional programmer's view of things.
- 8:33
So half the things I'm sure you encountered when you were, like, playing with this are the model did something that my code actually didn't expect. That might have been bad, but it might actually have been good, and it's a whole sort of adjustment to how we think about these applications.
- 8:45
Yeah. There's definitely pleasant surprises also.
- 8:50
I am ready.
- 8:52
Okay. Do I talk?
- 8:54
I think you can just talk, and I don't think I can mute it 'cause I don't know how to use the mouse backwards very well. [laughing]
- 9:01
Okay.
- 9:01
Yeah.
- 9:02
I just need to be a bit-
- 9:02
You can talk whenever you're ready. [laughs] No problem.
- 9:05
You can step around in front of the-
- 9:06
I'm here to help whenever you need. Yep. I can hear you loud and clear.
- 9:11
All right. Okay. So-
- 9:14
What can I help you with today?
- 9:16
Yes. Can you help me create a grocery list for asparagus pizza?
- 9:23
Absolutely. Got it. Pizza ingredients are added to your Groovy list. Anything else?
- 9:28
No. Can you list out the pizza ingredients for asparagus pizza, though?
- 9:35
Sure. I've updated the grocery list to include pizza crust, mozzarella cheese, tomato sauce, and asparagus.
- 9:42
Anything else? Would you add anything else to the list?
- 9:51
I've added garlic and olive oil to the list. Does that sound right?
- 9:56
That sounds perfect. All right. Now let's create my reading list. So-
- 10:01
Great. Uh-
- 10:03
Can you create a-
- 10:04
I'm just start adding to your reading list.
- 10:05
Okay, great. [laughs]
- 10:06
Absolutely. What books would you like to have?
- 10:08
Turn detection. Yeah. [laughing]
- 10:10
Yeah.
- 10:10
Uh, okay. Reading list. Um, uh, can you add the f- Could... Please add the first book, "Dream Count," and can you look up the author?
- 10:20
Okay. I've added "Quick" to your reading list.
- 10:22
No, that was not the book I wanted to add. Please add the first book, "Dream Count," and look up the author for me. [laughs]
- 10:37
Hey, are you there?
- 10:40
Segmentation fault.
- 10:43
Got it. "Segmentation Fault" is added to your reading list. [laughing] What's next? [laughing]
- 10:49
That is, uh... I guess I'll have to... [laughing]
- 10:55
It clearly couldn't find the author for that one. [laughing]
- 10:59
But let's try one last time. Okay. One more book, "Dream Count," and can you look up the author?
- 11:10
Hey, are you there?
- 11:13
Segmentation. [laughing] [laughs]
- 11:16
Shit.
- 11:17
Okay. "Segmentation" has been added to your reading list. [laughing]
- 11:21
It just likes your voice better than mine.
- 11:25
All right. No. "Segmentation Fault" is great, but I would really like to read "Dream Count." Can you add "Dream Count"?
- 11:39
All right. Hey, are you still there?
- 11:43
Yes. I'm still here. How can I help you?
- 11:46
Okay. Can you add the second and third books of the Three Body Problem trilogy?
- 11:53
Sure. The second and third, both "Segmentation Fault" and se- [laughing] While segmentation could be about dividing things into parts-
- 11:59
Can you add the second and third books of the Three Body-
- 12:03
... which can also be a computer science concept. Is there anything else I can help you with?
- 12:07
I swear it worked much better 10 times last night. Um-
- 12:13
I apologize for my performance isn't up to par today. [laughing] Is there anything specific you'd like me to try again?
- 12:19
Well, maybe you can try... All right. But, but, um, let's recover here.
- 12:26
Please add the second and third books of the Three Body Problem trilogy
- 12:35
You want to add the second and third books of the "Three Body Problem" trilogy.
- 12:40
I just-
- 12:40
I've added-
- 12:41
I just wanna point out that it did look up the names and find it. Uh, let's see if it'll call out-- co-co-corporate it.
- 12:48
It does a Google search, right?
- 12:48
Yeah.
- 12:49
A compound Google search with the-
- 12:50
This one might be in its training data 'cause it's a bit older, but the Dream Count one, which it was spectacular on last night, it did a Google search because it's a new book.
- 13:02
All right. Let's move on. Can you create a new list for me called work tasks?
- 13:10
So first item, I would like to say, "Create H2 roadmap by end of day Friday."
- 13:17
Got it. Create H2 roadmap by end of day Friday is added to your work list.
- 13:22
I don't see it. Can you show me on the screen?
- 13:26
I've displayed the work list for you.
- 13:28
All right.
- 13:28
You should see-
- 13:30
Let's add another item. Um-
- 13:32
Okay.
- 13:33
Fi-finish-
- 13:34
What's the next item?
- 13:34
... writing podcast script by end of day Thursday, and can you add the exact date?
- 13:42
Okay. I've added finish writing podcast script by end of day Thursday to your work list. You should-
- 13:47
What's the exact date for end of day Thursday?
- 13:51
Today is Wednesday, June fourth, twenty twenty-five. So end of day Thursday is June fifth, twenty twenty-five.
- 13:59
Can you add that to the list?
- 14:02
Yes. I've added the exact date to the work list.
- 14:05
All right.
- 14:05
It now says, "Finish writing podcast script by end of day Thursday-
- 14:09
Perfect. Perfect
- 14:09
... June fifth, twenty twenty-five."
- 14:11
Can you-
- 14:11
Is there anything else you'd like to add to the list?
- 14:13
Yes, actually.
- 14:14
Or anything el-
- 14:14
Yes, I would. Can you combine my grocery list, my reading list, and my work list and split them into two lists: personal tasks and work tasks?
- 14:29
I've combined your grocery list, reading list, and work list. They are now displayed on the screen as separate lists.
- 14:36
Why are garlic oil-- garlic and olive oil separate from pizza ingredients? Can you f-flesh out the grocery list, please?
- 14:47
You're right. I've updated the grocery list to include garlic and olive oil as part of the pizza ingredients.
- 14:52
All right. And can you consolidate the work list? W-- I see two on the screen.
- 14:59
You're right. I apologize for the duplication. I've consolidated the work list to a single entry. You should now see the updated list on the screen.
- 15:07
Okay. And then finally, can you do the following? Can you assign grocery list to Kwin, reading list to Shrestha, with the exception of Segmentation Fault, which Kwin will read, and work list to Shrestha?
- 15:25
I understand you want to assign the grocery list to Kwin, the reading list to Shrestha, except for Segmentation Fault, which Kwin will read, and the work list to Shrestha as well.
- 15:34
Is that correct?
- 15:34
Can you show that on the screen?
- 15:39
I've updated the lists to show the assignments you specified. You should now see the updated lists on the screen.
- 15:45
And you've spelled my name correctly, which was the harder name, but you spelled Kwin incorrectly. Um, but okay. We'll-- d- we're out of time here. Okay. So if we want...
- 15:57
The final thing here is we would like to have a nice background for this app. So can you-
- 16:04
I apologize for spelling your...
- 16:06
Okay. Could you help us generate... Actually, I'll be more assertive. Please generate a, an app that has the words "Hello World" jumping up and down on the screen with the colors of Google, and then we also want two neon green ASCII cats which are animated.
- 16:32
I'm now generating an app with "Hello World" jumping up and down in Google colors and two animated neon green ASCII cats. It may take a moment for the app to appear on your screen.
- 16:51
Detention. [audience cheering] [audience applauding]
- 17:07
So yeah, there you have the Google colors and the cats from Pipecat. So with that, I'll hand it over to Kwin.
- 17:16
So first, thank-- Or you should go back to your pride of place right in the middle. Um, thank you for being such a good sport. Uh, very messy code on my part, including things like basically no instructions to the LLM about how to display text on the screen, and just telling it it has a function that can
- 17:31
pla-- display text on the screen, and it sort of guesses and learns in context, as you can tell from Shrestha, about when it should clear the screen, 'cause there's an optional clear argument to the add text to the screen function.
- 17:43
And it's super impressive, but also super jagged frontier about whether it kinda can intuit what you wanna do in those contexts. So thank you for, like, doing this, 'cause this is what I do all the time with this code, trying to figure out, like, what these models can do and what kind of code you have to write
- 17:59
and what you don't have to scaffold for them to do well.
- 18:02
Yeah, and it's been, you know, p-playing with this, every turn is different, and it's interesting to see the things that it struggles with, like your name Even if I spell out the exact letters, it somehow really wants to spell Kwin the way it spells.
- 18:19
I think it also, um, I mean, turn detection, as we saw, there's a lot of work that can be done of, of course there. Um, and I'm trying to remember.
- 18:28
There's... And there's of course a lot of variation, and sometimes here, like, there are times when it gets the grocery list perfect and, you know, combines the list perfectly, and sometimes it's a bit in the middle, like here.
- 18:39
And the way this code works is it just, for a given, like, session, it loads lots and lots and lots of previous conversational sessions in user assistant, user assistant sort of messages.
- 18:48
Does sometimes, depending on the version of the code I've sh- like, got running, it summarizes a little bit. Sometimes it doesn't. So we really are leaning on the intelligence of the LLM to do all of the sort of contextual understanding about what we mean by a list, what we mean by the context in that, in which we
- 19:05
are talking about that list. It is super amazing that it, that it works at all, basically, in my mind. And it's all voice-driven, and it's all multimodal from the ground up.
- 19:13
Um, we have a whole nother video we can show, but I, I definitely think we're, we're out of time. So we will-
- 19:18
You have time to do the final talk, and everyone seems to be excited, so...
- 19:22
Gonna show one minute of video.
- 19:23
Maybe should we talk about our grandmothers?
- 19:25
Oh, yes. I totally forgot that part.
- 19:26
Sure.
- 19:26
Sorry. Let's skip past the demo where it gets the grocery list perfect. Um [laughs].
- 19:30
You th- I, I think maybe, maybe this crowd-
- 19:31
I am ready
- 19:31
... would like to see that demo, so. [laughs]
- 19:33
No, no, that was great. Um, so this has been fun for me to work on 'cause, like, it's so relevant to my everyday life. But and Shrestha and I were talking about it, and I think there's actually something else that really kind of hooked me that she said.
- 19:46
Yeah. So, you know, um, my grandmother was [REDACTED:origin], of course, and she used to wear this cloth garment called a sari, and her way of reminding herself when she had to do things was tying knots on the sari.
- 19:59
Uh, of course, um, and then I was chatting with Kwin, and what was incredible is apparently his grandmother in North Carolina, so very different from Calcutta in India, used to, uh, tie strings around her fingers.
- 20:12
Um, and firstly, you know, this is kind of incredible. You know, no matter how many continents separate us, like, smart people come up with the same generally intelligent patterns.
- 20:22
But it's also incredible how technology allows humans to evolve. Now, the, the one problem with either the knots or the strings is you knew you had to remember something, but you didn't know what it was, so you still relied on your memory.
- 20:37
And, you know, ultimately, that's why I do the work I do at Google because I want to build the technologies that enable, you know, an infinite world of creative possibilities tomorrow or even today across continents.
- 20:54
Um, and I just wanna say that we believe that voice is the most natural of interfaces, and there will come a world where most of the interaction with language models will happen via voice.
- 21:06
And the Gemini models are trained to be multimodal from the ground up, so of course they ingest text, voice, but also images and video. So if you have any questions about Gemini, please reach out to me on, on X, on LinkedIn, email, wherever.
- 21:24
Happy to work with builders like yourself.
- 21:27
Yeah. Thanks for coming to the talk, and we would love to see what you build with these models and APIs. [audience applauding] [upbeat music]