Building Conversational Agents

Talk transcript

1. 0:00

   [upbeat music] Hello, everyone.

2. 0:15

   Hello.

3. 0:17

   Uh, heute machen wir alles auf Deutsch. Is that-

4. 0:20

   Super. [laughing] [laughing] Perfect.

5. 0:25

   It's just that Philipp and I were both [REDACTED:origin], so we thought it was funny. Maybe we, we can do it in [REDACTED:origin]. Actually, it looks like there's a, there's a [REDACTED:origin] crew there, which is nice.

6. 0:34

   Uh, no, no worries. We'll, we'll, we'll do it in English. We'll do it in a couple different languages maybe. We'll find out. Do we have other languages in the room?

7. 0:43

   Other nationalities? Yeah, what do we have? Sh- shout it out.

8. 0:47

   Serbian. Italian.

9. 0:49

   Serbian. Italian. Spanish.

10. 0:51

    Spanish.

11. 0:51

    Any Icelandic? [laughing] No? Dane- Okay, close. Close.

12. 0:58

    Romanian.

13. 0:59

    Romanian. Nice.

14. 1:00

    Dutch.

15. 1:01

    Dutch.

16. 1:01

    [REDACTED:origin].

17. 1:02

    Okay.

18. 1:03

    [REDACTED:origin].

19. 1:04

    Uh, which?

20. 1:05

    Delhi. [REDACTED:religion].

21. 1:06

    Yeah, but [REDACTED:religion]. Okay. Canada. No?

22. 1:10

    No, no.

23. 1:10

    Okay. Bangalore. No? Uh-

24. 1:13

    Farsi.

25. 1:14

    Farsi. All right, nice.

26. 1:16

    Czech.

27. 1:17

    Czech, yeah. Brilliant. Okay, this is, uh, fantastic. [laughing] Well, thanks everyone for making your way over here with, with all your languages. Really appreciate it. We can, we can really put the model to the test today, which is great.

28. 1:30

    Um, yeah. Hi, I'm Thor, or Torsten for the [REDACTED:origin] speakers amongst you. Um-

29. 1:36

    Hi, I'm Philipp. Um, only Philipp. [laughing] So [laughing] for [REDACTED:origin] and English speakers, so. [laughing]

30. 1:43

    Yeah. It's nice. Uh, we... So we work on the developer experience at Google DeepMind, broadly covering kind of Gemini API, and also, uh, working Google AI Studio as a tool for-

31. 1:55

    Yeah

32. 1:55

    ... you know, developers to try out the models quickly, uh, and then also the, the API, um, interfaces. To use the API, you do need an API key. Uh, actually, who has an API key already?

33. 2:08

    Gemini API. Okay, a couple folks. Uh, who, who has used AI Studio before? Google AI Studio. Okay, a couple folks. Great. Um, maybe we can quickly-

34. 2:19

    Antigravity.

35. 2:19

    Sorry?

36. 2:19

    Antigravity.

37. 2:20

    Yeah, that's good. Uh, you actually don't need an API key for Antigravity. But, um, so if you don't have an API key yet, uh, if you have your machine on you, I hope you do, uh, it's, it's, it's a hands-on workshop.

38. 2:36

    I do apologize, they took away your tables.

39. 2:38

    Yes.

40. 2:39

    Um, so it's a very laptop situation. Uh, like literally, uh, laptop. Um, I hope you didn't bring your Mac Mini or, you know, whatever. [laughing] Uh, but so yes, if you just go to ai.dev.

41. 2:55

    Um, and I mean, there's... You can also go to ai.studio. You can go to aistudio.com, but we paid a lot of money for ai.dev, so please use that. [laughing] Uh, it just, it just redirects to AI Studio.

42. 3:08

    But, um-

43. 3:10

    Yes. Let me-

44. 3:10

    So ai.dev/api-key.

45. 3:13

    Uh, yes. Or on the-

46. 3:15

    Or

47. 3:15

    ... corner left side, there's a Get API key, and then api-keys. It's where you can-

48. 3:21

    Or in [REDACTED:origin], Schlüssel.

49. 3:22

    Uh, yeah. It's, it's my personal account, and we cannot change language, so [laughing] Uh, you need only a Google account, so no credit card, no nothing. All of the things we are going to do is part of the free tier.

50. 3:35

    So if you go to AI Studio or ai.dev, and there's some, like, sign-in form, you can, like, just use your Gmail, no worries. We will not charge you. Uh, and then on api.keys, uh, you find at the top right corner normally something called API key create or Create API key.

51. 3:53

    Uh, and when you do it for the first time, you, uh, might need... So similar for me, uh, I can import my projects or I can create my projects.

52. 4:00

    I can just click Create, give it any name. I mean, we can call it, uh, IEI

53. 4:09

    Workshop, uh, and then Create Project. Maybe I can slide it to English.

54. 4:19

    Okay. Takes a few second, and then you should be able to create key, and this key will be used for the demos or the hands-on things we are going to do use later.

55. 4:32

    Wait, it shows the API key?

56. 4:33

    Yeah, I will delete it, so don't, don't copy it. [laughing] [laughing] But you can like-

57. 4:37

    Copy very fast.

58. 4:38

    Yeah.

59. 4:39

    Take photo.

60. 4:39

    You can use that key. You can, like, uh, put it into your Bash RC or Zsh RC file, or later you can, like, directly inline it it, whatever you prefer.

61. 4:47

    If you have one, you, you can use just one. And since we have a few more minutes time, we really want to make sure everyone who wants to follow along, take your time, create your API key.

62. 4:58

    If there's an error appearing or something else is not working, feel free to raise your hand. Thor will come to you, and we'll help you. And-

63. 5:07

    And yeah, just a reminder, it is a secret API key.

64. 5:10

    Don't show it.

65. 5:11

    So don't do what Philipp is doing there. Uh-

66. 5:13

    That's why I delete my key again.

67. 5:14

    Yes. [laughing] So we do get that a lot, that people, um, leak their API keys. Mostly it's clock code that is, like, pushing the API keys to- [laughing] ... to, to GitHub.

68. 5:25

    So I recommend you don't do that. No, just, just kidding. Just remember, it is a secret API key. Um, so treat it like a secret. Uh, don't share it with your neighbor.

69. 5:36

    Uh-

70. 5:36

    Yes

71. 5:36

    ... um, but yeah, do create that API key now, and we'll give you a couple minutes to do that. And while you do that, or, you know, once you've finished creating your API key, I'd love for you, you know, to just briefly introduce yourself if you want to, and just sort of, you know, let us know what

72. 5:55

    you would like to get out of the workshop. Maybe there's a specific use case you're working on. Um- Yeah. We'd love to kind of get to know you all a little bit as well.

73. 6:04

    So while you create your API keys, uh, if you want to, feel free to just, you know, shout out, uh, your name or nickname, uh, and sort of what you're working on, what you would like to get out of the workshop.

74. 6:17

    Yeah. Wolfram Rebenwolf from the Thursday AI podcast, working for Corby for Weights & Biases, and a big fan of Gemma, uh, and, you know, Gemma Four and Gemini. I'm one of the few who are using it for coding and stuff as well.

75. 6:30

    Nice.

76. 6:30

    For agent use. And, um, I'm also running it on the glasses, actually.

77. 6:34

    Oh, nice. Mm.

78. 6:34

    Um, and I want to see more of this, you know?

79. 6:37

    Okay.

80. 6:37

    And I want Google Glasses.

81. 6:40

    Okay. Yeah. We'll, we'll see if we can get to the glasses by the end of the workshop.

82. 6:45

    Thank you.

83. 6:45

    Sorry. Yeah.

84. 6:46

    If you have a seat to your right, can you jump in so he can, uh, come up on the side here? Makes it easier to be able, um, to find one.

85. 6:54

    And while we are waiting, uh, we launched something cool on Chrome, which I haven't activated it, but when you go on your tab bar, click right, you can now move tabs to the side, and we have vertical tabs now, so... [clapping] [laughs]

86. 7:09

    Yay. So more screen. That's good.

87. 7:14

    Nice. Okay, cool. So glasses.

88. 7:18

    Gemma.

89. 7:18

    Are you, are you, are you running... What are you running on the glass? Well, technically on the phone, right?

90. 7:24

    Vision Claw is the software where you use Gemini-

91. 7:26

    Yeah

92. 7:26

    ... which calls OpenClaw-

93. 7:28

    Nice

94. 7:28

    ... customization for tool.

95. 7:30

    And then that's just using Gemini Life kind of through the WebSocket on the phone, right? Yeah, nice. Yeah, that's cool. Yeah, Vision Claw, if you haven't heard of that, a pretty, pretty fun open source project.

96. 7:40

    Um, and I think you can run it, yeah, sort of on the... Well, now that, uh, Meta has opened up the SDK for the, the Meta Ray-Ban, uh, you can actually hook in something like Gemini Life API, uh, into the glasses.

97. 7:56

    Well, the glasses connect to your phone, and then the phone actually connects to Gemini Life, which is cool.

98. 8:01

    An experiment, but, uh, it shows what will be possible.

99. 8:04

    Yeah, nice.

100. 8:05

     Any problems creating API keys? Everyone has an API key?

101. 8:11

     Should we check? [laughing] Okay.

102. 8:16

     Anyone else? Anything specific that... Yeah?

103. 8:19

     Uh, Michael. I work for GetYourGuide in Zurich. Um, we're building, like, our first kind of AI customer support agent. So-

104. 8:26

     Cool

105. 8:26

     ... just interested, I think, in the general how you guys are doing it, comparing.

106. 8:30

     Okay, cool. Nice. Last chance. Anyone else? Okay. All right.

107. 8:39

     So we can start. Before we go into the hands-on session, we... I have, like, ten, 15 minutes slides, um, to give a bit of a background what we are going to use to build.

108. 8:50

     The first session, which I'm going to do, is more on, like, the building an agent without any live au-audio input. That's where Thor is going to take over later, and then we are going to building some very nice conversational agents.

109. 9:03

     So, um, who of you has used the Gemini API to make an API call to Gemini before? Good. That's a few. Have any one of you used the Interactions API?

110. 9:15

     One, okay. Two, okay. At least some person. So the Interactions API is a new API we launched in December in beta, which hopefully will succeed generateContent soon. It's a unified API to use with models, uh, with agents, and is much more aligned with, I would say, the industry.

111. 9:31

     So much closer to what you are familiar with from open models, with the Chat Completions API, or from Anthropic, or from OpenAI. Um, and what we are going to do, the slides, then we build a coding agent, like a small little Claude code with reading, writing, uh, reading files, writing files, running bash commands.

112. 9:49

     And then we have some time at the end if you have questions around it. And then we do a short break, toilet, drinks, and then Thor will continue. We all have an API key.

113. 9:57

     So as I said, we want to build an API which works for both models and agents. And when we launched the Interactions API, we also launched Deep Research. So maybe you have used Deep Research in ChatGPT or in the Gemini app, where you basically start off a query, and then you get back a plan.

114. 10:12

     And then the model goes on and, like, does deep research for ten, 15 minutes, visiting hundreds of sites. And, um, the API, um, supports both models and agents, and it's very simple to switch between those.

115. 10:24

     You basically either define a model, which could be Gemini 3 Flash, or you define your agent, which could be Deep Research. And we are working on it to, for you to bring your own agent or you to define your own agent that you can customize all of these behaviors.

116. 10:36

     And it's the same surface. So we'll see later, but when you send a request to NanoBanana to generate an image, you can, like, basically chain those interactions to a Flash model to do something else, to Luria to have you, like, generate, uh, audio and, uh, or even to hopefully soon, um, Veo to generate you, uh, video.

117. 10:56

     And the interface is very similar to what you see from OpenAI. So we now have, like, those, uh, content blocks, basically. So every input you provide and output you provide is the same type.

118. 11:07

     It has a type field, which could be a function call, a thought signature, a text, audio, video, image, which hopefully makes it for you easier to build with the Gemini API.

119. 11:16

     And it is less, I would say, Google branded, less proto specific, less gRPC to make it easier for developers to build.

120. 11:24

     Um, the core primitives of the Interactions API, in addition to making it easier, we also introduced state on the server, which will be, which we will use for building our agents, so we don't need to manage our loop and always send back the whole history.

121. 11:37

     Very similar to, uh, Responses API. You now have a previousInteractionId you can provide, which basically attaches to the existing history, so you can, like, just send a new input.

122. 11:47

     Um, as mentioned, we have an agent with Deep Research and background true, so you can start your research, can poll it, or, um, soon use webhooks to get notified when your research is done, so you don't need to keep the connection open.

123. 12:00

     We have the type blocks, and then also we have the same, like, streaming pattern, very typical for web development using SSE, which also makes it hopefully easier for you to build.

124. 12:09

     We all support the built-in tools, but we also now have support for remote MCP. And I think two weeks ago we launched tool combination, so you can now combine Google search with your own custom function, which was one of the big features people were asking for years, I think, and now you can do this.

125. 12:24

     Um, and to summarize again the difference between... So generateContent is what we have today. Interactions API is what we will have tomorrow. We will have, um, server-side state management, but you don't need to use it.

126. 12:36

     So if you say, "Hey, I want to manage my turns, I want to manage my context. I don't trust you," or, "I need to do, like, context engineering. I will need to remove certain parts," you can do this.

127. 12:46

     Um, you send... Can... It's way easier to send new input, basically. We have the built-in agents. We have also background support, so you will get asynchronous execution. We all see with agents when you send the prompt, it might take one, two, three, four minutes to complete, and keeping HTTP requests or connections open for, I would say, more

128. 13:06

     than, like, 10 seconds is not a very good practice. So you want to, like, use asynchronous calls to either get notified or, like, do polling, uh, when it is done.

129. 13:15

     And it is less proto-oriented, which I really prefer. Um, it's much closer to what people know from, like, the, the developer ecosystem. And also a side effect of the state management is that the implicit caching for the API is much better.

130. 13:28

     So for the one of you who don't know what is implicit caching, so when you send a request to the model, the model needs to encode all of your input tokens.

131. 13:37

     And you can cache those encodings for a follow-up request to save cost. And cached requests, I think, are 90% cheaper for the input tokens. And when you have to manage your state or context yourself, you maybe strip out, uh, line breaks, remove, um, certain parts.

132. 13:54

     This breaks the cache. And using the server-side state, the server keeps the context, so the chances for your cache hit rate is much higher. And we see, like, two to three times better cache rates from, like, the, the startups using Interactions API today.

133. 14:10

     Quick code example on what we mean with making it simpler. So on the left side, you have, like, the very proto specific one-off input parts with inline data or text where the, the field basically describes the type.

134. 14:21

     And on the right side, almost looks similar to other APIs, I would say. So should be much easier if you decide after the workshop to give it a try.

135. 14:31

     Um, then I guess all of you know roughly what an agent is. We have a brain or our model which decides what it wants to do, calling a tool, generating text, doing something else.

136. 14:42

     We have tools which basically gives our brain hands and eyes to interact with the environment where it is in. We have the context. It's basically all of the... All the model knows, uh, what it has to do, what it can do, if there are certain preferences, if there are certain constraints.

137. 14:57

     And then we have the loop, which basically combines our model with hands and tools and runs it until the model no longer calls the tools and generates a text.

138. 15:06

     So some quick examples on how to use the API, and then we go into, like, the, the nice hands-on part. So basic chat usage with server-side state becomes very easy because we have our interactions.create call.

139. 15:20

     So we define our model, we define our input, what's the capital of France. We get back the output, and then we can just continue providing the previous ID. And the model behind the scenes or, like, the server behind the scenes basically has our user input, our model output, and then appends the new user input that you don't

140. 15:37

     need to have, like, a client-side history object where you append the user turns, the model turns, and then the user turn again. This becomes very helpful when you build agents where you have the loop and always need to append new user input.

141. 15:52

     And, um, as mentioned before, that also works for agents and models. So we really want to build this unified interface where you can continue your conversations no matter what model you use.

142. 16:05

     And in this example, we basically run a deep research request on research AI agents in 2026, and then we take the research output and just continue with the NanoBanana model to generate a visual for it.

143. 16:19

     And it's, like, four lines of code basically without for you to, what's the context, how do I provide the input, and hopefully makes it a lot easier. But as said, you don't have to do it.

144. 16:30

     So, like, the input field also accepts the same array with role user, role model, and all of the inputs.

145. 16:38

     Um, for tool use, it's also hopefully now much easier. We have a type function, our name for our function which we want to use, describe, and then we have the parameters the model needs to generate.

146. 16:51

     And then here's roughly what we are going to build. So we make our API call with the tools, and then we check what the, the output of the interaction is.

147. 17:02

     Requires action basically means you as a client, you need to do something. So the output generated some function call or some object which you need to react to. We iterate over our, um, output types, check for the function call, execute the function, append the result, send a new interaction until the model no longer decides it wants to

148. 17:23

     call a function and generates a text or something else. Okay.

149. 17:29

     Um, the last time I did a workshop, we were all still coding manually. So that's my first time doing a hands-on workshop where at least I don't code much manually anymore.

150. 17:43

     I'm not sure how you do it. So what we are going to do today is we don't code manually. We are going to use your preferred IDE agent CLI of choice.

151. 17:54

     I'm not sure how many of you are following, but, um, to make it easier, we created agent skills for our agents to use. So if you go to the Gemini, you can search for Gemini API docs coding agents.

152. 18:11

     Can you zoom in it?

153. 18:12

     Ah, yes. Sorry, of course. Um I can also... Can you see it?

154. 18:20

     Yeah, I think you can also just Google the- Yeah ... you can also Google, uh- I mean, I can- Gemini coding agent skills. Gemini

155. 18:30

     agent skills. Okay. Um, yes, this one it should be. The set up your coding agent with Gemini MCP and skills. I think that should bring us here.

156. 18:48

     And the- Very different results Really? Yeah. [laughs] Yes. Or if not, you can go to the documentation, and then on the left side in the Getting Started section, there's a coding agent setup.

157. 19:07

     Are you-- Are we successful? Okay. Um.

158. 19:13

     So one question.

159. 19:13

     Yeah.

160. 19:14

     Should we install this skill globally or just for the project?

161. 19:16

     You can-- You don't need to install it globally.

162. 19:19

     Okay.

163. 19:19

     And that's a good question. So how do you install? So we have multiple skills. So we have the Gemini API dev skill. We are going to use this one.

164. 19:26

     That's for the generateContent API. If you want to use that API or if you are familiar with it, go for it. But we have the, the Gemini Live API, and that's what Thor is going to use later.

165. 19:36

     So we also don't install this. But then we have the Gemini Interactions API. And here you can either pick the, the first command or the second command, depending on what you want, and then, like, just copy it, uh, open your workspace, um, where you're working in.

166. 19:53

     And then I already did it, but you can, like, add the command npx install, and then you should get a, a wizard asking you to install it. There are many preselected.

167. 20:06

     Don't get confused by it. It just means that all of those agents are, are compatible with the .agents folder. So you might have a .agents/skills/gemini-interactions-api, and in there we have our skill.

168. 20:23

     Yeah. I-- Let me also, uh, wait. So I tested with Cursor and in Antigravity and also Gemini CLI. So if you are using one of those three, you are good.

169. 20:33

     If you are using Claude Code, I think it should work as well. I'm not sure if they follow .agents best practice. If not, I mean, I'm sure you can- They invent- invented it, right?

170. 20:43

     Mm-hmm. Like the skills and then the- Yeah, the skills, but I think they only look at .claude. Uh, I think... Yeah, they don't look at agents. They don't look at agents.

171. 20:52

     But I think it installs into skills, no? I don't... Okay. So [coughs] - It should, it should work. Yeah, if you install npx skill with Claude, it doesn't really- Okay.

172. 21:01

     Yeah. Question? Uh, you showed two different commands. One said skills.sh and the other one Context7 or- Yeah. They both, both work the same. What-- But what's the-- what is Context7 mean?

173. 21:10

     Uh, so Context7 is, I think, a product based out of Upstash. So Context7 is-- has an MCP server in our skills CLI, which you can use to get access to, like, skills.

174. 21:22

     It's like a public repository. So- Okay ... and skills.sh is the same from the Vercel team. Okay. So it's like-- And it, it works-- both works on GitHub. So the google-gemini-gemini-skills is a GitHub repository, and the GitHub repository includes all of the skills we, we documented.

175. 21:45

     So you can also go there and find it there. There are also the deals and install commands, and it makes it much easier than cloning it and making sure you have it in the right directory.

176. 21:56

     And then what we can do is, um, to make sure our skill works, we can just, like, ask our agent, uh, "What skills can you

177. 22:07

     use?" And then it should-- If it-- If you have installed it correctly, you should see, yes, we have our specialized interactions API skill. So Antigravity here picked up our skill in the .agents/skill folder.

178. 22:24

     Okay. How are we doing? Can you zoom in just a bit? Zoom in more? One more. Yeah. I think just in the back. It's- Okay. Yeah. This, this should be.

179. 22:34

     Better? Yeah.

180. 22:35

     Um, I'm-- I was using Telegram with my agent.

181. 22:38

     Mm-hmm.

182. 22:38

     Is the result of this being run on the system, or would I have to have it run on-

183. 22:43

     No, you can-- Like, you should be able to do it on, like, your OpenClau kind of stuff if you tell it install the interactions API and then-

184. 22:49

     Just the final product of, of the workshop. Will it run?

185. 22:52

     Yeah, it should run. It's a Python script we are going to execute later.

186. 22:55

     So because I'm only running in a sandbox.

187. 22:57

     Yeah. No, that should work. Okay. Any, any difficulties installing the skill? Any questions what a skill is? Everyone familiar with skills? Okay. What we can do maybe, uh, to give you some, uh, insights on what skill we created or what it contains, right?

188. 23:17

     The importance when creating skills is it should be either something the model cannot do reliably, or if you have some personal preferences on, like, how to do a certain workflow, or I don't know, you always need to run tests using BAN or something like this.

189. 23:32

     And what we did with our skill is we made sure that the agent is aware of which Gemini models are available. A common issue we saw before that is, like, Gemini always use Gemini 1.5, which is no longer the latest model.

190. 23:45

     Um, we also included the agents here. Uh, we have some, like, very high-level information on how it works, but we did not include, like, all of the documentation. What we did instead is you should see, yes, a, a link to our documentation, which is available as Markdown.

191. 24:04

     So instead of the need to always update our skill with, like, for example, we added a new feature to Interactions API to combine tools, we would have needed to update our skill, and then every one of you also need to update your skill to be able to use it.

192. 24:18

     And then we are not making a lot of progress in terms of, like, knowledge cutoff. Instead, we provide the, the information as part of the skill. So all of the agents now have, like, web fetch tools, so they can query the information based on the skill.

193. 24:31

     And then, like, we only need to maintain, like, the documentation, which is mostly up to date. Yeah.

194. 24:36

     How do you find that works with, like, efficiencies? Sorry. How do you find that works with, like, tool calling efficiency and all that sort of thing? Do you find it, like, it having to go and fetch the page to then I mean, it- Add it ...

195. 24:45

     it normally you would provide it as a reference on your local file, right? Yeah. So it either needs to do a read file call or a web file ch- call, which is the same, I would say, in terms of, like, cost.

196. 24:56

     And it works very well. Okay. So what we are going to do as a first example, since we are not wipe coding, uh, we want to build something more substantial.

197. 25:09

     We not just say, "Build an agent." We want to be more specific, so we want to build, um, create

198. 25:17

     an agent class with a constructor and a run method.

199. 25:28

     The constructor creates a GenAI client. So the GenAI client is what we are going to use to call our model. We also need, uh, defines our model.

200. 25:46

     And we also need a global previous interaction

201. 25:58

     ID. And then add the main method to run an example.

202. 26:06

     Uh, do all of that in, uh, workshop.py. Okay.

203. 26:21

     So then as we can see, the model in this case, or the agent as a first step, read our skill,

204. 26:28

     analyzed our main file. It's implementing the skill. Okay. It checks. It should probably fail. Yes, because I'm using uv, so I stop, and I tell it, "Use uv, uh, from the

205. 26:42

     workspace env." And then we let it generate uv. Okay. It checks if we have installed the library.

206. 26:56

     We have. So maybe in your case, the agent still tries to install Google GenAI. If not, you can do it yourself with, like, pip install or uv pip install, um, google-genai.

207. 27:09

     And then- Just for this ... make sure to... Okay. So

208. 27:15

     we have our starting agent class. We have our GenAI client. We have our model ID. It defaults to Gemini Free Flash. It would never have defaulted to Gemini Free Flash if it would, uh, it had hadn't read the skill, right?

209. 27:30

     Because then we were stuck with Gemini 1.5. We have our run method, which calls, uh, makes our interactions.create, uh, call. We have the input text. We have our, um, previous interaction ID.

210. 27:43

     We set our new previous interaction ID, and then we return the text. And then nice as a main example, um, which we will run in a bit. We create our agent.

211. 27:53

     We have turn one, so my name is Phil. And then the agent, uh, runs it. And then what is my name? To... Now we can check if w- our

212. 28:02

     Gemini created Gemini agent, uh, works with, uh, multi-turn in our interactions API. You might see a warning similar to the one I got here. Uh, interaction usage is experimental.

213. 28:15

     That's... As we are still in beta, we really work hard to get the API out of beta to make sure you can use it in production. And our call was successful.

214. 28:25

     "Hi, Phil. Nice to meet you." Okay, maybe we weren't successful. I don't actually have a name. I s- "What is my name?" Hmm.

215. 28:34

     Maybe we should check. Did we do it correctly?

216. 28:39

     I mean, my name is maybe... Ah, okay, that's still the call. Sorry. I asked it what's your name. I mean, it's a language model for trained by Google. It makes sense.

217. 28:50

     And what is my name? "Your name is Philipp. How can I help you?" So yeah. You are using Gemini 3 Flash as a coding model? Yes. Respect. I mean, it, it, it works really well if you are providing good instructions with good skills and good context, and don't expect it to, I don't know, like, cure cancer.

218. 29:10

     Like, if you have a very good understanding of what you are trying to build, Gemini Free Flash is, like, very fast. I mean, it didn't take much long. It didn't consume credits, right?

219. 29:19

     Every one of us is somewhat token constraints at the moment. And it, it works really well. Okay. Use it for agentic use, yes. But for planning and coding, I'm not- Yeah.

220. 29:28

     No, it, it works. I mean, we will going to use free w- I mean, like, it also asks, like, if it wants to run. So we are closing the loop.

221. 29:34

     Even, like, with Gemini Free Flash, it tries to run our script to make sure it works. And then we will continue in a bit. How are we doing? Anyone making successful calls?

222. 29:46

     Yes? Yeah. Okay. Perfect. Back around. Okay. Any, any issues, any errors, any questions?

223. 29:54

     Yeah. I'm using TypeScript. Okay. It, it works? Yeah. Nice. Great. Works for me too. Mm-hmm. Awesome. Even coding on a phone, so... Okay. So normally the next step for our agent, right, we now have our, like, very basic run.

224. 30:11

     We can chat with the model. Now we need to add tools to it. And we want to build some kind of a coding agent. So, uh, first tools are, we are going to add is a read and write file tool.

225. 30:21

     And we just continue in our, um, main agent thread as like, okay,

226. 30:27

     uh, next we need to add a, a read file and a write file tool. Create the,

227. 30:39

     create a basic Python implementation. And also the

228. 30:49

     JSON schema definition. So when you use function calling or tool use, right, we need to create a JSON schema which we provide to the model so the model understands what it needs to generate.

229. 31:01

     Once it generates that schema, we also need to have some kind of a code implementation which we can then run on the client. So we ask it to create a Python implementation and also the JSON schema, uh, and, uh, map to...

230. 31:17

     for the key and key function comma schema.

231. 31:26

     Okay. Let's see what it will come up with.

232. 31:31

     Okay. It's cheating a little bit.

233. 31:51

     So I have like a, a solution folder, and it looked up the- [laughs] ... implementation, implemented. Yeah. I mean, that's why it's still important to, like, check your way to...

234. 32:00

     I can like... It, it, it found, like, the solution, and then it was like, "Oh, I got this solid Python example and solution.agent to guide me. The task is implement readFile and writeFile." [laughs]

235. 32:14

     Smart model for sure.

236. 32:15

     Yes. Uh, so what we got back is we have two new, like, very basic, very, very basic file implementations. So we have a readFile tool with a file path which uses, um, Python syntax to open it, to read it, and we have a writeFile tool with file path and the content and writes it.

237. 32:33

     And then we have our readFileSchema, um, reads a file and returns the content. writeFile writes the file and returns the content. And it made some updates to our

238. 32:43

     agent. So what did it change? Okay, our input is now a text, uh, string and a list. Makes sense since we now need to return a function call. We check.

239. 32:55

     What do we do? Okay, we create a tool definition for our model, which is the

240. 33:01

     t schema. So that's our tools map schema.

241. 33:07

     And then we have our loop, which you might be familiar with from the slide I showed. So after we run our request, we check the interactions.outputs. So the interactions.outputs include all of the events generated by the model.

242. 33:23

     And since Gemini is a reasoning model, it also includes, for example, the thoughts and the thought signatures, which we need to return since we are using the previous interaction ID and the server-side state.

243. 33:34

     That's done by us, and we only need to check, okay, do we have a function call? We have a nice debug, so we can check it. And for our outputs or our function call, we check our tools.

244. 33:45

     Do we have our tool or not? I mean, maybe the model wants to edit the file, but we don't have edit a file tool. We would catch it here.

245. 33:53

     It calls our model, and then it creates the tool results. So for function call, we have a function result. That's also part of the change. We really want to make it easy.

246. 34:04

     Um, we will see later when we use Google Search, there will be a Google Search call and a Google Search result to have very the, the same schema. And then we use recursion.

247. 34:13

     So if we have tool results, we basically call ourself the self run method again. If we don't have tool results, we return the interaction text. And it also updated our example.

248. 34:26

     Write Hello from the agent to a file named hello text and return it back. So we-- I mean, the changes roughly look good to me. We can try and run it.

249. 34:38

     So you will run Python workshop main. What do we get? We get our tool call with writeFile. We get our tool result. We get a readFile call again, and then the final agent response.

250. 34:54

     The file hello agent text was successfully created with the content. Okay.

251. 35:01

     Agents are not only like singleton, right? Maybe we wanted to respond. So next step is basically very ambitious, telling it, "I want to have a

252. 35:13

     continuous," I don't know, "stood in implementation to test."

253. 35:23

     Let's see it. Let's see where it will continue.

254. 35:32

     Um, I hope at least it should update our main function where we have an input, a while loop, basically waiting for the input using our agent. So yep. So we have our user input, uh, and then we have-- we always continue with our agent.run method, which then inside the agent runs it in a loops until there are

255. 35:52

     no tool calls anymore. And if there's a result, we basically get back our response. Okay. Let's quickly wait. Is it too fast, or are we still on track roughly?

256. 36:05

     Yeah. It's fast.

257. 36:06

     I think it's fast.

258. 36:06

     It's fast?

259. 36:06

     Yeah.

260. 36:06

     Okay. I mean, we will share the code later. Um, also very happy to answer questions. Uh, we have many, like, blog posts and examples of that online, so you can...

261. 36:16

     if you are interested in, like, rebuilding it later with less fast. Um, but I'll-- I try to be a little bit slower, but I also want to give Thor enough time to have you speak.

262. 36:26

     So what we can do now since we or, like, our agent implemented our, like, while loop to, um,

263. 36:34

     to, to provide input, we can say something like hello, right? Normally, we should now not do any tool calls because hello is like, yeah, our agent correctly kinda understood, or Gemini in this case understood, "Hey, hello is nothing I need to solve with a read or write file tool," right?

264. 36:50

     So I can say, "Hello. How can I help you? Can you write- Or maybe, can you create a CSV with a thumbs up?

265. 37:07

     Might take a while a bit. So it does the thinking, and it does the function call. And whoop.

266. 37:15

     Okay. Um, certainly, here's a CSV file, blah, blah, blah. Um, okay. Can you write it to disk?

267. 37:31

     Uh, yes I can if you allow... Wait. Maybe

268. 37:36

     our model is not having our tools. Let's check.

269. 37:40

     There's the tools map. It does. Why is it not using our tools? What tools can you use?

270. 37:50

     Okay, it has a read file. Yes. Maybe we are, we're not explicit enough.

271. 38:07

     And what we can do to improve this in one second is we can add a system instruction to tell the model, "Hey, you are a coding agent. You can use tools to write and interact with the..."

272. 38:20

     Okay. There we got our tool call write file with our CSV. Cool. Since we saw a mistake, we can now tell the model, "Hey,

273. 38:29

     add system instructions for the interactions API call, and add an example

274. 38:41

     prompt for a coding agent." Okay. And what's really nice now, since we, uh, loaded the interactions API skill in the beginning, the model still has the awareness of, okay, how do I add system instructions to the interactions API?

275. 39:03

     And what I can guarantee you is that, uh, Gemini 3.0 Flash has never seen any code of the interactions API, because the model was trained be- before we even released the API.

276. 39:13

     So all of the work we were doing so far is based on, like, the skills and, like, the decoding infrastructure and hasn't been part of the, the training. Okay.

277. 39:23

     So what did we get? Um, okay. We can provide it on the run command, or we have one when we create it. That's good. And okay. Coding persona. You are an expert software engineer and helpful coding assistant.

278. 39:38

     You have access to the local file system. Okay. Let's accept it. Let's start our agent again. Let's say hello.

279. 40:00

     Okay. Hello, how can I help you with your software engineering and coding tasks? I mean, definitely better than what we had so far. And what did we send before?

280. 40:08

     We said, "Can you create a SVG with a thumbs up?" So can you create an SVG with a thumbs up? And now let's see if it calls. Yes. It...

281. 40:21

     Now at this time, we got our write file tool call, and then also we got a, hey, I have created a thumbs up SVG file with a simple line art thumbs icon.

282. 40:31

     So, um, can I...

283. 40:37

     I think you should.

284. 40:38

     Yes.

285. 40:38

     Yeah.

286. 40:38

     There we go. We have got a thumbs up icon. Cool.

287. 40:43

     Um, and of course what's missing for a coding agent, right, we need to get some bash tools. That's now not part of the solutions folder. So let's see how we will get our bash tool.

288. 40:55

     Now add a similar run command tool that allows the model to execute

289. 41:07

     bash commands. Okay. Creates an implementation plan. [coughs]

290. 41:21

     It does hold its weight. You're asking to

291. 41:25

     do something.

292. 41:27

     Oh, Philipp, I think there's a question. Yep. Can you-

293. 41:29

     Yeah, I'd say, like I was just testing it, it does hold its weight tools a little bit. Like-

294. 41:33

     It does

295. 41:33

     ... if you try to do like with character and then-

296. 41:35

     Yeah

297. 41:35

     ... even with the

298. 41:36

     Okay. So we have our run command, which uses a subprocess, which in this case I guess is okay. We don't care too much about security for this example. And then our output is a std out.

299. 41:51

     Um, works. Edit. We have our run command tool. Yeah. [coughs] It even updated our system prompt. And now let's stop that. Let's clear that.

300. 42:06

     Let's run it again. Um, any suggestion on what we should test?

301. 42:15

     Use nana banana to create an image.

302. 42:17

     Uh, I'm not sure that will work because we don't have any skills or any information for it.

303. 42:22

     What's time?

304. 42:23

     It's time. Time.

305. 42:24

     Time. Yeah, I guess so.

306. 42:25

     Oh.

307. 42:26

     From bash.

308. 42:28

     Get the time. Tool call, run command, date, Wednesday, April 8.

309. 42:36

     It looks good.

310. 42:38

     It works.

311. 42:38

     Um, yeah. Cool. That's our small little coding agent. Delete all files. No, I mean, let's not do this. [laughs]

312. 42:49

     Try. Try. Any more questions? Any more ideas? We have, like, roughly five to seven minutes. Yes.

313. 42:55

     General question. So because the state is kept in the cyber-

314. 42:58

     Yeah

315. 42:58

     ... is, is it possible to, like, fork the history of-

316. 43:01

     Yes

317. 43:01

     ... different parts?

318. 43:02

     Yes. So- Um, what you always can do... So what we are doing here is, right, we always use the previous interaction ID from the previous term. So we basically, uh, stack it, and you can always go back to any index in the stack and branch from there.

319. 43:23

     So if you would keep the interaction IDs on your client side, you can always use those to, I don't know, branch out and, like, I don't know, have, like, a, a first, um, prompt on, like, do basic web search and then, like, use this as, like, a base for, like, five parallel requests doing some other work.

320. 43:41

     And you can always get the context. So we have an interactions.get method

321. 43:48

     which you can use to retrieve the interaction and then also get the previous interaction ID. So you can basically go back until the beginning and get all of your state if you want to save it for later.

322. 44:00

     And the default for those interactions being stored on the server for free tier is one day. For paid usage is fifty-five days at the moment.

323. 44:09

     Thanks.

324. 44:09

     Yeah, you had a question.

325. 44:10

     You answered my question. [laughs]

326. 44:11

     Sorry?

327. 44:12

     You answered my question. [laughs]

328. 44:12

     Okay. [laughs] Perfect. More questions. Yeah?

329. 44:14

     Does it mean then it can have infinitely long context window?

330. 44:19

     Uh, no. So once... So Gemini models have a million, uh, token context. What would happen now if you reach that, you will get an error. But we are working on context compaction techniques, but it's easier said than done, and, um, still something you currently need to maintain on, on your own client side.

331. 44:38

     So when you say that it will retain, the server will retain for one day, so that means every day it resets to zero and then the-

332. 44:44

     No, no. So when you send a request, you get an ID, the interaction ID. Uh, and the interaction ID stores your input and the output of the model. In the free tier, the input and output and the ID is stored for one day.

333. 44:58

     So meaning if you send a request now and you continue eight hours later from that point, the state or, like, the context is still available. If you would send a request tomorrow, it would basically say cannot find, um, request with the old interaction ID because it basically is pruned after a day.

334. 45:15

     But if you use a paid API key, it's stored for fifty-five days. And, um, the interactions API is also coming to Vertex, and I think there might be a little bit more flexibility in terms of, um, how long you want to store or customize it.

335. 45:30

     Yes?

336. 45:31

     Is there a specific API to rework the context class if you wanna do complex context or cache them on the client side?

337. 45:37

     We don't have one yet, but hopefully soon. Yes?

338. 45:43

     Maybe an idea, you could have the API joke being spoken out with the eSpeak tool.

339. 45:48

     Sorry?

340. 45:49

     The eSpeak tool. It could use the eSpeak tool to actually speak the joke about APIs mentioned in the chat.

341. 45:54

     Ah. And we can also use Gemini, which has a TTS model, which can speak to... But speaking and listening, I mean, tool will show many, many cool things. Any questions regarding the interactions API and the small little agent?

342. 46:09

     No? Okay. Then you get eight minutes.

343. 46:14

     And so five minutes.

344. 46:15

     Okay. Five minutes.

345. 46:16

     Five minutes bio break, and then we'll be back here-

346. 46:20

     With-

347. 46:20

     To make your agent talk.

348. 46:22

     Yes. Cool.

349. 46:24

     Cool. Thanks. [clapping]

350. 46:32

     That worked really well. [laughs]

351. 46:33

     Nice. Yeah.

352. 46:39

     I used the old one and super happy that now there's a new-

353. 46:42

     Mm-hmm

354. 46:42

     ... one version of it. It was two point five before.

355. 46:45

     Yes. Yeah. It's, it's been a while.

356. 46:47

     It's been a long time.

357. 46:50

     How much better is two point one?

358. 46:51

     How much better?

359. 46:51

     Yeah.

360. 46:51

     Much better.

361. 46:52

     Much... [laughs] We didn't even pay him for it.

362. 46:55

     So-

363. 46:55

     There you go

364. 46:56

     ... big upgrade.

365. 46:57

     Big upgrades, yes. Um, yeah, maybe, uh, I know there's a couple more minutes, but-

366. 47:06

     Can I, can I ask a question while we wait about caching?

367. 47:09

     Caching?

368. 47:10

     Yeah.

369. 47:10

     Yes.

370. 47:11

     The input tokens.

371. 47:12

     Yeah.

372. 47:12

     How is that, um, are those cached, um, in the, uh, context array element level or the whole context is cached? So like when adding, uh, the elements to the interaction, is each one of those individually cached or-

373. 47:26

     That's a good question. We probably need to find Philipp to answer that. I actually, I actually don't know. Philipp, caching question. Yep. Uh, the input tokens. Yep. What was it?

374. 47:40

     Um, is the context, uh, cached at the individual interaction level or is the whole context, uh-

375. 47:46

     It is not on an interaction level. It's more on an, like, object level. So it's, it's- For example, when you provide an input, like, no interaction ID, first input PDF, four thousand tokens, and the text input with ten tokens, and you do an follow-up interaction call, maybe only the PDF will be cached and not the other, like

376. 48:07

     the short text. And then if you do another one, maybe the PDF and, like, the follow-up turns will be cached. So it's, like, more on, like, an object level.

377. 48:15

     But since you... I mean, how was it? It's very easy to make a mistake in caching if you, like, even the slightest change in your prompt, removing white space, line breaks will break it.

378. 48:29

     So, like, having this rely on the server to keep it, it's more guaranteed that it's secure, and it could be as easy as, hey, my user says there's an empty line break at the end.

379. 48:40

     Okay, I remove it, and then I use that history again, and then it falls apart.

380. 48:44

     To do it right, ideally every conversation, the previous part of the conversation is cached.

381. 48:49

     Yes. Uh, optimally. But I mean, it depends on where your request kind of hits it and, like, how fast you follow up, but the cache rate should be pretty high.

382. 49:02

     Cool. Maybe, uh, to kick things off, we can sort of look at one of the examples that, that we can build, um- With the, the Live API, the new model.

383. 49:14

     So this is now, uh, Gemini 3.1 Flash Live, which came out two weeks ago, I think. Um, very exciting. It's been a while. I think the previous 2.5 native audio model was December, so it's been a while.

384. 49:31

     The reason being, we did kind of major underlying architecture rework to, you know, ideally

385. 49:39

     lower latency, you know, better scalability in the, in the future as this ramps up. So, uh, that's very exciting. And then, you know, in parallel, we're working on bringing, uh,

386. 49:49

     the Live API to the Interactions API as well. Um, so that's something that Philipp and I are working on at the moment. Not there yet, so we still have to use the, uh, Live API for now.

387. 50:00

     But let's do a little demo. And so this demo uses the Live API in combination with Lyria 3. Uh, so on the Gemini API... Have, have people played with Lyria?

388. 50:13

     So the new Lyria model is actually a music generation model, and so Lyria 3 can now generate songs. Uh, you know, including songs with lyrics. Uh, so this is the clip model, which is kind of a 30-second, uh, clip generation.

389. 50:28

     Uh, but then also we have a full song model now, where you can generate kind of full, full songs. So the idea is that we have our conversational agent, which is kind of the DJ that you're interacting with, and then that DJ has a tool called To Generate Music, right?

390. 50:44

     Um, and so you know, I... So I'm originally [REDACTED:origin], but, you know, growing up I learned kinda English through Harry Potter and BB- BBC Radio 1. And if you remember back in the day, you could like phone in and like wish for a song, and so that's sort of the, the, the idea of the experience here.

391. 51:04

     Right then, you're on Live Jukebox. What absolute banger are we cooking up today? Tell me, what genre are we leaning into, and what sort of vibe are you going for?

392. 51:14

     Let's make-

393. 51:14

     Can, can we get some sort of high energy [REDACTED:origin], um, techno schlager- [laughs] ... about the AI scene in the UK?

394. 51:26

     [REDACTED:origin] techno schlager about UK AI? That's absolutely brilliant, mate. Proper Radio 1 style. Any specific lyrics or themes you want crammed in there? Or sh-

395. 51:36

     Nah, just surprise me. Let's go.

396. 51:41

     Right, you beauty. Let's get those speakers rattling. Prepare for a burst of machine mayhem. Here comes [REDACTED:origin] techno schlager about the UK AI scene. Let's go. [techno music]

397. 51:53

     In London town, the future's bright and keen. The cleverest robots that you've ever seen. The computers wear a proper bowler hat, and the androids all enjoy a lovely chat.

398. 52:07

     They're drinking builder's tea with digital grace. A happy British smile on every metal face. Oh, wunderbar, the AI is super gorgeous. Ja, super gorgeous. [techno music] [laughs] [claps]

399. 52:25

     Yeah. Um- [claps] The, the TPUs enjoy when you clap for them. [laughs] So, um, appreciate that. Uh, any- anyone have any idea for another one? We c- we can do one more.

400. 52:39

     Just shout it out. Any languages? Any genres? Can we have a nursing,

401. 52:48

     uh, uh, song in Swahili? A nur- nursing song? Song, yeah. In- Swahili ... Swahili? Okay.

402. 52:58

     Right then, welcome to the Live Jukebox. You're on air with me.

403. 53:01

     Can we get a nursing song in Swahili, please?

404. 53:07

     A nursing song in Swahili. Brilliant. That's something we don't hear every day. [laughs] To make it an absolute banger, what sort of vibe are we going for? Upbeat and celebratory or something more relaxing and gentle?

405. 53:19

     And what specific bits about nursing should we squeeze in? Let me know, and I'll get this track buzzing.

406. 53:25

     Yeah, tech- techno hardcore- [laughs] ... about... Oh. [laughs]

407. 53:35

     Techno hardcore in Swahili about nursing. Wicked. Prepare to blast those speakers. I'm dialing that in now. Get ready for a total banger, mate. [techno music] [singing]

408. 53:49

     Okay. The vibe is wrong. That's true. [singing] I never know.

409. 54:11

     Well, we, we can feed it into Gemini to, uh, you know, guess the audio out. Yeah. Okay. [laughs]

410. 54:19

     That is the Live Jukebox DJ, uh, DJ. You, uh, if you want to, it's, it's on... So this was built fully with Google AI Studio. Uh, so it's, it's kind of a little vibe codus, um, demo.

411. 54:31

     I've published it in AI Studio as well, so you can try it out. Uh, you will need a paid API key for this one, because the music generation actually requires a billing account.

412. 54:41

     Um, so if you have, you know, an account with credits, uh, you can use that, uh, for example.

413. 54:49

     Uh, cool. Yeah, Gemini, uh, 3.1 Flash Live. You know, benchmarks, obviously benchmarks don't really tell you the truth, uh, as much. They are great for benchmarking things. Uh, the real world, especially in kinda live audio, does often look a bit different.

414. 55:08

     So, um, you know, ideally we'll just try it out ourselves. So Gemini 3.1 Flash Live, it's the model that is now in Gemini Live in your phone. So if you're using Gemini app, uh, on your phone, um, you're talking to that model.

415. 55:23

     Uh, as well as, I think, Search Live has it now in there as well. So if you're talking to Google Search- Uh, I think that's the same model. And then you can build applications using this model on the Live API.

416. 55:38

     So the Live API is a stateful kind of WebSocket API. Um, you are able to send real-time text, audio, video feeds, uh, to the model. So audio, you're sending in a kind of, you know, buffer chunks.

417. 55:55

     So if off the, the real-time audio, you're streaming that in. Uh, video, you can stream in, uh, at a maximum frame rate of one frame per second. So this can be, you know, a camera feed, this can be, uh, a canvas, so, uh, it could be like your, your, your screen share, right?

418. 56:13

     So you could share the screen with the model. So for example, uh, Shopify is using this for, uh, Shopify Sidekick, um, where it's actually kind of like a tech support walking you through.

419. 56:24

     You know, if you're like, "Oh, how do I set up a custom domain for like my Shopify store?" It would basically like talk you through how to do that, and it can see kind of where you are on the screen by sort of ingesting, uh, the frames of the screen.

420. 56:37

     And then in return, uh, the WebSocket gives you kind of real-time events back, and so these are basically streaming back audio buffers. Uh, and then also you can get the audio transcription, so that's kind of the text, um, of it.

421. 56:51

     And then we have, um, tool calling built-in, so Google Search grounding is built in by default. So if you need kind of real-time weather information, you can access that as well.

422. 57:01

     Um, yeah. Some key features. So what's really cool about this model, again, it's, it's kind of a native audio model. So what that means is we're not going through text, so it's kind of not a cascading pipeline where, um, you're transcribing the text, running the text through an LLM, and then generating speech.

423. 57:19

     Uh, but rather the model itself, um, is, you know, going sound token to sound token, and the intelligence is kind of baked into this audio model. Um, so it's based on, you know, Gemini 3.1, so decently intelligent.

424. 57:34

     Uh, you have different kind of thinking levels that you can enable. Um, and so the great thing with that is kind of the multilingual support. So it's, I think, 97 languages that are kind of supported in preview, uh, at the moment, which, uh...

425. 57:48

     And the great thing is because it is kind of a, you know, a native audio model, it, it can actually, it has sort of the audio understanding of, of Gemini built into it.

426. 57:56

     So, um, it can understand a mix of different languages. It can, you know, sort of a Denglish for example, which is like a mix of [REDACTED:origin], Deutsch and, and English, right?

427. 58:06

     So it, it would be able to sort of n-n-naturally switch be-between kind of different languages as well, which is really great. Um, yeah, barge-in. You know, obviously there's kind of automatic, um, uh, voice activity detection sort of built into the model, so you can interrupt it.

428. 58:22

     You saw it earlier with DJ, you know, I was kind of trying to have a conversation, but we're trying to get it to, to, you know, uh, use the tool.

429. 58:28

     Tool use is the other big thing. Um, so major improvements in kind of tool use and instruction following here with that model. Um, and so you can build some, some really cool things with that.

430. 58:42

     Um, so obviously we currently give, only give you, uh, a WebSocket API. So that is kind of a downside. If you used something like gPC Realtime before, you know, you'd get a direct WebRTC, um, kind of infrastructure which, which can be helpful.

431. 58:59

     So we have partnered with, you know, a lot of, uh, sort of integration partners like Livekit, Pipecast. Uh, Software Mentioners, uh, in Poland, they, they built a great service called Fishjam.

432. 59:10

     Uh, Vision Agents, Vox Implant. So these partners have integrated kind of the Live API directly, and then give you sort of easy WebRTC integrations if you, if you, um, you know, want that or need that kind of for your system.

433. 59:24

     Um, yeah. Let's, let's try it out. So you can try it out yourself, uh, and it's gonna be interesting if we all try it out in this room at the same time.

434. 59:31

     So, uh, we'll see how that works. But, uh, again, ai.studio or ai.dev and then /live. Uh, you can try out the model, and so you can, you know, ingest your webcam in here as well.

435. 59:43

     Uh, so we g- we can give it our webcam feeds, um, allow this time. Uh, and then, you know, we can send text as well. So we could send like, uh, "How is my outfit?"

436. 59:56

     Uh, so in this case, you know, I'm not ingesting any-

437. 59:59

     You're wearing a green jacket over a blue T-shirt paired with a black cap. The combination looks casual and comfortable. Is there a specific occasion you have in mind?

438. 1:00:09

     Yeah. Okay. So, um, you know, that obviously is a bit further away from our upbeat sort of British, Australish, uh, Australian, uh, uh, live DJ. So what we can do is we can, um, you kind of adjust our, our voice, so through the system instructions.

439. 1:00:28

     Um, now in terms of the base voices, we don't have that many. There's kind of 30 different base voices.

440. 1:00:34

     Got a project in mind?

441. 1:00:35

     And they are, you know, fairly generic sort of... But because, you know, Gemini has very deep audio understanding, what we can do is actually we can modify the voice through prompts, through system instructions.

442. 1:00:47

     So what we can do is actually we can give it, um, kind of a system instruction here, for example, and we can just say, "Okay, speak in a friendly Irish accent."

443. 1:00:56

     Right? Um, and so now we have that base voice, Puck, and, uh, speak in a friendly Irish accent. [device beeping]

444. 1:01:05

     Hey, can you hear me?

445. 1:01:08

     Well, hello there. Loud and clear, so I can. What can I do for you on this fine day?

446. 1:01:13

     What do you think of my outfit?

447. 1:01:16

     Well, now, you're looking very smart, so you are. That green jacket suits you well, I must say. A grand casual look. Were you thinking of heading out somewhere?

448. 1:01:28

     Uh, no, I'm just here at the AI engineer... Okay, anyway. So I, I realized I didn't put on... So what I wanted to show you as well is, um, this grounding with Google search So what we can do is, um, you know, obviously the, the model itself is trained to a certain knowledge cutoff date.

449. 1:01:46

     And so, um, you know, if we need kinda real-time information like this gorgeous weather that, you know, no one would actually believe us, the model wouldn't believe us- [laughs] ...

450. 1:01:54

     you know, if it, if it didn't have, um, Google Search grounding. But now, you know, we can, we can have Google Search grounding. We can have different thinking levels as well.

451. 1:02:04

     So, um, no thinking is actually very minimal thinking. Um- [coughs] ... but it means, you know, the model will respond faster. It's kinda lower latency because the more thinking we have, uh, obviously the more thinking steps are kind of involved and, um, it'll, you know, increase the latency.

452. 1:02:24

     So you can see in the benchmarks as well, there's kind of thinking high, uh, thinking low, uh, sort of different thinking levels. Um, but so now if we turn on kind of the, the Google Search here.

453. 1:02:37

     Oh, hello there. What range of shenanigans can I help you with today? Anything at all, just give a shout.

454. 1:02:44

     Uh, yeah. What's the weather right now in London, actually?

455. 1:02:51

     Well, now, it looks like it's about nine degrees and mostly cloudy in London right now. [laughs] Might be a bit chilly, so you'd be wanting a jacket for sure. Anything else you need to know about the goings on over there?

456. 1:03:03

     What, what are you talking about? Can you check again? What date is today?

457. 1:03:08

     Ah, apologies there. I seem to have gotten ahead of myself. Today is Wednesday, April 8th, 2026. Is there anything specific you needed to know about today?

458. 1:03:18

     Yeah. Now look up today's actual weather. [laughs]

459. 1:03:23

     Right you are. For Wednesday, April 8th, 2026, in London- [laughs] ... you're looking at temperatures between 5 and 13 degrees with a decent chance of rain, so don't forget that umbrella.

460. 1:03:34

     Anything else on your mind?

461. 1:03:37

     All right. Uh, here we are. It looks like I didn't pray to the, the demo gods. Okay. [laughs] There seems to be something going on. I wonder if we messed up the UI there.

462. 1:03:46

     Uh, it should, it should [laughs], it should work a lot better than that. Um- [coughs] ... I think the Google Search grounding for some reason isn't, isn't working. So but what we can do now is we can obviously try it out, uh, ourselves in an application.

463. 1:04:00

     So the easiest way we could do that is, you know, like the Life Jukebox D- Life Jukebox DJ, um, we could use Google AI Studio to kind of vibe coach our integration.

464. 1:04:13

     Um, so we have this pill here, which is called, uh, add, you know, voice conversation real-time voice with, you know, Gemini Life API. Uh, and then we could say, um, build kind of a multilingual interview assistant that allows me to, you know, train for interviews in different languages, uh, like [REDACTED:origin] and English and Spanish and, you know,

465. 1:04:36

     what have you. Uh, and so we can now kind of fire this off. So this uses Gemini 3.0 Flash Preview. Um, it is limited to, uh, kind of JavaScript full stack, um, environments at the moment.

466. 1:04:54

     So I think you can choose between kind of Next.js, Angular. Um, there's like XR building blocks as well, if you're building for, for kind of glasses, um, sort of web VR experiences.

467. 1:05:06

     Um, so, you know, feel free to kind of fire one of these off, uh, right now, or you can also clone, uh, the Life Jukebox DJ that I shared with you earlier, and you can try that out.

468. 1:05:17

     Um, it'll, it'll take a little time, uh, and you'll hear like a little chime once it is, uh, ready. So in the meantime, what we can do is, uh...

469. 1:05:26

     So if you go to the Gemini Life API docs. Gemini Life API. Uh, there we are.

470. 1:05:37

     So, you know, we've done this. We tried out kind of the Life API in Google AI Studio.

471. 1:05:43

     Um, we also can use the coding agent skill. So, uh, Phil showed you this earlier. Um, we have dedicated coding agent skills also for the, the Gemini Life API, so you can install that.

472. 1:05:56

     It'll help, you know, your coding agent, uh, integrate the Life API, um, more easily, more quickly. Uh, but then also, you know, we have good old example apps on, on GitHub, which can be very, very helpful.

473. 1:06:10

     So what you can do is, um, you can clone these example apps. Uh, so in GitHub, you know, you can, uh, like you do in GitHub, right? [laughs] Uh, you can clone this.

474. 1:06:25

     So we'll open kind of new terminal, and yes, feel free to follow along. Let me do that a bit bigger. Uh, we'll make a new directory. We'll call it AINGEU.

475. 1:06:40

     Uh, I like that they call it Europe, right? But it's, uh... I mean, I guess, yeah, the UK is still- [laughs] ... part of Europe, just not the EU. Fair enough. [laughs]

476. 1:06:51

     Um, and so we'll, we'll go in there, uh, and then we'll just do a Git clone, um, of our app. Uh, and so now we have our app in here.

477. 1:07:02

     Um, so there's actually a couple different examples that we can use in here. Uh, so if you're using Antigravity, there's a handy agy command to, uh, open your examples in Antigravity.

478. 1:07:16

     Uh, and so we can, you know, look at our different examples here and how we kinda need to set that up. So we have, you know, two different scenarios.

479. 1:07:28

     So the Gemini Life GenAI Python example uses the Gemini Life API on the server. So it creates a WebSocket connection from your server to, uh, you know, Gemini Life API, and then- On your front end, you basically set up a proxy, um, to proxy the WebSocket

480. 1:07:53

     connection to your client side, right? Because your, your browser window is kind of your client, and so that is what is capturing, um, your, your, uh, audio feed, your video feed.

481. 1:08:05

     And so in this example, we're just using FastAPI, uh, and we're basically, uh, just setting up kind of a WebSocket here, uh, that our client can connect to. Uh, and then we're basically just receiving sort of our, um, you know, audio cues, our video cues from the client side, uh, or, you know, our text input queue as

482. 1:08:29

     well. Uh, and so we're receiving that. We're then setting up, um, a Live session. So, um, we're using our Gemini client, so we kind of abstracted sort of all the Live API stuff into this Gemini-Live file here.

483. 1:08:45

     Uh, and you can see, like, starting the session, we're basically setting up kind of our LiveConnect config. Maybe I close this for now, uh, so you can see that.

484. 1:08:55

     Um, we're setting up some system instructions. So that was, you know, earlier, kind of set a helpful assistant. We also said, kind of, "Speak in a friendly Irish accent," for example.

485. 1:09:05

     Um, you know, this is where we put our, uh, sort of system instructions, our guardrails. We can, you know, make that pretty long in terms of covering sort of what we, what we want.

486. 1:09:16

     Uh, and then we can define kind of our tools here. Um, and then we're basically setting up our, um, you know, session, and, uh, our session sort of is, you know, the WebSocket session.

487. 1:09:29

     Uh, and then we're just receiving our audio and video cues from the client side, uh, and proxying that through. So that is kind of one approach. That's sort of the server-to-server approach.

488. 1:09:40

     Um, and, you know, we're just using kind of UV here. Um, so if we're setting this up for the first time, uh, we can go into... So this is our Gemini-Live, um, GenAI Python, uh, example here.

489. 1:09:54

     So we can set up, uh, our virtual environment. We can then, um, activate our source here.

490. 1:10:03

     Uh, we can install our dependencies. Uh, again, I might have... This is a, a fun part of, uh, uh, Google Laptop

491. 1:10:19

     security. Come on. All right. Just look away. Don't look, don't look at that. [laughs] Um, and then, yeah, install our requirements. And so we'll need, uh, an API key.

492. 1:10:36

     Uh, so the API key, you can see kind of here, uh, how the conf-fun-configuration is. So we basically just need our Gemini API key, uh, and we need to set up, uh, an environment variable for that.

493. 1:10:48

     So you can see we have an example file here. Not a lot in there because it's basically just that. So we can copy our .env.example into our .env,

494. 1:11:01

     uh, file here. And then do you remember how to get your API key? Where do you get your API key?

495. 1:11:10

     AI.dev. Yes, AI.dev. There we are. Fantastic. Love it. Um, okay. AI.dev. Uh, so that is where you get your API key. Uh, I think I have a couple API keys, so it takes a little while to, um, load them here.

496. 1:11:28

     Uh, which one is... Maybe we'll, we'll use this one. So once you've created your API key, uh, you can copy the A- API key from here. Well, actually, maybe I should create a new one, because later I'll need to delete it.

497. 1:11:42

     Uh, so we'll say ai-eng-europe. Uh, we'll, we have a couple projects here. We'll just use...

498. 1:11:51

     We'll just... Uh, which one? Which one are we using? Too many projects. Okay, we'll just use this one.

499. 1:12:00

     Uh, and so now... [laughs] I really don't like us actually returning the clear API key here. I think we learned something today. Um, okay, we save that. Uh, and so now we have sort of our API key set up, and now what we can do is, uh, we can run our demo, and that was just the main.py.

500. 1:12:22

     And so now our demo will be up and running on [REDACTED:url] here.

501. 1:12:30

     And so we can see, um, it's just kind of a basics of demo. Uh, and when we connect...

502. 1:12:37

     Top of the morning to you. I'm Gemini Live, a little demo of what this API can do. Why not try out some fun features, like hearing me speak in different accents?

503. 1:12:47

     Hey, can you see me seeing?

504. 1:12:52

     Aye, I can see you all right. You're sitting there with your handsome face looking straight at me.

505. 1:12:57

     Yeah.

506. 1:12:57

     Is-

507. 1:12:57

     Can we speak in [REDACTED:origin], please?

508. 1:12:58

     Of course. I'm happy to speak [REDACTED:origin] with you. How can I help you today?

509. 1:13:02

     Sorry, I can't speak [REDACTED:origin]. Can you speak Chinese?

510. 1:13:05

     Of course. No problem. What would you like to chat about?

511. 1:13:09

     Have you eaten? [laughs]

512. 1:13:09

     I am an AI, so I don't need to eat. You? Have you eaten? Do you have any good experiences?

513. 1:13:09

     Ah, that was, that was a tough, uh, tough diss.

514. 1:13:30

     Uh, I was asking- It understood you, though. Sorry? It understood you. Ah, yeah, yeah, yeah, but it didn't transcribe it. Oh. [laughs]

515. 1:13:40

     All right. Uh, we're finding out a lot of things here, uh, to improve, which is nice. Um, but yeah. So what we can see is, um... So this is...

516. 1:13:50

     You could actually notice that the latency ... is a bit worse because we're actually having that jump from our client to our server. Um, I would love to blame the Wi-Fi, but, uh, so with the server-to-server setup, you just have that additional latency of sort of going through, um, your client.

517. 1:14:09

     So what we can do as well is we can go directly from our client to the server. Uh, and so this is the other example that we have, um, which is this one here using ephemeral tokens.

518. 1:14:25

     So, uh, for ephemeral tokens, we can just kinda look into, uh, the setup here. Uh, very similar. We'll just go, uh, back. Uh, actually, let me get a new terminal up here.

519. 1:14:41

     So we'll say, uh, ephemeral tokens. So our ephemeral tokens are basically short-lived tokens, uh, that we generate with our API key on the server side, and then we send that ephemeral token to our client, so, you know, our phone, our browser, uh, to then initiate the WebSocket connection directly from the client to the Live API.

520. 1:15:07

     Uh, again, similar setup here. We want, uh, a virtual environment. Uh, we then, uh, activate our virtual environment, and we install our dependencies.

521. 1:15:23

     Uh, we also need our, uh, API key, uh, again. So I think what we can ch- do is just use the same one. So we'll copy... Maybe just copy this one here and then paste, uh, paste that in.

522. 1:15:39

     Is it big enough? I don't know. Can people see that? Maybe we'll zoom in a little bit more.

523. 1:15:46

     So now we have our key here, um, as well. Uh, and so

524. 1:15:52

     now that we have our dependencies installed, uh, we can run our server. Uh, and so our server here,

525. 1:16:01

     um... We can look at the server real quick.

526. 1:16:08

     So this server is basically just, uh, a very, you know, slim sort of back end that just has our Gemini API key.

527. 1:16:18

     Uh, and then it generates an ephemeral token for us. So sort of, um, ephemeral token's currently on the V1 alpha API, so you'll need to use actually a different API, uh, at the moment for this.

528. 1:16:31

     And then you would pass in kind of this expiration time, uh, because ideally, you know, the token should be short-lived. Uh, so should the token ever leak, you know, uh, it shouldn't be too costly because it'll expire, um, pretty soon.

529. 1:16:47

     So there we are. That's our token, and then we return our token back to our client. So on our front end, we then have our Gemini Live kinda integration here.

530. 1:16:58

     And so this is an example of just a pure WebSocket integration without sort of any SDK. Um, so, you know, if you... You can use kind of any sort of WebSocket framework, um, here, and you can see sort of, you know, how all the, the different sort of raw WebSocket events, um, are handled.

531. 1:17:19

     Uh, and so you can see here we have, um, you know, our... This is kinda our WebSocket, uh, API. So here we need to use kind of the V1 alpha.

532. 1:17:29

     Um, and then this is the bi-di, so bidirectional, um, you know, we're streaming in both directions. Uh, and here we, we pass our access token, which is our short-lived token.

533. 1:17:42

     Uh, great. So what we can do now is now our server's up and running. Uh, so we can see, um, this beautiful interface here, which was, uh, handcrafted. Um, you know, no agent involved in the creation of this one, um, back in the day.

534. 1:17:59

     And so we can just see kind of all the different, um, knobs here that we have. So we'll try this kinda Google grounding as well, and hopefully it was, uh, just a thing we need to fix in the UI.

535. 1:18:11

     Um, here's our Flash Live, uh, preview model, so that's 3.1 Flash Live.

536. 1:18:17

     Uh, and then we can, uh, connect here.

537. 1:18:22

     And so we see kind of the, the server events. Uh, actually, let me maybe, uh, do that again. So we say enable Google Search grounding, connect. So what we can see is kinda our WebSocket, uh, setup.

538. 1:18:38

     So if we look into the network, we now see we have kind of this, um, WebSocket connection here. So we see we have our access token. So our token request to the back end...

539. 1:18:50

     Actually, let me... Let me see that. Uh, okay. So you can see we're just getting our token. So, um, the response was just our auth token and when it expires.

540. 1:19:03

     Uh, what we sent in terms of the payload. Where's our payload? Mm. Okay, actually, there was no payload. Uh, yeah. We're, we're just getting a token back. So once we have our token, we then set up our, um, WebSocket, and so we can see our messages here.

541. 1:19:21

     So basically, the first message in our WebSocket, um, we're sending our setup, our configuration. So you see we're, you know, sending our model as, like, Gemini 3.1 Flash preview.

542. 1:19:34

     We have some real-time, uh, input there. We have the tools. So hopefully now the Google Search, uh, should work. And so now what we can do is we can, um, start our audio.

543. 1:19:49

     Hey, can you hear me?

544. 1:19:51

     Yep. I can hear you loud and clear. How can I help you today?

545. 1:19:55

     Um, I'm in London right now, and I'm wondering, do I need an umbrella? What's the weather today?

546. 1:20:04

     I'm not getting the current weather for London right now. Sorry about that. Your best bet is probably to quickly check a weather app on your phone or search online.

547. 1:20:12

     That'll give you the latest on rain.

548. 1:20:14

     Oh, you're killing me. Just use Google Search.

549. 1:20:20

     My apologies. I seem to be having some trouble pulling that up right now. I know that's frustrating. You can get the forecast easily by just asking your phone or searching online.

550. 1:20:29

     I'll try to get this sorted on my end.

551. 1:20:32

     Gemini, why are you doing this to me? That's just, uh...

552. 1:20:37

     I understand your frustration. Sometimes technology gives us hiccups, and I'm sorry I couldn't get that weather for you right now. I'm always learning, so hopefully next time will be smoother.

553. 1:20:47

     Was there anything else I could help you with?

554. 1:20:49

     All right. That was painful. Uh, I do apologize for that. Okay, so something is definitely going on. That is, um, not good. Not, not nice. Um, let's see if we have more, uh, luck with the custom tools.

555. 1:21:07

     So we're also passing in... Do we pass in some custom tools, or is there actually some issue with our, uh... Da, da, da, da, da.

556. 1:21:24

     Hey, can you show me a Hello World alert?

557. 1:21:34

     There you go. A Hello World alert has been displayed. Anything else I can help you with?

558. 1:21:40

     Yeah. Can you change the background, uh, of the page to green, please?

559. 1:21:49

     Ooh. [clapping]

560. 1:21:49

     You got it. The background color should now be green. Let me know if you need anything else.

561. 1:21:54

     Yeah. Can you tell me what's the weather right now in London? [laughing]

562. 1:21:58

     I understand you're asking about the weather in London, but unfortunately-

563. 1:22:02

     I did not enable the Google grounding, but for some reason it's... Yeah. So with the Google grounding it should work, so I, I wonder if I messed up something.

564. 1:22:13

     No worries at all. The weather information isn't something I can grab right now, but if you have any other questions or need help with different functions, feel free to ask.

565. 1:22:22

     Let's try this again. Hey, can you hear me?

566. 1:22:32

     Ah, yeah, we need some audio. Hey, can you change the, the background back to white, please?

567. 1:22:40

     I wish I could, but I don't actually have control over your display settings. You might need to look in your device's settings menu to change that. Anything else I can help with instead?

568. 1:22:51

     Oh, no. Sorry, of the page. The, the background of the page back to white.

569. 1:22:57

     Oh, I see. Sadly, I still can't change that for you. Those appearance settings are part of the website or app you're using.

570. 1:23:06

     Oh yeah, you're right. Um, no, but I th... Okay.

571. 1:23:12

     We'll disable custom tools. That shouldn't be the case anymore. All right. Um, lots of work to do, [laughs]

572. 1:23:20

     but, uh... So yeah, this is sort of how you can, um, get up and running.

573. 1:23:27

     Uh, did anyone manage to get it running on their machine?

574. 1:23:33

     Yep.

575. 1:23:33

     Yeah? Is, uh... Is Google Search grounding working?

576. 1:23:37

     No.

577. 1:23:38

     No?

578. 1:23:38

     Sadly, no.

579. 1:23:40

     Okay. That's, uh... That's a shame. All right. Um,

580. 1:23:50

     what else did we have? So we need to fix, uh... Lots of takes away from this. Um, yeah, if you didn't get the link, so this is the link to the Live API examples.

581. 1:24:02

     So, um, they also link from the docs. So if you go to kind of the Gemini Live, uh, API docs, you can find them there.

582. 1:24:12

     Um, is that the end of my slides? I do think we have... Yeah, the agent skills as well. So I mean, if you just go to the Gemini Live, uh, docs.

583. 1:24:30

     Uh, Gemini Live API. Uh, and in the... On the main docs, uh, we've actually linked it from, uh, the top there. Uh, try the Live API, Google AI Studio, uh, the GitHub examples, or use the coding skills.

584. 1:24:47

     Um, so that is... Yeah. That is how you can get started.

585. 1:24:54

     Cool. That was, uh, not how I was hoping this would go. Um, yeah. We'll, we'll figure out what, what went wrong there. But, um, yeah, appreciate you all joining.

586. 1:25:05

     And yeah, we'll, we'll take questions.

587. 1:25:07

     Yeah. So, uh, I was just wondering, uh, how does context look like in this, uh, Gemini?

588. 1:25:14

     Yeah. So you can look at kind of the session management. So, um, there's actually sort of...

589. 1:25:22

     So the Live API has kind of this context window compression. Um, so without compression, kind of audio-only sessions are limited to 15 minutes. Um, audio-video sessions are limited to two minutes.

590. 1:25:36

     Um, so what that means is that sort of

591. 1:25:41

     you will... The session will kind of terminate. It will give you sort of this go-away ping. Um, and what you can do is sort of enabling context window compression.

592. 1:25:53

     So context window compression is kind of like a, you know, sliding window where you basically say, you know, like, "I want to keep that much context sort of in my window."

593. 1:26:02

     And sort of as the conversation progresses, it will then actually, like- Forget kind of the previous context before that window. Um, so yeah. So context window compression is something that you can, uh, enable kind of, you know, to have that sliding window to, uh, make the sessions longer.

594. 1:26:22

     But then, you know, there's only so much context depending on, um, the frame rate that you're feeding in in terms of images, depending, um... Yeah, mostly, mostly it's images.

595. 1:26:33

     Audio is sort of, uh, audio-only sessions, there's more kind of context you can keep in the window. But then as you're adding kind of video frames to it, uh, it does,

596. 1:26:46

     yeah, compress it. Uh, yeah. In the... Oh, yeah, yeah.

597. 1:26:57

     Um, is there any real-life use cases that this is being used for that's very creative from a, like, a business context?

598. 1:27:06

     Very-

599. 1:27:06

     Obviously, what you've done today is a lot of fun, but, like, applying it to, to business. Is there any businesses that are actually doing this really well right now?

600. 1:27:14

     Yeah. Um, I mean... So Shopify has it in production with Shopify Sidekick.

601. 1:27:20

     Mm.

602. 1:27:20

     Um, there is, uh, a bunch of... So, like, uh, actually one that I really enjoy if you Gemini Life API blog.

603. 1:27:32

     Uh, we had kind of a, a case study, which is, um, this startup. Yeah, so I mean, Stitch is using it as well. You can, like, vibe code w- w- vibe design with your voice in Stitch.

604. 1:27:47

     But then, you know, Stitch is built by Google, so, um, you know, you probably gotta discount that. Um, Waymo is also integrating it, so, um, you know, we... First we got rid of the drivers in the cars, and then, you know, you do wanna talk to someone in the car, you can then, uh, talk to Gemini, uh,

605. 1:28:06

     in the future. So they, they are working on that. Um, I like this one. So this is, uh, Hey Ado. Um, it's a, it's a great startup from Argentina.

606. 1:28:17

     Uh, so they are building these, um, voice, uh, sort of companions for the elderly, uh, in combination with, uh, kind of an app, uh, for sort of the caretakers or, you know, the, the, the, the children of the elderly, where, you know, they can get notifications sh- yeah, should the elderly mumble about something that...

607. 1:28:41

     No. But so it's, it's a, it's a really nice interaction where, like, the multilinguality, um, really shines, you know. Because, like, in Argentina, uh, you know, a lot of it would be Spanish.

608. 1:28:53

     Um, but there's a nice example you can kind of look at, um, which is really sweet.

609. 1:29:00

     I came here... I came to visit my grandma, Maria. I want you to say hi to her but remember she-

610. 1:29:03

     Yeah. You might... You, you can, you can look at it in your, in your own time. Uh, and then I think there was another one. So, um, yes, it is somewhat

611. 1:29:14

     more of a future music use case, just kind of looking at, you know, some of the rough edges and limitations. And I think for now, if you're,

612. 1:29:24

     you know, like in, in a real business use case, you might be, you know, better off with kind of the cascading pipeline because it gives you sort of the observability at each step of the pipeline.

613. 1:29:39

     Um, which kind of with the, the real-time, you know, native audio, um, you don't really have that fine-grain control and observability in terms of, you know, plugging into, say, like rewriting the response before it's being said.

614. 1:29:56

     Um, you know, obviously there are certain benefits with that in terms of the natural flow of the conversation, but for certain business use cases it might, might just not be there yet.

615. 1:30:06

     So it is somewhat, you know, a bet in the future of, like, what, uh, kind of real-time conversational interactions will look like in the future. Um, but yeah, depending on your business use case today, uh, it might not be the best fit just yet.

616. 1:30:25

     Are the transcripts all stored and available?

617. 1:30:29

     Uh, no. So you... In the session, you can't, uh, retrieve them, so you would have to store them on your, on your end. Um,

618. 1:30:37

     a- again, so that is sort of where the, um, integration partners come in. So if, uh... Yeah. So LiveKit, Pipecast, they all have, like, really good offerings to, you know, store the entire audio as well as the entire transcript and sort of give you additional observability tooling on top of that as well.

619. 1:31:01

     Um, so it's not something that is kind of all available sort of,

620. 1:31:06

     you know, from just the Google site. Uh, so that's something we're currently, we're relying on kind of the partner integrations to, to sort of give you that additional functionality.

621. 1:31:17

     Yeah. The issue I always see- Oh, sorry. Uh, behind you. Okay. Yeah.

622. 1:31:22

     Do you wanna go first?

623. 1:31:23

     Uh, no. [REDACTED:gender] first.

624. 1:31:25

     Okay. Um, so you said that this is not, you know, like super business ready for more s- more complex use cases, but what are your thoughts, for example, in, um, replacing interview, interviews for recruitment using this?

625. 1:31:40

     Yeah. I mean, I'm not sure I would, you know, replace your entire interview pipeline with that, but I think it is a great scenario, you know, for certain steps in the interview process or, you know, being able to screen more candidates, for example.

626. 1:31:58

     Um, yeah. I do-

627. 1:32:01

     Do you think it's, is that ready?

628. 1:32:04

     Do I think that's ready? Um-

629. 1:32:07

     Because there's a lot of, like- Um, context that you also wanna give it, right? Especially even if it's like a first screen or the quick screens that you're mentioning, you still want to put some criteria that would guide guardrails on how to guide the, the conversation.

630. 1:32:22

     And then the second part to this is also,

631. 1:32:26

     can it work, for example, with if my company's in a, a Google environment, Google Meets, right? Right now we have auto transcript, so can you put these tools together?

632. 1:32:39

     Um, yeah. So it is, uh, a preview, so it is something you can use in production. Um, depending on your use case, I think you need to evaluate, like, you know, do you need like SOC 2 to- z- sort of there's potentially a bunch of things that you need to build around it for it to actually fit

633. 1:33:00

     your business use case. Um, so yeah, it is ready for experimentation. [laughs]

634. 1:33:12

     Does that help?

635. 1:33:13

     I mean, I think that's what we're doing a lot in the company, is experimenting.

636. 1:33:18

     Yeah. Yeah. Uh, oh, yes.

637. 1:33:22

     Um, an issue I always have is when I'm demonstrating voice agents, a lot of people are talking, [laughs] and it's pr- uh, it can't differentiate the speakers between each other.

638. 1:33:33

     So is there a solution for this already?

639. 1:33:37

     Uh-

640. 1:33:37

     Like, train it on a voice sample or something

641. 1:33:39

     ... in terms of like identifying the different speakers?

642. 1:33:42

     Yeah, and so it only listens to you if it's your agent, for instance.

643. 1:33:46

     Oh, interesting. Um, it only listens to you, so it needs to-

644. 1:33:54

     Imagine you have a coding agent and somebody makes a prank and says, "Delete the f- file" or-

645. 1:33:59

     Yeah

646. 1:33:59

     ... stuff like that. You don't want that. [laughs]

647. 1:34:01

     Yeah, no, that's, that's interesting. No, I don't think there's any specific ability to sort of like say, "Just listen to me." Um, so there is sort of,

648. 1:34:15

     kind of this proactive audio where you can tell it to kinda only respond in certain,

649. 1:34:23

     you know, to certain contexts, so like ignore things that aren't relevant to the conversation. Um, so to some extent, that works. Um, but I don't think it's super reliable at the moment where you'd say like, "Only listen to me, ignore anyone else."

650. 1:34:39

     I've done that with Parakeet from NVIDIA, where you can train a little 10 seconds, and it can differentiate the speakers that way.

651. 1:34:46

     Nice.

652. 1:34:46

     But it would be nice to have something like you talk to it, and then it recognizes your voice-

653. 1:34:50

     Yeah

654. 1:34:50

     ... and ignores the other voices for the rest of the session.

655. 1:34:53

     Ignores everyone else. Mm, that's cool. That's a great, great idea. Yeah. Thanks.

656. 1:34:58

     Uh, yeah, I think in, in the... Yeah.

657. 1:35:02

     What does thinking look like? Uh, is it, does it think in text, or is also thinking in speech?

658. 1:35:09

     Yeah, so you get the thinking, uh, only in, in text. So there's, uh, text events, uh, on the WebSocket channel that, um... So you can, you can opt into getting the thinking, um, as text.

659. 1:35:25

     Yeah. It wouldn't speak out the thinking. Yeah.

660. 1:35:28

     Uh, thanks for the demo and, uh, for being brave to go up against the demo gods. Um, I had a sort of question around the multimodality side of things.

661. 1:35:37

     Mm-hmm.

662. 1:35:37

     Uh, one of the areas that I really want, uh, good text to... No, speech-to-text models is having them be grounded in what I'm looking at. Like, when I'm coding, I wanna just, you know, word vomit into Cursor, or sorry, Antigravity, uh, and have it understand the context and, you know, when I say something that is a specific

663. 1:35:57

     class name, it should just actually use that. Um, how does that work inside of this framework? Like, would, would this be a way to actually do, do that grounding, or would you recommend some other API for that?

664. 1:36:09

     Um, and this is for... So, so general Gemini models are really good at audio understanding. So, um, if your use case doesn't require like fully real-time transcription, I would actually recommend using like Gemini 3 Flash, um, to basically

665. 1:36:33

     transcribe but also get, you know, like ingest context and sort of basically get contextually aware transcription. Um-

666. 1:36:44

     Does that go before the text, uh, before the sample, or... You can do it at the same time. So when you are in your editor, right, you have maybe like two files open with main.py and util.py, and you can provide...

667. 1:36:58

     So since we are multi-modal, you can provide the file as text input and your audio as audio input in the same request. And then it has like awareness of both, and it can use both.

668. 1:37:10

     And it can, I don't know, like generate new code or do something else. I think Stitch does this in a way. I'm not sure like if you can show it or not.

669. 1:37:17

     But in Stitch, you have like, uh, Figma-like mock-ups, and you can speak to it. So you can like... It has the visual awareness plus the audio input on, I don't know, which button you should create.

670. 1:37:29

     And if you only have one button, it knows which button it has to change.

671. 1:37:34

     Yeah. There is, uh... So I mean, depending on your use case, if you need it to like be fully real-time conversational, um, then yeah. So you can kind of use text to sort of ingest additional input or, you know, imagery if it makes sense.

672. 1:37:51

     But then again, that, you know, reduces the context, um, window size. Um, or, you know, if you don't need kind of the fully real time, um, then like using Gemini, you know, just Flash to, you know, transcribe is actually pretty good

673. 1:38:18

     ... every second I send a new image. But if you need, like, visual awareness and your visual doesn't change, you can send the visual as, like, the first input, then speak for five seconds and not send an additional image, then you are not using so much context.

674. 1:38:34

     I think, like, one image is around 1,200 fil- uh, 1,200 tokens.

675. 1:38:38

     Yeah.

676. 1:38:39

     So not too much. Um, so if you are, like, in your editor, you want real time, you basically can use the API. First input is send real-time image, and then send real-time audio, and then you stop basically, and the model has the image input and the audio input and can respond to it.

677. 1:38:54

     So you... There's no need to stream the image consistently if you don't, if it doesn't change or if you don't need, like, it to react to it.

678. 1:39:03

     Yeah. Cool. Yeah. Uh, I think we have a bit more time. Uh, there in the back. Yeah. How do we get you the microphone?

679. 1:39:17

     Sorry.

680. 1:39:17

     Or do you just wanna sh- shout it out? Oh, yeah. [laughs] We'll do a...

681. 1:39:23

     Thank you for your interesting presentation. Uh, I have a question about the, uh, personalization or adaptation. Can it, uh, recognize the speaker's level or the knowledge during the interaction, and then based on the speaker's knowledge, produce the result or not?

682. 1:39:40

     Sorry, can you repeat? Can it, uh, g-

683. 1:39:42

     Can it, uh, recognize the speaker's knowledge or the background during the interaction to produce the response based on the speaker's knowledge or something like that to personalize itself to the-

684. 1:39:54

     So you, you want to ingest kind of initial context? Is that what you're saying?

685. 1:39:59

     Not context. For example, suppose you are talking to that about the civil engineering, and can it recognize I'm a civil engineer, and based on my knowledge, produce the result, use the advanced keyword in civil engineerings or not?

686. 1:40:13

     Based on my knowledge. So you would, you would have to, if it's, like, special- specialized knowledge, you would have to give it a way to access that knowledge.

687. 1:40:23

     Oh, that it doesn't have any memory to recognize the human's background or to find the main context of the information, uh, of, of the interaction, and based on that, produce the next result.

688. 1:40:38

     So you, you have to somehow feed in that knowledge. So you could either do that sort of before you, you know, like, as you set up the session. You can ingest kind of the, the knowledge as initial context, for example, and then it has that in context to talk about.

689. 1:40:55

     Or you would give it kinda function calls to access knowledge sort of during the session as you, as you converse.

690. 1:41:03

     Yeah, I see. Thank you.

691. 1:41:04

     Does that...

692. 1:41:05

     Yeah, no. I was to find that, uh, look at the GPT, for example. During the some turns, it can find the speaker's or the user's knowledge or the main concept of the information, and based on that, the GPT can produce some results.

693. 1:41:21

     That means step by step, gradually it personalize to the main context of the conversation. So my question is, can it, step by step, personalize itself to the main context of the in, uh, of the interaction, and then produce the result to point to the...

694. 1:41:39

     Yeah.

695. 1:41:40

     So, like, as long as the, the context stays within the context window during that conversation-

696. 1:41:46

     Yes

697. 1:41:46

     ... yeah. It, it would... It can-

698. 1:41:48

     Okay

699. 1:41:48

     ... just, like, it can identify different speakers and sort of remember what was said in the conversation. And, like, if they introduce themselves with their name as well, it can remember kind of that person's name.

700. 1:42:03

     Yeah.

701. 1:42:03

     And so, like, yeah, that's kind of the, the, the audio understanding sort of within Gemini.

702. 1:42:09

     Thank you.

703. 1:42:09

     Okay, cool. Uh, yeah. Do you just wanna pass it forward there?

704. 1:42:15

     Yeah, forward.

705. 1:42:16

     Yeah. [laughs] [coughs]

706. 1:42:18

     Yeah. Uh, thanks, uh, for the nice presentation. Uh, could you share maybe some, um, of your experience on how to evaluate these, um, live, uh, voice apps? Because I can imagine that this becomes a lot more complicated than typical apps.

707. 1:42:34

     Um, yeah. I've... So i- it definitely depends on your use case in terms of, like,

708. 1:42:43

     uh, you know, like, what are your requirements in terms of, you know, do you have HIPAA? Do you have SOC 2? Like, what is, uh, the amount of function calls, the amount of guardrails?

709. 1:42:56

     So there's definitely a lot. You know, these demos are nice and, and fun, but, like, to bring that in a bus- business context, there are definitely a lot more, um, steps involved.

710. 1:43:09

     Uh, and so that is kinda where the partner integrations come in. So, uh, you know, LiveKit has built kind of their entire business around sort of giving you all the batteries around sort of voice agents.

711. 1:43:21

     And so I would recommend if, you know, sort of looking at the partner integrations for the sort of real business use cases potentially.

712. 1:43:31

     Thanks.

713. 1:43:35

     Cool. Yeah.

714. 1:43:41

     I hope you don't mind if I just ask a simple question about the Interactions API, going back to a previous talk.

715. 1:43:46

     Yes, please.

716. 1:43:47

     When's that gonna be available on Vertex?

717. 1:43:49

     Um, hopefully soon. I mean, if you speak to some Google Cloud person, some Vertex person, the more you tell them, "I need it on Google Cloud," the, the easier it gets. [laughs] [laughs]

718. 1:44:02

     Will do.

719. 1:44:02

     Yeah, it's certainly not in our control. Um-

720. 1:44:04

     Okay. Yeah, that's fair. That's fair. That's cool. Thank you.

721. 1:44:05

     But, but, I mean, it, the API will be the same, so you can start today on Gemini API, uh, start testing. If you need higher rate limits, anything else, you can, like, always reach out.

722. 1:44:17

     If you need Vertex enterprise specific features, then you might need to wait a little bit.

723. 1:44:24

     Uh, can I ask, like, in terms of, like, PII in any, like, conversation history, do you know how that's, like, stored in terms of, like, like, data sovereignty? Can you specify your own data sources, or is that all handled, like, back-end, um, with- with- like, within the API?

724. 1:44:37

     So you can always disable storing anything. So we have a store equals false flag, so we would not store anything. But not storing means no server-side state. So if you would like to use this, that's a bit difficult.

725. 1:44:49

     For other Vertex features in terms of data sovereignty, where you call the model, I would expect them to be, like, similar to generateContent. So if they have it today, they will have it there in the future as well.

726. 1:45:00

     Okay. Cool. That's great. Thanks. Appreciate it.

727. 1:45:07

     Cool. Uh, one last one. No? Okay. Ah. [coughs]

728. 1:45:13

     Sorry.

729. 1:45:13

     No, please.

730. 1:45:16

     Thank you. So I have a question about, uh, hallucinations. So we-

731. 1:45:21

     Sorry, the what?

732. 1:45:22

     Hallucinations.

733. 1:45:23

     Yes.

734. 1:45:23

     So you've have shown some examples, uh, with the weather that, uh, [laughs] didn't work so well. Uh, but, uh, how do your clients actually deal with that stuff on production?

735. 1:45:38

     Because I can imagine that for, like, some examples that we've seen here, this is fine, but, uh, in real life this is a different story. So can you give some best practices or how to deal with that?

736. 1:45:49

     Yeah, definitely. Um, so I mean on, uh, for the demos, the, there's definitely a lack of best practices in terms of, like, system instructions and, you know, there's a lot that you can do sort of with- [coughs] ...

737. 1:46:04

     uh, the, you know, better system instructions to, uh, have the agent actually follow, um, the system instructions and not, you know, go off and, like, hallucinate the weather, for example.

738. 1:46:16

     Um, so yeah, I think we, we have, like... There's some best practices docs, um, so I'd recommend kinda going through, through those. We have an example as well, sort of, you know, how to sort of structure, um, your, your system prompt and sort of put, you know, your guardrails in there, guidelines and, um, kind of the tool

739. 1:46:37

     definitions as well. And so once you've built that up, uh, the agent gets a lot better, you know, at following the system instructions and kind of staying within those, those parameters.

740. 1:46:53

     Cool. Cool. Um, yes, thanks so much everyone. I, I do apologize for the hiccups, but we, we learned something, and we'll, we'll improve upon it. Uh, but yeah, would love for y'all to test it out and, um, you know, let me know over the next couple days, I'll be around, what you find, and let me know your

741. 1:47:12

     feedback. Thank you. Cheers. [applause] [upbeat electronic music]