AI Engineering with the Google Gemini 2.5 Model Family

Talk transcript

1. 0:00

   [upbeat music] Okay.

2. 0:16

   Hi, everyone. Welcome, welcome. Um, so welcome to our workshop, AI Engineering with the Google Gemini 2.0 family. So as it is a workshop, we are going to keep it super hands-on, so please ca-keep all computer open.

3. 0:31

   Uh, you don't need any Google account, um, like Google Cloud account. You can use your personal Gmail. It will be completely free for you to use. So that's the point.

4. 0:39

   Um, before we get started, can you maybe help me understand, how many of you has used Google Gemini before?

5. 0:48

   Oh, wow. That's, that's cool. That's a lot of hands, more than the last time I gave a talk like this. Um, so what, um, we are going to do, we have, like, three slides, so don't worry, not too much.

6. 0:57

   Uh, but we're going to focus on Gemini 2.5. So there's a Gemini 2.5 Pro model and a Gemini 2.5 Flash model. We're going to use the Flash model as it's available, uh, for free tier via API access.

7. 1:09

   So we are going to do coding. And both models are multimodal by default, meaning they can understand text, images, audio, videos, documents, and can generate text. We also have Gemini models which can generate images, which we are also going to use, and we have now Gemini models which can generate, uh, audio, so you can create speech from,

8. 1:29

   from text. Um, if you are curious where you can find those nice model cards with all of the feature, the model context, the output tokens, it's on the, the Gemini docs.

9. 1:40

   Um, as mentioned, we have two new, um, text-to-speech models since Google I/O last week, no, the week before. Um, those are really cool. You will try and see them later.

10. 1:50

    And now for the important details. So I created a Slack channel. If you are on the AI engineering Slack channel, you should be able to find it. Um, feel free to use it during the workshop.

11. 1:58

    You can ask questions. I try to regularly check, uh, to answer them, or even afterwards if you have questions, complete a workshop at home or next week, uh, I will take a look and make sure that you get all of the answers.

12. 2:10

    And then, uh, we have one QR code. Uh, it's AI Studio. You can also just enter in your browser ai.dev, uh, or ai.studio. It's which will brings you to AI Studio.

13. 2:19

    And the other link is, so there's a GitHub repository, uh, with the workshop we are going to do. The workshop, I can, like, now switch directly to it. So, um,

14. 2:31

    let's hope if the Wi-Fi give us some freedom. So the good part about a workshop is that we have Google Colabs. So there's not a lot of downloading, uh, happening, and it will all run in the Colab environment if you have a Google account.

15. 2:45

    And the other thing is what we need to do is in AI Studio to generate an AI key. So the GitHub repository is now loaded. In the GitHub repository, we have a notebooks-

16. 2:53

    Can you use the slides please?

17. 2:55

    Uh, yeah, of course. Uh, sorry. So in the, uh, GitHub repository, we have a notebooks, uh, folder, which includes all of our four workshops, uh, plus a, a zero one, which is basically some minor instructions how to set up AI Studio, how to get an API key, and how to send your first request.

18. 3:13

    And then, uh, we have one, or the beginning section will be all about text generation, um, getting started, getting familiar a bit with the SDK. Uh, the second part will be all about multimodality.

19. 3:23

    How can Gemini understand images, video, audios? How can we generate images or audio? And then the third, um, section will be about function calling, structured outputs, the native tools.

20. 3:33

    How can I integrate Google Search into it? And then I guess, um, with all of the hype currently going on, we, we look at how you can integrate MCP servers together with Gemini using it as a model to call the different tools.

21. 3:44

    Um, also very nice, so there's a solutions, um, folder. The solutions folder includes the same notebooks, but with the solutions. So all of the notebooks include to-do text and also some code snippets, um, and some comments.

22. 4:00

    So there's a, a mix between working code snippets, code snippets which has some pointers, and straight up exercises with to-dos for, for you to do. Um, I will work with you through, like, the existing, uh, snippets, and then everyone can work on the exercises.

23. 4:16

    The idea is to, that we try to maybe use 30 minutes per different section. If you, for example, are already very familiar with how what I can do with text generation, and would la-rather, like, look at the multimodalities parts or at the function calling parts, feel free to, like, directly jump into the section.

24. 4:33

    And in general, we want to keep it very open, very dynamic. If you have questions related to the content, maybe unrelated, please keep them coming. Ask them in Slap.

25. 4:43

    Raise your hands. I'm not sure. Maybe we have some microphones here as well, so we can, like, give it to you to, uh, make it super interactive. So I guess let's get started.

26. 4:52

    So if you go to the notebooks, there's also a Colab, um, button you can click, which opens the notebook directly in Google Colab. And if you prefer, like, a local Jupyter environment, you can try to clone the repository.

27. 5:02

    I'm not sure if it works with Wi-Fi or not. Um, I guess Colab will be the easiest. And as mentioned before, the only requirement you basically have is a, a working Google account.

28. 5:13

    Can be your corporate one, can be your private one, can be one you create in the next five minutes. And the first step is, uh, what we need to do is basically go to AI Studio.

29. 5:23

    For the ones who-- you who are not familiar with AI Studio, AI Studio, um, is our developer platform to quickly test the models, to experiment with the models, uh, and also keep it very similar to the development code you will be use.

30. 5:37

    So if I try to run a request like, um, maybe let's ask something. What's the AI Engineering Summit? I can, on the right side, for example, enable a native tools connected with Google Search.

31. 5:51

    I have our Flash preview model. I can run the request, and we'll see how fast, yeah, the model is thinking. And the nice part here is, uh, I can also directly get the SDK code from Our request as soon as it's ready.

32. 6:06

    So if you are experimenting in AI Studio and you want to convert it into a Python script or want to play around with it, extend it, um, that's all possible.

33. 6:15

    So the AI Engineering Summit refers to in- several events focusing on artificial intelligence and engineering. That's great, and that also matches, like, the one from New York, which was done this February.

34. 6:27

    It's cool. So what you need to do to get your API key, at the top right is, uh, I can also make that bigger. Maybe it's easier. So we have, uh, Get API key at the top.

35. 6:38

    Um, you go to it. Um, sorry for ... It's German, uh, for being it's German. But on the top right corner, there should be A- a Create API key, a blue button.

36. 6:47

    And in there you should-- it should open, um, some, uh, I can click it. Should open some modal where you can either select your Google Cloud project. If there's none, you should be able to create one.

37. 7:01

    If ... Yep. Can you switch to light mode maybe? Uh, yeah. Yeah.

38. 7:06

    Of course. Um, appearance. Light. Okay. Good idea. Okay. Uh, once you select your Google Cloud project or create one, you should be able to create one.

39. 7:27

    And once it is created, um, you should have it available as a pop-up. If not, you can scroll down a bit. There are your API keys. Um, and then the second step would be to go into Colab and to go on the left side, uh, in the navigation.

40. 7:42

    So I can also ... Let me quickly change that to light mode as well.

41. 7:46

    Um, but anyways, left side there's a key, which is called [REDACTED:password]. And then you enter a name, which is Gemini API key, and then the value of your API key.

42. 7:58

    All of what I walked through is also part of the zero zero, um, setup and authentication notebook. So if it was too fast, you can, like, look up. There should be a screenshot of it where I clicked on where to add it.

43. 8:09

    If you are running locally, um, you need to expose the, um, API key as environment variable with the same name. Should be also part of our, um, notebook. So in the first cell, so basically what we try here is we check whether we are in Google Colab.

44. 8:25

    If we are in Colab ... Yep. Can we please go on to the next slide so we can follow along? Yeah. We ... No, don't worry. Like, we go through it one and then you have enough time to, like, five to 10 minutes to set it up yourself.

45. 8:36

    Okay? Quickly that we- Can we increase the font? Yep. Is there any PowerPoint presentation? No. No PowerPoint. Code only. Okay. Um, so we, we go through the API key setup in a minute.

46. 8:49

    You have plenty of time to do it yourself, and if you have questions, I'm very happy to come to your place and help you get it created. Just to complete the, the setup, so once you have created your API key, made it available to Colab or made it available in your environment, best is to open the first,

47. 9:04

    uh, notebook. It has a super small code snippet in it which uses your API key and generates, um, uh, uses Gemini 2.0 Flash to generate a first string. So our goal next five to 10 minutes is trying to get this working.

48. 9:19

    Okay? And again, going back, so we have those QQR codes. One is for AI Studio. It's the left one. The other one is for the GitHub repository. You can also go to AI Studio, uh, ai.dev to enter AI Studio or go through Google Search.

49. 9:37

    And you can find the GitHub repository on my GitHub account. Um, it's like gemini.2.5.aiengineering.workshop. And, um, I will in the meantime change the appearance.

50. 9:51

    Can you go back to the previous slide? Yep, sorry. Sorry.

51. 9:55

    Where, um, where is the Google Colab link?

52. 9:59

    Um, so they are in the r- GitHub repository if you go to the Notebooks section. In each of the notebooks at the top there's a button which opens Colab directly with the notebook.

53. 10:12

    Okay. Thank you. Okay. Okay, quick check. Are we, are we ready?

54. 10:19

    Yeah. Any notes? No. I'm just looking for the- Okay, cool.

55. 10:26

    Sorry to interrupt. So the first section will be all about the, the defaults basically. LLM started with being text only. We generated text, so what would ... The first section basically covers, um, all of how can I generate text?

56. 10:44

    How can I, uh, take, uh, generate text and have, like, a streaming response? How can I account my tokens? It's, like, always important, right? To understand how many tokens did I use?

57. 10:53

    How much will it cost? And there are, like, a few exercises for you to try out different models, try out different prompts. It will also go a bit into detail on how the SDK works in terms of, like, which inputs you can provide.

58. 11:07

    So in the Google AI SDK we have, um, this, uh, concept of a client, and client has the, the models abstraction. The models abstraction has the method, uh, generate content or, uh, generate stream content.

59. 11:20

    And I can also make it maybe a bit bigger. And, uh, each of the ... Or it has parameter for the model and the model ID is basically the Gemini model we want to use, which is defined at the top.

60. 11:31

    Uh, so all of those cells use the, the same concept. So all of the workshop section have the same. Um, if you think, okay, 2.5 Flash is not the right model for you, you can change it to a different model ID.

61. 11:44

    If you have, like, an paid account and want to use the Pro version you can also change it. Um, and, um, content is basically our way to provide, uh, data or conversation to chats and messages to, um, Gemini.

62. 11:59

    So the first, um, test basically is we ask it to generate three names for a coffee shop that emphasizes sustainability. And we use the client models generate content. We have our model ID, our prompt, and then we get our response from, from Gemini.

63. 12:17

    And, um, if you have already set up everything, you can try prompting a few things, um, ask it to explain some terms or maybe, like, just change the model ID, and then we continue with, um, counting tokens.

64. 12:32

    So there are exercises in there, uh, which don't have any code snippets. The solutions part of the workshop has the code. So if you are getting stuck or if you want to look it up, uh, what I edit, uh, feel free to take a look there.

65. 12:47

    But, um, definitely try it first yourself. If you want to get familiar with the SDK, there are plenty of other snippets or exercises basically just to, uh, make sure that you understand the concepts and, um, can practice it.

66. 13:00

    And there are other cells which are partially done. So for example, the one we have here, which has some code comments and also some to-do calls. Here, the idea is really to, to h- force you not to learn new APIs.

67. 13:13

    So next to the, um, generate content method, there's also a count token, um, um, API, which we can use to count our tokens. So similar to our generate method, we provide our model ID and then our prompt here.

68. 13:28

    And basically what the, uh, API does is now it counts only the tokens for our prompt since we, we haven't generated something. So we can run it, um, and we get an input tokens of 11.

69. 13:41

    So the Gemini tokenizers are basically converted those two, four, six, eight, nine words plus a, a full stop to 11 tokens, uh, which is then, um, an estimate of roughly $0.00002.

70. 13:59

    Um, the count tokens API doesn't expose the pricing, so basically what I did here is, like, looked up the 2.5 Flash pricing and calculated it. And similar to only counting the input tokens, we oftentimes also want to count the output tokens to understand, okay,

71. 14:20

    um, how much does it cost? So in the next, uh, example, we basically generate content, and each response has a very nice method which is called... Or like an abstraction which is called a text, which allows us to exily, uh, easily access the generation, but also has a usage meta data object.

72. 14:39

    And a usage meta data object includes all of our consumed tokens and generated tokens. So we have input tokens, uh, we have thought tokens. So Gemini, uh, 2.5 is a thinking model.

73. 14:51

    So before generating your response, it first generates thinking tokens, basically abos- an abstraction, where it uses more compute to have, like, more room to generate a good answer for you.

74. 15:02

    And then also the candidate tokens, uh, which are the response tokens at the end. And that's how we can calculate the total cost of a request, where we use the input, uh, token price and then our candidates tokens and the thought tokens.

75. 15:15

    And, um, for this case, it would be, um, less than, like, 0.2 cents. Yeah.

76. 15:22

    Is 2.5 Flash also a thinking model?

77. 15:25

    Yeah.

78. 15:26

    Okay.

79. 15:26

    Yep.

80. 15:27

    I just see different multipliers.

81. 15:30

    Yeah.

82. 15:30

    Is there, is there a different price for thoughts token in your estimate versus-

83. 15:34

    No. So, um, input and output tokens are calculated different. Um, as-

84. 15:39

    Input and output.

85. 15:40

    Yes. So we have prompt tokens. That's basically the input tokens of your prompt, and then we have the candidate tokens, which is the, your response, and the thought tokens.

86. 15:48

    And those basically have the same pricing, and the output tokens are much more expensive than the input tokens because that's where the computation mostly happens, and the input is just one encoding.

87. 15:59

    So that's why you always see, like, the big difference in output, uh, input pricing versus output pricing.

88. 16:04

    Is that an order of magnitude more expensive?

89. 16:08

    More.

90. 16:08

    More than order.

91. 16:08

    Yeah. So for Gemini 2.0 Flash, which is our, our most cost-effective and cheapest model, the input price for 1 million is 10 cents, and the output price is 40 cents.

92. 16:20

    Some of it, uh, 639 thought tokens. Like, is that data specific or is it a function of, uh, the prompt?

93. 16:31

    So you mean, like, why we got 639 thought token? So that's sadly not, like, directly visible to us. So we can, like, look up thought summaries. But basically, the model generates, first of all, like, a lot of, like, reasoning.

94. 16:46

    Of course, in our case, like, we, we ask it to generate a haiku. Might be not, like, the most, uh, [coughs] sorry, most difficult, um, prompt. Um, you can control the thought tokens with something called thinking budget, where you can limit how many tokens the model has to think or to reason.

95. 17:02

    So you have a, some sort of a cost control, but it's basically done dynamically based on your prompt. Okay. Yeah, there's also a question.

96. 17:10

    Yeah. I was just curious about the price for the output tokens.

97. 17:12

    Yeah.

98. 17:13

    I, I was just looking at the documentation that says 350 for thinking.

99. 17:17

    Yep.

100. 17:17

     60 cents for thought tokens.

101. 17:20

     Yep.

102. 17:21

     Why is... It, it is, it seems different from what you've got here.

103. 17:23

     Yes. So, um, I can... Let me open the docs and make, make it easier. Um, so, um, Gemini 2.5 Flash is a hybrid model, so you can use it with thinking and without thinking.

104. 17:36

     And without thinking, basically the computation is much more cost-effective. As, um, you might know, like, the transformers is all based on intention, which is, isn't like,

105. 17:48

     uh, N by NX, like, kind of calculation, so it gets bigger and bigger, which means it gets more and more and more compute intense. And, um, without thinking, it's, for us, much easier or inefficient to run.

106. 17:59

     So if you set, um, thinking to, um, zero or, like, thinking budget to zero, uh, you are, you will have zero thought tokens. But you have, like, your candidate tokens, and those candidate tokens will then be charged with 0.60 cents.

107. 18:15

     But once you, once you use thinking, meaning a th- a thinking budget greater zero, you will, um, pay the price for the thinking tokens and for the output tokens, and that's where's the, the $3.5.

108. 18:29

     Is the thinking budget shared between sites? Is the thinking budget for, for, uh, for the whole thing?

109. 18:36

     Yes. Le- I will open the documentation, so we'll see it in, in one second.

110. 18:41

     And how do you turn thinking on or off? Do you have to control it with the budget or is there a flag? I saw you have some constant control, no constant-

111. 18:48

     Yeah. So, um, in any case, if you have, like, any questions, the Gemini docs are a great way to, like, find the answers or, like, just ask Google or Gemini directly.

112. 18:57

     And on the model capabilities, we have the thinking section, and there are, like, the thinking budgets. And if you want to disable thinking, you basically set the thinking budget to zero.

113. 19:06

     Uh, you can control it. It could be, uh, integer between, like, zero and 24,000. And then setting thinking budget to zero disables thinking. So that's your way to disable thinking.

114. 19:18

     Is the thinking budget like a number of, of reasoning traces to do? So if I say four, how does it know?

115. 19:23

     No, it's tokens. So-

116. 19:25

     Oh, it's tokens

117. 19:25

     ... yeah. So we, we have-

118. 19:27

     Okay.

119. 19:27

     Yeah. So we have seen in our example here, we had 600, a bit more than 604 tokens. So if we would set our thinking budget to 512, it would be a maximum of 512 thinking token.

120. 19:38

     Just cut it off. But how, how does it know how many reasoning traces to do or is that not controlled?

121. 19:45

     Uh, mm.

122. 19:45

     Doesn't it, does it spin out other reasoning traces or does it find some-

123. 19:50

     No.

124. 19:51

     No.

125. 19:52

     Okay.

126. 19:53

     So that's just the intermediate number of tokens for the chain of thought then?

127. 19:56

     I guess.

128. 19:58

     Oh.

129. 19:58

     Okay. Continuing w- uh, with our, um, notebook. So, and please continue yourself. Like, I'm ... Like, you can do it with your own tempo. Even, like, do it faster, slower, just to make sure that we are all on, like, the same page.

130. 20:12

     And I, I guess, like, the most interesting part about, like, streaming and, like, LLMs in general, as we all have seen it with ChatGPT, is that waiting for the whole response is a very bad user experience, right?

131. 20:23

     Like, who wants to wait, like, 60 seconds, two minutes for a response? So that's why, um, everyone now kind of uses streaming. And with the Gemini and the Gemini SDK, it's, like, super easy.

132. 20:34

     So instead of having just generate content, we have generate content stream. Same input parameters, except that we now get, um, an iterator back from our, um, call, which we can loop over, and we can, like, print our, uh, chunk or, like, stream it back to our user using, uh, an HTTP service.

133. 20:58

     Okay. Cool. And then similar to other models, Gemini is a, a chat model, right? And, um, maybe you are familiar with the OpenAI SDK where you have the concept of messages, where you have, like, a different, like, inputs per user turn, assistant turn, user turn, and it makes it very, I would say, still

134. 21:23

     complex to manage yourself because you need to keep track of it. To make it easier, we, um, added something which is called a Chats API, um, which basically does all of the state management, uh, on the client, but as part of the SDK.

135. 21:37

     So you can create a chat, uh, with your, uh, model, and then you can basically send messages into, uh, the chat session. And the user, um, like in this case, let's, like, we are planning for a trip.

136. 21:50

     Um, we send a message, we get back the response. But also our chat session includes the user prompt and the assistant message. So instead of needing to create this object of user turns and model turns, we can directly, uh, continue with, like, sending our next message, um, asking, um, for some,

137. 22:15

     um, good food, uh, recommendations. And since we are in, like, a conversational setting, the model knows that we are, um ... Or, like, it mentioned for us to go to different European cities.

138. 22:29

     And based on, like, the next request, it uses, like, the whole, like, conversation as history, um, to get our response. Uh, can, like, also quickly

139. 22:39

     print the response here. And it gets some different examples. And then if you, of course, if you need to store it to a database on general, um, you have ...

140. 22:57

     We have a nice get, um, history method available, which allows you to basically retrieve the complete current state, and you can, like, store it or update it or whatever you wanna do.

141. 23:08

     Yeah?

142. 23:14

     Is the chat session history stored on the client or on the-

143. 23:14

     That's only a client abstraction. So the back end receives the same request if you would send it, like, as a single request, uh, with an array. It's only a client abstraction to make it easier for developers, people to quickly build.

144. 23:28

     So if I turn on my laptop, will the chat session be loaded?

145. 23:33

     Yep. And then, um, similar to OpenAI or to other models, you can, like, give the model some kind of system instruction to have it behave differently, respond it in a different language, make sure it respects, like, policies or guideline you provide.

146. 23:49

     Um, this can be done through a generations, uh, config. So we have another argument now in our model call. Um, next to the model ID and the content, we have a config that we can provide our systems instructions.

147. 24:02

     And similar to the systems instruction, we can provide, um, other, um, generation configurations. So temperatures can be used to make the generation more creative or more deterministic. So if you, for example, build a retrieval augmented generation where you want the model really mostly trying to use what you provide as a context, there you would normally set the

148. 24:23

     temperature to a very low, um, value. If you work on some content writing, marketing, you would set the temperature to a very high value. We can control the max output tokens to make sure that we are not exceeding some budget, some length.

149. 24:35

     And top P and top K are also ways to make our generation more diverse.

150. 24:44

     Yeah. Um, so it-- Let me open. So here we have the, similar in the config, we have the thinking config, and in the thinking config you can basically set the thinking budget or also include defaults, uh, in your request.

151. 24:59

     Okay. And then, um, I think what's very more unique about, uh, what we can do with Gemini is that we have direct, direct support for files. So in this case, I download, uh, Adventures of Tom Sawyer, uh, a book completely, store it in a file, and I use the files, uh, API to upload the file to a

152. 25:24

     Google Cloud, uh, storage bucket, um, that is, um, free for you. So like if you, uh, don't want to use your own corporate bucket or whatever bucket with each AI Studio account, basically there's your personal, uh, bucket which stores the file for, I think, one day.

153. 25:41

     Uh, but you can control the, the time to live. And, um, instead of, um, needing to provide the whole file with your request, which can be very, um, intensive, you can upload it and then instead just provide the reference to the file.

154. 25:57

     And what, um, the Gemini API does behind the scenes, basically it downloads the file on the back end and makes it available inside your prompt. And similar here, we can, um...

155. 26:09

     We uploaded our book, we parsed it into our contents, uh, list here. So we don't no-- no longer have like a single prompt. We have now an array with our file, and we ask it to summarize the book.

156. 26:20

     And it was also done while I'm talking. And then we can also see, okay, the token usage now we had for, uh, was one hundred thousand tokens. So much bigger than what we tested before.

157. 26:32

     And using the file API makes it very easy, uh, also to work with, with PDFs, which we'll, um, see in like the next chapter. And then, um, as an exercise for you is, um, to combine a bit all of the things.

158. 26:45

     So how can I use the book to use the chat JSON to chat with our, uh, model to help me better understand it? Yeah.

159. 26:52

     So the file, uh, it will be-- It's not a private system, it's just in the context window?

160. 26:58

     Yeah. [whispering] So what you do is basically you upload the file from your client to the cloud into a bucket. And when you send a request, part of your... I mean, I can like show it.

161. 27:10

     Um, part of your request, uh, would be only the reference to where the file is stored. And what the Gemini API does behind the scenes is it loads the file into where the request runs and then puts it into the context.

162. 27:24

     So you can use now this file pointer for all requests, and you don't need to send it every time.

163. 27:33

     And here we also have like the URI. So that's basically where,

164. 27:38

     um, our file, um, is stored or can be accessed as well. Yeah.

165. 27:43

     Um, how do you deal with that-- I-is that o- That file is own-- Is that, um, hidden through obscurity? So is it only available to the chat agent reference to itself or-

166. 27:56

     It's only available to your user. So when you send a request, you send an API key, and this API key is basically used to get the file so nobody else can access the file.

167. 28:07

     Oh, so it's hidden by the API.

168. 28:08

     Yeah.

169. 28:10

     Nice.

170. 28:10

     Okay, cool.

171. 28:12

     Question.

172. 28:12

     Yeah.

173. 28:14

     What happens the file is too big for the, the, bigger than the context window? How Gemini handles that, you know, those basically problems?

174. 28:24

     So if you would use a PDF which has like more than a million tokens, basically what would happen? You would receive an error most likely, uh, with like saying it, uh, the file is like too big in terms of like, uh, context.

175. 28:37

     What you can do is basically you can use the, the file, uh, the file to count the tokens. So we have the file now here, and we can client, uh, models count tokens.

176. 28:56

     But because we need the whole file, so what would we do?

177. 28:57

     You would need to chunk it then. So if it, the file doesn't fit, it doesn't fit, and you would need to like think about, okay, can I chunk it?

178. 29:04

     Can I summarize it? Can I maybe do-- use other techniques to first extract the important information? And once I have, um, context, which is, um, smaller than the maximum context of my model, you can provide it again.

179. 29:19

     Yeah.

180. 29:20

     It, it doesn't have like those context-

181. 29:22

     No

182. 29:22

     ... because it chunk it?

183. 29:25

     No.

184. 29:25

     Are there any costs associated with file transfer?

185. 29:28

     With the files, f-files upload?

186. 29:30

     Uh, if there are any costs associated-

187. 29:31

     No

188. 29:32

     ... with file transfer?

189. 29:33

     No. So it's like... But it will be deleted. So don't expect the file you upload now to be there an hour or a day. Um, but you can like use the same concept with, uh, Vertex in your own bucket where you have like more control over it.

190. 29:47

     Where you can say, "Okay, maybe I'd want to upload it using a different API call or already have it available." That also works.

191. 29:56

     Thank you.

192. 29:56

     Oh, do you-

193. 29:56

     So there's an entry point to saying, "Use this in the Google storage bucket"?

194. 30:00

     With Vertex AI, yes.

195. 30:01

     With Vertex AI.

196. 30:02

     Yes. It's roughly the same, but you need to set up the client differently.

197. 30:06

     It's a different entry point.

198. 30:08

     Yeah.

199. 30:08

     I see.

200. 30:08

     Okay. Yeah.

201. 30:09

     So for the PDF-

202. 30:11

     Yeah

203. 30:11

     ... does it extract, uh, just the text, or would it also

204. 30:17

     Yeah, good question. So I mean, we can maybe directly jump into like the, the, the, the PDF section. So, um, continue with section one or jump to section two or three directly.

205. 30:27

     What I will do is like in the section two, which is all about multimodality, meaning we will cover, uh, visual understanding, audio understanding, videos, and document processing, and that's where I will like jump to.

206. 30:40

     Um, and part of the, um- Okay, being connected. Part of the, uh, working with PDFs basically. So similar to what we have seen a minute ago, we have a PDF.

207. 30:55

     In this case, it's basically an invoice from a supermarket. Um, I upload it, and I ask the model, like, what's the, the total amount? Uh, we can run it, and what happens behind the scenes I can show...

208. 31:07

     Oh. It's not here. Wait one second. Okay, we don't have the file here. I'll upload it quickly. But what happens, um, behind the scenes is we run OCR on your, [clears throat] on your PDF and provide the PDF as image.

209. 31:26

     Um, so you don't need to do it manually, so there's no like, "Hey, it's a PDF. Let's convert it to an image and then run OCR, and then I provide the image and the OCR."

210. 31:36

     That's not needed. Um, we are doing it for you. Okay.

211. 31:44

     Can we take questions?

212. 31:46

     Yep.

213. 31:46

     There's some questions.

214. 31:47

     Yeah, yeah, yeah, yep.

215. 31:52

     Hey. I'm just curious what the, what the point is of passing in the OCR text if you're passing the PDF as an image already.

216. 32:01

     Yeah. So OCR's, or, like, the image understanding is not perfect yet, right? If we reach a point where the model understands it, like, in the same way as without the text, then I guess there's no point.

217. 32:15

     But, like, based on, like, what we have seen and also what the industry does is you receive better results when you provide the OCR plus the image. Um, yeah, that's- that's basically it.

218. 32:29

     Yep.

219. 32:30

     Maybe you kinda missed that. So the PDF itself,

220. 32:33

     do you actually look at it as an image? So if there's tables and diagrams-

221. 32:37

     Yeah

222. 32:38

     ... and you extract it in image space, or you just take... There's also sometimes text hiding in the PDF for cut and paste.

223. 32:45

     No, I, I think... I, I mean, I don't know exactly, but I think it's just basic OCR. Nothing special. No magic. And then a screenshot of the PDF.

224. 32:55

     Okay. So it is an image space.

225. 32:56

     Yeah, so it's both. Okay. We can try again. We have now the PDF available. And in case, um,

226. 33:03

     the workshop has multiple sections with files which are being part of the repository. So if you run into, like, a similar issue, especially for the image understanding part or the audio understanding part, uh, and you use Colab, you might need to download the files manually and then upload it.

227. 33:20

     But in our case, so we have now our invoice. I think we can, like, uh, we can quickly show it.

228. 33:32

     So I was shopping. In Germany we have a co- a supermarket called REWE, and I basically bought some butter, um, some bread, like, some sweet potatoes, and, uh, we prompted it and asked it, "Okay, what's the total amount?"

229. 33:46

     We can see here the total amount is, like, 20.20. Now let's see if we got it correctly, and we got it correctly. And it also correctly extracted it in German even if I prompted it in English, which I think is, like, pretty cool.

230. 33:59

     Okay. Let's start with the image understanding part.

231. 34:04

     Can I ask just one quick question?

232. 34:05

     Yeah, please.

233. 34:06

     Sorry. About the thinking budget.

234. 34:08

     Yeah.

235. 34:09

     Is, so it's not just omitting the thinking process that you see, it's actually changing the chain thought?

236. 34:15

     I'm not exactly sure what happens. I only know that by defining the thinking budget, you can limit how many, how many tokens will be used or generated as a maximum, and very similar to what OpenAI has with, like, low, medium, high effort, we have a bit more granular control.

237. 34:33

     So we could technically do the same, say, like, low would be, like, 1,000 token thinking, medium would be maybe, like, 12,000 token thinking, and, uh, high would be 24,000 token thinking.

238. 34:45

     Uh, and it would then use, like, at a maximum those tokens to, before generating your response. But

239. 34:51

     what exactly happens I can't tell you.

240. 34:54

     'Cause you said that you could set it to zero. Like, the model is trained to go through that chain thought process, right? So that might actually skew the results or ...

241. 35:03

     Yeah. So without thinking, what we have seen, especially on more, like, math type of question where the model benefits from, like, the reasoning, uh, the performance is a bit worse.

242. 35:13

     But for, like, general everyday use, especially image understanding or, like, OCR, you can, like, easily run it without it. The truth it would be you need to try, and I think the, the real benefit here is that you have those granular controls so you can run evaluation of thinking budget zero, 1,000, 2,000, 4,000, and see how it

243. 35:34

     impacts your evaluation. And then you can, like, calculate for yourself, okay, how much am I able, or, like, what's my, my maximum cost of it, and, like, what's the, the accuracy I need to reach?

244. 35:47

     Okay. More questions or should we continue? There's a question. Yep.

245. 35:53

     Do you support any other file types or-

246. 35:56

     No. It's like, uh, there's a documentation for it, but it's like, uh, JSON, PDFs, all different image types, all different video types, uh, all different audio types. So all of the multimodal, um, features we support.

247. 36:12

     Um, if it gets a bit more specific with, like, .jsx file and .vue files for, like, web development, we are working on it, but it might ... I mean, you will see an error, and the easiest way is to just replace it with a .txt.

248. 36:25

     Like, if it was a Microsoft Excel file, you just-

249. 36:27

     No. I would say, like, those, um ... Like, you would need to, like, use, like, Markdown or, like, another library to convert it, and then, or, like, copy, paste the input and

250. 36:38

     ... Yep.

251. 36:39

     I actually have two questions. First question, is it because including the OCR along with the images-

252. 36:55

     So I'm not exactly sure what our researchers did. The only thing I know is that we get better performance when you provide the image plus the OCR.

253. 37:04

     So I guess there's a benefit of having both.

254. 37:07

     Awesome. And the second question is, the script you just showed us-

255. 37:11

     Yeah

256. 37:11

     ... is it essentially the same thing as we saw on the UI interface of Gemini, like going after the PDF files?

257. 37:21

     Yes, kinda. So the UI or the AI Studio, of course, doesn't use Python. Um, but it calls the same APIs behind the scenes. So the API behind the file upload one is the same API we call from AI Studio.

258. 37:35

     We call the same exact model. Both have the ex- same, uh, exact parameters, so it should be easily what you test and experiment in AI Studio, can you convert into code and run it locally.

259. 37:47

     There's also this, um, Get Code, um, button. So if you're on AI Studio and, I mean, we can quickly try uploading our invoice again.

260. 38:00

     And acknowledge. And use our prompt. So tier.

261. 38:15

     And now we run basically the same request, and we can now... We also have this, um... It's a bit hidden. It's like this code, um, button at the top, which, uh, where you can get the exact same code basically.

262. 38:30

     Takes a few seconds as the WiFi is super bad. But here you get, like, the exact same Python code where you create your client. We have our model. In this case, as we uploaded it manually, we provide the document not as a file URI.

263. 38:42

     We provide it directly as base64. We have our prompt. Uh, we have the model request already since we generated it. And then you can continue.

264. 38:51

     Okay, cool.

265. 38:53

     Uh, question.

266. 38:54

     Yeah.

267. 38:55

     Uh, can we still use the count tokens, uh, function for when we use a file upload?

268. 39:00

     Yeah. So I mean, we can quickly so try it. So we are here. We have our PDF. I think, I guess it's very interesting to know, like, how many, um, tokens will we use.

269. 39:10

     So, um, let's quickly count tokens. So if we go here,

270. 39:25

     we have our count tokens, and now we use the same contents. We have the same model ID, we have our prompt, and we have our PDF.

271. 39:33

     And let's print our token count. And alternatively, what we also have done, so if you use the,

272. 39:44

     uh, response, so you run a request already, you should have access to the resp- or usage metadata. Yes. And so we have our count tokens. So our PDF here is converted into, like, um, uh, roughly 500, uh, tokens.

273. 39:58

     Um, and the, the prompt we have is, like, around 20. And if we compare it to, like, the request we run, we see, okay, we have the same exact amount of prompt tokens.

274. 40:07

     And we have prompt details. We have our output tokens.

275. 40:13

     Four tokens of 42 and candidate tokens of

276. 40:18

     78. Okay. More questions or... Yeah. Okay.

277. 40:24

     Is the 560 here based on the OCR content of the files?

278. 40:28

     I think we don't charge for the OCR. I think one image is roughly 500 image, uh, like 500 tokens.

279. 40:35

     Gotcha. Okay.

280. 40:36

     Um-

281. 40:37

     This is really very much depending on the-

282. 40:40

     Not for you, no.

283. 40:41

     Thank you.

284. 40:44

     Okay. Yeah.

285. 40:46

     Um, can you share more about the parsing? Um, is it just OCR text or does it understand hierarchies and images inside the PDF as well?

286. 40:55

     So the PDF is converted into an image, and we provide the image. And if the image or, like, the PDF has tables, visuals, mind maps, uh, the model is trained on similar data, so it'll definitely understand parts of it.

287. 41:11

     I mean, we can, we can try it. I mean, maybe we can, like, search for some mind map and ask it something.

288. 41:21

     Um, you, maybe you can start thinking about a prompt

289. 41:25

     while I'm searching for a mind map. If WiFi allows us.

290. 41:33

     Okay. Okay, we have our mind map. I guess it's

291. 41:44

     just a mind map on how to do mind maps. Um, any, any idea what you would like to know?

292. 41:50

     I mean, what's the most central concept or idea?

293. 41:53

     Sorry?

294. 41:54

     What's the most central concept in the mind map? Yeah.

295. 42:02

     Like this?

296. 42:03

     Yeah.

297. 42:03

     Okay. I mean, I'm opening it, but I think on the small scale it looks correct with how to mind map.

298. 42:25

     Answers. Perfect. [laughs]

299. 42:28

     I think the, the, the issue that really tends to happen, right, like, for example, there's a table. Will the fidelity of the table, in fact, because we're gonna capture it as an image.

300. 42:37

     Will the fidelity of the image able to represent that table well? Because for example, that's actually a complicated table. So those are hard challenges, right? If you can basically keep it as a PDF, and if you're saying it's only Google OCR, and only that normal OCR, and then only through the image.

301. 42:51

     Like, if anything gets messed up, for example, from before, even if the mind mapping algorithm-

302. 43:00

     Yeah. So where we have seen the most success is to-- when you, like, separate a bit. So there are already, like, ex- very good existing methods which allow you to extract tables or other visuals from documents, and when you then, like, work with those images and tables directly and not, like, you...

303. 43:18

     The way how we work changes from, like, previously we provided a table, run OCR, then ask our question. Now we directly ask the question based on the table as the models get so good in, like, the multimodal understanding that it can combine, like, the different aspects.

304. 43:33

     I mean, we can try it maybe as a good example as well. Um, maybe we find some nice, I don't know,

305. 43:41

     invoice image or something. And then we can, like, ask it maybe to add something or to combine it, which would be very hard, I guess, for a, a normal model.

306. 43:50

     But in general, like, especially Gemini is so good with, like, the multimodal understanding. It also... Videos is like, uh, that's like my most new favorite thing, is like take a YouTube video which is, um, below one hour, put, put it into AI Studio, and, like, have it summarize it.

307. 44:07

     Or if you have, like, any specific question, like, it, it's so much faster than, like, sitting there even, like, watching it in, like, two speeds. You get a response in, like, 80 seconds or something, and you can, like, even, like, ask it to extract specific timestamps on when somebo- something was said or to help you, like, section

308. 44:24

     it very easily.

309. 44:28

     Is the nature of, uh, picture being sometimes [inaudible]

310. 44:32

     Sorry, can you speak a little bit louder?

311. 44:34

     Is the nature of prediction, I guess, not being just for the files provided by the user?

312. 44:39

     I don't think so. So AI Studio is, like, very developer centric, and we don't want to do too much black magic. There are safety filters and control which you can configure in the SDK or in the AI Studio.

313. 44:53

     And in AI Studio is under Advanced Settings. I have safety settings. I have all of them off. But if you, like, want to filter on very explicit content or hateful content, we run some classifications basically before and after to make sure that you are not creating, uh, for your users harmful content.

314. 45:15

     Okay. I have our invoice image. Okay. Okay.

315. 45:35

     What should we ask? How much it would cost if we subtract the pedal maybe?

316. 45:44

     Excuse me. I have a quick question.

317. 45:46

     Yeah.

318. 45:48

     I want to look into this. Uh, so does the PDF need to be really well formatted? Because the kind of problem that we face, generally the PDFs are auto-generated, right?

319. 45:57

     And that's why it's not always, uh, very well formatted. Like there is headers, footers, whatever on top of each other. Images, you know, on top of each other. Even then, I mean, ha- have you, uh, guys, like, seen the performance or done any calibration on those type of PDFs are uploaded or anyone [inaudible]

320. 46:19

     what type of PDFs do you, uh, use? How do you [inaudible] those?

321. 46:24

     So I sadly don't have an answer for this. I think we-- you can always think a bit about, like, would we as humans have struggled to understand those PDFs if they are, like, switch, switched up?

322. 46:35

     If yes, then most likely the model will do as well. I think one very nice part about Gemini AI Studio and the Gemini API is i- you can get started very quickly.

323. 46:46

     Like, all of us kind of set up within, like, twenty to thirty minutes a free account, uh, with access to Gemini 2.5 Flash via API, 2.5 Pro in the UI.

324. 46:56

     So the best thing always is, like, to test and to explore and evaluate. And even if you n- need to run like 1,000 PDFs, it's not very cost or, like, expensive anymore.

325. 47:06

     So it's really best thing is to run your own evals to get some more than, like, I try five EV, uh, PDFs in the UI, really look into it.

326. 47:16

     And if you have, like, any questions or problems, like, best way is to reach out to us. We have teams helping and building with customers, and then we can, like, iterate it on it together.

327. 47:26

     Uh, going back to your PDF, any question? Any prompt idea?

328. 47:31

     Yeah, I have a question.

329. 47:32

     Yeah.

330. 47:34

     I know you explained about security, but there are mandated, uh, security by law, which are, uh, LLM firewall or, uh, benchmarking automations or LLM guardrails or s- SCM posture management, uh, mandated by CISA.

331. 47:55

     So when I implement those rules, um, for, uh, either, uh, a defense or, uh, for financial applications,

332. 48:08

     how do you just a small set of... I know that Google is pretty strong in setting up those, um, security measures.

333. 48:18

     From a different perspective too, how do you integrate this Colab and the other security measures?

334. 48:29

     So I guess for those type of environments, uh, where you have a lot of, like, compliance regulations, best is to work with Google Cloud. So everything we do in AI Studio is also somewhat similar available in Google Cloud in Vertex AI, and Vertex AI provides more features for those kind of use cases.

335. 48:51

     They... I'm not exactly sure, but, um, they definitely have more, like, information they can provide on how to handle all of those things and those guardrails with Gemini.

336. 49:03

     So that-

337. 49:04

     So, so the go-- Yeah. Yeah. So does team need to

338. 49:10

     Global Google, uh, Google GCP environment. When I go global on GCP environment, then there are certain,

339. 49:22

     uh, performance and, uh, cost being more here, such as, um, um... So there are-- there's a guardrail for bot-based, and there's a guardrail for organization-based, and there's a guardrail based on the industry, industry-based.

340. 49:44

     So we have to fit those platforms into, um, sm- uh, from gran- most granular level to all the way to industry-wide level. So, um, that's becoming... When I do it globally at GCP level, then it becomes, um, more expensive and as well as, uh, there can be a performance issue.

341. 50:08

     Yeah. So I know that there are regional endpoints for Gemini and Vertex AI as well, and also at Cloud Next, um, they announced a new Gemini on-device kind of thing, where big companies can buy basically a huge box where Gemini is pre-installed, and it gets delivered to your environment.

342. 50:30

     I'm not exactly sure about the details. Easiest is like to, to do a quick Google search and, and look for it. But those is exactly where, where Vertex provides you more features and support than AI Studio.

343. 50:42

     Yeah. Thanks. One of those last tech, uh, 2025 conference, uh, someone from Google, uh, also represented and, uh, that, that's a good platform, but I didn't know how you can integrate from here to there.

344. 50:58

     Yeah.

345. 50:59

     This is the most granular level.

346. 51:01

     Yeah.

347. 51:01

     How do you integrate from the most granular level to the global level? That, that's the part that I was... But anyway, thanks for it.

348. 51:08

     Okay. Maybe back to your PDF question, uh, regarding, uh, what to ask. So instead of like asking what, uh, the total sum was, I asked it to, to sum up the unit prices, which worked very nicely.

349. 51:20

     Of course, it's a, a very well-formatted PDF in this case, but, uh, the image understanding is like very good. And the new way of how we should think about it is like, I should ask the question directly based on the image before doing too much of processing we, we have been doing in the past.

350. 51:38

     Cool. Okay. We are now at like half time almost. I guess it's time we move maybe a bit away from all of the multimodality part into more about, uh, I guess, agentic, um, parts, which are, I would say, definitely more interesting, at least to me, um, especially if you combine in with like the multimodality parts.

351. 51:58

     So, um, okay. Um, so part three is, um, all about structured output and function calling. Do you know what structured outputs and function calling is and like how it roughly works?

352. 52:17

     Any hand signals yes or...

353. 52:20

     Yes.

354. 52:20

     Everyone? Okay. Not bad. Um, okay. So the part three, um, goes-- continues with PDFs, um, as they are kind of very interesting and also, um, we want to do structured outputs.

355. 52:45

     So structured outputs is for us a way to create more structured data structures from text, so which we can use to work way more easily afterwards, right? And at the end, we prefer structured output much better or much more because we can integrate it into other APIs.

356. 53:02

     We can connect APIs. And Gemini supports, or the SDK supports Pydantic. So Pydantic is a very nice Python library, which lets you create those data structures. And also we can create na- nested data structures.

357. 53:14

     So here we have a recipe, uh, with a name, ingredients, which is a list of strings, and then we have a recipe list, which is basically a list of our recipes, and we can provide it in our configuration.

358. 53:26

     So similar to the generation arguments or our thinking budget, we have, um, a response type and a response schema we can provide. Here we ask it, okay, can it generate two popular cookie recipes for us?

359. 53:39

     And we basically force it to use our, um, structure. And I already like nicely, um, printed it here. But if we look at the, the raw response, um,

360. 53:54

     of our model, we get back a, a JSON, uh, with, um, all of the different input fields. And there's also a nice to parse method, um, which allows us to convert it back into our Pydantic, uh, schema, and then we can access all of the,

361. 54:14

     the data points. And as we had our invoice, uh, we can now maybe like, like complete the exercise with you. Um, what if we like-- So we asked about the total amount, right?

362. 54:29

     But when working with PDFs, normally we want to have structured data as a result, right? Text is not very helpful for us when we want to put it into a database or want to work with it.

363. 54:39

     We really need those data schemas. And what is very nice about Gemini is that we can basically combine both of it. So we use our, um, structured output method with our multimodal capabilities for files, and we can

364. 54:56

     provide our, um, file. Oops.

365. 55:33

     Mm. Okay. Oh.

366. 55:58

     We need our invoice there. So maybe it's a good example. So I didn't change the recipe, uh, from our, like, data structure, whi-which we want to create, and we ask it to extract the information from our PDF.

367. 56:11

     Our PDF is an invoice from a supermarket, so it doesn't have a recipe name, ingredients, so Gemini did not generate or hallucinated something, so we get back a, a empty recipe list.

368. 56:25

     So if we now change it to our invoice data, um,

369. 56:30

     we should hopefully then see the correct extracted information.

370. 56:44

     Yes. So we extracted the date-

371. 56:48

     Mm-hmm

372. 56:48

     ... all of the items we bought, and all of the different prices. And with that data now, it makes it much more easier for us to work with, right?

373. 56:57

     If I have some kind of automated system where I need to, like, take in invoices, any kind of, like, PDF document, I can now provide, uh, a structured

374. 57:09

     to what information I want to extract, and Gemini does basically all of the matching for us, which is, like, super nice. And function calling basically is the same idea, but instead of, um, having, like, a data structure, our output is a name and the argument.

375. 57:25

     So similar to what we have, um, uh, we, we create, uh, this case it's like a function declaration. It's a structure of, like, how a function signature is done.

376. 57:35

     So we have a weather function, which has a name, a description, and, like, the properties, uh, which we need to provide. Um, there's the same function, but just as Python code.

377. 57:45

     And with function calling, we provide the function declaration and our prompt, and then the model generates a stru-a structured output which has the function name it wants to call and the, um, input and environment or the input arguments for how to call it.

378. 58:00

     So in our case, we have a weather function. We only have one location. Um, we provide it, uh, similar to all of the other configurations in our configuration, uh, argument.

379. 58:11

     This time we have tools, and we want to know what the weather is in Tokyo. And obviously, what's the weather fits into the description of our function as it r-helps us retrieve the current weather.

380. 58:24

     So if we run it, the model, uh, instead of generating a nice response, wants to call the get weather method and with the location Tokyo. If I change the prompt to hello...

381. 58:39

     Oh. Makes sense. Uh, one second. So if I change to hello, we don't have a function call, right?

382. 58:55

     Like, the model correctly understands, A, that's just a greeting that responds and, like, how can I help you? So-- But we want to call a function. So

383. 59:05

     we have what's the weather is in Tokyo. And then the next step is for you as a developer, uh, to call the function, right? The models cannot call or invoke directly the function.

384. 59:15

     What you would normally then have in your, um, code or in your application's a way to identify, okay, which function is called. Could be a simple switch statement, um, to check, okay, what's the name?

385. 59:25

     And if you get a name, call the method with the provided argument. And then what you do is the output of your function is the next user input. Uh, so the model generates this name and arguments object, and we provide it as a user the output.

386. 59:41

     And in our case, it's the result. And if we look at the weather method we have, it's basically some dummy data about the temperature, the condition, and where it is.

387. 59:54

     And then the, the model generates a very nice response. So we have user input, model has a structured output, user provides the structured response, and then the model generates a very nice, user-friendly response.

388. 1:00:05

     So we call our function, the function returns to weather, and then the model generates a very nice response, which is the weather in Tokyo is sunny with a temperature of twenty-two degrees Celsius, and it feels like twenty-four degrees Celsius.

389. 1:00:17

     So you can think about it, okay, that's how I can integrate tools or make it or co-convert my LLM into an agent more or less or a way to call something.

390. 1:00:27

     Um, the get weather method can be anything. It can be a database call. It can be a real API call. It can be, I don't know, reading emails, sending emails, all of the things we currently see with all of the MCP hype basically going on.

391. 1:00:43

     And MCP servers have tools as well, and those tools of MCP servers basically expose the same, um, declarations. So an MCP servers has the tools defin-- uh, defined. Get weather, for example, if it is in weather MCP server, and it has an endpoint or a method which you can call, which is list tools.

392. 1:01:04

     And this list tools, uh, method would then return the schemas of our functions. Which look very similar to what we have created here. And then what you do on your LLM side or client side is you t-take those schemas from the MCP server, provide it into your LLM call, and then the LLM generates, depending on the context,

393. 1:01:25

     uh, the output as well, which is structured. And then instead of calling the method on a client, as we did with our get weather, you would use the MCP client again and then call the remote tool.

394. 1:01:36

     So very similar to what we have done here on the client side manually, but more abstracted away and more managed. And of course, the benefit here is that not every one of us needs to implement the get weather method.

395. 1:01:49

     It's way easier to use, like, the weather MCP service from, I don't know, some weather provider. Similar, um, you don't want to use maybe your own personal Google Drive MCP server.

396. 1:01:59

     Best would be if Google creates those. So that's the whole idea why MCP servers is kinda so cool. So, and, um-

397. 1:02:08

     I think

398. 1:02:08

     Yeah, yeah, yeah.

399. 1:02:09

     How many tools can we have for the LLM tool?

400. 1:02:11

     So currently, we are working on improving and extending. Um, but currently the suggestion is probably between five to ten. And, um, if you have more tools, you can use embedding models to basically filter.

401. 1:02:24

     You have to use a prompt, and what you would do is basically run some similarity matching between the descriptions and, like, what makes sense, what doesn't make sense. And then you only, um, put the top tools so to speak.

402. 1:02:37

     A follow-up question. I think so Anthropic, they're, uh, this year they produced a solution of, like, they trained their model and everyone in a loop for about two hours.

403. 1:02:44

     Like, are they similar metrics for even, uh, Gemini?

404. 1:02:49

     I'm not sure about the claim from Anthropic and, like, how you do it. The only thing I can share is that Gemini 2.5 Pro was the first model to complete Pokemon Blue.

405. 1:02:58

     Uh, which ran, I think, for, like, two hundred hours straight. Um, the only, like, the, the big challenge here is, like, so we have, like, a limited context, right?

406. 1:03:06

     Which is for Gemini, one million. So if you would continue for two hours, you would definitely run out of, like, those two, like, one million tokens. And Anthropic, I'm not sure what their context currently is, but I'm pretty sure two hours is not enough to run out.

407. 1:03:19

     So what you most likely do is you summarize, you compress the context and the conversation when what you provide to your model. So yes, I'm pretty sure Gemini can run for more than two hours, but it depends on, like, what you want to solve and how you are going to solve it.

408. 1:03:35

     Yep.

409. 1:03:37

     If you call a native tool, it seems like the background traces are all hidden. Is that correct? And is there any way to access them? Because you can imagine we use them for managing performance.

410. 1:03:48

     Yeah, that's correct. So native tools are the next section. We can go into it, um, in one second. And yes, the, currently the native tools are not, like, being returned in a way we had with, like, user assistant.

411. 1:04:00

     That's basically happening behind the scenes. And what the user gets is the final, uh, good assistant response. Um, that's great feedback, um, regarding, like, can we have them or not.

412. 1:04:11

     Like, very happy to take it to the team. I can definitely see why it would be helpful for u- for, like, people to directly build with it. Um, but for now it's not the case.

413. 1:04:21

     And speaking of native tools, so, um, Gemini can... It's basically trained to do native things. So function calling is very generic. We generate the de-declaration and can try to do everything.

414. 1:04:35

     Uh, but native tools are much easier to use, as you don't need to define or create a declaration. And they work basically on the back end side, so you don't need to execute anything.

415. 1:04:45

     And as native tools, we currently have Google Search available. Um, basically all of the native tools are in AI Studio here. So we have structured output. This is not a native tool which we used to get back the structures.

416. 1:04:56

     Code execution is a native tools, and basically means that we or Gemini runs code for us. So,

417. 1:05:06

     uh, function to sort, uh, top five cities based on population.

418. 1:05:24

     So what it can do is, like, run Python code for us. So if you prompt it to solve a task using Python, um, it should run for us the Python code.

419. 1:05:37

     So it generates the Python code, and it did not run it. That's a bad example.

420. 1:05:43

     You said create function and it didn't run.

421. 1:05:45

     Sorry?

422. 1:05:45

     Your prompt said create a function.

423. 1:05:47

     Oh. Did I? Ah. So okay, my bad, not Gemini's bad.

424. 1:05:54

     Use Python. Let's see what it does. Um,

425. 1:06:02

     okay, yeah, now it gets it. Thanks. Perfect. So we have some reasoning, and then it generates executable code. Executable code is also provided via the API, um, which runs for us Python.

426. 1:06:15

     So it writes the Python script, it executes the code, um, and it generates the matplotlib chart. And normally we... Okay, w- in the notebook, we have an example available on how you get the chart if we look into the code execution tool.

427. 1:06:31

     So similar story here. We run it, and it returns the markdown, and you can also... Like, it can re-generate and return, um, images. So next to code execution tool, there is the URL context tool, which basically allows you to provide a URL as part of your prompt, and we, like, extract the information from the website behind the

428. 1:06:51

     scenes and make it available into your, uh, context. So instead of going to a website, Command A, Command C, Command V, uh, we do it for you. So in this case, I ask it, okay, like, what is the other benefits of Python?

429. 1:07:06

     Um, from like the URL. You can provide up to 20 URLs in one request. Behind the scenes, it goes to the URL, extracts the information, provides it as part of your prompt.

430. 1:07:16

     And if we look into our prompt, yeah, here, uh, it returned our nice smart plot, plot, um, chart, uh, which is very cool. And then a final

431. 1:07:27

     tool is, of course, Google and Google Search. Kinda makes sense. Um, so let me find it. Yeah. Uh, you can allow or enable Google Search, which then, um, allows Gemini to use or do a Google search behind the scenes.

432. 1:07:45

     And what happens here is that, uh, it takes our prompt,

433. 1:07:48

     um, in this case, what are the latest developments in renewable energies? And first, it converts it into one or multiple Google Search queries. Then it executes those, those Google Search queries, provides it back to the model, and then the model generates your final response from your user input and from all of the search results.

434. 1:08:07

     Currently, we don't, as mentioned, don't export or expose those tool calls. But, um, I guess especially helpful or interesting for, um, Google Search is that we have, um,

435. 1:08:25

     a grounding metadata object. Yes. So we have grounding support, uh, which basically allows or points exactly to where it got or, like, which information ref- uh, refers to which, um, source, and also grounding meta

436. 1:08:50

     information, which websites there is. Um, which websites were crawled. And in our case, so we have the... Yes, there.

437. 1:09:00

     Does the cost go up with these tool usage, or it's just part of the regular token?

438. 1:09:04

     So, um, for both code execution and grounding and URL context, we increase the context size, right?

439. 1:09:13

     Yes.

440. 1:09:13

     Those contexts will be billed. Uh, tokens will be billed. Uh, for Google Search, um, there's a free tier from, I think, fifteen hundred Google searches which are free. And then afterwards, I think they have thirty-five dollars per one thousand searches.

441. 1:09:28

     Uh, code execution, the Python running is free. Um, URL context is currently in preview, so I'm not sure what the, if and what the pricing will be. Um, yeah.

442. 1:09:42

     And I think, like, what's, what's very cool and, um, is part of the exercise if you already have done it or are going to do it later is you can combine those.

443. 1:09:49

     So you can use the Google Search tool with the URL context tool or the code execu- execution tool with the Google Search tool to basically have it done more agentically.

444. 1:10:00

     Like, first search for, like, what's the latest React or Python version, then write a Python script and run it and return it, um, which makes it very nice to use.

445. 1:10:10

     Are you guys planning to also include the researcher tool in that at all?

446. 1:10:14

     Sorry?

447. 1:10:14

     The deep research tool that Google has, is that also going to be released then?

448. 1:10:18

     Yeah. We, we have heard, like, a few people would like to have a deeper s- research API. I think the more people ask for it, the more likely it will going to be.

449. 1:10:27

     Um, maybe something I can, like, plug and share is so I, um, as part of, like, our team, we, um, yesterday we open sourced an example for how you can build your, um,

450. 1:10:42

     your own deep research using Langcraft with Gemini. Very similar to, um, wait. Very similar to what basically the, the whole deep research agents do. We have a, a question.

451. 1:10:55

     We generate queries. We then run, like, multiple web searches, and then, like, reflect and see, okay, was the user question already asked? Do I need to do more research?

452. 1:11:04

     And then you have this kind of loop for, okay, do I need to search other tools? Uh, it's completely open source. It uses all of the things we have currently seen today, Gemini 2.5 Flash, Gemini 2.0.

453. 1:11:15

     Um, so if you want your own deep research, that's probably the best way to start.

454. 1:11:21

     And just with, uh, coding, right, in there, are you still getting, uh, charged based on the, uh, tokens,

455. 1:11:29

     uh, or the LLM calls?

456. 1:11:31

     Yeah.

457. 1:11:32

     So there's no additional, uh, um, price for tools calling, uh, of course, if those tools are calling some APIs, then there will be

458. 1:11:41

     an additional pricing for those APIs, right?

459. 1:11:42

     So, so there is additional pricing for the Google Search.

460. 1:11:48

     Uh, we had it, like, two minutes ago. Basically, one thousand five hundred, uh, searches are free. If you use the native tool, then it costs money. And all of those tools, in a way, enrich your context.

461. 1:11:59

     So if you use the URL context tool which goes to a blog post and the blog post has ten thousand tokens, you pay for those ten thousand tokens as it's included into your prompt and then tries to answer your, your question.

462. 1:12:12

     For, like, the function calling we had, it's a bit different. Of course, we generate tokens, but, like, the structured output tokens are

463. 1:12:20

     less. Uh, and you do the function calling or, like, the, the Python call on your side.

464. 1:12:26

     One, one quick. Like, there are two tools which have overlapping functions like, uh, uh, in the same weather tool one, one tool which is giving... And the temperature one tool which is giving temperature and humidity.

465. 1:12:38

     How does the LLM figure out which one to call or, uh, call both of them or call one of them and-

466. 1:13:40

     Inputs and outputs of the tools are not dependent of each other, right? I can start my disco ball and the music at the same time, and not, okay, I first need to start my disco ball, and if it runs, I can, like, start the music.

467. 1:13:52

     Um, if you have more of those sequential tool calling, um, basically you need the input-- Uh, sorry, you need the output of the first function for the input of the second function.

468. 1:14:01

     Basically, if you... I don't know, you have, like, some kind of, uh, smart home system, um, and you want to set the temperature based on, like, the outside weather.

469. 1:14:10

     You would first need to check what's the weather and then, like, set your temperature inside of your house. Um, here would you basically provide instructions, uh, using the system prompt to your model to say, "Okay, um, to change the weather, you first need to look up the weather and then set the temperature."

470. 1:14:26

     And what Gemini would do, basically you have your user prompt. It generates the function at a func- structured function. You call the function, and instead of generating a user-friendly response, it generates another function call.

471. 1:14:39

     And then you can call it. So basically, it continues this function calling loop before generating a very nice user-friendly output.

472. 1:14:47

     Can you say that again? So can a function call be in a loop?

473. 1:14:52

     Yes, kinda. I mean, I can like... So there's an-- I put up an example, uh, for, like, the sequential, um, function calling. Uh-

474. 1:15:03

     Is there a way to manage multiple function calling timeouts and user responses? Uh, is there a library that helps us manage that? Should we do or should we have to write if there are multiple parallel function calls and some sequential and some may take longer?

475. 1:15:19

     Uh, is that workflow something we have to code for?

476. 1:15:23

     Yeah. So currently there's not a good way to handle, like, functions who could time out. Or, like, the best way-- So it's a, a bit like, I would say parallel calling is like you need to wait for all of the results to put in into the next one.

477. 1:15:38

     If one takes longer, you can basically provide an error message or not ready yet or something. But you would need to, to explore how it works on the live API, which is basically our way to create real-time agents.

478. 1:15:51

     They are working something called asynchronous function calling, where, um, the conversation continues. So when you think about a customer support agent, which you can-- Which you talk to, right?

479. 1:16:02

     It would be very weird if the agent stops for, like, three minutes and doesn't say something because it needs to look up your information. So that, that's what they call asynchronous function calling.

480. 1:16:12

     So you can continue the conversation with the agent, but the agent runs a tool call and then, like, injects the response, uh, later. But it's only for the live API.

481. 1:16:21

     And is that available as a Python library or how do we-

482. 1:16:25

     Yeah. So, um-

483. 1:16:26

     The live API

484. 1:16:27

     ... documentation live API. Um, it was launched on, I think, in I/O or at Cloud Next. Um, so yeah. There, there you have, like, uh, tool use with live API, and there's asynchronous function calling, um, with code snippets for, for Python and JavaScript.

485. 1:16:50

     Cool. Yeah.

486. 1:16:52

     Is asynchronous function calling a synonym for, uh, parallel? Parallelize?

487. 1:16:57

     No. It's, it's more like I start a function call now. I need to continue my conversation. I get back the response ba- later. So instead of, like, getting the response, what you get back-

488. 1:17:07

     In the back model.

489. 1:17:08

     Yes. What you get back from the model could be, for example, interrupted or scheduled so that you as a developer know, okay, I started something. The model knows it started something.

490. 1:17:19

     And then you can inject once the, your function is, the output is ready, put it back into the conversation, and then the model uses that information to continue.

491. 1:17:30

     And what about, uh, logs of, uh, all the API calls?

492. 1:17:34

     Yep.

493. 1:17:34

     Uh, so like the OpenAI Playground-

494. 1:17:37

     Yep

495. 1:17:37

     ... we can see. What, what's the equivalent?

496. 1:17:40

     Um, so in AI Studio, we don't have a feature like this. Um, but you can, like, use third-party tools like LangSmith or Arize AI Phoenix. Uh, but Vertex AI supports those features.

497. 1:17:54

     Okay. So if we have a-

498. 1:17:57

     Yeah. There was a question. Yeah.

499. 1:17:58

     Um, i- in live API-

500. 1:17:59

     Yeah

501. 1:17:59

     ... previously, um, especially understanding with model boxes and coordinates was bad.

502. 1:18:04

     Yeah.

503. 1:18:05

     In the live API, we had to, like, transfer. Is-- Has it gotten better?

504. 1:18:09

     I don't know. The only thing I know is that, so previously there was only Gemini Two dot O Flash available in the live API. But now we have Gemini Two point five Flash, so basically the model we used in, like, the Jupyter Notebooks.

505. 1:18:22

     Um, you-- I guess you have to try or maybe, like, reach out to the people who are working more with it.

506. 1:18:32

     Okay. More questions? Yep.

507. 1:18:39

     How is URL context different from search? How is URL context different from search?

508. 1:18:42

     So URL context tool basically works based on the link you provide. So if you provide your own personal blog, it goes to that link and tries to extract the information and uses it in the context.

509. 1:18:55

     And search basically uses Google search, creates one or multiple search queries based on the prompt, searches, and then provides the outputs into your prompt. So if you already know where the source is for your prompt to create the best answer, URL context could be a good use case for it.

510. 1:19:17

     This agent was asked about real-time

511. 1:19:19

     Yeah

512. 1:19:20

     ... functionality. Does it also access, uh, paywalled content? Does it have access to paywalled content?

513. 1:19:29

     I don't know. You-- Paywall content, you mean like pages?

514. 1:19:35

     Websites.

515. 1:19:37

     I don't know. I would not expect to have it ex-- Like- But the Live API supports function calling. So if you have a subscription or, like, a way to access just paid content, you can create your own function, which the model can then invoke and use as a context for, for its, um, information.

516. 1:19:59

     Okay, cool. Then last section, I guess every one of you might have heard about Model Context Protocol by now, and that's, like, either our solution to how we all align on how to call agents or, like, what's the right thing to do.

517. 1:20:14

     I'm a big fan of it. Um, I think it makes it way more accessible for people to build agents, especially if we get more first-party remote MCP servers, um, where you can basically focus on building your agent instead of creating all of those functions.

518. 1:20:28

     I mean, I'm pretty sure there are, like, now a million get weather functions which can be used, and I hope with, um, MCP we can, like, fix this. Uh, what we, um, shipped and announced at Google I/O is, um, native integration of MCP servers inside the Google Gen AI SDK.

519. 1:20:44

     So the SDK which we have used during the whole workshop, um, allows us to directly use MCP servers and sessions, which makes it even easier for all of us to integrate it.

520. 1:20:55

     So when we, um, look back at our function calling example, we needed to call a function, create a declaration, and all of those different things. And now with the MCP, uh, integration, we can basically, uh, only start or initialize our client.

521. 1:21:12

     Here I created an MCP weather service which kinda has the same functionality. And, um, all of the things we now need to do is, like, we use our generate content method, and in our tools argument, we provide our session.

522. 1:21:28

     So we start our server, we create our session,

523. 1:21:34

     and we provide the session to, um, the Gemini SDK. And then what happens behind the scenes is basically the same function call loop we did manually. It's like, okay, what are the tools available?

524. 1:21:48

     Get all of the tools from the MCP servers, put them into the LLM call. If the LLM call makes a function call,

525. 1:21:56

     extract the function call, call the MCP server, get the response from the MCC-- MCP server, put it back into our conversation, have the model generate final response. Um, I guess it's, um, probably time to find out if Colab has Node installed and to see if it works.

526. 1:22:20

     Okay. Okay, it doesn't. What I can do is I can quickly set it up locally and show you in one second on how it

527. 1:22:45

     works.

528. 1:22:55

     Okay.

529. 1:24:16

     Okay, so we are back. I'm in Cursor, same notebook,

530. 1:24:21

     same setup, and now I can, like, use it. And now basically what happens if I-- Let's not ask about London, right? We are in San Francisco. Um, let's ask it about the weather in San Francisco.

531. 1:24:33

     It now does all of, like, the different function calls and loops behind the scenes. The MCP server is very simple. It uses the OpenMeteor API, which is kind of free to use on a small scale for testing.

532. 1:24:46

     Um, so we generate the output, run all of the MCP calls. Might take a bit longer for the API call, and then we get back. Okay, the weather in San Francisco today will be seventy degrees Celsius at zero, and then all of the other numbers.

533. 1:25:05

     And that's basically now all you need to, to combine or connect an MCP server with the, the Gemini SDK, which is like... I mean, it fits on a single screen, like, which is easy enough for you to get started.

534. 1:25:17

     Um, we use here, like, a, a local running MCP server, but it works the same way with a remote MCP server. That's, um, also the exercise, uh, for, for this, um, part of the workshop.

535. 1:25:28

     So basically, uh, [REDACTED:username] from, uh, the Cognition AI guys, so Hint or Devin, they had a-- have a very nice remote MCP server which, um, can talk to GitHub repositories.

536. 1:25:39

     So instead of, like, creating a sid IO client, you can use the streamable HTTP client. And connect a remote MCP server to Gemini to talk to it. And, um, yeah, basically benefit for one of the advancements currently happening in AI.

537. 1:25:55

     Do we have more questions? Yep.

538. 1:25:58

     Yeah. Don't you also have the ADK that is launched recently?

539. 1:26:01

     Yep.

540. 1:26:02

     And there's Gemini stack. How close these two things interplay, and what is the first thing I understood in SDKs?

541. 1:26:09

     So ADK is, um, for those of you who don't know, is like Agent Development Kit. It's an, uh, agent library which adds a lot of more abstraction on top of like the, the client, uh, SDK we used.

542. 1:26:22

     It makes it a lot easier to do all of the tool calling, um, integrates MCP server as well, makes it much more easier to manage like multiple agents or deploy it to cloud.

543. 1:26:33

     So it has a lot of, I would say, batteries included. Uh, question again, depends on you if you would like to use, uh, frameworks or rather prefer building it yourself.

544. 1:26:43

     I see a bit benefit of like getting started very quickly using agentic frameworks. But the more abstraction you add, right, the more-- the less you know on the first time, and then maybe you need, you need to dig a bit deeper later and all of the, the advancements or speed you get in the beginning will be your

545. 1:27:00

     slowdowns in the future. But ADK is definitely a great way to start. They also have like tons of examples. It has support for, uh, Gemini and the Gemini API, so definitely a good way to, to take a look.

546. 1:27:11

     Yep.

547. 1:27:14

     When you use a YouTube in URL to better tools, is it turned back on the ground and taking significant people search around like this?

548. 1:27:20

     Um, no. I don't think so. Uh, but what you can do with, uh, YouTube links or YouTube videos is you can ask it to return timestamps for you as the response.

549. 1:27:32

     And it works very nicely. Uh, so it, it's very accurately because so basically a video is more or less just many images behind each other, right? And currently, uh, when we process videos, it will be done at one frames per second.

550. 1:27:48

     And you will always have like the timestamp, the image, the timestamp, the image. That's how Gemini exactly knows at what place in the video is, I don't know, a man jumping or dancing or something.

551. 1:28:03

     So it's analyzing one frame per second-

552. 1:28:06

     Yes

553. 1:28:06

     ... and then describing it.

554. 1:28:08

     It depends on your pro-- I mean, we can-- I mean, it's part of the section, um-

555. 1:28:13

     Yeah, I use that all the time-

556. 1:28:14

     Okay, cool

557. 1:28:14

     ... to like summarize videos. But yeah.

558. 1:28:17

     Yeah, but yeah. Basically what happens with videos is, I mean, videos are on twenty-four frames per second or even more that like easily lets the, the context explode.

559. 1:28:26

     Yeah.

560. 1:28:26

     So currently what is done is that is we have one frame per second. So if you have sixty seconds, it would be sixty images, uh, which makes it also easy to count how many tokens it would be, and it's also how we can fit in, um, a one-hour long video into one million tokens basically.

561. 1:28:44

     Is it using the transcript from YouTube or is it doing, uh-

562. 1:28:50

     No, it's-

563. 1:28:50

     ... speech to text or-

564. 1:28:52

     It's all multimodal natively.

565. 1:28:55

     Oh, okay. It's not using-

566. 1:28:56

     No. I mean, you can like easily upload normal .MP4 files, and you can ask it like what was saying or said in, in the video.

567. 1:29:04

     Yeah, I tried that for Dutch-speaking stuff. It was quite good.

568. 1:29:07

     Okay, cool. Yep.

569. 1:29:12

     I was wondering if you've, uh, played around with any of the memory in some ways. I saw you had the memzero-

570. 1:29:17

     Yeah

571. 1:29:18

     ... people in there. Just 'cause Gemini seems to struggle a bit with the

572. 1:29:21

     memory stuff that they have known, so.

573. 1:29:25

     So I'm also like, maybe for you if you don't know, is like memory is basically next to tools what makes an agent an agent. And for memory, we have short-term memory and long-term memory.

574. 1:29:35

     Short-term memory is basically your conversation, which is part of your current like state when you talk to an agent. And long-term memory is basically memory or like information about a previous interaction or something about the user.

575. 1:29:49

     And long-term memory is not like part of the LLM, which we need to provide externally. And memzero is like something I looked at it because it's very nice. It's does it implicitly.

576. 1:30:00

     So basically it takes the conversation and tries to create a nice abstraction for you, we-- which you can include. Um, I'm not sure like what the current state is like, how or what works well with Gemini.

577. 1:30:12

     I'm like, what I always see or where Gemini really shines is the long context. So the better your information extraction is and like what you provide to the model based on the memory, I think it works very well.

578. 1:30:24

     I'm not sure about those tool calling memory kind of systems.

579. 1:30:29

     Yep.

580. 1:30:30

     In terms of context window, is there other plans-

581. 1:30:33

     Yeah

582. 1:30:33

     ... for using memory?

583. 1:30:34

     Yeah. So we are working on two million. Um, not sure when it is like generally available, but.

584. 1:30:40

     I know internally you basically go to ten million for that.

585. 1:30:42

     Yeah, like there, there was research, but you know, like the bigger you go, the more expensive you get. And do you want to pay like fifty dollars per one million tokens?

586. 1:30:52

     You're not paying for token, July.

587. 1:30:53

     Yeah. [laughs] More questions or everyone working?

588. 1:31:01

     Just a question.

589. 1:31:01

     Yep, then.

590. 1:31:04

     So before MCP, we need to define the functions-

591. 1:31:09

     Yeah

592. 1:31:09

     ... and to call it ourselves.

593. 1:31:11

     Yeah.

594. 1:31:11

     And we need to decide the logic how to call the tool, when to call the tool ourselves as well.

595. 1:31:16

     No. So the, the difference with MCPs now and, uh, what we had previously is that we don't need to write and define the functions and the function declarations. The LLM sees at the end the exam sact-- exac-exact same thing.

596. 1:31:35

     So if I create my function declarations manually with adjacent schema, or if I retrieve them from an MCP server, those are the same things for the LLM. It's just makes it much easier to Not need to rewrite the same functionalities over and over again, right?

597. 1:31:53

     If you work at a company and you want to integrate APIs which you have internally, there's a pretty big chance that two teams write the same function declaration and the same wrapper to call it.

598. 1:32:06

     And the idea here is to only have one team needing to write it, and everyone to benefit from it. The same is with, like, public APIs like Google Maps, G Drive.

599. 1:32:16

     So MCPs gives us a way to collaboratively create, like, the best or, like, the standard as a way to how to call it. And then everyone can, can im- can implement the MCP tools, uh, which you need for your use case.

600. 1:32:29

     Yes. But with the MCP given a prompt, how does it decide which tool to use under the MCP?

601. 1:32:38

     It's-- So an MCP server, for example, exposes four tools. Um, you provide all of those four input schemas, which are like function declarations with a name, a description, and the parameters to your LLM call.

602. 1:32:53

     And then based on the prompt and those declarations, the LLM, in this case Gemini, decides if it should call a tool, which tool could to call, or should it call multiple tools.

603. 1:33:05

     And that's the same logic we had for normal or have for normal function calling. So there's no, no difference in that way. The only difference we see is that

604. 1:33:18

     MCP service being easily available or implemen- implementing many tools, people start to be very easy on how many to add, right? So we see people add, like, fifty, a hundred tools.

605. 1:33:29

     So we need to improve our LLMs to be able to have them really know, okay, which is the right tool to use when you have like fifty different tools available at once. [coughs]

606. 1:33:38

     Google accuracy way, is that an, um, like, part of the Gemini SDK or ADK, or is that maybe separate?

607. 1:33:50

     So there are examples for ADK to work with A2A. So A2A stands for Agent to Agent Protocol, which is done by the Google Cloud team. The idea here is, uh, to allow companies to build agents in different frameworks like LangChain, LangCraft, LlamaIndex, and then have a, uh, an easy way to create multi-agent systems that one agent can

608. 1:34:13

     call the other agent without needing to implement compl- complex logic. Um, but the Gemini SDK is like not directly, but there are great examples for it.

609. 1:34:27

     Cool. Yeah.

610. 1:34:30

     Is there a way for Gemini to, like, natively use website search data source, like in Colab, the same way, like, this is called

611. 1:34:41

     So we are working on basically browser use or computer use, use cases, which is available in preview, uh, which we are currently testing with a few companies, um, would-- that would allow Gemini to control UI.

612. 1:34:56

     So it could basically go to whatever website you want it to go to. And then there's, [coughs] sorry, the URL context tool where you can provide a website, and we would programmatically try to extract information from it.

613. 1:35:09

     Of course, it's like a-- If it's a super heavy JavaScript website, there's not much to extract. That's where, like, the, the browser use agent would then, um, be useful.

614. 1:35:19

     It's like an API for Navina or something?

615. 1:35:22

     Yeah, kinda. Yeah. Hopefully coming soon.

616. 1:35:29

     Does browsing use, is there currently a spin up of a, a, a cloud, uh, browser environment? Or does it work more like

617. 1:35:39

     So it will return structured outputs again based on what you provide. So you can control a local environment. But we are also working with the Cloud Run team to make it super easy to you-- for you to run so that you can, like, run a Chrome instance on Cloud Run, uh, which Gemini can talk to and control,

618. 1:35:56

     or you can, like, control a local instance.

619. 1:36:01

     Both. It's gonna be a way.

620. 1:36:01

     Yeah. Yeah. Yeah.

621. 1:36:04

     How do you manage the authentication for the MCP?

622. 1:36:09

     That's a good question. I think the whole industry is currently trying to answer, like, what's the best way to handle agent authentication. So the MCP, um, as a protocol itself supports OAuth.

623. 1:36:20

     So basically, when you want to connect to an MCP server which is protected, you would get back a 403 with, like, not authorized, which can then trigger an OAuth flow on your client.

624. 1:36:31

     Maybe you've seen it in, like, Cloud Desktop, where you get a pop-up, like, to log in with, I don't know, like, your, like, uh, Atlassian account or something. So that's, like, one way of doing it, but it definitely needs more work, right?

625. 1:36:45

     Do you want your agent to access all of your emails or all of your GitHub repos- repositories? Or how can you, like, scope it down to, like, only the one specific repository?

626. 1:36:55

     But they are currently, like, actively working on it. I know that Auth0 will be here tomorrow as well. Those guys are very great to talk to. I think I know that they're doing a lot there.

627. 1:37:04

     So... Yeah.

628. 1:37:11

     I was wondering if you could say something about the citation, citations and how that's different from, like, URL context. Maybe I was just struggling with trying to get application on it.

629. 1:37:21

     And also just sort of curious, like, how, like, how you trigger it as well as there can mess around with prompting and then sometimes the outputs can have their own

630. 1:37:30

     You mean, um, citation in when you just send a regular prompt or a citation when you, like, use Google search that have citations?

631. 1:37:38

     I think very specifically citation metadata, like the output field.

632. 1:37:44

     Yeah. So that's available at, like, the Google Search tool. Um, and it has information about which websites was used to, to retrieve. You can, um, click on the link directly to see where you land.

633. 1:37:58

     And then there's this, um, metadata or like chunking sup- or like crowning support, which basically has a start index and an end index from the answer, and then also which sources were used.

634. 1:38:11

     So you technically can highlight or put numbers behind it to help users understand, okay, that part of the response is generated based on those two, um, links.

635. 1:38:25

     I think the, the part I'm confused about is the, it seems like if you use the Google Search tool that the ground truth, the, those citations that you get there, but then like the citation metadata, it seems like it may be like guessing from the training data.

636. 1:38:41

     But-

637. 1:38:42

     Y- you mean in, in general, there are sometimes citation metadata or?

638. 1:38:47

     Sorry, what?

639. 1:38:48

     Wh- which citation data do you mean?

640. 1:38:51

     The one literally called citation metadata.

641. 1:38:54

     W-

642. 1:38:54

     Not grounding-

643. 1:38:55

     Okay

644. 1:38:56

     ... URL and such.

645. 1:38:56

     I know, or I only know that's part of like legal and like training, that if there's something referred directly, you need to like provide it. That's why it is there.

646. 1:39:07

     That's why it is not always there. Um, it can be there, but it doesn't must be. And it's not like for a user to, "Hey, that's coming from there," but it's more like, um, compliance basically why we need to have it.

647. 1:39:20

     So, all right. So follow up that, the thing I'm, the thing I'm interested in is you're familiar with like Anthropic's citations?

648. 1:39:29

     I only know about like their web search feature.

649. 1:39:32

     They have a citation thing where it's just whatever you pass as input, it'll give you direct quotes from that. So it's kinda like in their response-

650. 1:39:40

     Ah, okay

651. 1:39:40

     ...

652. 1:39:41

     So you provide as a context, like a document.

653. 1:39:45

     Yeah, you pass a local as input-

654. 1:39:45

     Okay

655. 1:39:45

     ... and it lists the stages where it's getting this from. Um, I imagine you could kinda like forge that-

656. 1:39:51

     Yeah

657. 1:39:51

     ... through structured outputs. But I'm curious what like the best practices are with doing that, and if, if you could use, you know, structured outputs versus the, the URL context seems like another avenue you could go with that.

658. 1:40:07

     No, that's like a very good question. Currently, we don't have like the same experience as Anthropic has. Um, I guess the base is like to really try and like different prompting strategies and like what you want to achieve.

659. 1:40:20

     Um, yeah, no, but it's like good, good feedback.

660. 1:40:24

     Is that the approach NotebookLM is gonna take next?

661. 1:40:28

     Right, yeah. Are there, uh, are there any, um, public MCPs you can think of that are public so we can use them now?

662. 1:40:36

     Sadly, not yet. I hope, hopefully one day that you can like just, here's Gmail, here's Drive, and let's chat. Um, but currently there's no public remote MCP server.

663. 1:40:51

     And if, and if we wanted to, can you, can you, uh-

664. 1:40:52

     I don't know exactly what, what are the reasons. I think the more people ask for it, the easier it will be that we get one. So if you... Like, that's great feedback.

665. 1:41:01

     More people, um, if the more people are going to use MCP servers, the higher the chances are going to be. And I think in general, for MCP to succeed, we need more first party remote servers, right?

666. 1:41:13

     Because we cannot build a secure GitHub MCP servers. That's something GitHub needs to do because they know how their OAuth system works, how you can scope it. Um, so we really need those first party MCP servers in the long term.

667. 1:41:28

     Yep.

668. 1:41:29

     How should websites, prepare websites for the, the, the AI mode? Is there some changes required on the schema or anything else?

669. 1:41:36

     For, for AI mode?

670. 1:41:38

     Yeah.

671. 1:41:39

     I have no idea. I think, like, so AI mode, you mean about like inside Google Search?

672. 1:41:44

     Yeah.

673. 1:41:45

     I think in general, overall, same goes for like normal Google Search. It's like have high quality content, um, try to stick to like, um, web standards. Um, I guess same works here as well.

674. 1:41:59

     But I don't know. Sorry. Yeah.

675. 1:42:02

     When making like multiple calls with the files API, should it trigger the prompt caching for Gemini 2.5, or that's important to be clear?

676. 1:42:12

     It should. So if you uploaded a PDF, you need to make sure that you put a PDF at the first part of your prompt. So you'd like the, the automatic, um, caching works from the beginning to the end, right?

677. 1:42:24

     If you change the beginning, you can never cache like the long document. But if you put, uh, the PDF in the beginning and change the prompt behind it, it should work.

678. 1:42:33

     Yes. Yep.

679. 1:42:36

     I had a question, um, Google Search grounding as well. So sometimes there's a case where, um, the index that it provides, you know, the start, uh, and index-

680. 1:42:44

     Yeah

681. 1:42:44

     ... is actually out of range for, you know, like the length of the-

682. 1:42:47

     Really?

683. 1:42:48

     ... yeah, answers provided. Yeah. I was, I was curious if you had had any similar experiences.

684. 1:42:52

     No. But if you like have an example for us to reproduce and to share, that would be very helpful. The only thing I know is that, so sometimes there can be a start index, which can be null, which basically means it's zero.

685. 1:43:04

     Like, it starts at the beginning.

686. 1:43:05

     Yeah. Sometimes I get for even start and end actually outside of the range, 'cause I assume it's basically just a character count, right?

687. 1:43:12

     Yeah.

688. 1:43:13

     Um, the outside of the range of the input.

689. 1:43:16

     No, like if you have an example, please like send it to me on Twitter or somewhere, like-

690. 1:43:20

     I can actually do that

691. 1:43:21

     ... okay. That, that would be very helpful, because that should not be the case.

692. 1:43:24

     And then for the, um, excuse me, the ground- grounding metadata is two parts, the grounding supports and the grounding chunks.

693. 1:43:30

     Yeah.

694. 1:43:30

     Is there any way to actually, um, get the, the citation, or maybe that was the question that the gentleman asked, the citation from the website, basically the text that it refers to or not?

695. 1:43:41

     So that-

696. 1:43:42

     'Cause you only get the website actually. So you just get the URL.

697. 1:43:44

     Yeah. No, like, so currently you get like the, the start and end of the response, which refers to something, but not the web part. But it's like, like please s- put it all together and like send it to us, like very happy to like talk to the Google Search team who is like building the native tool.

698. 1:44:02

     Um, that's super helpful with like to better understand like what you need.

699. 1:44:08

     Yeah. Thank you.

700. 1:44:08

     Okay. Cool. Then thanks all for coming. Um, please continue with the workshop. Try it out. If you have like any questions, we are very happy to receive any positive, negative feedback, any ideas, any pain points you have.

701. 1:44:26

     Um, we are available on like social channels. You can like find me, Philipp Schmid, basically everywhere. If not, we open a GitHub issue or be very noisy about when, when something doesn't work.

702. 1:44:37

     Uh, we, we always try to make sure it, we fix it. Cool. Thanks. [clapping] [outro music]