Build & deploy AI-powered apps

Talk transcript

1. 0:00

   [upbeat music] My name is Paige.

2. 0:15

   I started doing machine learning a long time ago, um, around two thousand and nine, two thousand and ten. Um, was primarily working with, um... Though it feels like forever ago.

3. 0:26

   I was just, uh, talking with a friend about this recently. Um, back in two thousand and nine, two thousand and ten, it was kinda wild that companies would even trust open source software to do business-critical work.

4. 0:38

   Um, and so, uh, I was contributing to things like NumPy, SciPy. [microphone interference]

5. 0:44

   Like little, like little antenna, these microphones.

6. 0:49

   You can start. You can start.

7. 0:51

   Yeah.

8. 0:51

   Say what.

9. 0:51

   Sure. Cool, cool, cool. Um, so NumPy, SciPy, um, Matplotlib, which is still just as excruciatingly painful to use. Um, uh, scikit-learn, uh, kind of the early days of the scientific computing stack.

10. 1:05

    Um, and, uh, eventually started working as an engineer. Um, my background is geophysics and applied math for undergrad, and then computer science and carbonate geology for grad school. And so I started at Chevron doing work in subsurface geosciences, um, doing a lot of things like velocity modeling, drilling optimization using very, very basic machine learning models.

11. 1:28

    Um, and, uh, also some large-scale compute. So if folks in the audience, um, also have [REDACTED:physical_attribute] like me, you might remember Cloudera, uh, uh, which was one of the first companies that made some of the open source, uh, frameworks and tools available, um, for consumption of, like, Fortune one hundred companies.

12. 1:49

    Um, so things like Spark. Um, a lot of the Databricks team came from the Cloudera world. Um, and so I was spinning up clusters of machines there. Um, eventually TensorFlow got released in the open source world around twenty fifteen, towards the end of it, um, and I started contributing to that.

13. 2:05

    Um, the geosciences world is really big into GPUs. They were even into GPUs before, um, the machine learning world. And, um, uh, so, so I had experience working with CUDA and, and all of the, the associated tools.

14. 2:21

    And TensorFlow, when it was first released, um, only supported CPUs. I'm not sure how broadly that's known, but, uh, but it was, like, distributed deep learning across CPUs, and they needed somebody to help with, like, getting GPUs, um, to, to work with TensorFlow.

15. 2:38

    Um, that's also why there are three different code paths in the original TensorFlow one framework, um, because they had to, like, gut the back end and replace it for CPUs, for GPUs, and for TPUs, both for single node and for distributed computation across multiple nodes.

16. 2:56

    Um, so that's basically it. Like, I owe my entire career to open source software, um, and open source models. And, uh, I got hired at Google for specifically that reason.

17. 3:06

    Um, and then I left to go work at GitHub for about a year on VS Code, which is also open source, um, and early user experience testing for Copilot, and then came back to work on our large models.

18. 3:17

    Um, and so I was part of the original PaLM 2, Gemini, and Gemma teams.

19. 3:22

    Yep. So that is journey. I'm not sure-- Like I said, I'm not sure it's interesting.

20. 3:27

    And you need, you need to leave, uh, soon you said? 'Cause I need to have a talk with you.

21. 3:31

    Oh, uh, four-four fifteen because I've got meetings. Uh, we have leadership team meetings from, like, four thirty to five thirty.

22. 3:37

    Then I'll take over.

23. 3:38

    Yep, and Guillaume will be here, and then also potentially, uh, potentially our colleague Ian will come to, uh, will come as well.

24. 3:45

    Yeah, we need to be... Potentially, because I would not say the same thing if I have half an hour or almost an hour. [laughs]

25. 3:52

    I think, I think there will be, uh, there will be, like, enough good stuff to, uh, enough good stuff to talk about. Um, but yeah, like, if, if you're-- if anybody's curious, like, open source is a great way to, like, work with a team to figure out if you would want to work with them more, um, and

26. 4:10

    then to also make sure that your work is public so that other folks are interested.

27. 4:14

    I'd love to show you my, my projects. It's open source.

28. 4:16

    Cool. Awesome.

29. 4:19

    Guess how many stars it got?

30. 4:21

    How many?

31. 4:21

    Seven hundred.

32. 4:22

    Awesome. Wow, excellent. Good job.

33. 4:26

    I'm catching up with you guys. [laughs]

34. 4:42

    Right? Like, there... Um, usually if you see everybody sprinting to do the same thing, that's a great indication that it's the wrong thing. Um, and so, so like, as-- Or, or like a thing that eventually the model will, will have that capability to do.

35. 4:58

    Um, so, so one of my favorite examples of this is that, um, you know, when the models were first released, they had context windows of like eight thousand tokens, sixteen thousand tokens, and so everybody was sprinting to build a vector database, um, because they were like, "Oh, well, we have to work around this constraint.

36. 5:13

    The models have this very small context window." Um, and then, you know, obviously that's expanded over time. Um, there's another great, uh, example of, you know, everybody sprinting to build fine-tunes of models to support different languages.

37. 5:29

    Um, though now models support a variety of languages. If you've been, uh, kind of listening along for some of the Google DeepMind sessions today, you've seen that in action, and then also many of our competitor models also support multiple languages.

38. 5:42

    And so I think, like, we, we also saw everybody sprinting to build an agent framework, when the reality is, I think that the m-- like all of that will probably be absorbed into the model eventually.

39. 5:53

    Um, and also everybody sprinting towards things like building MCP servers, whereas now mostly people have kind of moved away from MCP servers and are adopting skills which are just fancy markdown files.

40. 6:04

    Um, and then obviously longer term, you could imagine like, "Oh, hey, I just want a little listener, um, watching everything that I do, and it will automatically create skills behind the scenes for me to use."

41. 6:14

    And we're starting. Yep. Yep.

42. 6:16

    No, I was gonna say, um-

43. 6:18

    agree because you guys will always cater the sort of most generalized applications, but very, very specific ones, like for instance, bioinformatics, which is what I'm working on, Google is not gonna work on that.

44. 6:32

    I don't think that they will absorb th-those kind of functionalities.

45. 6:35

    So, so counter, counter-example, um, our first implementation of PaLM 2 and our first implementation of Gemini, we had to have a fine-tune, one called MedLM and one called Med-PaLM to support medical use cases.

46. 6:49

    Um, now we see that all of the people who had previously needed to rely on those fine-tunes are just using Gemini out of the box with either retrieval or with a custom prompt, because all of the data that we used for those fine-tunes is just incorporated into Gemini itself.

47. 7:04

    But, uh, you have the issue of reproducibility. You're not, you're not gonna have the same results all the time. You're gonna have a lot of stochasticity as a consequence of that.

48. 7:12

    So, so, so no large language model is deterministic. Um, so, so that will be a problem regardless. Um, but, uh, but I do agree, like there, there are, um, you know, I think a lot of the magic is going to come from having like a very opinionated view of use cases, um, and being able to work directly

49. 7:32

    with customers and solve their problems. Um, so with that-- and that's a perfect segue into, uh, into the session today. So greetings, everyone. Thank you for being brave and for coming back.

50. 7:46

    Um, I know that there were a lot of people who were, uh, unable to, to join us earlier this morning, um, for some of the sessions that we had.

51. 7:55

    Um, so just show of hands, how many folks came back this afternoon, were not here this morning or, uh, like were, uh... Cool, cool, cool. Excellent. So, uh, we have a treat for you today.

52. 8:07

    Um, for the sessions that we delivered earlier, um, we're going to do basically a recap of them. So we'll be walking through all of the, all of the examples.

53. 8:16

    Um, you won't have, uh, you won't, uh, uh, kind of walk away feeling like you've missed anything. Um, and, uh, I'll be doing some within AI Studio and Antigravity.

54. 8:28

    Um, and then my colleague, uh, my colleague Guillaume, and you can also see one of my agents that's kinda doing its thing with computer use and browsing. Um, uh, the, uh, I'm going to disconnect for a second so we don't accidentally see something that perhaps should not be shared.

55. 8:45

    Um, and then, uh, going to go straight into slideshow view.

56. 8:53

    Awesome. So, uh, greetings everyone. My name is Paige. Um, so, uh, I, I am really excited to be here today, um, and I am especially excited to show you all of the things that we've been doing at Google DeepMind over the course of the last year, um, the last, uh, six months.

57. 9:11

    Uh, it's been a wild ride, and never in my career have I been more excited to be a m- machine learning engineer working in this space. Um, just over the last month and a half, uh, it feels a little bit like whiplash.

58. 9:23

    Um, we've released a ton of different things. Um, so Gemini 3.1 Flash Live, um, which we'll take a look at in a second, uh, which gives you the ability to have a real-time conversation with the model.

59. 9:33

    Um, Gemini 3.1 Pro and Flash Lite, um, respectively our largest and our very, very small but very capable model, um, that are also very cost-effective, very performant. Um, Augment Code, if you've heard of them, they're a company over in the Bay Area.

60. 9:49

    They've just recently replatted their entire agent infrastructure to default to Gemini 3.1 Pro, specifically because of performance and cost ratios. Um, uh, it can do a lot of really good work in a very, like, um, small number of pennies.

61. 10:06

    Uh, also NanoBanana 2 for image generation, image editing, including reverse image search, which we'll also take a look at in a second, um, and you'll be hearing about from my colleague Guillaume, who's our expert in generative media.

62. 10:18

    Um, our Embeddings 2.0 model, which allows you to embed video, audio, images, code, and text in the same embedding space. Um, so you can say like, "Show me everything you have related to llamas," and you will see everything from, um, like stuffed llamas, to pictures of llamas, to videos of llamas, to things of like what does it

63. 10:38

    sound like when a llama makes noises? 'Cause I don't know, but like the model, the model somehow does. Um, Lyria 3 for music generation, Genie 3 for world model building, um, our full stack runtime with AI Studio, which allows you to incorporate database, OAuth, um, uh, custom API keys if you want to use other services, and a

64. 10:58

    whole bunch of other things. Gemma 4 for open models, um, we just released it last week under an Apache 2 license, which is really exciting if you care about open models.

65. 11:08

    And then also Veo 3.1 Lite for video generation. Um, so massive number of things across a broad spectrum of surfaces. Um, and part of the reason for this is because Gemini is kind of unique in the industry in a couple of ways, one of which is that it's natively multimodal, so it can understand video and images and

66. 11:28

    audio and text and code and all of the above all at once. Um, but it can also output multiple modalities. So it can output text, it can output code, just like all of the other competitors on the market.

67. 11:39

    Um, but it can also output images, it can edit images, it can output images and text interweaved, and it can also output audio tokens. Um, so, uh, so pretty, pretty compelling use cases.

68. 11:51

    Um, but I think it's a lot more interesting to see it rather than to just have me waxing poetic about it. Um, and for that, I am going to go into AI Studio real quick, um, and we're going to take a look at some of the things that Gemini can do.

69. 12:06

    Um, so first off, how many people have used AI Studio before? Um, excellent. I'm very, very glad that all of the DeepMinders have raised their hands. Um, but, uh, for folks who have not, uh, AI Studio is kind of the best place to go to get access to DeepMind's models as soon as they're released.

70. 12:24

    Um, we have a playground feature. We also have a build feature, which we'll see in a second, which is very similar to [REDACTED:url].dev or Lovable. Um, you can select different models here off to the right.

71. 12:34

    Um, so you can see if I click on the model name, um, we, uh, see some pills, um, everything from Gemini to live capabilities, um, image ch-- uh, image gen-- uh, generation, video generation, audio generation, and music, um, and then also our Gemma open model family, which we'll also take a look at in a second.

72. 12:55

    Um, you can select different models. For the purposes of speed, I am selecting Gemini 3.1 Flashlight preview. I am also on my personal instance of AI Studio, and this is, uh, an attempt to drive down costs.

73. 13:09

    Um, so Gemini, uh, 3.1 Flashlight is around twenty-five cents per million tokens analyzed, um, which is really, really good, uh, almost an order of magnitude lower than our Gemini 3.1 Pro model.

74. 13:22

    Um, and it can still analyze video, audio, et cetera. Um, uh, it's just, uh, a lot more lightweight, which means that you might not always get the, the same great capabilities, um, but it usually does a pretty good job.

75. 13:34

    Um, and you can do everything from analyzing images to analyzing video. So if I, as an example, um, was going to look up a, uh, was going to look like a dinosaur YouTube video, um, we already used this Rexy, the, the little T-rex before, um, so I know it works.

76. 13:52

    Um, but, uh, but I am going to find, um, maybe this one, uh, Carnotaurus, uh, which is- [exclaiming] -uh, which is-- it looks like a very, very long video, so around an hour long.

77. 14:07

    Um, we might, uh, chop it up a little bit, but if you click this plus sign, you can see that you can add different files. You can either add files from Drive, everything from text files to, um, uh, to PDFs.

78. 14:20

    Um, you can upload files directly. You can record audio live, um, add camera footage, add a link to a YouTube video, which we'll do right now. Um, so I paste in the link to the YouTube video.

79. 14:33

    I'll do a start time of zero seconds and an end time around three hundred, um, and it samples at around one frame per second. Um, and then what you see is this is, you know, thirty thousand, um, uh, thirty thousand nine hundred tokens.

80. 14:50

    I'm going to make sure to turn on Google Search grounding off to the right, and I'm going to say please create a table with timestamps for all of the kinds of dinosaurs that you see in this video.

81. 15:06

    Um, make sure to include a fun fact about each dinosaur. Um, for all of the other dinosaur experts, um, like armchair dinosaur experts in the, in the room, you might have noticed that it said Carnotaurus.

82. 15:19

    I am skeptical that that's an actual dinosaur. Um, so, uh, we'll see, uh, we'll see what happens. Um, but what's, uh, going on behind the scenes is that, uh, the model is getting sent this YouTube video for inference, and it's not just the metadata associated with the YouTube video.

83. 15:37

    It's also frame by frame, um, the video itself. And, uh, question?

84. 15:41

    Can you just put it in light mode?

85. 15:44

    Um, light, uh, does anyone else have, uh, does anyone else have issues with seeing the settings? Or, like, uh, could we, could we change the lighting? Oh, is this better?

86. 15:57

    Yeah.

87. 15:57

    Oh, excellent. Awesome. Um, so, so that's, uh, th-th-- thank you for, thank you for the request. Like, it's-- I, I didn't, I can see it perfectly on my screen.

88. 16:07

    Usually, my eyes get a little ir-irritated by light mode, but this is, um, as long as it's better. And so, um, so it took in the video. Um, it d-defined the different dinosaurs that it sees.

89. 16:21

    Um, uh, it's says that the name means "meat-eating bull," um, mentioned Triceratops, um, and then also mentioned that Pteranodons are not necessarily dinosaurs. They're a group of flying reptiles.

90. 16:35

    Um, and then if I wanted to, uh, if I wanted to get the code that was used to generate this experience, um, and I wanted to replicate it in my own app, all I have to do is click Get Code, and it automatically configures the model.

91. 16:49

    Um, it configures the, uh, the URL that I have inputted, um, the kind of the offset for the video, the, the prompt that I used, um, and it's in TypeScript, Python, or whatever your favorite language might be.

92. 17:03

    So the TLDR is that if you can get it working in AI Studio, you can get it working as part of your app. All you have to do is click the Get Code button.

93. 17:11

    Um, so that is, uh, Gemini 3.1 Flashlight for analyzing video. Um, you can also use it to analyze images with a couple of other baked-in tools. Um, and if we look over to the right, we can see structured outputs, code execution, things like function calling, even for custom functions.

94. 17:31

    Um, also some things like URL context. Um, and all of these are very, very special, but they're also just one-liners if you want to use them as part of your API.

95. 17:42

    Um, so as an example, if I turn on code execution, um, I'm going to see a Gemini 3.1 Flashlight selected. Um, I'm going to go into compare mode, so I want to compare it, um, with maybe the Gemini 3.0 Flash preview, also with code execution turned on.

96. 18:00

    Um, and I'm going to look for a picture of Lego bricks. Um, this one,

97. 18:09

    and copy the image. Uh, and right now we're in compare mode, so I'm comparing two different models, both with the same tool turned on, um, and say something to the effect of, "Draw bounding boxes

98. 18:23

    around all of the green Lego bricks, uh, using Python. Um, make sure, uh, or display the image

99. 18:33

    with bounding boxes." And code execution is sort of giving Gemini the ability to, uh, to sort of stand up a makeshift Python environment that's sandboxed, um, use a whole bunch of data science libraries that are pre-installed, um, and use those, invoke them as tool calls, um, uh, kind of like writing the code and incorporating anything that you

100. 18:55

     might share in. So I shared in, um, this image, uh, and very, very quickly, I'm not sure if you saw how quickly, um, but it was able to draw bounding boxes around the, the green Lego bricks.

101. 19:07

     You could also ask for segmentation masks. Um, and even more excitingly, so Gemini 3 Flash is still plugging along, um, but if you look at how much this cost, um, to do, to define the bounding boxes, you could have also asked for things like, um, "Tell me how many green Legos there are," or, "Tell me, uh, um,

102. 19:30

     you know, what are the orientations of the Lego bricks?" "Tell me, uh, how many, uh, you know, uh, what are they called, the little funny rabbit things? Labubus. Tell me how many of those you see in all of the frames of this video, um, and at what timestamp."

103. 19:44

     Like, those, those are the kinds of tools that can be invoked, um, via the sandbox environment. Um, and again, like, very, very inexpensive in order to do this work with Gemini 3.1 Flashlight and code execution turned on.

104. 19:57

     Um, and it's just a one-liner to stand up that sandbox Python environment, um, with compute that Gemini can use to do that work.

105. 20:07

     Cool. And it looks like the, uh, it looks like the Gemini 3 Flash preview was able to do it. It was checking its work, so that was the iteration that it was going through along the way.

106. 20:19

     So it got this first result, and it said, "All right, I wanna double-check and verify that what I did was right." It drew the segmentation mask with, uh, to, like, define all of the, the, like, green spectrums, um, that it saw in the image, uh, and then it, uh, kind of verified that those were the correct coordinates.

107. 20:38

     Um, and still, if I look, um, it's significantly more cost with the verification, but still on the order of pennies in order to draw the bounding boxes. It just took a little bit more time and also a lot more tool calls to invoke.

108. 20:51

     Um, so I strongly, strongly recommend, um, playing around with Gemini 3.1 Flashlight for your use cases, especially if previously you were relying on Gemini 2.0 Flash or 2.5 Flash.

109. 21:04

     Cool. Um, so we also have a feature called Build. Um, Build, again, is kind of similar to [REDACTED:url].dev or, uh, or Lovable if you've played with that before. Um, uh, we've recently added a feature where you can add database and auth to your apps within Build.

110. 21:24

     Um, and so as an example, today, if I wanted to click this guy, um, and say something to the effect of, um, "Create an app that gives the user the ability to upload an image of their bookshelf.

111. 21:39

     Um, the bookshelf should have a whole bunch of books on it with, uh, so we can see their spines, um, uh, things like titles and author names. Um, I want you to use Google search grounding to, uh, to fill in the blanks, uh, for the, for the books, um, so make sure that you have information about the

112. 21:56

     title, um, the author name, the description of the book, and then also what kind of genre, uh, it might be. Um, and I want you to save it all to a database, um, so the user logs in with, uh, with their Google account, um, they upload a picture of, uh, of their bookshelf, um, and it saves all

113. 22:17

     of their books, uh, in, like, this database format, um, so they know what books they have and it's attached to their account."

114. 22:24

     Which is a tall ask. Um, so, so it's, uh, kind of defined that work. I'm going to... I've got 3.1 Pro Preview selected. I'm going to do the default instead in the hopes that it might be a little bit faster.

115. 22:38

     Um, and I'm going to click Build. And what happens behind the scenes is we get, um, we get put into this IDE, uh, like environment, um, where you can see the, the model kind of going through the thinking process, figuring out what it would need to do in order to spec out the assignment, um, build the plan,

116. 22:55

     how long it's working. Um, you can also upload files. So you can upload, uh, files that you might have, like PDFs or specs for apps that you would like to create, um, and give them to the model as well.

117. 23:08

     You can connect to your Drive instance. Um, and we also have a settings section off to the right where you can define custom secrets. So right now I have a Gemini API key that I've pre, uh, pre-added.

118. 23:20

     Um, but you could also add, like, a Supabase API key or, um, an API key for NADen or whatever your favorite flavor might be. Um, you can see a new version history, which if you've played with AI Studio before, is definitely, uh, something that is much appreciated.

119. 23:36

     Um, and then also integrations, so things like OAuth, um, as well as GitHub, so you can sync to a public or a private repository. Um, but while this is working, um, I'm going to go ahead and show off, uh, something called, uh, Genie 3.

120. 23:53

     Um, or actually before that, I'm going to show off Gemini Live real quick, um, just in case folks haven't seen it. Um, so how many people have heard of Gemini Live?

121. 24:03

     Few hands. Cool, cool, cool. Um, uh, we also happen to have the expert for Gemini Live. Uh, Ian, come on down. Uh, Ian is the, uh, the Gemma, the Gemma 4, uh, team member that I had mentioned, uh, that I had mentioned before, and he'll be doing some live model demos, which makes me really excited.

122. 24:22

     Um- But, uh, Gemini Live gives you the ability to have a conversation with the model in a variety of languages. Um, but you can also share video feeds, you can share your screen, um, and all of this is kind of stacked together in one speech-to-text LLM understanding and text-to-speech pipeline.

123. 24:40

     Um, so as an example, um, uh, we've still got our Lego bricks and pieces pulled up, so I can share my screen to say, [notification sound]

124. 24:52

     uh, "Hey there, Gemini, uh, what do you see on the screen?"

125. 24:57

     I see a Google search for Lego bricks and pieces. There are image results showing various kinds of Lego pieces, sets, and different color combinations. On the right, there's a larger preview of some brightly colored Lego brick illustrations from Freepik.

126. 25:12

     Anything specific you're looking for?

127. 25:15

     Does anybody speak a language other than English?

128. 25:18

     [REDACTED:religion]. [REDACTED:religion]? Oh, excellent. So, so I'm going to, I'm going to ask for you to fact-check something, um, uh, and also spell-check something. Um, so only respond to the user in [REDACTED:religion].

129. 25:33

     Is that correct? Uh, I meant Spanish. Oh. It is Spanish. Oh, sorry. [laughing] Like the... Oh, gotcha. So Spanish? Um, yep. Oh, there we go. Uh, only, uh, Spanish. Excellent.

130. 25:45

     Um, and then, uh, I was about to say, I don't know that language, or I haven't had that before. Um, but only respond- You know Spanish, right? I know S- I know Spanish. [laughs]

131. 25:52

     Like, I grew up in Texas, so it's kind of like prerequisite to know Spanish. Um, but the, uh, but I'm going to share again. Um, [notification sound]

132. 26:05

     Hey, Gemini, could you tell me what you see on the screen?

133. 26:19

     Or hopefully... Let me see. No, I don't think it-

134. 26:27

     Claro. Veo una página de resultados de búsqueda de Google con muchas imágenes de ladrillos de juguete, sobre todo de la marca Le-

135. 26:35

     I think you have to click the microphone again. That's not Spanish, that's [REDACTED:origin]. Oh. No.

136. 26:38

     Que no-

137. 26:39

     It's the... Mm. There you go. [laughs] Like, the, uh... So one of the things, one of the things that you can do, so you see that I've modified the system instructions, um, but, uh, like, do you speak a specific dialect of Spanish?

138. 26:54

     Castilian. Castilian Spanish. Castilian Spanish? Okay. [laughs] Excellent. So, um, I removed the system instructions, and I should be able to do this just within the span of conversation. Um, so, [notification sound]

139. 27:11

     hey, Gemini, could you tell me what you see on the screen, but could you do it in Castilian Spanish?

140. 27:18

     Por supuesto. Veo una página de resultados de búsqueda de Google llena de imágenes de ladrillos de Lego. Hay de muchos-

141. 27:26

     That's better

142. 27:26

     ... colores y tamaños.

143. 27:27

     Oh, excellent.

144. 27:28

     Y algunas muestran construcciones ya hechas.

145. 27:30

     Very, very cool.

146. 27:31

     ¿Estás buscando algo en específico?

147. 27:33

     Yep. Awesome. So, so you can, uh, you can ask within the span of the conversation, you can, um, modify the system instructions to select different languages or different dialects.

148. 27:45

     Um, and then again, the same thing, if you click Get Code, um, it gives you the code that you would need to use to replicate whatever you just did, um, so the model name, any configuration settings, um, as well as any tool calls that you might invoke.

149. 28:00

     Um, and it works with sharing your screen. Um, you can kind of interact with your screen as you share it. Um, but it also works with video feeds. So you can say, [notification sound]

150. 28:12

     um, "Hey, Gemini, how many fingers am I holding up? And also compose a poem about me."

151. 28:21

     Well, I see two fingers up, like the peace sign, and here's a little poem for you: With [REDACTED:physical_attribute] and an open heart, you come to learn, to play your part.

152. 28:32

     The camera's focus, moments start, a creative spirit, a work of art. How was that?

153. 28:38

     That was very sweet. Thank you. Ain't the... And so the, uh, the models are able to, to kind of interact, uh, to view video feeds, to, to view the, the screens, um, and you can stitch them together in your own projects.

154. 28:56

     Um, just taking a look, I'm going to enable Firebase real quick. Um, so it should be setting up the database for that app that we were building. And then the other thing that I wanted to show is something called Genie.

155. 29:08

     Um, so if you haven't heard of Genie before, this is a world model that DeepMind has created. Um, it's actually a composition of models. Um, so, uh, NanoBanana, Veo, a bunch of Gemini used for prompting, um, and it's stitched together into a system that allows you to describe something, um, uh, like a, a game, an environment with

156. 29:30

     a character, um, that you can interact with, that you can, uh, play this game,

157. 29:36

     uh, and, and sort of do it pixel by pixel. So it doesn't generate a Unity environment, it doesn't generate an environment for Unreal Engine, um, it just generates this, uh, kind of frame-by-frame experience, um, for anything that you can imagine.

158. 29:49

     And co- so it could look something like this volcanic landscape, where you're navigating with a little rover, um, with your arrow keys, um, or something like this jet ski, where, like, you hit a light, um, and it feels like the physics is happening or like the physics is, uh, like responding in a real way.

159. 30:07

     Um, if you knocked that light into the water and then circled back around, it would persist throughout the duration of your 60-second experiment, um, but there's no physics engine behind the scenes.

160. 30:18

     Um, and then even things like experiencing a hurricane in Florida, you can start with a static image or, like, a family photo, um, and see how, uh, see how some of these things get created.

161. 30:31

     Um, but for this, I am going to go back to my other browser. Um, I'm going to pull up Project Genie. I'm going to click Explore Now

162. 30:43

     Then I'm going to say something like, uh, a world, um, maybe a, a Regent's Canal on a sunny, um, day. Um, but with dolphins,

163. 30:58

     uh, swimming in the canal and, uh, all of the boats have pirate flags, um, which is hopefully not part of the training data. And then, uh, the, the, uh, sort of character description would be, um, you know, something, again, kind of atypical, maybe a pink sparkly squirrel,

164. 31:23

     um, uh, with, um, a pink sparkly squirrel with, uh, purple, um, purple feet and, uh, um,

165. 31:37

     uh, a pirate hat. Uh, and then create the sketch. And what should happen is that it uses NanoBanana to, to kind of ideate on that first frame. It'll show it to us to, to make sure that it looks consistent with what we had described.

166. 31:53

     Um, uh, clearly, like Regent's Canal is getting overtaken by pink sparkly squirrels with pirate hats. Um, and then what, uh, what we should see is kind of this first iteration and then a playable world that we can interact with for just sixty seconds, at least for the, um, at least for the first implementation that we've released to

167. 32:12

     the public. You're able to access Genie-3 through, uh, an Ultra subscription in some parts of the world. Um, not every part. Um, but, uh, but hopefully, hopefully I haven't overbooked my, uh, my GPU or TPU allotment or allocation.

168. 32:29

     Um, if we have, uh, we can take a look back at the, um, we can take a look back at Shelf Scan. It looks like the model is doing the work of creating the, creating the Firestore rules for us.

169. 32:44

     Oh, there we go. So pink sparkly squirrel, um, pirate hat, Regent's Canal, dolphins. That looks pretty good. Um, so let's go ahead and create this world. Um, you can use the arrow keys to move around the, uh, or the WASD keys to move around and then the arrow keys to change the perspective.

170. 33:02

     Um, and then we should also be able to use the space bar to jump. Um, but let's see, let's see how this works. Uh, oh gosh. Whoa, whoa, whoa, squirrel.

171. 33:15

     Um, and so it looks like it's walking on water, this squirrel, um, or like, uh, hopping along. Um, uh, you can also jump. Um, so jump on top of the boats.

172. 33:26

     Uh, you can see the, the little bicycles. Um, you can see some of the people along the way. Um, and it does look like all of these boats on Regent's Canal, um, have, uh, have pirate flags, uh, and dolphins, um, that are not currently moving, which is, which is pretty wild. [laughs]

173. 33:46

     Um, and then if I click, uh, space bar, you can see the squirrel jump. Um, it looks like it doesn't realize that Regent's Canal has pretty deep water, so I probably should have specified that in my prompt.

174. 33:57

     Um, and then you can also see it, uh, attempt to jump, um, uh, attempt to jump into the, uh, onto the sidewalk, uh, and do, do its work. Um, so it's wild to be able to see the things that you can create, um, the different, uh, the different experiences that you can construct.

175. 34:19

     Um, and again, it's even more bonkers to me, um, that each part of this is being generated dynamically, um, as you're moving your arrow keys around. Um, so it creates a video at the very end that you can download, that you can, uh, kind of see and review and interact with.

176. 34:36

     Um, and, uh, this is again, just using Genie-3 and this composition of models as opposed to a singular model. Um, other world model building companies, so things like World Labs, um, that's Fei-Fei Li's company, are taking a slightly different approach.

177. 34:50

     They're, they're building out actual Unity environments or Unreal Engine environments. Um, none of these things are stored as 3D game assets. Um, they're just kind of like raw pixels, um, that are incorporated into the, into the experience.

178. 35:07

     Cool. So going back to AI Studio, it looks like the app is still getting, uh, the app is still getting cooked. Um, let me, uh, and hopefully, hopefully we'll be able to see it.

179. 35:19

     Usually whenever it starts working on config files, that means that it's almost done. Um, I also really love looking through it to see, um, like what its approach towards the, the construction of the Firestore rules were, um, what its approach towards prompting the model might be.

180. 35:36

     Um, it looks like it's confirming the app. Um, and then, uh, once it's done, it should make a little bitty noise, um, to, uh, to talk through, um, to talk through the, the app itself.

181. 35:50

     Um, so it looks like it's ax-asking to allow my camera. Um, so I'm going to allow. Um, we have this Shelf Scan AI experience. I'm going to sign in with Google

182. 36:04

     with my personal account. We can see that it's connected to Firestore. Um, I'm going to find very quickly a bookshelf, um, with books on it.

183. 36:19

     Let's see. Those don't look like real books because many of them are hanging, like suspended, uh, below the shelf. Um, like AI, AI image generation makes it hard.

184. 36:34

     But this is, uh, this one looks decent. So like let's save this image. Um, looks like somebody has a whole bunch of cooking books. Uh, I'm going to upload the photos, so this pixel photo, um, and hope that it can understand, uh, hope that it can understand web pages or web- dot WEBP, um, rather format.

185. 36:59

     Um, it's gathering data via Google Search, um, and then the books should be populated in the library, hopefully. If not, we can try with a JPEG.

186. 37:12

     But it does have pretty good branding. Um, it was able to identify the seven books, it looks like, um, or at least, like, it identified the books. Let me try, uh, let me try again, but just with a JPEG image.

187. 37:26

     Um, so I'm going to just take a screenshot. So same image, just stored as a screenshot.

188. 37:38

     I'll find in here. So Desktop, Screenshot at four seventeen.

189. 37:50

     And then identify books. And if not, we can try fixing the errors as well, um, to see what might have been going wrong.

190. 38:11

     Oh, so, so it looks like the, uh, there are insufficient permissions, um, for saving in Firestore. Um, so, uh, so it looks like it's going ahead and fixing, fixing those issues.

191. 38:27

     Um, but as it does, um, you can also see that you can log in, you can log out, you can share the app, um, so you can specify who has access to it, share full screen.

192. 38:38

     Um, and one of the things that I also really, really love about AI Studio, um, is that, uh, it's figuring out, uh, where the files should be modified in order to make those changes.

193. 38:50

     Um, so it's figuring out the validation logic. It's figuring out that the, that the size, um, might have been the culprit, um, so the, the image URL size. Um, and then it's figuring out where it would need to modify in order to, uh, in order to make that change.

194. 39:08

     Um, so it looks like it's in the Firestore rules. Um, some of the other nice things about AI Studio's build feature is that we have, um, an app gallery.

195. 39:19

     So if you need to get inspired for some of the apps, uh, that are using our models, you can kind of review them. Everything from Lyria for music generation to, um, multiplayer experiences with games.

196. 39:31

     So you can see this multiplayer Neon Snake or this Mandelbulb Explorer. Um, you can see a design with NanoBanana, so, like, being able to change and modify images. Um, you can take a look at this sick, uh, MediaPipe example, um, which allows you to, to kind of play this game where your hand is detected, um, but you

197. 39:54

     take this little, uh... And everybody can find out that I play this game really poorly. Um, uh, uh, how to, uh, s- uh, move one of the little, one of the little marble dealums.

198. 40:10

     Um, oh gosh, I am really horrible at this game. Um, but the, uh- [chime]

199. 40:19

     And then it also sounds like the other app, uh, finished getting created. Um, perfect timing. Um, but if I, uh, if I sort of upload the image, let's try to identify the book again.

200. 40:36

     Fingers crossed. Yep, and then it automatically populates all of the books. So they all got, uh, they got, all got cataloged with the date, the, the type of book, the details associated, um, the name of the book, the author, even though some of those were not,

201. 41:01

     uh, available in the, in the spines of the books that I uploaded. Um, if I log out and then log back in, um,

202. 41:12

     it keeps all of the, the books that I had added persisted. Um, and if I wanted to share this with all of you, because clearly I want to know what all of y'all have on your bookshelves, um, I could copy this link and then do a QR code generator.

203. 41:28

     Um, and, uh, if you use this QR code, you should be able to access the app that I just created, upload your own bookshelf images, and then have them cataloged, um, to your own, uh, to your own apps.

204. 41:44

     Um, next feature would be, like, finding a way to give my friends the ability to request them. Um, because every time I give my friends a book, um, they, they have a tendency to keep it, which I understand, but is also, like, exhausting.

205. 41:59

     I have so many copies of Infinite Jest out in the world. Um, but the, uh, but this is the, um, like, uh, just kind of a whirlwind tour of some of the things that you can do in AI Studio, some of the new features that we've added, the new models that we have available.

206. 42:14

     Um, and with that, I am going to welcome my colleague Guillaume, um, who is going to tell you all about our generative media models. So everything from music generation to image creation, image editing, to video generation.

207. 42:27

     Um, and it should be a fun time. Um, so thank you so much. Thank you for coming. Come. [audience applauding]

208. 42:41

     Hello, everybody. Um, this is the right one.

209. 43:04

     You're gonna be awesome. And then do I have a... Where did I leave my pen?

210. 43:07

     You can, you can show my screen now. [microphone feedback]

211. 43:11

     Okay, good. Um, so as, as Paige said, we are go- it's going to be the same talk as I did this morning, the same workshop. Uh, we have slightly less time, so I'm gonna go faster on some things and maybe not run things, uh, in real time.

212. 43:25

     But you, like, y- the, the content I'm going to show is on this, uh, this thing. So if you just... You can, you can just open it and, uh, and run it yourself at the same time.

213. 43:35

     Um, so, uh, I'm going to talk about generative media. Um, generative media is everything about creating, um, images, videos, uh, or text, uh, um, like sp- spoken text and, uh, and things like that.

214. 43:54

     I think I would s- I would fit Genie into gen media as well. Um, so yeah. Now... And we have plenty of models like that at DeepMind, so let me, let me go through all of them.

215. 44:06

     Um, so very quickly, my name is Guillaume. Uh, I've been at Google for six years now, two years at DeepMind doing, uh, developer advocacy, uh, mainly on most, on most of the Gemini model until last year and since- And this year I'm focusing more on the, on the gen media models because that's the funniest model to play

216. 44:24

     with. Um, I've been, I've been working in the video game industry before, um, and that's how I joined Google initially.

217. 44:33

     So, uh, yeah, open your phones. Everything is a media. I already said that, basically. Um, Paige talked a bit about our vision of, uh, what, um, world models are.

218. 44:46

     Um, my, my definition of a world model is something that can, uh, ingest as many modalities as, as it, as it can and understand them, uh, kind of for five senses and to, uh, to also talk or output things in different modalities as well.

219. 45:00

     So that really has been at the core of D- DeepMind vision of what, uh, generative AI should be. Uh, the, the first Gen, um, Gemini model, like it was only two years ago, but it seems old, but it was only a text to text, uh, to text to text model.

220. 45:17

     But actually, behind the scenes, it was a already, um, a multimodal model because... And that, and you could have been sending it ma- images, but it was blocked. It was, they did, uh, for testing reasons.

221. 45:28

     I don't, I don't remember why. But basically, they blocked it in the model with, uh, with post-training. And then when it released a few months afterwards, we released 1.5.

222. 45:37

     This one, the new... The novelty was that it was, uh, the first multimodal model. Uh, sometimes when you were giving it a picture, it was, it was answering, "I'm sorry.

223. 45:46

     I'm just LLM. I can't do anything with images," because some of the, that training was still, still in there. Um, but basically, yes, we, we want to create those models that can understand all of the physics of the world from videos, audios, and, and all of that.

224. 46:01

     Uh, and the only reason why we have so many different models is that it's easier to ship one models that only does video and one models that only does, uh, images and one models that only does text than to have like one models that does everything.

225. 46:14

     And then it m- it becomes a problem every time we want to update one of the feature, we have to release a whole new model and everybody has to, to convert and all.

226. 46:23

     So that's, that's just the, the reason for that. Um,

227. 46:27

     everybody knows us for the Gemini models. We have lots of other models. I will go over... I will skip that for now. Um, quick, um, quick timeline. Uh, we release things all the time.

228. 46:40

     Uh, I think Paige says it, said it earlier. On, on average, we are releasing a new model or new capabilities every five days. Uh, that's just the gen media models.

229. 46:50

     Uh, there's all of the other models on top of that. And if you add all of the changes we are doing in AI Studio, all of the pricing things, like small features here and there, we are releasing like two to three new th- new things every week.

230. 47:03

     Um, and, um, so yeah. It's, so it's basically, it's, it's hard to keep track and, uh, like, it's even hard for us to keep track of everything we have to offer.

231. 47:14

     So I know that's, that for people like, uh, like you, it's even harder because you also have to look at what the, the competition is doing. Um, so that's also why we are doing those, those, um, those talks.

232. 47:26

     Very quickly, uh, the, the updates on NanoBanana. We released NanoBanana 2 a few month ago, um, in

233. 47:35

     Jan- end of January, the beginning of February. Uh, the main thing is that you can output different, uh, as- aspect ratio and, uh, and sizes. Um, it adds search grounding, so that's basically how I...

234. 47:48

     Eh, you don't see my screen anymore?

235. 47:53

     Yeah.

236. 47:58

     Why? Okay, just imagine in your head. That's, that's what media apps are for. Um, so the, the main thing about...

237. 48:15

     Oh, thank you. NanoBanana, uh, Pro was that it w- it adds search grounding, so you could ask it to search for the internet, and that's how I made this image.

238. 48:23

     Like, just look for what you can find about my footprint on Google and make, uh, an image about me, which is kind of scary in a way. Um, and but the new things with NanoBanana 2 is that you can do the same thing with image grounding.

239. 48:37

     So you can s- you can, uh, talk about a specific places, a specific, like bridge like that, and it will look for, for an image on, uh, on the web and then create image based on that.

240. 48:49

     So that's without grounding, and that's without grounding. So you can see that it's, uh, looks a bit more than the, uh, than the normal building

241. 48:58

     We also have Veo. Very quickly on Veo, the main novelty in the past, uh, week is that we released Veo 3.1 Lite last week, uh, which is the, the cheapest model for generating video we have.

242. 49:11

     So it's only five cent per, per image. So it... Which is way, way cheaper than what Veo 3 was a year ago. So the idea is that you can use that to prototype, test your prompts and so on, and then, uh, if you want better quality, then you can move to the, uh, to the better models.

243. 49:27

     Uh, and then Lyria 3 is the, the, the, the coolest model this year. Uh, it's our music generation model. Uh, so you can either generate, uh, 30-second songs or full songs of three minutes, uh, with the lyrics, lyrics and, on all of that, and I will, I will show you some demos afterwards.

244. 49:45

     Um, and I've... As far as I know, w- we are the first one to offer such a music generation models through API, so that's really cool for any kind of, uh, agentic or whatever workflow you, you might have.

245. 49:58

     So if you want to be woken up with a song about the latest news every morning, you can do it now. Um, and also another one very quickly, uh, that I love but nobody knows about, is that we have actually another Lyria model that is Lyria real-time, and this one is basically a live model, so it's creating

246. 50:18

     music, uh, indefinitely. And, and you can just prompt it differently, so it's just going to change what you are... uh, the kind of music it's generating in real, in real time like a DJ.

247. 50:30

     So it's, uh, it's pretty fun to play with, but, uh, some hope, like, people don't know about it. Um, as I said, I'm going to... Because it's, it's meant to be a workshop, so the idea is that you test things yourself.

248. 50:44

     Uh, so you can open this, uh, this link, and that will show you the, the content I prepared. Um, while you take pictures and all, uh, just one disclaimer, it's gen media models, so they are all paid models.

249. 50:57

     So they... Like, running the notebook actually costs a bit of money. Um,

250. 51:04

     the video, the, the, the video generation is going to be the most expensive thing, so you can just skip that. Uh, the rest is pretty cheap, so I think you can run the whole, the whole, uh, notebook for something like one, one euro, so it should be, uh, should be fine.

251. 51:18

     But just I prefer to, uh, to be, uh, to be clear with everybody. So let's, let's move to it. So

252. 51:27

     what's the idea of this workshop? So the, the idea was to showcase all of the gen media models with this, uh, this example of we are going to take a book, uh, and I'm going...

253. 51:38

     we are going to take a book from an, uh, an open source library. So, uh, we are allowed to use it. And then we are going to create, uh, images to illustrate, uh, what the characters look like, what the...

254. 51:49

     and what's happening in each chapter. And then we are going to move to the other gen media models. So, so creating videos, uh, about, uh, about the chapter, and then creating musics and having, uh, Gemini, uh, tell us about, uh, what's happening in the,

255. 52:05

     in the chapter. So up... This is just set up. You need to install the SDK. You need an API key. I guess you ca- you could have guessed that.

256. 52:13

     Uh, I'm initializing the client. Uh, and something that, uh, I didn't know about until recently is that you actually have a way when initializing the client to, uh, implement some retry system.

257. 52:26

     So that... Which is kind of, uh, very useful, uh, nowadays with whenever you use NanoBanana too because, uh, especially when the US wakes up, it's becoming harder to, uh, to get, to, uh, to get something out of it.

258. 52:42

     Uh, and we're working on a, um, yeah, we are working on getting more capacity, but still, uh, it's... having some kind of retry system alway- always- always helps. Um, so import, and then we are selecting the models.

259. 52:56

     And I usually when, when I create content for... that are using paid models, I usually have some kind of checkbox that, uh, you can... If you, if you're opening, it's already checked.

260. 53:07

     It should be, it should not be, but I made a mistake yesterday evening, uh, so because I don't want people to run the models and have to pay, uh, especially the, the Veo, [chuckles] the Veo notebook, for example, example, like, cost something like $20 to run, so I don't want anybody to run it by mistake.

261. 53:24

     Um, just for the sake of the, of the demo, I'm also limiting the number of, uh, characters, images, and chapters images we are creating just so that it's, it's faster to run and all.

262. 53:35

     So the book is, uh, named "The Wind of the Willows" from Kenneth Grahame, and I'm... I took it from the Project Gutenberg libr- libraries, which is an open source library where you can download open source books.

263. 53:48

     For some reason, it doesn't work, uh, since I've been in the UK. So there has to be something about this library not being available in the UK. Um, but if you run the notebook, it should work because it's very likely the server is in, in another country, so it will be able to recover the book.

264. 54:05

     Um, so, so what I'm doing is that I'm just downloading this, uh, this book here, like a, like with a, with a rougets URL. And, um, I didn't say, I didn't talk about that and Paige neither.

265. 54:20

     So, uh, very quickly, uh, because everybody's always asking that question to us. We have a specialty at Google, is that we are always creating multiple projects that are doing the same thing, multiple products.

266. 54:32

     And we are doing-- we did the same with messaging apps. We are doing the same with, uh, with Gemini. So that's can, that can be a bit confusing, so just very quickly to, uh, to clarify things.

267. 54:44

     Um, so if you, if you look at the graph, like, basically we have the consumer's apps. So on the, on the, on the left, it's the apps that are for everybody.

268. 54:54

     You can do plenty of things with them. You can, uh, you can ask any question to Gemini, you can generate images and all. But as a developer, like, I guess everybody in the room, you can be kind of frustrated because you don't, uh, you, you can't really access the parameters, select the exact models that it's doing.

269. 55:10

     You can't... It... You know that it's, it's likely doing cool tool, uh, cool, uh, tool calls, but you don't know which one it's doing. So that's, uh, that's nice for the, for the broad public, but it's, uh, it's not really for us.

270. 55:23

     Um, on the other end, we have, uh, Vertex AI, which is our enterprise, uh, offer. Uh, that's the exact opposite. You have a lot of control. You can decide in which data center your, well, your prompts are going to be run.

271. 55:36

     So especially in Europe, a lot of people are looking to be certain that, uh, the data is not going to, uh, to leave Europe. So that's, um, that's kind of convenient for that.

272. 55:46

     But it comes with, uh, like, great responsibilities as well, so it's kind of hard to set up. So I usually only recommend people to start with Vertex if they are already using GCP or if they have a team of, uh, DevOps who can do the setup for them.

273. 56:02

     And, and in the middle ground, we have, uh, AI Studio and the developer APIs that we made for developers. But basically, the idea is that it's as easy as possible to start to play with the model and, and do stuff w- by just creating an API key and then using it.

274. 56:18

     Um, and as part of making it as easy as possible, uh, yeah.

275. 56:26

     As part as making it as easy as possible, we have this, uh, client file upload API that is basically a way to not have to set up buckets to store your, your files.

276. 56:37

     And, uh, um, we are going to u- I'm using it in this example so that we are uploading it to this... Behind the scene, it's creating a bucket, but you just...

277. 56:45

     The, the idea is that you don't have to bother with how it works. You just upload the file, and then you can use it in your, in your Gemini prompts afterwards.

278. 56:54

     Um, I'm also going to use, uh, structured outputs because I want to be certain about what exactly the model is going to, to do because I, I'm going to have it, uh, to, to ask it, uh,

279. 57:06

     to generate a lot of prompts. So I want to know exactly that, uh, it's just re-returning the prompt and not some, like, uh, introduction or, um, text or something.

280. 57:16

     Like, like the, "Oh, yes. Sure, I can do that." And then it's, it's hard to pass, so that's why we are, um, having this, uh, this structured output. And I'm using chat mode, which is basically a way to, uh, um, to chain requests to, uh, to the model so that you can, uh...

281. 57:35

     It, it basically saves the history and resend history so that the models knows what happens before, which is quite convenient in this case because we don't want to upload the book all the time.

282. 57:43

     We just want to... the, the book to be in the history and in the, in the context so the models can ask, uh, can answer new questions and ge-generate new prompts about the book.

283. 57:52

     And for the images, it's going to be quite the same. By, by using this system, we will have all the previous images that are going to be in memory, so the model will be better at, uh, keeping the consistency of the characters and better...

284. 58:06

     and, uh, and the consistency of the style.

285. 58:09

     So, uh, I'm just giving it the book. Here's the book, ill- uh, to illustrate with NanoBanana. That's all. Then I'm defining a style. So I, I went, I, I made it so that you can just do nothing and the, the model will co- will come up with the style, but then I wanted to try something else earlier.

286. 58:26

     So it's going to be a dark fantasy style with black and white background and colored characters. Um, and basically, I am also adding some system instructions for, for NanoBanana because I, uh, from my earlier tests, I think it's better with NanoBanana 2 and Pro.

287. 58:42

     But with the first NanoBanana, um, whenever I was asking its images about books that were in the portraits, uh, format, it, it tend to cr- to thought that it was, uh, book covers that I wanted, so it al- it was always adding titles and things like that.

288. 58:58

     So I had to add some system instructions to make sure that, uh, I, I don't want borders. I don't want titles. I don't want description. I, I... It stays time- family-friendly, which is likely to not be very aligned with dark fantasy, I guess.

289. 59:12

     Um, and, uh, and no panels as well because I don't want, uh, I don't want a comic book. I want just an image for, for each chapter. And then basically, I will use that chat to ask the model to describe each of the main characters.

290. 59:27

     Uh, I initially wrote only the adults be- because at some point NanoBanana could not generate, uh, kids' images in Europe, but that's not the case anymore, so we could remove that part.

291. 59:38

     And I, I'm getting, uh, this, this list of characters and a prompt for each of them. And then I can just go over, uh, each of the, each of the prompts and ask NanoBanana to create an image that...

292. 59:53

     to illustrate that, um, that, uh, that prompt. So that's how I initia- I initialize the image chat. So it's, it's going to be a separate chat. Um, and then I'm sending all of the, all of the, uh, the prompts to create, to, to be created.

293. 1:00:08

     So we can see that, uh... So that's, that's Mole, the main character. So the... As, as requested, the background is black and white, and the character is colored. Um, not very dark fantasy, but yeah.

294. 1:00:23

     Um, that's, uh, the water rat. That's the toad.

295. 1:00:32

     Uh, the, the, the badger. The, the... Yeah, the badger. And, uh, that's, that's it for now. And then I'm going to do same thing again. Like, uh, now you, you have the full book in your, in your history, so give me prompts for each chapter to illustrate them.

296. 1:00:48

     So up I get a prompt for each of those, uh, chapter. And I'm creating images for, for each of the prompt.

297. 1:00:58

     So here we have the, the characters having a picnic next to the river. Then another one's on the road and something is happening. Um, and here in the forest.

298. 1:01:10

     That's, uh, that's the third chapter. And if you look closely, you can see that there's, there was a problem here because the toad was not, uh, represented using the, uh, the character, uh, the, the images that we, we created before, um, likely because the prompt was not clear that it was exactly the same character.

299. 1:01:29

     Um- And so which is why, I mean, I have a second way of doing it, which is actually cleaner. And if you were to do that, to do that at scale, that's, that's how I would do it.

300. 1:01:39

     Um, and in... I'm creating another type of structured output, which is, which is a chapter, which is- which has a name in the prompt as well, but also a list of characters that are, that are meant to appear in the, in the image.

301. 1:01:53

     And so I'm, I'm running- I'm asking again the model to, to come up with something, but this time it's, it's giving me the list of the character. And that way I can, um, basically here, um, for each, uh, for each image that I'm going to create, also give the reference images of what the character should be looking

302. 1:02:11

     like. So that it only has that, those images in the, in its context instead of having all of the possible images that we did before. In this case, we have something like five cha- five characters, so it's okay.

303. 1:02:22

     But if you are in a real book with like 40 characters, that, that wouldn't be sustainable to, uh, to expect the models to actually manage all of the, all of the context perfectly.

304. 1:02:32

     Um, and, and honestly, if I was to, to do it at scale, I think I would even create more than one image for each character. Maybe one, one image from the front, one from the back, one from each side, so that...

305. 1:02:44

     And, and, and then I would pass exactly the one that I, that I need in the, in this case. So we can see that the image are-- looks kind of the same, but, uh, this time the, the toad is the right one.

306. 1:03:01

     Uh, yeah, this one's very lookalike. Um, so now we can move, uh, to the next step, and we can use Veo to create, like, small videos based on those images.

307. 1:03:12

     So I'm, I'm using the, the Veo, the largest model because it's, uh, because I don't have to pay, honestly. Honest- uh, otherwise, I would use the smallest one. Um,

308. 1:03:24

     but basically, we are going to do the same thing. We are going to ask the model to generate a video using the Veo model. Uh, the prompt, I'm going to use the same prompt as the one that we used to, to generate the image.

309. 1:03:35

     And I'm, I'm passing the last generated image as, uh, as a starting, um, frame. And then I want a portrait, and I want 720p because it's going to be smaller.

310. 1:03:48

     Uh, and here we go. So I actually haven't checked what the sounds looks like because I-

311. 1:03:54

     Stand back. You shall not pass. Leave this place, little ones. You don't belong here

312. 1:04:03

     I think it's quite good. Um, but sometimes it's not, not very good because the model doesn't like... The, the prompt was just about creating the still image and the, and, and nothing-- and the model doesn't know what exactly is expected to be happening afterwards.

313. 1:04:18

     So a better way of doing it is actually to reuse the chat mo- the chat to ask the model to come up with the, um, with the, uh, with some explanations about what's happening after the image.

314. 1:04:31

     And I'm also passing it the image again so that it knows exactly which part of the, of the chapter we are talking about so that it can come up with the, with the right prompt.

315. 1:04:41

     Um, and so it, like, it came up with this, uh, this prompt, "Mole shivers and clutches his scarf in terror," blah, blah, blah. Uh, "Water rats bravely draws his cutlass and steps forward to protect him."

316. 1:04:54

     So up we can see how it goes. [dramatic music]

317. 1:05:07

     And somehow, like, it's, uh, like the, the, the... Like, in this case, the results are not really as good. Uh, and, uh, yeah. And every time, every, every test I've done, somehow when I use the same prompt, it's, it adds the, it add text.

318. 1:05:22

     And, and not when I, when the- when it generates another, another prompt. B- but that it should not have any imp- impact because in both cases, I'm just giving a prompt and giving an image.

319. 1:05:32

     But I don't know why it's happening like that. Um,

320. 1:05:37

     um, and then we can use the new Lyria models that we generated, like, that we, um, shipped last week, uh, to, uh, to create songs for each model. So s- still the same, the same trick again.

321. 1:05:49

     We are going to ask the model to create the, the prompts for that. Uh, and then I'm going-- we are going to use generate content with the Lyria model, uh, ask it to create, uh, the song, uh, thanks to the chat, the, the prompt, and that's basically it.

322. 1:06:05

     So let's see. The first, uh, the first song should be orchestral acoustic folk music, peaceful flowing acoustic guitar and, and flute duets. Let's see. [gentle music]

323. 1:06:26

     I think it fits. Then the second one, the open road, is, uh, jaunty, adventurous, rhythmic with fiddle and acoustic bass. [upbeat music]

324. 1:06:46

     Fits as well. And then the last one is suspenseful, creepy melody with staccato, pizzicato strings. [tense music]

325. 1:07:07

     And I think it fits as well. And I, I, honestly, I think the, the model is really good because I... It's not very often that you can do those kind of demos without actually checking what's, what it's going to be, because I ran the, the notebook before, but I didn't check what the, what the music were going

326. 1:07:23

     to be looking like. And that's, uh, so far it worked all the time. Um, one of the things that I saw-- I forgot to say about all of that is that the way we are training our model internally is that a lot of the training data for the gen media models is actually made using Gemini.

327. 1:07:41

     So that's the reason why Gemini is quite good at generating those prompt for the gen- uh, the, the generative media models because it's already trained on, uh, on understanding what Gemini is ask- is asking for.

328. 1:07:53

     So that's, uh, that's what makes, uh, the Gemini very good at, at generating those, um, uh, those prompts. And then, uh, the last, the last model that I wanted to show is a text-to-speech model.

329. 1:08:06

     So I, I'm pretty certain that you all have, have heard of this one because, uh, everybody loved the, uh, integration in NotebookLM where you could create a, um, a, um, a podcast based on, on your documents, and that's basically the same model that, uh, you can create...

330. 1:08:24

     You can ask it to, to talk, uh, to tell, read, and, uh, what the text you, you give it, and you can have it... You can have two characters with different voices and, and so on.

331. 1:08:33

     That's, that's what makes it, uh, like, nice to listen to. But what I wanted to show you is that, uh, there are actually tricks to, um, to have more than two voices actually.

332. 1:08:44

     Um, and, uh, and the reason for that is, uh, I- I think Paige, uh, showed that a, a bit, but you can, uh, when you're using text-to-speech or live, you can actually do a lot in the prompting to make it speak in, in different ways with different accents, with, uh, uh, and so on.

333. 1:09:00

     And we are going to use that to our advantage to basically have more than two, have more than two voices in, uh, in the same, um, in the same generation.

334. 1:09:09

     So what I did, be- because I was lazy, I would not do that if I was... I wanted to do that at scale. But I basically ask it to extract one of the dialogue from the book, and then to rewrite it as, um, as a transcript of, of a play, and to replace the name of the character

335. 1:09:26

     by nar- narrator if that's just a narrator talking. And for all of the others, is, are just going to be named character, but with a specific style for each of them.

336. 1:09:38

     And if the same character comes back, it's, it's going to reuse the same style so that it's going... Like, the same characters are going to have to, to speak the same way, have, and have the same voices.

337. 1:09:49

     Um, and then I'm defining the voices. So the na- narrator is going to use a Sulafat voice, whatever it is, and then the, the... all of the others are going to share the Fenrir voice.

338. 1:10:02

     Um, and then I'm just asking it to, uh, to, uh, to generate the thing. And just one trick here with the TTS model. You always have to start with "Read this," or, uh, "Tell me this," or whatever.

339. 1:10:16

     If you just send the text, it's, it's going to ignore it for some reason. So, uh, you, you need to prompt it to read the, the text all the time.

340. 1:10:24

     Um, but that's also where you can add some, uh, some context and, like, the equivalent of what would be a system instructions like, "Read this in a, a scary way," uh, or the, the characters are very excited.

341. 1:10:35

     So that's where you can, you can add those details as well. Um, and so you can see the, the, the text that we got is narrator is saying that, then the first characters is going to talk very fast-paced and with a British accent and, uh, posh British accent.

342. 1:10:52

     Um, and then, and so on, and you can see that the same, uh, the same way of speaking is, is coming often. So it's when, uh, the two characters are going back and forth.

343. 1:11:01

     So let's see how it goes.

344. 1:11:03

     Small, neat ears and thick, silky hair. The two animals stood and regarded each other cautiously.

345. 1:11:11

     "Hello."

346. 1:11:12

     "Hello."

347. 1:11:13

     "Would you like to come over?"

348. 1:11:14

     "Oh, oh, it's all very well to talk." He spoke rather pettishly, being new to a river and riverside life and its ways. The second animal said nothing, but stooped and unfastened a rope and hauled on it, then lightly stepped into a little boat which had not been observed.

349. 1:11:35

     It was painted blue outside and white within, and was just the size for two animals, and the first animal's whole heart went out to it at once-

350. 1:11:44

     Yeah

351. 1:11:45

     ... even though he did not yet fully understand its uses. The rower sculled smartly across and made fast. Then he held up his forepaw as his guest stepped gingerly down.

352. 1:11:57

     "Lean on that."

353. 1:11:58

     Yeah.

354. 1:11:58

     "Now then, step lively."

355. 1:12:00

     To his surprise and rapture, he found himself actually seated in the stern of a real boat. "This has been a wonderful day. Do you know, I've never been in a boat before in all my life."

356. 1:12:15

     "What? Never been in a... You never... Well, I... What have you been doing then?"

357. 1:12:21

     He was quite prepared to believe it as he leant back in his seat and surveyed the cushions, the oars, the rowlocks, and all the fascinating fittings, and felt the boat sway lightly under him.

358. 1:12:33

     "Is it so nice as all that?"

359. 1:12:37

     "Nice? It's the only thing. Believe me, my young friend, there is nothing, absolute nothing-"

360. 1:12:43

     It is the fourth British accent

361. 1:12:43

     ... "half so much worth doing as simply messing about in boats. Simply messing, messing about in boats. Messing-"

362. 1:12:52

     "Look ahead." It was too late.

363. 1:12:55

     Well, we can stop here.

364. 1:12:56

     The boat struck-

365. 1:12:56

     But ba- basically, you couldn't guess that it's actually using the same voice for the two characters. They are so much different. And so that's, that's, that's, that's a really, actually a really cool way of, like, uh, uh, creating those discussions with multiple characters so I can use it.

366. 1:13:12

     And, uh, the last thing is that, uh, as we said, like, the Gemini models are multimodal by, by default. So I was, I was using a book, uh, but you can also send an audiobook or movie or some, like, not too long movie.

367. 1:13:26

     But you could, you could do the same things with, uh, with using all of the multimodal input possibilities to, uh, to have Gemini create things, uh, for, uh, to illustrate other, um, other kind of modalities as well.

368. 1:13:40

     Um, how much time do we have? Um-

369. 1:13:44

     I, I said I was going to show you the, uh, Lyria real-time model. So this is a very good, a very cool example that... Oh, no. Stop. Stop. I'm going to show you something be-before.

370. 1:13:57

     Uh, Paige made some, uh, example of cool things that you can build in AI Studio. So one of the thing I wanted to try was basically to do the same thing as what I do in the, in the notebook, which is kind of lengthy and quite complicated stuff.

371. 1:14:12

     So what I did is I basically copy-paste it. "Can you build an app that, uh, that illustrates a book using Genmedia models as described in this Python notebook?" And I just paste all of the content of the notebook and, and build it, and we'll see how it goes.

372. 1:14:28

     Um, it can take quite a long time. Like earlier, it took fifteen minutes, so we... You might not want to wait for that. But, uh, we ca... I can show you Space DJ in the meantime.

373. 1:14:38

     So, uh, as I said, the model generates, uh, music in real time, and this is, uh, this is a, uh, a demo where they made a, a star, like, a universe of stars and the, the planets are prompts.

374. 1:14:51

     So if we go closer to some planets, it should start playing music. [ethereal music]

375. 1:15:01

     And then if we move slowly somewhere else... [upbeat music]

376. 1:15:14

     Yeah, we are... Yeah, I guess in the middle, uh, area. [upbeat music]

377. 1:15:25

     You see, you, you, uh... I-I'm really surprised, like, people are not doing more things with this model because it's, uh, it's so funny to look. [upbeat music]

378. 1:15:38

     And there's an autopilot, so you can just let it move around for, uh, for ten minutes and then, uh, listen to the music changing in real time. Um, so that's kind of cool.

379. 1:15:49

     Oh. Let's see how it goes here. Like, yeah, it's thinking. But let me show you what it, what it did before when I ran it. Oh.

380. 1:15:59

     And I [chuckles] I ke- I kept my laptop open for half an hour just so that it would not do that and refresh the page, so I guess I lost.

381. 1:16:08

     Um, but that's basically what it did in, uh, in, uh, like earlier. Just, uh, as you can see, just the prompt and nothing more. The, like, very long prompt.

382. 1:16:21

     Up. Uh, and, uh, it took something like, like a thousand seconds, so like fifteen minutes if I'm not wrong, um, to create the app. But that's basically doing exactly the same thing, so I can choose the file and upload it, and it's going to take some times.

383. 1:16:38

     But basic- Like, trust me, it's doing exactly the same thing, uh, as, uh, as what the notebook was doing, and that's really, like, quite impressive that it's doing that, uh, on the first try right away.

384. 1:16:50

     Um, and just to finish, I... One of the thing I'm doing when I'm vibe coding, uh, while it's working, is that I always have those, uh, instructions on how to get, uh, to get it to generate, um, applets or apps.

385. 1:17:07

     Uh, and one of the reason for that is that I want the apps to be as easy for me as to review as possible because I want to avoid exactly what happened to Paige earlier is that something was not working and then you, you need to figure out what it is and ask the model to fix it.

386. 1:17:21

     Um, so one of, one of my key tricks is that I'm asking the model to create different files for each, uh, each feature so that, uh, whenever I need to review something, I can qu-quickly check.

387. 1:17:33

     Like, I ask it to modify this feature. It's, it's updating so-something that has nothing to, to do with it, so I know from the start that there's something wrong going there and I will be more thorough in the, in my, in my reviews and all.

388. 1:17:47

     And also some instructions that I've... I don't understand why it's not by default in any vibe coding, uh, tool, which is add logs. Because when you need to, uh, when you need to debug, you, you just not...

389. 1:17:59

     The error message is not enough. You need to know what's happening before and what's happening, like, bec-because you know... You need to know where, where it's happening, so that's, uh...

390. 1:18:07

     I really recommend you to, uh, to, to use those kinds of, uh, of guidelines whenever you, you vibe coded things. And we can see how it goes. No. Oh, and something there, I don't think Paige showed it, but if you are, uh, if you're willing to pay, you can add an API key here, and then you are

391. 1:18:23

     going to have more, uh, more quota to use Gemini when, when you are vibe coding on, on AI Studio.

392. 1:18:31

     Uh, I think I will just let you show your demos, and maybe I can show you afterwards how it looks, uh, after.

393. 1:18:39

     Yeah, sure.

394. 1:18:40

     Um, for the Veo model, when you generated the, the video-

395. 1:18:45

     Yeah

396. 1:18:45

     ... it came with, of course, the music as the background and, of course, the text. Is it using your other models to generate that, or is it just Veo?

397. 1:18:54

     Yeah, it's just Veo at the moment, so.

398. 1:18:56

     So is there a way to kind of orchestrate all the models and put them together, or do you have to just use FFmpeg to actually stitch those two-

399. 1:19:05

     Uh-

400. 1:19:05

     ... differently generate?

401. 1:19:06

     Oh, if you, if you want to use the music generated by Lyria in the Veo m- in the Veo video, for example? Uh, no, you don't have a way to do that.

402. 1:19:14

     Uh, it's, it's more bec- i-it's more because of the model limitations that the Veo three generation was not meant to be able to, uh, to ingest audio files. So, uh, so that's why it's, it's limiting.

403. 1:19:27

     Uh, but I guess, like, the future is that we want every model to be able to ingest all modalities, uh, so that we... you can, you can do that.

404. 1:19:36

     Uh, uh, same thing, I think, uh, it, it would make sense even for Lyria model to be able to, uh, ingest audio models that it can... so that it can be used as references as well.

405. 1:19:47

     Or even if you want to do multi-turn and say, "Okay, I lo- I love your song, but the, the ending was a bit, uh, uh, not, not epic enough, so make it, make it more epic," so that it would maybe just, just update the, the end of it.

406. 1:19:59

     So yeah. It's not possible at the moment, but that's, that's the direction it's, it's going, uh, to.

407. 1:20:03

     How is the performance compared-

408. 1:20:05

     Really specific for the music in Veo, whether you're generating the video

409. 1:20:11

     No, I-

410. 1:20:12

     Or not as well

411. 1:20:12

     ... f- I wouldn't try Veo for, for any kind of music. Like, I-

412. 1:20:16

     I mean-

413. 1:20:17

     The, the, the training data for, for the background music and, like, is very, is likely very light because it's always doing the same kind of non-music things. Um, so, uh, no.

414. 1:20:29

     But once again, all of those models are trained more or less together, uh, and, and share some training data, so I'm sure the next generation of Veo is going to be better at that.

415. 1:20:38

     It's just that the current one, it, it shipped a year ago, and a year ago there was plen- like, we didn't have music generation models. Uh, Nano, like, uh, we talked earlier about text in videos and, uh, and, uh, like a year ago, Nano- NanoBanana was not even called NanoBanana.

416. 1:20:53

     It was Gemini 1.5 image gen- no, 2.0 image, uh, generation something. Uh, and it was, uh, it was not as good at, as, as text. So that's, uh, yeah, it's just, uh, just a question of, of generation of models, uh, there.

417. 1:21:09

     Oh, and I forgot something to, uh, while it, while it work. Um, the, if you want to run the, to run the notebook, uh, because you will have to remove it as well.

418. 1:21:20

     Um, here. Here. When I started... Yeah, I created the chart. I, I added these lines that say, "Service tier priority." It's actually something that we shipped last week. Um, and that's a way for, uh, to indicate when you are prompting the models to...

419. 1:21:37

     If you want, uh, if it's not that important, you don't, you're, you don't care about the latency, but, uh, but you're, but you care about price. So there's, uh, a service tier that is called Flex, and basically that says, uh, the, the request can take a few minutes to go through, but, uh, it's going to be half

420. 1:21:54

     price. So it's kind of, uh, the same thing as, uh, as using the batch API. And on the other end, because I want it to be so that-- certain that it would go through today, you can, you can say that it's a priority request.

421. 1:22:06

     So it's going to have slightly higher priority and that, uh, it should not, uh... And it's, it's going to be more reliable, but you're going to pay twice the price.

422. 1:22:17

     So, uh, if you are running it by yourself, you might want to remove that line to save a bit. Um, and we can see how it goes here. Oh, and see?

423. 1:22:25

     As I said, like, this model is in, uh, high demand, so

424. 1:22:31

     yeah. That's, um... I'm, I'm sorry about that. It's, uh, it's, uh... The model is too, uh, too, uh, too good. People... Everybody wants to use it, and we don't have enough capacity. [laughs]

425. 1:22:40

     So if you have, if you have any spare TPU to, uh, to share, um, we can make a deal. You have a question?

426. 1:22:47

     Do you plan to, uh, put, like, variations of this model open, like Gemma, but for, uh, for art and-

427. 1:22:59

     Um, so the question, like, for the video, for... The question is, uh, will, will, will there be any open-weight, uh, Gen Media model, basically?

428. 1:23:08

     Yeah. Yeah.

429. 1:23:09

     Um, so I don't think so for image and video generation, uh, to be honest. Um, one, one of the reason I see behind that is all of the, um, not sec- like security, but like the, um, when, when you, when you generate images or videos, um, we do a lot of checks about what you are asking for

430. 1:23:30

     and what, uh, what is actually being generated afterwards, and we are blocking a co- a couple of things that are not aligned with our, our visions. And I, I feel like whenever you have an open-weight model for that, it's, it's more like open bar, and you can, you can have it generate whatever you want.

431. 1:23:44

     Yeah.

432. 1:23:44

     So that's, that's... I think that's, uh, that's going to be where our, uh, our company values are going to be, uh, to be limiting us. Um, that said, for example, for music, it would make sense to, uh, like especially if you want real time, it would make sense to be, uh, to be, uh, to be on device

433. 1:24:03

     so that it's, it's, it's doing it faster. So that, that's something that might come at some point. Um,

434. 1:24:12

     cool. Yeah. I think we'll see after the Gemma demos if, uh, [laughs] if it works better. Um, yeah. Hope it was interesting.

435. 1:24:22

     Yeah. Thank you. Thank you. [clapping]

436. 1:24:26

     And so, uh, Ian is, uh, basically my, my counterpart. I work only on Gen Media models. Uh, Ian is working on Gemma models and he's going to... Like, the biggest release of last week was Gemma 4, and he's going to show you how cool the model is.

437. 1:24:40

     Okay. Thank you.

438. 1:24:43

     Yes.

439. 1:24:44

     So as Guillaume mentioned, my name's, uh, Ian Ballantyne. I'm a developer relations engineer, uh, working on the Gemma models, and this is an impromptu talk. Uh, we... Obviously, because everything that's happened today, uh, we've done...

440. 1:24:56

     Paige mentioned that we've, we're gonna do two sessions, just kind of go through a whole load of stuff. We have a, a number of Gemma talks for the rest of the week as well.

441. 1:25:03

     Uh, Omar's gonna do a keynote on Friday, I believe, which is gonna cover a lot of, like, the what Gemma is and why you should be interested. So I'm gonna, like, speed run that section.

442. 1:25:12

     Um, and we have another talk, uh, tomorrow on, uh, what we've called sovereign escape velocities. So, like, how you take ownership of AI and run things on your device or on your own cloud or on phones.

443. 1:25:24

     And hopefully, some of what I talk about here is gonna be, like, a realization of some of that. So that's a little bit more about, like, the why. I'm just gonna show you the what it is, if that makes sense.

444. 1:25:33

     So let's get plugged in. While we do that, uh, which one do I need? Sorry.

445. 1:25:38

     One for-

446. 1:25:39

     This magic one. Yep. Soon as I get a video in.

447. 1:25:43

     Actually, I think audio.

448. 1:25:45

     Uh, no. Let's go with no. [laughs] We'll, we'll try and show the ones without audio first of all.

449. 1:25:52

     Uh, okay. So as I mentioned, this is super impromptu, so we're gonna go with whatever we get. Uh, but I will go through... I'll just show you a couple bits when we get the screen up.

450. 1:26:03

     Oh, do you want me to move it to one side, or can I do a mirror?

451. 1:26:07

     Let's do mirror.

452. 1:26:08

     Let's do mirror. Uh, where? Yeah, you go for it.

453. 1:26:13

     Perfect. Yeah, so what I'm hoping to show you a little bit is, uh, you know, some of the things that-- uh, some of the devices that Gemma models can run on.

454. 1:26:20

     Like, I'll talk very briefly about the different size of Gemma model, uh, what we released in Gemma Four last week, 'cause it's brand spanking new. Um, and hopefully show you-- give you kind of a feel of, like, the kind of capabilities that you can now do either locally or that you can run on, like, a single GPU

455. 1:26:36

     that maybe you couldn't, like, six months ago. Okay. Here we go. Perfect. Ta-da. So Gemma Four, hooray. Uh, so this was released last, uh, Thursday. Um, it's a family of four models.

456. 1:26:49

     So we have what we call the effective models, the E-two-B and the E-four-B models, and these are models designed to run, like, on mobile phones, Raspberry Pis, Jetson Nanos, kind of like very small, low-end hardware.

457. 1:27:02

     Uh, the E part of it is the question we always get asked. The effective is, uh, because the model architecture has a per-layer embeddings structure, which means that the embeddings actually don't need to be loaded as part of the model.

458. 1:27:15

     So you can have them running on flash, um, and then you can page in, uh, the embeddings as they're needed for the model. So the actual-- what we kind of describe as, like, the brain of the model is about a two billion parameter, about four billion parameter.

459. 1:27:27

     But if you put all of it in RAM, it's a bit bigger than that. It's kind of more like a five and an eight billion parameter model. Um, so that's why it's called Effective Two-B.

460. 1:27:36

     Uh, we also have [REDACTED:generic_id], which is a mixture of experts model with f- uh, four billion activated, and we have a thirty-one billion parameter dense model, which is our, like, our flagship kind of big model.

461. 1:27:47

     Um, and both of the two bigger models, the twenty-six and the thirty-one, are designed to be able to run on, like, laptops and desktops or, like, single instance GPU clouds depending on, like, the precision and quantization you need.

462. 1:27:59

     So I'm not gonna get into that, but just imagine that a lot of the capabilities you can do right now, you could do on, like, a MacBook with, you know, enough RAM or a, you know, a f- uh, a fifty-ninety or something like that.

463. 1:28:11

     That's kind of where we're sitting right now. Um, why are these kind of exciting and interesting to us? Because with these models, we're focusing kind of on the agentic side.

464. 1:28:20

     So they now have thinking built in. They are multimodal, so they can understand image, audio, video. Uh, audio, by the way, is just for the two smaller models. The effective models that run on the phone can understand audio, but the rest can understand image and video.

465. 1:28:35

     And we're seeing, uh, performances of the models that are in the range of, like, ten X bigger than it in terms of parameter size. So what might have required a cluster, now you could do same kind of capabilities for, uh, for a single GPU.

466. 1:28:49

     So that's kind of really where we're seeing, um, some innovation here. Um, and what I'm gonna jump straight into is I wanna show you the demo section. Oh, I sh- I'll show you this one slide.

467. 1:29:00

     I d- I like this one. This is quite-- kind of explains a little bit where, where we've come. So the Gemma Two models you can see, like, right in the middle were kind of, you know, particularly good at creative writing, but, uh, uh, not so kind of evenly spread abun- uh, bunks amongst other different capabilities.

468. 1:29:16

     And as we've kind of gone through Gemma Three and then now through Gemma Four, we've kind of evened out the overall capabilities of the model. We've got things that are much better at coding, much better at function calling, action taking.

469. 1:29:26

     It's all kind of built into the design of the architecture of the model rather than relying on, uh, like, strong, um, instruction following capabilities, which we find with bigger models, uh, is, is kind of more important.

470. 1:29:38

     So the models are designed with that from the ground up.

471. 1:29:42

     So, uh, let's go over to... Uh, this is all spoilers from Omar's talk, so I'm gonna go straight to the demo section. Here, I wanna show you this one.

472. 1:29:52

     So this is the, uh, Google AI Edge Gallery app. Uh, has anybody tried this app yet? Uh, have-- Uh, one, two. Okay, like a handful of people. So you can download this on Play Store or App Store.

473. 1:30:06

     Um, it works on Android and iOS. And, uh, what it is, is it's, uh, it's a way that you can test and try the models. So we released a new feature called Agent Skills, and what Agent Skills allows you to do is it allows you to set up skills for the models that run physically on the phone.

474. 1:30:23

     So this model, this is the E2B model, is running literally on, uh, a Pixel, I believe, in this case. And you can prompt it, and you can test out the different capabilities.

475. 1:30:33

     So it's effectively given a set of, um, uh, skills that define things like Android Intents, where you can actually trigger and call other apps, or you can write your own JavaScript skills or, like, run, like, a WebView.

476. 1:30:47

     And, uh, you can instruct the model, and it will make a decision about how to actually trigger these. So you can get it to, for instance, add things to this, uh, research tracker, or you can ask it questions about the research tracker, and it will call the correct function to pull that data back.

477. 1:31:02

     So we've gone from, like, a world of just being able to chat with it on your phone to now that you can give it some more ambiguous input, and it's able to make a decision about, uh, what functions to call.

478. 1:31:12

     Uh, the next one, if it plays. Yeah, so in this, uh, in this example, we've got a number of different... I'll show you that one last. There we go.

479. 1:31:21

     Uh, a number of different skills that you can try out, so everything from just, like, you know, loading locations in maps to, uh, like, uh, uh, using APIs and other services.

480. 1:31:33

     Um, so y- you can build these things yourself, or you can just use some of the preloaded ones to try and understand what it's kind of capable of, and this is, like, our playground for kind of doing so.

481. 1:31:43

     And then lastly, I wanted to show you that, uh, this is the 2B model on the left-hand side. So you can-- you could also try things like vibe coding on device.

482. 1:31:52

     The model's actually quite capable of doing this. You're not gonna build, like, you know, a big architectural system, but it can do, you know, small, like, web apps. It understands lots of different languages.

483. 1:32:02

     Uh, you can write things in Python, HTML, TypeScript. Uh, it can validate some of the stuff and, uh, you know, you could build, like, little apps that can run on the, on the phone itself.

484. 1:32:13

     So you kind of got the whole feedback loop just on-- in the palm of your hand, basically. So this one, they just generated a, like a really simple calculator, and importantly, it does actually support divide by zero and gives an error correctly. [chuckles]

485. 1:32:26

     So it's even able to reason about those kind of things. And to, uh, to kind of add, I think this at the moment has got with thinking turned off.

486. 1:32:34

     But if you add thinking turned on, we see that you get a bump in terms of the quality of the output in term-- uh, in terms of what the model's actually able to reason about 'cause it adds that planning step before it actually does the execution.

487. 1:32:47

     Okay. So next I'm gonna show you a demo running on the device. Um, who here has used or uses LM Studio or, uh, Ollama or any of those kind of tools?

488. 1:33:00

     Okay. So like, maybe like three or four people. Um, so actually this is useful. So one thing that's really important to us when we build these models is making it compatible with a lot of the tools in the ecosystem.

489. 1:33:10

     So we partner with, uh, folks like Ollama, LM Studio, vLLM, SGLang to make sure our models kind of, uh, like work really well with them and that they can be deployed efficiently.

490. 1:33:21

     Um, LM Studio is kind of like, uh, a tool for running local model instance. So in this case, I'm just got the -[REDACTED:generic_id] model, uh, running on my device, and you can, uh, you can load it from here, and at the moment, this one is configured.

491. 1:33:36

     This one will take about, uh, at the moment it says seven-- eighteen gigabytes. But if you add, for instance, the, uh, enough memory to do the kind of context, you're looking at maybe like twenty-two gigabytes, uh, of, uh, RAM required to do it.

492. 1:33:49

     This is an M4 Mac, so obviously it has unified memory. So you can, you can run that, but you would need... If you wanted like full performance, you'd need to have a GPU with enough RAM to be able to run the whole model.

493. 1:34:01

     And as I mentioned, the [REDACTED:generic_id] model is a mixture of experts, so it, uh, it actually only needs f- uh, four billion activated parameters. So it's much quicker than the thirty-one-B model, but it is more intelligent than just a four billion parameter model would be.

494. 1:34:16

     So, uh, one cool thing about this is that with these models, you can serve them, uh, on your local machine, uh, via a compatible endpoint. So, uh, like an OpenAI compatible endpoint or an Anthropic compatible endpoint, and then you can use them with other apps.

495. 1:34:32

     So what I'm gonna do here is I'm just going to serve this model, uh, on port one two three four, and then I can use other apps to call, uh, the chat completions API directly on this one.

496. 1:34:43

     So the code to do this would be like OpenAI, uh, open chat completions, and then point it at this server, and it would be the equivalent of just using your local machine rather than using a cloud machine.

497. 1:34:54

     So I'm gonna run a, uh, terminal here. I'm gonna do demo SVG space. So what I'm gonna do is I'm gonna create, uh, one orchestrator instance, and then I'm gonna create ten sub-agents, and they're gonna generate SVGs for me.

498. 1:35:11

     So all they're gonna be told to do, uh... Actually, let me make sure I turn thinking off to make sure it's super fast. Uh, glad I remembered that. Right.

499. 1:35:21

     So each of these terminals here is a separate agent, and you can see the one, the orchestrator is this one right here. Uh, so it's farming out those decisions to all the different sub-agents, and each of those sub-agents has begin-- been given a different thing to draw.

500. 1:35:35

     So you can see this would be like a very good way for you to do some kind of quick prototyping on your local machine. Uh, you don't need the internet to do this.

501. 1:35:42

     And the throughput in the corner up there is how much combined token generation is being done by each individual, um, sub-agent. So when they finally put it all together, the orchestrator will just compile this into a page, and you get like a lovely array of different SVGs that they've generated.

502. 1:35:59

     So you can imagine this being any kind of agentic task that you want to be able to do on your machine. You know, sorting out files, implementing bits of code, doing like, uh, subdividing research, analyzing data, that kind of stuff like that.

503. 1:36:11

     Um, and you can give them all very different jobs to do this. So some of them have finished already. That's good. So something hopefully will pop up in a second.

504. 1:36:20

     There you go. So there are my SVGs. So, oh, they're not too bad, are they? And as I mentioned... Oh, this one's animated. Ooh, a spinning planet. So it's somehow fig- figured out how to make that look as though it's, uh, animated, which is quite cool.

505. 1:36:34

     Yeah. And this is the thinking turned off. So if I had thinking turned on, it does a little bit more thought in the planning stage, and you generally get better SVGs out.

506. 1:36:41

     Like, more time spent thinking about it results in better SVGs. Um, so the next thing I wanted to show you is, uh, I'm gonna jump over here to open code.

507. 1:36:54

     So as I mentioned, uh, you can use any application that can use an OpenAI compatible interface. So if you've got like a programming environment, uh, you can just basically point it at that, and you can try the model out to just see how well it performs.

508. 1:37:07

     So in this particular example, the... All you do to configure this in open code is you just... I'm gonna kill all my terminals here. Bye. Goodbye.

509. 1:37:20

     Let's get rid of these. If you go to... There is a single file. Uh, if I go vim .config/open code/config.json, and I scroll down to...

510. 1:37:37

     Where is it? This section here, this is all you need to specify. So you give the name of the provider, what the schema is, the models that you want to expose, uh, and any additional parameters that you need for the endpoint, and then you just literally point it at a URL, which in this case is just my

511. 1:37:54

     local machine, and that's all you need to do. And I have another example. I'm not gonna run this right now, but we have a guide for how you deploy Gemma four on, say, something like Cloud Run or like a cloud provider.

512. 1:38:06

     So if you don't have enough RAM to run the, the quantization that you want, you can just throw it up to a single DPU, and you get this, this single command here, this one right here, the beta cloud deploy, and this will put the model on a, an RTX Pro six thousand.

513. 1:38:23

     So you can just-- If you just wanna test it out, that's one way that you can test whatever model that you've fine-tuned or run yourself. And we have another way to access it as well.

514. 1:38:32

     If you go to ai.dev, that, uh, page in GM we've been showing you, we've also got the demo models are available here. So if you just want to prompt them, we've got the two big models, and you can test them out there, and they support like, you know, video and file uploads.

515. 1:38:47

     Grounding research is all kind of built into that too. So you can test to see how they perform in isolation as well. So really there's no reason whether you have the hardware or don't have the hardware, uh, for you to be able to give it a shot.

516. 1:38:59

     Uh, and yeah, what I wanted to show you was that I had, uh, generated a spec for, uh, a little game. Uh, where is it? Okay. So I made a game called Nebula Drift.

517. 1:39:12

     So what I did is I used the thirty-one billion parameter model to give me just, like, a spec for a game, and now I'm just gonna give it to my local model to go and implement.

518. 1:39:20

     So if I go implement spec, uh, and then I just give it the file, this one, and, um, make sure it's the right model, and let's make sure we turn thinking back on so that it's gonna actually reason about it too.

519. 1:39:40

     Uh, there we go. Uh, and we'll run that. And so why is this interesting? Because we haven't changed anything about open code at all. We've just literally given it the model to try on the local machine, and in the open code spec, it's gonna give it all the different tools that it needs to be able to run

520. 1:39:58

     open code and to behave in that environment. So it should think about, uh, the task it needs to do, and then it w- needs to look at the file system to either read files from the file system or write files to the file system.

521. 1:40:10

     And if it goes wrong, I can-- similar, like, to any coding harness, I can just prompt it and say, "Oh, did you check this," or, "Can I feed the error back into you," and have it kind of reason through it.

522. 1:40:20

     So this is just it following these instructions. So yeah, it's decided to make a directory. So it's gonna use the shell directly-- directory for that. So if you've used any coding tools, this shouldn't be too, uh, unfamiliar.

523. 1:40:32

     But maybe you've used it with another model, maybe you haven't used it with a local model or an open model, um, like, running on your hardware. So I just wanted to show you that this is kind of possible.

524. 1:40:41

     Oh yeah, there you go. It's already created an index file. That was quick. Um, it's gonna create... If it reads the spec correctly, it should generate two more files.

525. 1:40:49

     It should generate a JavaScript one and a CSS one. Uh, and then what we'll do is we'll just try to see what game it managed to make from the spec.

526. 1:40:57

     Um, and to kind of give you an idea, it should be like a in-infinite racing game where you've got to avoid some asteroids. So if it finishes off the code,

527. 1:41:10

     um, there we go. Shall we give that a shot? That looks like it's actually done something correctly. So if I go run that... Where did it write it? What was it called?

528. 1:41:20

     Nebula dr-- Oh, nebula_drift with an underscore, it decided. Oh, I think it's even trying to change the file path as I look at it right here. It's trying to make edits to it.

529. 1:41:29

     Stop. Stop editing. Um, let's just, just try and run that.

530. 1:41:34

     Okay. We have a game. Uh. Oh, it doesn't run. Okay. So this is interesting. So what we can do is if we just go,

531. 1:41:45

     uh, da, da, da, da. Let's try and run that again. Ah. So there's a syntax problem. So if I say-- If I just literally copy this. So imagine in most harnesses, you'd have a feedback loop where you could feed it back straight into the coding, but I'm just gonna do it manually and see if it can spot

532. 1:42:02

     its own error. I, uh... Stop. Stop. I found an error.

533. 1:42:13

     Ch-ch-ch. Let's check that. Did you actually read that or not? Let's try again.

534. 1:42:28

     Okay. So now it should, sh- now it should, uh, do some investigations. So probably reread the files, see if it can spot what the typo is or what the problem is, and then it would-- should try and edit it or fix it.

535. 1:42:39

     So actually, in open code, they give you two tools, one which can edit, like, individual lines and one that can just, like, rewrite the file. So it, it can even make a decision about which one of those it needs to use to do it.

536. 1:42:50

     Uh, so hopefully it's gonna edit it correctly. Um, and then write that back out. So it's updated the game.js, and it's editing...

537. 1:43:02

     Uh, what are you editing? Something else. Oh, um, it can also kick off sub-agents too. So if you give it, like, specific skills or tasks to do, it can basically, uh, run sub-agents.

538. 1:43:16

     Uh, the 31B is pretty good at that. The 26B depends, like, on the task. Sometimes it understands, like, what sub-agents to call, and sometimes you have to give it a bit more prompting.

539. 1:43:26

     But similarly, with an environment like this, you can expose MCP servers, you can create skills, you can do all sorts of stuff to kind of, like, help guide the model.

540. 1:43:35

     Um, shall we see whether that's fixed anything?

541. 1:43:43

     There we go. I don't even see. Did that refresh?

542. 1:43:47

     Nope. Not yet. It's trying to edit the game file.

543. 1:43:55

     Must match exactly. So okay, so it's trying to edit a part of the file.

544. 1:44:00

     I could also just say, like... Uh, no. Okay. I think it probably got unloaded. Let's put that back in. Let's load that back up.

545. 1:44:11

     Try again, Gemma. Right. So if I tell it just output the full file again

546. 1:44:22

     for game.js. Uh, where did it put it? It was in that one. So you can tell it, like, specifically which file it should be reading and writing to. So hopefully it should listen to what I say and then, uh, regenerate that file.

547. 1:44:43

     While we're waiting, does anybody have any other ideas for a game to, to make?

548. 1:44:49

     Paige is obsessed with, like, pink squirrels, but, like... So you could try something in that vein. Any, any ideas for a game?

549. 1:44:55

     A game where you could build your own game within the game.

550. 1:44:59

     Whoa. Let's try it. So I'm gonna use a game.

551. 1:45:04

     Yeah.

552. 1:45:05

     Right. Let's go ask... We'll ask the big model. So if I go, uh, okay, write a spec for a game where you can build your own game inside the game,

553. 1:45:25

     and use this as a reference. So we're gonna build a spec first. Uh, let's just go here, and we'll use... Yeah, we'll use, we'll use the spec for the other one as a reference.

554. 1:45:42

     Let's try and run that. While we see what's going on with our open code.

555. 1:45:53

     This looks like it's editing the file right now.

556. 1:45:58

     Yeah, you can see it's processing the tokens right there. And if I go to this one,

557. 1:46:05

     Omniforge. So- [laughs] ... what it's, what I've done here is I've told it to follow a particular spec, uh, so it keeps the same pattern. So like for instance, if I had like a format that I needed it to do in, it's pretty good at following those instructions and kind of keeping to that, uh, that record.

558. 1:46:23

     This may not be impressive if you've used any model in twenty twenty-six, other than to know that it's something that you can actually run at home. That's, for me, is the interesting part of this.

559. 1:46:33

     Like, the, you know, all the top models can do this. Uh, but yeah, up until recently, not, not that you could run yourself. So, uh, let's leave that doing its thing there.

560. 1:46:45

     Has it fixed my file yet? It's made more edits. We can go and see what it changed. So it's starting.

561. 1:46:52

     Oh, it added some more delta time. Oh, it's changing the input.

562. 1:46:59

     Okay. Well, you can see what it's generating here, like it's got like these little player cases and all sorts of stuff. Shall we see if it runs again? Uh, while we're waiting for that.

563. 1:47:09

     Right. Are you gonna run yet? Ah. Do you know what I could do, is I could just say to it, um,

564. 1:47:19

     uh, let's say, pressing the button doesn't start the game. Regenerate the files.

565. 1:47:37

     And hopefully it's just gonna write them back out.

566. 1:47:40

     Let's do that. Okay, it's thinking about it. See how our spec is getting on. So let's just get thirty-one B, can you just implement

567. 1:47:52

     this spec? Uh, but as a single index.html with the CSS and JavaScript included.

568. 1:48:09

     See whether it can do that. Okay. Uh, and there's one other thing I wanted to show you. We'll do that in parallel. AI.dev. One thing that the model's actually quite good at doing, being multimodal, is to understand, um, the context of the input that it's given.

569. 1:48:32

     So I could, for instance, take a screenshot of a website, like, uh, if I go to Gemma for DeepMind.

570. 1:48:47

     Let's go to this one. And if I just go like, uh,

571. 1:48:54

     let me just grab this, and I could just go to the model, uh, im- implement this web page,

572. 1:49:06

     web page as a single index.html. And if I just upload that file.

573. 1:49:16

     That one. There you go. And try and run this. Yeah.

574. 1:49:24

     How's our open code getting on? So it's written the game file, hopefully with no mistakes. Written the style file, and hopefully written the index file.

575. 1:49:37

     So let's give that another whirl. Right. There we go.

576. 1:49:47

     So there's our game. Except I don't see any asteroids. [laughs]

577. 1:49:53

     Okay. Well, we've got a star field, and we've got a movable ship. So again, this is just what the model has come up with. You could just iterate on this.

578. 1:50:01

     You could kind of add to it. Uh, I'm not gonna-- I could ask it for more asteroids, but I think we're done with this one. Uh, let's see how the other ones are getting on.

579. 1:50:09

     So this is the, this is the game about making a game. I'm really curious just to see how, what it does for this, 'cause that's gonna be pretty nuts.

580. 1:50:17

     And then this one, this is generating the... Oh yeah, this is generating the recreated webpage. So, uh, this is all free to, to play around with. Um, so just be aware there's, like, usage limits in terms of like, like shared ownership.

581. 1:50:36

     Um, but you can just use it, like, to test stuff out pretty easy. Uh, that's why it's a little bit slower than the, um, the Gemini models, 'cause it's the, uh, there's only one poor Gemini model serving everybody.

582. 1:50:48

     Um, let's see. Is this one done yet? Still going. I don't know. Maybe it's made like a really epic adventure. I think it might just be... This might be, you know, ship it as soon as this is done.

583. 1:51:00

     Uh, what else did I want to show you? Oh, okay. While we're doing that, I want to show you this cool demo. So one of our team, uh, is really into their robotics.

584. 1:51:10

     So they, uh, they made a version. If you've ever used, um- Any of the robot simulation tools. So this is called Open Duck, uh, which is just like a little simulator of duck, and you can ask it questions, or you can talk to it via a, uh, the E2B model, which is running in the browser.

585. 1:51:29

     So it's-- you don't-- you actually download the model, and it runs in WebGPU straight in the browser page. So when it tells it to do these actions, Gemma is interpreting, like, what it's asked the model to do.

586. 1:51:40

     It looks at what the, the robot can do and instructs it to... If you look really carefully, you can see where it says perform action. Yeah, before it actually does the action, it tells you what command it's trying to trigger.

587. 1:51:50

     So you can imagine this actually running embedded on the device, or in this case, it's in the simulator to just, like, kind of like prove the point. Um, so that's quite a cool one.

588. 1:52:00

     The other one I wanted to show you was, uh, the Android Studio team have integrated it into making Android apps. So you've got like a little chat window at the side, and you can, uh, build the apps from there.

589. 1:52:13

     So they, again, they have like a little agentic builder setup. Uh, so if you were just building phone apps, you can, um, use the model. Uh, I think they're using the 26B again.

590. 1:52:22

     Yep, they're using the 26B for that one. And then, uh, we also have it working with, um, the, uh, ADK, so our Agent Development Kit. Again, d- it exposes like different functionality.

591. 1:52:37

     It's got a thinking loop. It's got like a, like a feedback system, so you can give it more longer-running task, and it can make decisions about, like, pulling information and stuff like that.

592. 1:52:45

     So all these different environments that you can kind of run it in. Shall we see how our final-- Okay, this one is done. So this is gonna be-- this will be our webpage.

593. 1:52:56

     Oh, that's not bad. That looks pretty good. I don't imagine these buttons go anywhere. But for, uh, what's new? Oh, they've-- I thought-- I literally thought it had made a video. [laughing]

594. 1:53:07

     That would've been a bit in-- That would've been a bit intense. But yeah, but you can see even just from like the layout, it's pretty close. It's kind of matched the font correctly.

595. 1:53:14

     The understanding of the actual page is pretty good, and this is just kind of one-shotted. So point it at random web pages and see what it can build. And let's see whether this...

596. 1:53:23

     I'm, I am now really interested in this game. Let's see whether this actually works. Oh my gosh. Right. Is this-- Oh, I've got like a full editor. [laughing] So I can build like a little...

597. 1:53:34

     Oh, I like that. So the goal, can I run the game? Uh, I can put things in.

598. 1:53:44

     Oh my gosh. Please tell me that... Oh, yes, the triangle moves. Oh, I love it. [laughing]

599. 1:53:51

     Way. [audience clapping] Okay. There you go. Well, that's, that's a good point to end on, I think. So yeah, uh, Gemma models, uh, you know, they run on your phone, they run on your laptops, they run on GPUs.

600. 1:54:03

     Uh, go try them out. Try the AI Edge Gallery if you wanna explore. Uh, I'll be around today and tomorrow if you have any questions about how. Uh, but yeah, uh, really excited to share that with you and, uh, enjoy the rest of the conference. [audience clapping] [upbeat music]