Any-to-Any: Building Native Multimodal Agents

Talk transcript

1. 0:00

   [upbeat music] Okay.

2. 0:18

   Yeah, go. Awesome, yeah. Thank you everyone for joining the session. Uh, if you joined the previous session, we're switching slightly the topic and talking more about Gemini now. But I have two slides about Gemma as well, so to make Gus happy.

3. 0:31

   Hey. Um, I'm Patrick. I'm a member of the technical staff at Google DeepMind. I work on the Gemini API and AI Studio. And today, I want to talk about any-to-any building native multimodal agents.

4. 0:43

   So I wanna talk about multimodal understanding, multimodal generation, real-time interactions, and then also build an example app together. So at the end of this session, you should be able to build this for yourself, a little NotebookLM clone.

5. 0:58

   Um, so what does any-to-any mean? Uh, these are all the capabilities you can do, uh, with the Gemini API. So there's a lot of use cases this enables, uh, because Gemini does not only understand text, right?

6. 1:12

   It's natively multimodal, so you can also feed in code, image, audio, video, and then some more like URLs and also Google Search. And then it can not only generate text, but now we're also able to generate images, uh, speech generations, video generations, uh, function calls, and of cour- of course, code generation.

7. 1:34

   So yeah, this enables a lot of really, really cool stuff. Um, but this slide is, is sl- slightly giving the wrong impression because actually there are still different models.

8. 1:46

   It's not one multimodal model yet. Uh, this is kind of a bigger vision that we have at Gemini, uh, at Google DeepMind to bring more of the generation capabilities also into Gemini.

9. 1:57

   But currently, it looks a little bit like this. This is an ugly slide, I know. Um, but yeah, we have the main Gemini model, a series right now, Gemini 3, and it's able to understand multiple modalities, but it only outputs text.

10. 2:12

    And then we have different specialized native generation models. For example, NanoBanana for native image generation and speech generation based on the main Gemini models. And I wanna talk about this in a moment a little bit.

11. 2:26

    And also, I mentioned Gemma here. Uh, correct me if I'm wrong, but it allows text, image, and video input, and the smaller models also audio inputs, so you can also build multimodal agents locally.

12. 2:37

    Um, yeah. So I wanna focus on, on four things, uh, four models. Uh, the multimodal understanding with Gemini, native image gener- generation, native speech generation, and then also if we still have time, a little bit about, uh, the Live API.

13. 2:56

    Um, and then build something together or at least give you the building blocks how you can build this, a little NotebookLM clone. Who has used NotebookLM before? Oh, [chuckles] almost everyone.

14. 3:07

    Okay, so I don't think I have to explain it. Um, but yeah, you can feed in multiple different sources. Um, and then the audio overview is pretty popular, where you can generate a podcast, uh, to explain topics for you.

15. 3:19

    And then also infographics are pretty cool. So we want to build the same thing, and we want to build this as an agent rather than a workflow. So this means that the agent should be able to decide what to create rather than where we hard code the pipeline.

16. 3:35

    Um, here we are having a reasoning model that can decide, um, what to create, and then it's hooked up via tool calls or function calls, and then calls the other specialized models.

17. 3:47

    Um, so this is roughly how the, the app or the agentic architecture looks like. We have the phase one for multimodal understanding, and then we have the phase two.

18. 3:59

    This is where we have the agentic loop, where we use Gemini as the reasoning model, and it can then call different tools, and these will then generate different modalities for us.

19. 4:10

    And then it, um, acts in a loop and reasons if we need more assets or if it's good enough. And then in the end, we get text, speech, um, and, and infographics as an output.

20. 4:23

    And yeah, I wanna do this as an example with, uh, some learning about Attention Is All You Need paper. So we want to be able to feed in PDFs, images, videos.

21. 4:34

    This can be a lecture, for example, or tutorial, and then voice memos. And ideally, we also want this cross-modal understanding, right? That we can, uh, draw information from all the different sources together and ma-- let the model make connections.

22. 4:50

    And it's actually extremely easy with Gemini to achieve this. This is basically the code you need. Uh, who has built with the Google Gemini SDK before?

23. 5:03

    Half. Half. Almost half of the room. Yeah. Basically, this is how you set it up. You get your API key, uh, for free at ai.studio, and then you install the SDK.

24. 5:13

    We have it available in different languages. And then you can simply upload different files. Uh, like here we are uploading a PDF, a video, and an MP3 file. Or you can also for smaller files directly use it as inline data.

25. 5:28

    And as a tip, uh, on the right side, I mentioned, uh, Gemini API skill. So you don't have to know this code now by heart. You can just, yeah, uh, hook up your agent with the Gemini skill and then tell it, um, to create this, and then it should know how to work with the Gemini models.

26. 5:46

    Um, yeah. The-- This is basically everything we need. And then we call client.models.generateContent, and here we're using gemini-3-flash. And then we can put everything together into the contents list and tell it, for example, "Analyze all these resources," give it a little bit more information what these resources contain, and then it should generate a summary.

27. 6:09

    And then a little bit of, of practical tips or nice to knows for, for understanding. Um, you can also use it to transcribe audio actually. Uh, Flash, and even the smallest ones, Flash Lite, is pretty good at transcribing audio if you just tell it in the prompt, generate a transcript of this file.

28. 6:29

    And then maybe nice to know is for audio, uh, one minute of audio translates to, uh, 1,920 tokens, and Gemini has a token, uh, limit of one million. So if you do the math, it translates to more than nine hours of content you can feed in, audio content.

29. 6:45

    For video, it's roughly one hour. Um, but there are, um, configurations you can tweak that give you more control, and you can even feed in longer audios. Then you can tell it to, um, look at only different timestamps.

30. 7:00

    So for example, only analyze from minute five to minute 15. Um, and then, yeah, you can use the File API that easily lets you upload larger files. You can even pass in URLs, uh, YouTube URLs directly.

31. 7:14

    And what's also nice to know is you can, um, combine it with context caching. This is built into the API. Uh, this is especially useful if you're, um, loading longer files into Gemini and doing repeated queries, because then it saves you 90% of the costs.

32. 7:33

    Um, so yeah, this is multimodal understanding in a nutshell. So here, doing a quick checkpoint. We're now able to use Gemini to understand all these different resources and generate a summary.

33. 7:47

    Um, and actually, the timer is not working, so I don't know how much longer I have. But I think that we are still a little bit good. Uh, so yeah.

34. 7:56

    Then the next phase is the, the multimodal generation part. Um, so for this, we're using the agentic loop, where we use Gemini, um, as the brain behind this, and we combine it with function calling.

35. 8:11

    And I will show you how to do this in a moment. And then these function call the specialized, uh, native generation models. And then it can reason if these assets are enough or if we need more.

36. 8:24

    And the way to do... to use these specialized models is also basically the same code. Once you have the SDKs, you call, again, client models generateContent. In this case, we're using Gemini 3.5 Flash image preview. [laughs]

37. 8:40

    It's not the nicest model, but this is actually Nano Banana 2, the more famous model, uh, famous name for it. And then we tell it to, yeah, create a picture.

38. 8:49

    Or in this case, we can... It's pretty good at creating infographics, which is pretty cool. Just give it in your prompt, uh, create an infographic, and then it's creating these, these nice, uh, slide graphics for us.

39. 9:03

    And similar for text-to-speech. There we have a text-to-speech model, which currently is still based on Gemini 2.5. Um, and you can combine it with different configurations. You can also do, uh, two, uh, speaker, um, audio files.

40. 9:20

    So for example, this podcast style. And here is a nice example if the sound works.

41. 9:33

    Does it work?

42. 9:38

    Neural network architecture.

43. 9:40

    Yeah.

44. 9:40

    Introduced in a 2017 Google paper called Attention Is All You Need.

45. 9:45

    So this is an output of the text-to-speech model we have that then explains, uh, transformers in only two minutes for you.

46. 9:55

    And then the function calling, like I mentioned. So basically, to combine Gemini with function calling or tool calling, what you need to do is you create your function declarations.

47. 10:08

    So there you give it a name and a description. This helps the model to understand what this function is used for. And then also the different parameters. So in this case, we only want one string, which is, uh, then used for the prompt.

48. 10:23

    So the detailed description of how the image should look like. And then you do the same for, uh, the audio, um, generation function. And then if you set up, um, your model call, client.models.generateContent, you configure the tools.

49. 10:39

    And then you also need to add this to your prompt. So this is a small example prompt, an agent prompt, um, where you tell Gemini, "Hey, here's the study we synthesized before from the different modalities, and now you're a research agent partner.

50. 10:55

    Your job is to enhance the study guide with multiple materials." And then you do... Uh, you tell it the two functions. So, um, decide which concepts are complex enough to need a visual diagram, and for this, call generateImage.

51. 11:10

    And which sections would benefit from an audio s- audio summary, and for this, uh, call generateSpeech.

52. 11:18

    And yeah, this is basically everything you need to set up the agentic function calling for the multimodal generation part. And I also quickly wanted to touch on why native generation mo- uh, matters.

53. 11:32

    So we call this native image generation models, for example, because they are based on Gemini. So all the training or a lot of the training that goes into the main Gemini models are now also available in these models.

54. 11:46

    And this allows a, a lot of really, really cool use cases because these models understand the world. Uh, this on the left side is, for example, I found on Twitter, and, uh, example I really like from Nano Banana 1, where you can draw arrows on maps and just tell it, "Hey, um, create a picture of what you

55. 12:05

    see here." And since Gemini understands the world here, it's able to, to correctly, um, create a picture of the Golden, uh, Gate Bridge for you. And then on the right side, this is a nice example on the o- uh, from the educational space.

56. 12:18

    So you can use Nano Banana In this case, it was Nano Banana 2 to directly, um, correct your math homework, for example, and create pictures with the corrections because it understands math.

57. 12:31

    It can even generate code on images for you. So lots of nice use cases. And for the audio models, um, these are multilingual, and they understand accents and tone.

58. 12:43

    So you have

59. 12:44

    Ay-up, lads and lasses. We're getting stuck into building these multimodal agents today. No faffing about.

60. 12:50

    And one with a-

61. 12:50

    Let's just get 'em sorted so we can all nip to the pub for a proper pint. [laughs]

62. 12:55

    Was it a good British accent? Good. Yeah? It was a British accent. A one. Is anyone speaking German here? One, two, three. I have a [REDACTED:origin] accent because you can actually also tell it to create different accents.

63. 13:10

    Of course, not every accent in the world, but still. So here's one with a [REDACTED:origin] accent. [speaking [REDACTED:origin]] Was

64. 13:19

    it a good [REDACTED:origin] accent? [speaking [REDACTED:origin]] [laughs] So yeah, I would say this is pretty- That's better [REDACTED:origin] than British. [laughs] Okay. Uh, yeah.

65. 13:35

    And yeah, again, so quick checkpoint. You now know how to do un- the understanding part and the generation part. And this is basically already the, the NotebookLM clone. And now I quickly wanted to mention now, uh...

66. 13:50

    or we also have, um, a model for real-time interaction with it via our, what we call the Live API. And for this, we have a very new model which is also based on Gemini, uh, Gemini 3.1 Flash Live, and we call this an audio to audio model.

67. 14:06

    So native, um, audio generation. It's only one architecture. Uh, audio goes in and audio goes out, so you no longer have this cascaded pipeline with different models. And this allows a lot of really, really cool, um, natural sounding interactions.

68. 14:22

    I think I don't have time for a live demo, but you can try it at ai.studio/live. And here is a quick video from one of our colleagues, Thor. Um.

69. 14:36

    Hey, Gemini. You kind of, yeah- How are you today? ... just start it and start talking to it, and then you can also activate your camera. Ah, top of the morning to ya.

70. 14:42

    I'm doing grand. Thanks for asking. And it's also able to understand your video feed in real time. Just enjoying the chat, you know. This is, to me, one of the most impressive features.

71. 14:47

    And how are things with you? Um, can you see me?

72. 14:54

    Well, as plain as day. I see you there with your [REDACTED:physical_attribute]. I think you can also configure the accent or something. Wearing a grand dark jacket over a blue shirt.

73. 15:01

    But yeah, try it out for yourself, uh, at ai.studio/live. And I think that's almost it. Um,

74. 15:12

    uh, yeah, this is again how you can do it in the code. But there again, we have a skill for it that you configure, configure, and this... Yeah, now we are at all the three checkpoints.

75. 15:22

    And yeah, the pattern is transferable to every, every other field. Um, and maybe a few, uh, shout-outs as well to some other models. I'm not sure if you've seen the keynote this morning.

76. 15:34

    Um, but we now have a multimodal embedding model where you can combine all the different modalities into one unified vector space, which allows applications like multimodal search. And then again, you can go local with Gemma 4, and again, have this multimodal understanding, and Veo for, for image, uh, for video with native audio.

77. 15:54

    And yeah, so that's it. Uh, thank you, and have fun building multimodal agents. [audience applauding] [upbeat music]