Any-to-Any: Building Native Multimodal Agents

How would you build a small NotebookLM-style application that understands source material and turns it into useful learning materials? Patrick Löber, who introduces himself as a member of the technical staff working on the Gemini API and AI Studio at Google DeepMind, approaches that problem through multimodal understanding, generation and real-time interaction.

Gemini accepts more than text: the API supports code, images, audio and video, alongside capabilities involving URLs and Google Search. Its output capabilities span text, images, speech, video, function calls and code. That breadth makes an any-to-any application possible, but the diagram describes an API ecosystem rather than a single model with every capability.

In the model lineup described here, Gemini 3 handles multimodal understanding and returns text. Specialized models handle native image and speech generation: Nano Banana, for example, supplies image generation based on Gemini. The distinction matters architecturally: an application must connect the reasoning model to the appropriate generator. Löber also points to Gemma as a local option, tentatively describing text, image and video inputs, with audio available on smaller models.

NotebookLM supplies the product reference: feed in several sources, then generate an audio overview or an infographic that helps explain them. Here, the goal is to let the agent decide what to create. The distinction between an agent and a fixed workflow is who chooses the next asset. A reasoning model makes that decision and invokes specialized models through function calls.

The architecture has two phases:

1. Understand the sources. Turn the mixed inputs into a synthesized study guide.
2. Generate supporting materials. Give Gemini the study guide and tools for producing images and speech. After a tool returns, the model considers whether the materials are sufficient or whether another asset would help.

The loop ends with a combination of text, visuals and audio, rather than a requirement that every source always produce every output type.

The concrete study topic is Attention Is All You Need. Inputs might include the paper as a PDF, images, a lecture or tutorial video, and voice memos. The useful capability is cross-modal understanding: the model should connect information across those sources, not merely summarize each file independently.

The SDK path starts with an API key from AI Studio and an SDK installation. Upload the PDF, video and MP3 through the File API, or supply smaller files as inline data. Löber also recommends the Gemini API development skills, which give a coding agent guidance for working with the API.

The central call is client.models.generateContent: place the sources and the instruction together in contents, explain what the files represent, and ask for a synthesis. Löber names gemini-3-flash in the walkthrough. The JavaScript below retains that Generate Content API style while taking the callable model identifier from configuration. Current documentation has newer model names and, in some examples, the Interactions API; those examples are not the API sequence shown in this recording.

import { GoogleGenAI, createPartFromUri } from "@google/genai";

const client = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const model = process.env.GEMINI_MODEL;
if (!model) throw new Error("Set GEMINI_MODEL to an available model ID");

async function uploadReady(path) {
  let file = await client.files.upload({ file: path });
  while (file.state === "PROCESSING") {
    await new Promise(resolve => setTimeout(resolve, 2000));
    file = await client.files.get({ name: file.name });
  }
  if (file.state === "FAILED" || !file.uri || !file.mimeType) {
    throw new Error(`File unavailable: ${path}`);
  }
  return createPartFromUri(file.uri, file.mimeType);
}

const sources = await Promise.all([
  uploadReady("attention-is-all-you-need.pdf"),
  uploadReady("transformer-lecture.mp4"),
  uploadReady("study-notes.mp3"),
]);

const response = await client.models.generateContent({
  model,
  contents: [
    ...sources,
    "Analyze these resources together: the Transformer paper, a lecture, " +
      "and my voice notes. Create a study guide that connects their " +
      "explanations and identifies concepts needing further explanation.",
  ],
});

console.log(response.text);

The prompt supplies the relationship between the files. That relationship is what turns a collection of uploads into a request for a coherent study guide.

Audio understanding can also serve a simpler task: transcription. Löber recommends asking Flash or Flash Lite directly to generate a transcript of an audio file. A separate transcription pipeline is not required for this use of the model.

Löber gives an audio rate of 1,920 tokens per minute and a one-million-token context budget, then describes that as more than nine hours. The arithmetic instead gives approximately 521 minutes, or 8.68 hours, before other input overhead. The audio documentation separately lists a 9.5-hour maximum per prompt; that stated limit does not make the two numbers in the talk reconcile.

For video, Löber gives a rough capacity of one hour and notes that configuration can extend it. The video documentation makes the relevant condition explicit: for a one-million-token context window, it lists up to one hour at default media resolution or three hours at low resolution. Resolution therefore belongs in the input-budget decision. You can also narrow the task to a time interval—for example, analyze only minutes 5–15—use the File API for larger uploads, or supply a YouTube URL directly.

Context caching becomes useful when the same long sources support repeated questions. Löber reports 90% cost savings for repeated queries over long loaded files using context caching. Treat that as a caching claim, not a guaranteed reduction in the total application bill: the caching documentation makes savings dependent on cache hits, while explicit caching also incurs storage costs tied to its lifetime. At this point, the application has the first essential capability: reading the source collection and producing a summary.

The second phase uses Gemini as the reasoning model, with function calls connecting it to specialized generators. Once an asset has been generated, the loop can assess whether it needs more material. Image generation uses the same client.models.generateContent entry point, with a different model and a prompt describing the desired picture.

The image model is Nano Banana 2. Löber calls it Gemini 3.5 Flash image preview, whereas Google’s launch announcement identifies Nano Banana 2 as Gemini 3.1 Flash Image. Use the model’s documented identifier rather than turning the spoken name into an API string. The demonstrated task is straightforward: ask for an infographic and receive a slide-style graphic that explains the material.

Speech generation follows the same division of labor. Löber describes a text-to-speech model based on Gemini 2.5, with configuration for two speakers so the result can resemble a podcast conversation. He plays an excerpt that introduces the Transformer architecture and the 2017 paper, describing the complete output as a two-minute explanation of transformers. This is the study guide becoming another kind of learning material, not merely being read aloud in the interface.

A function declaration tells the reasoning model what a tool does and what arguments it accepts. Give each tool a name, a useful description and a parameter schema. The image example needs one string: a detailed description of the desired image. Define an analogous speech tool and register both in the generation request.

Then pass in the synthesized study guide and assign the model the role of a research partner. Its job is to enhance the guide with materials where they help:

• Visual explanation: call generateImage for a concept complex enough to benefit from a diagram.
• Audio explanation: call generateSpeech for a section that would benefit from an audio summary.

The tool descriptions advertise capabilities; the prompt provides the criteria for choosing them. This JavaScript function constructs that decision request from the study guide produced in the first phase:

function buildAssetRequest(model, studyGuide) {
  const functionDeclarations = [
    {
      name: "generateImage",
      description: "Generate a diagram or infographic for a study concept.",
      parameters: {
        type: "object",
        properties: {
          prompt: {
            type: "string",
            description: "Detailed description of the image to create.",
          },
        },
        required: ["prompt"],
      },
    },
    {
      name: "generateSpeech",
      description: "Generate a spoken summary of a study-guide section.",
      parameters: {
        type: "object",
        properties: {
          prompt: {
            type: "string",
            description: "Content and delivery instructions for the audio.",
          },
        },
        required: ["prompt"],
      },
    },
  ];

  return {
    model,
    contents: [
      "You are a research partner. Enhance this study guide with useful " +
        "materials. Call generateImage for concepts that need a visual " +
        "diagram. Call generateSpeech for sections that benefit from " +
        "an audio summary.\n\n" + studyGuide,
    ],
    config: { tools: [{ functionDeclarations }] },
  };
}

These declarations let the model request an asset; they are not the image or speech implementations. The application connects the selected function to its specialized generator.

Löber attributes native generation’s usefulness to the training it shares with the main Gemini models. The generator can use knowledge of the world and the meaning of the input, rather than treating a request only as a description of visual appearance.

Two examples make that distinction concrete:

• A viewpoint from a map. In a Nano Banana 1 example Löber found on Twitter, arrows drawn on a map specify a viewing direction. The request asks for a picture of what would be visible there, and the output depicts the Golden Gate Bridge. The task requires interpreting the location and direction before generating the view.
• Corrections on homework. A Nano Banana 2 example returns an image of math homework with corrections. The desired output depends on understanding the mathematics as well as producing the annotated picture.

Löber adds that the image models can render code in images. For speech, the corresponding expressive capabilities are multilingual output, accents and tone. The British-accent sample uses regional wording about getting started on multimodal agents and then going to the pub; another accent sample follows. The room’s reaction favors the second example, while Löber explicitly notes that not every accent is supported. These examples show why voice instructions are part of generation, alongside the words to speak.

Understanding plus generation now supplies the core NotebookLM-style application: source material becomes a study guide, and the guide drives the creation of supporting media. Real-time interaction is an additional capability on top of that foundation.

The Live API adds real-time interaction. Löber introduces Gemini 3.1 Flash Live as a native audio-to-audio model: audio enters and audio leaves a single architecture, rather than passing through a cascade of separate models. He connects that architecture to more natural-sounding conversation.

With limited time, he plays a prerecorded demonstration from his colleague Thor rather than attempting a live session. The interaction starts with speech, then adds a camera feed. When the demonstrator asks whether the assistant can see him, it describes a dark jacket over a blue shirt. The useful transition is from conversation about an unspecified scene to an answer grounded in the current camera view. Löber also tentatively mentions configuring the accent.

AI Studio’s Live interface is the suggested place to try the interaction. Löber then shows a code slide and points again to development skills for implementation guidance. This completes the progression from understanding uploaded sources, through generating assets, to discussing material in real time.

The study application is one use of a transferable pattern: understand heterogeneous inputs, reason over the result, and invoke the output capability the task needs. Löber closes with three extensions. A multimodal embedding model maps different modalities into a shared vector space for multimodal search. Gemma 4 offers a local multimodal-understanding option. Veo adds video generation with native audio. Each expands a different part of the application—finding source material, running understanding locally, or producing another output medium—without changing the central separation between understanding, deciding and generating.