AI Engineer World's Fair 2026
An Interaction Is All You Need — Ivan Leo, Google DeepMind
Slide chapters
19 slide and demo groups · Timestamps are approximate. 4 additional groups were omitted from this pass.
Read the talk
An Interaction Is All You Need
Ivan Leo introduces Gemini’s Interactions API and Managed Agents through the work an application used to manage itself: conversation state, multimodal handoffs, tool execution, persistent workspaces and credentials.
From a talk by Ivan Leo
At a glance
Ideas worth remembering
Use
previous_interaction_idto continue server-managed conversation context, including the state associated with thought signatures. Do not assume it also carries tools, system instructions or generation settings into the next request.Multimodal chaining keeps the calling pattern familiar while changing the model and requested medium. A photograph can branch into location-specific image interactions, and a linked image interaction can supply context for video generation.
Conversation continuity and filesystem continuity are separate: retain the interaction ID for context and the environment ID for the sandbox’s files and packages.
A single Managed Agent request can launch substantial internal work. The repository analysis consuming more than two million tokens illustrates delegation scale, not a fixed context-window size or guaranteed analysis quality.
Proxy-side credential injection keeps a token value out of the agent’s working context. Safe use of the authenticated capability remains a separate concern.
Skills and named agents package different reusable assets: skills carry procedures and supporting files, while named agents save a prepared configuration. Maintained API skills also help coding agents avoid stale model and method choices.
A weather lookup grows into an execution loop
“Tell me a joke” fits comfortably into a request and a response. A weather question adds a dependency: the model requests weather data, receives the result, and uses it to answer. An agent takes that pattern further, choosing among several tools, inspecting what happened, and deciding what to do next before returning to the user. Ivan Leo, a developer experience engineer at Google DeepMind, opens with this progression to explain why Gemini needs an API designed for more than a single completion.
Function calling supplies an important intermediate step: predictable JSON objects that application code can consume. Leo compares this to a registration form sending a username and password to a backend. The server can act because it knows the structure of the payload. Similarly, a model’s structured output can support extraction, parsing and tool requests without asking the application to interpret arbitrary prose. Once tool results feed subsequent decisions, however, the application must manage a continuing process rather than just parse one object.
A language model running in a loop is a useful shorthand for an agent, but it leaves out what the loop needs. The model interprets intent; functions and resources let it act; memory carries information forward; an environment executes the work. A coding agent without somewhere to execute code cannot complete the execution part of its assignment. Leo also observes that more capable models can need less specialized scaffolding, though that does not remove the need for tools and a runtime.
The first practical example is small and personal. Unable to use presenter view, Leo asked AI Studio for a mobile-responsive website that could display his speaker notes. Brief instructions became a deployed application he could access during the talk—he says he “vibed it out” two minutes beforehand. AI Studio serves here as the low-friction place to try model capabilities and tune parameters before moving into API integration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One interface must accommodate very different workloads
An image-generation request and a research agent do not have the same shape of work. In Leo’s example, Deep Research can take three minutes or more: the user interacts with a research plan, the agent investigates independently, and a detailed report comes back later. Yet an application may want to combine that research with image generation and other agent tasks. Separate endpoints make those combinations harder to assemble.
Deeply nested response objects add another integration cost. Each capability can require its own extraction logic before its result becomes useful to the next call. The Interactions API is presented as a common interface for models and agents, including the combinations Google itself wants to build across its products. Its first concrete simplification is to move conversation-state handling onto the server.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep an interaction ID instead of reconstructing state
Gemini’s thought signatures make state handling more consequential than remembering the visible chat. Leo describes them as opaque values returned alongside model activity, including function calls and responses, that must be preserved for later turns. He reports two practical failure modes: missing signatures can reduce model performance, and changes as small as an accidentally inserted whitespace character have caused customers to lose cache reuse. Manually rebuilding the conversation therefore creates opportunities to alter information the model expects to receive intact.
What changes in the request? The demonstration starts with “Hi, my name is Phil.” The response supplies an interaction ID. The next request passes that value as previous_interaction_id, allowing the service to recover the earlier context. The important code distinction is between sending a new prompt alone and sending a new prompt linked to the earlier interaction; the client no longer has to reconstruct that history and its signatures by hand.
The handle preserves conversation history, not every setting used to create it. The linked state-management documentation clarifies that tools, system instructions and generation configuration remain scoped to each interaction and must be supplied again when needed. It also makes the storage tradeoff explicit: server-side continuation depends on stored interactions; stateless use remains an option. An interaction ID is a simpler way to carry context, not a promise that configuration or history persists indefinitely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry a photograph through image and video generation
The next demo turns state preservation into a multimodal handoff. Starting from one photograph, Nano Banana generates variations of the same person in different countries and locations. Each generated photograph is its own interaction, branching from the initial input. A later video-generation call uses an earlier interaction’s ID to continue with its image and context. The location variants are separate branches; they do not have to become a serial chain in which one destination depends on the previous one.
Which parts of the call change when a still image becomes a video? Leo’s concrete example first requests an image of himself in Venice. The application then calls client.interactions.create again, selects a video-capable model, and links the image interaction. The earlier image and conversation become context for the new generation task. The model choice changes while the API method and continuation mechanism stay familiar. This still requires compatible capabilities: the supplied documentation notes that a later model must accept earlier output modalities as input.
Audio and image outputs follow the same effort to simplify handling. Instead of navigating a different deeply nested object for each medium, the response exposes a type discriminator such as output.type. Application code can inspect whether an item is audio or an image and send it to the appropriate handler. For image generation, Leo changes the requested response modality and generation configuration while keeping client.interactions.create. The benefit is shared calling and dispatch logic across media, rather than identical model capabilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine tools, then represent the work as typed steps
The security-report example combines three distinct capabilities for investigating a React application:
- Google Search: Discover relevant security reports through the search index.
- URL context: Retrieve information from web pages for closer inspection.
- File Incident: Expose a custom application action alongside the built-in research tools.
The model can decide what information it needs and use the available tools to develop a response. Search and page retrieval extend the information available beyond what was present at training time.
Leo describes this as bringing the capabilities together in a single API call. That should not be read as automatic execution of arbitrary application functions. The supplied Antigravity function-calling example distinguishes a requested custom function from its execution: the client handles the request and returns a function result in a continuation. Mixing tool definitions makes them available to the model; an application still needs an execution path for its own actions. The talk does not establish that an incident was actually filed.
How can a response represent all this activity without pretending it was one assistant message? The Steps Data model replaces the legacy outputs array with typed execution steps. Discriminators distinguish model output, thought signatures and function calls, while content types identify media such as audio and video. This gives asynchronous tool activity and work involving multiple models a more explicit representation. A consumer can tell what kind of event it is handling rather than infer that from position in an alternating user–model transcript.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Conversation state and workspace state need different IDs
Managed Agents adds an execution environment to the interface. The Antigravity agent runs remotely using the harness—the machinery around the model that manages its work—also used in the Antigravity product and AI Studio. Leo says Gemini was co-trained with this harness and expects benefits from using it. The immediate engineering proposition is concrete: developers do not first have to tune a harness, choose a sandbox provider, manage infrastructure and invent persistence between runs.
The first Managed Agent demo asks what is in a GitHub repository. An API request starts a remote sandbox; the agent lists workspace files, reads individual files, and produces a report. Those operations happen inside the hosted environment without Leo intervening between them. One caller request can therefore contain a substantial internal sequence of model decisions and filesystem operations.
What must a follow-up preserve to resume that work? Two IDs carry different kinds of continuity:
| Handle | What it preserves or selects |
|---|---|
interaction.id, passed as previous_interaction_id | Earlier conversation context and the state needed to continue it |
| Environment ID | The original sandbox, including its files and installed packages |
The interaction ID tells the service which conversation to continue. The environment ID tells it where the work should run. Keeping one is not a substitute for keeping the other.
Sources populate the workspace with the material the agent needs: GCS buckets, GitHub repositories and inline files. These can supply helper files and other project context. A continuation demo passes the environment ID and finds the same files where the earlier call left them. Installed packages and newly created files remain available across turns when the conversation and environment are resumed. The Managed Agents quickstart develops this separation further, including keeping a workspace while starting a fresh conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A two-million-token investigation still starts with one request
The larger repository example came from a hackathon project: a programming language for creating reinforcement-learning environments, described through the analogy of Scratch’s drag-and-drop programming. Its repository combined a custom domain-specific language, a web page, schemas and an RL training loop running inside HUD. Leo reports that one analysis run consumed more than two million tokens. This is total work during an agent run, not evidence of a two-million-token context window or a measured accuracy result.
The example makes the cost distinction important: a single API call is a convenient unit of delegation, not necessarily a small unit of computation. The harness may perform many operations before returning. The supplied Antigravity guide provides practical controls for this kind of workload, including a best-effort total-token budget, progress monitoring and cancellation. Managed infrastructure removes setup work; it does not make an open-ended investigation cheap by definition.
The local-to-cloud path reuses more than a prompt. Workflows and skills can first be developed in the Antigravity IDE, then packaged in a folder and uploaded through sources into .agents. Leo presents the shared harness and prompt as the reason this can carry familiar behavior into the remote agent. The useful unit of customization is the skill package and its supporting files, rather than a separately rebuilt agent loop for each deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let code use a credential without giving it the secret
A remote agent may need authenticated access to the Gemini API or a private GitHub repository. Giving the agent a raw credential creates a second problem: malicious instructions could induce it to disclose the value. Managed Agents introduces a credential-injecting proxy between sandbox traffic and the external service. The proxy inspects outbound request headers and applies configured replacements.
Where does the GitHub token enter the request? In Leo’s example, code in the sandbox makes a call to the GitHub API, and the proxy injects the token on its way out. The diagram separates the request path from the secret’s path: the token joins at the proxy, after the agent has constructed the request. This addresses exposure of the credential value to the agent; it does not by itself establish that every authenticated action is safe or prevent misuse of whatever access the configuration permits.
The request and the credential meet at the proxy
Read the diagram as text
- Agent code in sandbox. request-path — Constructs and sends a GitHub API request without needing the real token value.
- Credential-injecting proxy. request-path — Inspects outbound headers and applies the configured token injection.
- GitHub token. credential-path — Added by the proxy rather than exposed to the agent.
- GitHub API. request-path — Receives the forwarded request with authentication injected.
- Agent code in sandbox → Credential-injecting proxy: Outbound API request.
- GitHub token → Credential-injecting proxy: Injected into request headers.
- Credential-injecting proxy → GitHub API: Authenticated request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Save a prepared configuration as a named agent
Once the agent behaves as intended, a named agent makes its setup reusable. Leo describes two ways to prepare one:
- Define the starting environment: Specify configured files and network sources. His example creates a data analyst from the Antigravity preview base agent with a GCS-supplied skill.
- Prepare an environment interactively: Chat with an agent to install the dependencies and packages it needs, then freeze that prepared environment as a specific agent.
Both approaches save the configuration that future work should start from.
How do these creation paths differ? The first describes the setup up front; the second captures a setup reached through actual preparation. Neither should be confused with resuming one live workspace by environment ID. The supplied quickstart clarifies that a saved agent uses a base environment and that a fresh invocation forks it. That lets a prepared starting point be reused without making every new job share the files modified by the last job.
At the time of the presentation, Leo reports support for up to a thousand named agents and says users pay for the model rather than sandbox or storage. Treat that as the offer described in the recording, not an indefinite pricing guarantee. The supplied Antigravity guide scopes unbilled environment compute to the preview period and describes charges based on model tokens and tools used. The architectural benefit is reusable preparation; the operating cost still depends on what the agent does.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Iterate locally—and give coding agents the API knowledge they lack
The closing tools shorten the path from experimenting to deployment. The open-source Gemini API CLI lets developers try model calls through shell commands. Leo also describes using it to iterate on an Antigravity agent locally, package the result as a folder, upload it and create an agent. This continues the earlier local-to-cloud approach: develop the workflow where it is easy to inspect, then transfer the prepared files to the managed runtime.
Migration has a different failure mode. A coding agent asked to implement a Gemini integration may keep choosing older model names and familiar methods, even when the developer intends to use the new API. Leo introduces an Interactions API skill to supply the agent with updated model lists, methods and implementation guidance. The team maintains that material and regularly evaluates whether the skills help agents build with Gemini as intended.
That is a practical ending for an API talk: changing the interface is only part of making it usable. The developer’s coding agent also needs enough current procedural knowledge to choose the interface correctly. Interaction IDs reduce hand-managed conversation state; environment IDs preserve the workspace; the migration skill addresses the knowledge gap in the tool writing the integration. Each takes over a different piece of work the application builder previously had to keep straight.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains interaction chaining, per-request configuration, storage and retention, typed execution steps, and compatibility limits when changing models across turns.
Walks through a first sandbox call, continuation with separate conversation and environment IDs, and saving a reusable agent with a base environment.
Details the managed harness, custom-function execution, background runs, token-budget controls and capability limitations. Useful for scoping the work hidden inside one agent request.
Provides shell access to model and media calls and managed-agent operations, with discoverable command schemas and dry-run request inspection.
Related talks
- Why AI Agents Should Have Their Own Sandbox — Philipp Schmid, Google DeepMind
Extends the same API and sandbox architecture with a fuller agent demonstration, filesystem inspection, environment download and distinctions between the shared harness and its configurable tools.
- Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff
Shows interaction-ID chaining in a story canvas and connects multimodal generation to a live voice agent’s tool calls.
- The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Provides a complementary security discussion about private data, untrusted inputs and external communication, helping distinguish secret protection from broader control of agent actions.
Read the complete timestamped transcript
16 slide groups · 3 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 4 additional groups were omitted from this pass.
- 0:12
Hey guys, my name's Ivan. I'm on the developer experience team at Google DeepMind, and today we're covering the brand new Interactions API and our new offering, Managed Agents. And so to take a quote from a very famous paper, "An interaction is all you need." Before we move on to what the Interactions API is and what the new Managed Agents offering is, let's take a trip down memory lane, and let's kinda trace how models have changed and how we've worked with models. So when we first started working with models, a lot of times you would have a single interaction. You would send a message, "Tell me a joke," and then you
An Interaction is all you need
Ivan introduces the Interactions API and Managed Agents, setting up a look back at how interactions with models have changed.
Text in image
An Interaction is all you need
Brief History
The diagram traces a shift from text completion and chat to function calling, then goal-driven work: a model uses multiple tools, reflects on their results, and compiles a response rather than answering in one step.
Text in image
Brief History Completion In a shocking finding, researchers discovered that a common household chemical drastically accelerates aging in lab mice. Instructions Tell me a joke. Why don't scientists trust atoms? ... Because they make up everything! User / Model User: Hi! Assistant: Hello! How can I assist you? Instructions + Functions Functions: get_weather(location) User: What's the weather in ny? Model: call get_weather("New York City") System: It's 72° and sunny in New York Goals + Thinking + Functions Functions: get_weather(), search(), browser() User: Find adoption rates, % who want to learn another language... Model: thinking ..., search(...), browse... Model: Compiling comprehensive...- 0:42
might get a response. For example, "Why don't scientists trust atoms? Because they make up everything." After a while, as models got more and more capable and they understood more about the world, this became a bit more complicated. We wanted to take these models, put them in different applications, but to do so, we needed them to be reliable. And so to do so, we invented function calling, where models could create individual JSON objects with predictable structures. Think like how when you wanna register for a website, your webpage sends a JSON payload to the back end
- 1:12
with a username and a password, and that enables the back-end server to say, "Okay, we have a new user, and now we can actually create a new account." That made it possible for us to actually use models in different parts of this application, whether it was extracting user information, parsing data, and more. But right now we're seeing this incredible cusp where models are getting more capable. We're seeing models being able to reason, go for very extended periods of time before they come back to people. And so you're moving from a place where we went from completions, instruction and
- 1:42
functions where we might ask, "Hey, what's the weather in New York City?" And the model has a call weather tool that it can get that data, ingest it, and generate a response to now a case where models have multiple tools under their belt. They're able to call tools, think and reflect on what the response meant, interact with the environment, and ultimately generate a response. And so this just brings us back to what is an agent? Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it. A lot of
What is an Agent
An agent combines a model, tools, short- and long-term memory, and an execution environment. Tool calls loop back into model inference, letting the model use their results before responding.
Text in image
What is an Agent Agents = Reasoning + Tools + Context (Memory) + Environment ✓ LLM used to understand the users request ✓ Functions or external resources that the agent can use to interact with the world. ✓ Short-term memory keeps track ✓ Long-term memory to recall past ✓ Environment to execute functions User Input Model Inference Agent Response Tool Calls
- 2:12
people have reduced this to just a language model running in a loop. And for most cases, this is true. If you have a really capable model, you'll find that as we've got models that become more and more capable, a lot of the scaffolding has fallen away. A lot of models, for example, the latest Opus models or Fable models, are now just immediately just using a batch tool instead of individually specialized tools like a edit file or a read file tool like we used to do. And so really what we have is a language model that's used to understand the context, the user's intent,
- 2:42
that's given a bunch of different functions, all different resources that it then interacts with its environment in. It has some short-term memory and some long-term memory to keep track of what it's encountered, and then an environment to execute functions in. If you're building a coding agent, but you don't give it an environment where it can actually execute code, for example, it's not gonna be able to do anything. And so if you'd like to try any of our models, the easiest way to do so is actually with AI Studio. And so I actually like to point out that my speaker notes today were actually coded on AI Studio.
Get started with just a few clicks
The checklist pairs no-setup, free-tier experimentation in AI Studio with billing and spend management. Ivan recommends it for trying Google’s models and says he built his speaker-notes app there.
Text in image
Get started with just a few clicks 01 Visit Google AI Studio (ai.studio) 02 Get your API key 03 No setup required 04 Generous free-tier 05 Start experimenting 06 Set up billing 07 Manage spend Interface text: https://aistudio.google.com/api-keys Google AI Studio Playground History Build Apps Gallery Dashboard Documentation Search What's new Get API key Settings Playground Explore Google models Models Agents Featured Test out our most advanced and newest models. Code and Chat Build chatbots, agents, and code with Gemini 3. Image Generation Create and edit images with Nano Banana and Imagen. Video Generation Generate videos with Veo models, our state of the art video generation models. Speech and Music Explore our text to speech and music [illegible] Real-time Real-time voice and video with Live API Start typing a prompt to see what our models can do Run settings Gemini 3.1 Pro Preview gemini-3.1-pro-preview Our latest SOTA reasoning [illegible] unprecedented depth and [illegible] powerful multimodal [illegible] coding capabilities System instructions Optional tone and style [illegible] model Temperature Thinking level High Tools Structured outputs Code execution
- 3:13
I vibed it out at two minutes before this talk. Uh, you see the, um, as... You can see the conversation history over here. Um, I literally said, "I need a mobile responsive website that I can pass in speaker notes," because I found out I wasn't gonna be able to use presenter view. And so what I did, I just gave it some brief instructions, and before you know it, you have a simple application that you can use without any hassle deployed on a website that's accessible to all. So we have a really generous, uh, free tier. For most of my presentations, I've just been using the free tier.
Presenter Notes Reader demo
Ivan shows the speaker-notes app he built in AI Studio after learning he could not use presenter view. The preview advances from page 1 to 2 of 21, pairing large white-on-black notes with timing estimates and navigation.
Text in image
SLIDE 1 Hi, I'm Ivan, and I'm on the Developer Experience team here at Google DeepMind. Today, we're covering the brand new Interactions API - an interaction is all you need Interface text: Presenter Notes Reader Back to start Remix Share Publish Preview Code Gemini Gemini 3.5 Flash · Ran for 187s Edited 7 files metadata.json src/types.ts src/utils/parser.ts src/index.css src/components/NotesInput.tsx src/components/NotesPresenter.tsx src/App.tsx Built I have successfully designed and built the Presenter Notes Reader, a mobile-responsive stage reader designed specifically for high-legibility and touch navigation under spotlight environments. Design & Functionality Overview • Dark Stage Aesthetic: Styled in a pitch-black theme to minimize blinding backlights and preserve professional stage presence. It utilizes highly legible typography pairing Space Grotesk headings with Inter body text. AI Features Add presentation timer Add font size togg[illegible] Make changes, add new features, ask for anything EXIT STAGE MODE Slide: 1 / 21 Est. Slide speaking: 12s Left: ~16 mins 01:35 Outline 1 / 21 Touch Helper: Off Chrome File Edit View History Bookmarks Profiles Tab Window Help Thu Jul 2 3:23 PM Presenter Notes Reader | [illegible] aistudio.google.com/apps/[illegible]?showPreview=true&showAssistant=true Ask Gemini Action Required JULY S M T W T F S Ghostty Screen Studio demo.mp4 Ghostty.dm[illegible] Mirror for [illegible] Display (P3-1600 nits) ProMotion Wallpaper Notifications Advanced… Night Shift… Arrange…
3:17Demo milestone
Presenter Notes Reader demo
Ivan shows the speaker-notes app he built in AI Studio after learning he could not use presenter view. The preview advances from page 1 to 2 of 21, pairing large white-on-black notes with timing estimates and navigation.
Text in image
SLIDE 2 Before we look at the new syntax, let's trace how drastically developer expectations have shifted. Originally, we had simple instructions. A strict, predictable turn. You ask for a joke; the model generates a punchline. Linear and contained. Then, we moved to instructions plus functions. You ask for the weather. The model recognizes the intent, pauses, generates a tool call, swallows the external data, and synthesizes a final Interface text: Presenter Notes Reader Back to start Remix Share Publish Preview Code Gemini Gemini 3.5 Flash · Ran for 187s Edited 7 files metadata.json src/types.ts src/utils/parser.ts src/index.css src/components/NotesInput.tsx src/components/NotesPresenter.tsx src/App.tsx Built I have successfully designed and built the Presenter Notes Reader, a mobile-responsive stage reader designed specifically for high-legibility and touch navigation under spotlight environments. Design & Functionality Overview • Dark Stage Aesthetic: Styled in a pitch-black theme to minimize blinding backlights and preserve professional stage presence. It utilizes highly legible typography pairing Space Grotesk headings with Inter body text. AI Features Add presentation timer Add font size togg[illegible] Make changes, add new features, ask for anything EXIT STAGE MODE Slide: 2 / 21 Est. Slide speaking: 49s Left: ~15 mins 01:44 Outline 2 / 21 Touch Helper: Off Chrome File Edit View History Bookmarks Profiles Tab Window Help Thu Jul 2 3:23 PM Presenter Notes Reader | [illegible] aistudio.google.com/apps/[illegible]?showPreview=true&showAssistant=true Ask Gemini Action Required JULY S M T W T F S Ghostty Screen Studio demo.mp4 Ghostty.dm[illegible] Mirror for [illegible] Display (P3-1600 nits) ProMotion Wallpaper Notifications Advanced… Night Shift… Arrange…
3:30Demo milestone
Get started with just a few clicks
The checklist pairs no-setup, free-tier experimentation in AI Studio with billing and spend management. Ivan recommends it for trying Google’s models and says he built his speaker-notes app there.
Text in image
Get started with just a few clicks 01 Visit Google AI Studio (ai.studio) 02 Get your API key 03 No setup required 04 Generous free-tier 05 Start experimenting 06 Set up billing 07 Manage spend Interface text: https://aistudio.google.com/api-keys Google AI Studio Playground History Build Apps Gallery Dashboard Documentation Search What's new Get API key Settings Playground Explore Google models Models Agents Featured Test out our most advanced and newest models. Code and Chat Build chatbots, agents, and code with Gemini 3. Image Generation Create and edit images with Nano Banana and Imagen. Video Generation Generate videos with Veo models, our state of the art video generation models. Speech and Music Explore our text to speech and music [illegible] Real-time Real-time voice and video with Live API Start typing a prompt to see what our models can do Run settings Gemini 3.1 Pro Preview gemini-3.1-pro-preview Our latest SOTA reasoning [illegible] unprecedented depth and [illegible] powerful multimodal [illegible] coding capabilities System instructions Optional tone and style [illegible] model Temperature Thinking level High Tools Structured outputs Code execution
- 3:43
I haven't been using my Google unlimited tokens, thankfully. And I think it's the best way for you to try our models, whether it's our new speech-to-speech translation models where you take in some... You speak in, let's say, English, and it can translate to any language, or, you know, you wanna try our latest models like the Gemini 3.5 Flash, et cetera. It makes it very easy for you to do so with simplified UI toggles to just be able to tune all of the different parameters. So let's talk about the Interactions API now. One of the things that we've seen, especially as we've moved from
Introducing the Interactions API
The model and agent examples use the same client.interactions.create call. The API aims to unify model inference and longer-running agent work, with optional server-side state and combined tools.
Text in image
Introducing the Interactions API A unified API to support inference of models and agents. Built for: Complex context management Long horizon background tasks Optional server side state management Easy interpretability and composability Combined-tool use (Google Search + Custom Functions) For Models from google import genai client = genai.Client() interaction = client.interactions.create( model="gemini-3.5-flash", input="Who won the last euro?", tools=[{"type": "google_search"}], ) For Agents from google import genai client = genai.Client() interaction = client.interactions.create( agent="antigravity-preview-05-2026", input="Research the history of Google TPUs.", environment="remote" )- 4:13
models to a dynamic where now we have models and agents, is that the kind of workloads that we want to support have changed significantly, right? Through the Gemini API, you have Nano Banana, you have, you know, Gemini 3.5 Flash, but you also have very capable agents like the Deep Research agent that's gonna run for anywhere from three minutes and more. You interact with it. You have some sort of research plan. It's gonna go out and do its own thing, and it comes back with very deep and detailed research reports. Now, it's pretty difficult then to
- 4:43
think about consuming models and agents through a whole bunch of these different endpoints. Additionally, for a lot of the older kind of API endpoints, a lot of the data was in, in very deeply nested objects. And so you can imagine the kind of difficulties that we had ourselves when we were building in first party integrations, and we wanted these use cases where you might be able to generate a Nano Banana image, do some research, generate a VO image from there, or spawn out all these different agentic, kind of like agents that go out and do their own thing. And so the
- 5:13
Interactions API is our sort of solution to this as we look to build an API that can support all of these different use cases and ones that we can build on ourselves as we look to bring Gemini across the entire surface of the Google ecosystem. So the best place to get started is just with server-side state. One of the things that you realize, especially as you use the new Gemini series of models, is that we essentially have this thing called thought signatures, where if you interact with the model, if it calls a function, if it gives a response, it's gonna pass back
Server-side State
The example introduces Phil, then asks for his name. Passing the first response’s ID as previous_interaction_id lets the second request reuse server-side context instead of resending the conversation.
Text in image
Server-side State from google import genai client = genai.Client() # 1. First turn interaction1 = client.interactions.create( model="gemini-3-flash-preview", input="Hi, my name is Phil." ) print(f"Model: {interaction1.output_text}") # 2. Second turn (passing previous_interaction_id) interaction2 = client.interactions.create( model="gemini-3-flash-preview", input="What is my name?", previous_interaction_id=interaction1.id ) print(f"Model: {interaction2.output_text}")- 5:43
a very opaque series of these numbers. For a lot of people, manually managing this was very difficult, and sometimes when we talk to different startups, we found that they would lose their cache just with a single white space that they mistakenly added in. So now with the Responses API, you can see on the first turn, "Hi, my name is Phil." And in the response, we give you a interaction ID. And as long as you give us back that same interaction ID under the parameter previous_interaction_id that you see in the second turn, all that context is preserved. And this is
- 6:13
really important with the new Gemini series of models because if you don't give us back those thought signatures, you're gonna see a decrease in performance. But what else does this unlock? Here you see a demo built by Vamsi from our team, that did an incredible demo where using a single photo you could generate with Nano Banana all these different variations of yourself in different countries and different locations. Each of these individual photos are a single interaction, but you can think of it from an initial API call, it spawns all these different interaction calls. Using those interactions, the
Venice Photo Demo
The demo turns one photo into portraits in different locations, including Venice. Ivan explains that each image is an interaction; reusing its interaction ID lets the demo generate a video.
Text in image
Venice Canals, Italy 🇮🇹 Interface text: × ‹ 7 › ⋮
6:40Demo milestone
- 6:43
same interaction ID, we then generate a video using the interaction ID and the new Omni model that we just launched. Something like this would not have been possible without the Interactions API built by Philip over there. So actually, it's a really incredible piece of work, I will say. So to give you an example, an idea of what it would take to build something similar, this is all it would take. First, you pass in Gemini 3.1 Flash-Lite Image, generate an image of me in Venice. You take that interaction ID, you chunk it
- 7:13
back into the client.interactions.create, and this time you specify that you want the Omni Flash model instead that we launched on Wednesday. And that allows you to now use the same Omni model with the same information, the same kind of images, and the same context throughout all these calls, making it possible to build more complex, rich, and just these very complex agentic applications. The same thing occurs with audio generation. If you look at the bottom, you see that now we have output.type. In the past, you would have to
Audio Generation
The Python example requests speech through client.interactions.create, then checks output.type for audio. It decodes the returned data and saves a WAV file, replacing nested response handling with a typed-output check.
Text in image
Audio Generation import base64 from google import genai import wave # Set up the wave file to save the output: def wave_file(filename, pcm, channels=1, rate=24000, sample_width=2): with wave.open(filename, "wb") as wf: wf.setnchannels(channels) wf.setsampwidth(sample_width) wf.setframerate(rate) wf.writeframes(pcm) client = genai.Client() interaction = client.interactions.create( model="gemini-3.1-flash-tts-preview", input="Say the following: WOOHOO This is so much fun!. [laughs]", response_modalities=["AUDIO"], generation_config={ "speech_config": { "language": "en-us", "voice": "kore" } } ) for output in interaction.steps: if output.type == "audio": print(f"Generated audio with mime_type: {output.mime_type}") # Save the audio as wave file to the current directory. wave_file("generated_audio.wav", base64.b64decode(output.data))- 7:43
take this very nested object, like .completions.audio, and it would be very tricky for you to nest it out. I can't even remember it off the top of my head. But now every single bit of the outputs that come out are clearly demarcated by a strong type. If it's an audio, as you see over here, all you gotta do is just parse it and handle it accordingly, right? But what happens if now I wanna generate an image? It's the same method over there, client.interactions.create. All you gotta do is just modify the response modality and the
Image Generation
Image generation uses the same client.interactions.create call as audio. The example requests a 9:16, 2k city image, then decodes the image-typed output and saves it as a PNG.
Text in image
Image Generation import base64 from google import genai client = genai.Client() interaction = client.interactions.create( model="gemini-3-pro-image-preview", input="Generate an image of a futuristic city.", generation_config={ "image_config": { "aspect_ratio": "9:16", "image_size": "2k" } } ) for output in interaction.steps: if output.type == "image": print(f"Generated image with mime_type: {output.mime_type}") # Save the image with open("generated_city.png", "wb") as f: f.write(base64.b64decode(output.data))- 8:13
generation config, and now I have an output.type of image, and this makes it very easy for me to start building and experimenting with Gemini. The same thing comes with tool use. You can mix and match different inbuilt tools that wasn't possible before. For example, over here, we're trying to get an agent to look up the internet to find the latest security reports for the React application. We've given it access to the same index that Google has with type Google Search. We've given it the ability to be... essentially retrieve information from web pages using the URL context tool that we ship with, and
Tool Use
The example combines Google Search, URL context, and a custom file_incident tool in one request to investigate a React CVE. It then inspects typed steps for function calls and prints their names and arguments.
Text in image
Tool Use from google import genai client = genai.Client() file_incident = { "type": "function", "name": "file_incident", "description": "Files a security incident in the internal tracking system.", "parameters": { "type": "object", "properties": { "cve_id": {"type": "string", "description": "CVE identifier"}, "severity": {"type": "string", "description": "Critical, High, Medium, or Low"}, "summary": {"type": "string", "description": "Brief description of the vulnerability"}, }, "required": ["cve_id", "severity", "summary"], }, } interaction = client.interactions.create( model="gemini-3-flash-preview", input="Search for the latest critical CVE affecting react, read the full advisory page, and file an incident for it.", tools=[ {"type": "google_search"}, {"type": "url_context"}, {"type": "function", "name": "file_incident", "parameters": ...}, ], ) for output in interaction.steps: print(f"{output.type}...") if output.type == "function_call": print(f"Function: {output.name}") print(f"Arguments: {output.arguments}")- 8:43
we've given it a custom tool here called File Incident. The model can reason autonomously on its own, figure out what information it needs, and then actually generate a response to you, all in a single API call. This makes it very powerful because now you have models that not only can work with information at the training cut-off date, but beyond, because now they have access to the entirety of the internet. And all this boils down to the new steps data model that we have, where instead of a world where we have a single message sent to a model and a response, we're gonna have very
The Steps Data Model
Steps replaces the outputs array with a typed timeline of model activity. The JSON separates a thought from a model output; content holds raw modalities, while typed fields carry names, arguments, and signatures.
Text in image
Part 1: Interactions API & Agents The Steps Data Model A typed timeline of everything the model did ✓ Steps [ ] replaces the legacy Outputs [ ] array ✓ Type discriminator Model_output, thought, Function_call, google_search_call ✓ Content [ ] is reserved for raw modalities (text, image, audio, video) ✓ Structured data lives in typed fields (name, arguments, signature) ✓ Step Status: done, waiting, in_progress { "id": "int_456", "status": "COMPLETED", "steps": [ { "type": "thought", "status": "done", "summary": [{"type": "text", "text": "Analyzing request..."}], "signature": "abc123..." }, { "type": "model_output", "status": "done", "content": [ {"type": "text", "text": ”Here are the results…”} ] } ] }- 9:13
complex things like async tool calls, models working together, right? And we wanna make sure we have a strong kinda data structure to support this. And so we introduced the Steps Data model, which replaces the original legacy outputs array, with type discriminator that makes it very clear at each step what the model output, what the thought signatures were, what the functions called were, and especially when it comes to the content that's being generated. Earlier you saw a content.type where it was audio and video, and this makes it easy to build these multimodal pipelines in a way that only the Gemini
- 9:43
model families can. The second thing that we've launched is Antigravity as a remote agent, as what we like to call Managed Agent. So we've standardized around using the Antigravity harness across all of our Google Suite products. In AI Studio, which you saw just now, websites are built with the Antigravity agent. In the Antigravity application, you're also using the Antigravity agent. Now, this is a very capable harness that we've co-trained Gemini on. And moving forward, I think you're gonna see many gains if you use this harness out of the box. If you wanted to
Antigravity as Remote Agent
One API call runs the Antigravity IDE's harness in a managed Linux sandbox with code, filesystem and web tools. The example requests a CSV dashboard, then reads the text from the final interaction step.
Text in image
Part 1: Interactions API & Agents Antigravity as Remote Agent General purpose agent running in a managed sandbox Same harness that powers the Antigravity IDE Rich Toolset: code execution, filesystem, web search, URL context, Agent Skills Managed Linux sandbox ensures secure and isolated execution One API call: no complex orchestration to build yourself from google import genai client = genai.Client() interaction = client.interactions.create( agent="antigravity-preview-05-2026", input="Analyze this CSV and build a dashboard.", environment="remote", ) # Access the result print(interaction.steps[-1].content[0].text)
- 10:13
build a coding agent in the past, you would first have to tune the harness, then you had to find a sandbox provider, manage the infrastructure, and then kind of figure out a way how to preserve the context, especially in between runs. But with the Antigravity Managed Agent, you get a persistent sandbox out of the box. With a single API call, you get a sandbox that you can keep hitting and treat as a personal claw without any sort of modifications. You can see an example over here where we wanna analyze what's in a GitHub repository. We then pass it over in an
Sample Starter Kit: Project Submission and Repository Review
The demo moves from a project-submission form to a repository review: Antigravity lists and reads files in a remote sandbox, then generates an evaluation report without manual intervention.
Text in image
Submit a project Project name Sample Starter Kit GitHub repo https://github.com/octocat/Spo Description This is a project which demonstrates how to get started with Git Cancel Submit for evaluation Interface text: TanStack Start Starter octocat/Spoon-Knife: This [illegible] Ask Gemini 14
10:40Demo milestone
Show 2 builds
- 10:40
Browser demo showing a project-submission dialog with project name, repository, and description fields plus a Submit for evaluation button.
Text in image
Submit a project Project name Sample Starter Kit GitHub repo https://github.com/octocat/Spo Description This is a project which demonstrates how to get started with Git Cancel Submit for evaluation Interface text: TanStack Start Starter octocat/Spoon-Knife: This [illegible] Ask Gemini 14
- 10:47
Browser-based repository review with file-reading tool calls, a file list, and the beginning of a project evaluation report.
Text in image
Sample Starter Kit STATUS Verified SUBMITTED Submitted June 22, 2026 REPO octocat/Spoon-Knife INTERACTION [illegible] ENVIRONMENT [illegible] [illegible] tokens /workspace/repo/README.md tool_call read_file Read HTML file /workspace/repo/index.html tool_call read_file Read CSS file /workspace/repo/styles.css tool_call list_files List directory /workspace/repo/.git Generating the Hackathon Review I'm starting with a review of the "Sample Starter Kit" hackathon submission. The project name is clear, taken directly from the user's prompt. The GitHub repository is accessible, and the description provides basic context. I'll need to assess its functionality and completeness based on the description, and consider what the user might expect in a final report. Assessing the Repository Content I've examined the repository contents: README.md, index.html, and styles.css. The project's simplicity is intentional, serving as a GitHub fork/pull request practice example. It perfectly fulfills its stated objective with minimal HTML, CSS, and a guide in the README. Now, I'm ready to craft the markdown report, ensuring it's valid and adheres to the supplied formatting guidelines. Project Evaluation Report: Sample Starter Kit FILES 3 indexed index.html README.md styles.css Select a file Linked report paths and repository files open here. Interface text: Back to submissions Message the review agent... 14 trace events captured. TanStack Start Starter octocat/Spoon-Knife: This [illegible] Ask Gemini 14
- 10:40
- 10:43
API call over to the Antigravity agent, and it then boots up a remote sandbox and then starts investigating what the repository's about. You can see it calls a list file to get what's in the workspace. It then reads each individual file, and then it generates a final report. All of this is happening in a remote sandbox without any sort of intervention from our side. The agent's able to research, reason, and handle all of this autonomously. We also handle multi-state persistent sandboxes, like what we mentioned earlier, where as long
- 11:13
as you preserve the environment ID that we return to you, you're able to route it right back to the original sandbox that you had. So these are two primitives that we've given you, the interaction.id, where you're able to preserve and work with context and ensure you're not busting your cache, and the environment ID that enables you to route it to the exact same context. And so this way you can use the same sandbox. You don't need to manage any sort of persistence of state, and all you need to do is just add a few lines of code. The other way we've tried to make it easy is for you to be able to load in sources. A lot of times you might have custom
- 11:43
dependencies, you might have .EMV files, helper files that you wanna show on the, on this specific sandbox. So we support Google's GCS buckets, we support GitHub repositories, and inline files.
- 11:56
And so what you see over here is a demo whereby it continues off the previous call that we did. We're passing in the environment ID, and the model has the exact same files right where we left it. Any sort of packages you install, any sort of files you throw and create, the model has access to it through each ev- and every turn as long as you pass us back the interaction ID and the environment ID. This isn't just the sort of constraint to very simple tasks with the Managed Agent. What you're seeing over here is a run where the model burnt over two million tokens trying to analyze a repository that we got for a
- 12:26
hackathon, where someone built from scratch an entire programming language to code reinforcement learning environments. If you've ever used a scratch programming language, it's a drag and drop language where you can kind of get simple characters to move. But over here with a single API call, we're able to analyze an entire repository that has a custom DSL, a webpage, a whole bunch of like schemas, and an RL training loop running inside HUD. And so it really opens up this kind of opportunities for you to build these very custom and performant agents without you having to handle the underlying
- 12:56
infrastructure or tune the harness. The best part about using the Antigravity agent is that fundamentally, it's the same agent running in your Antigravity IDE, which means that if you have certain workflows that, uh, you've tuned locally, you have skills that you've made sure are great, you can... Once you're happy with those skills, you just package it into a single folder. We allow you to upload the same way using the sources into s- into a .agents folder, and that gives your agent the exact same skills with the same agent harness and the same prompt running locally and in the cloud.
Network Proxies
The proxy injects a GitHub token into outbound request headers so the agent can clone a private repository without seeing the credential. The code scopes the header transformation to github.com.
Text in image
Part 1: Interactions API & Agents Network Proxies Protect your secrets Network Proxies client = genai.Client() interaction = client.interactions.create( agent="antigravity-preview-05-2026", input="Clone my private repo test/private-repo and analyze its contents.", environment={ "type": "remote", "network": { "allowlist": [ { "domain": "github.com", "transform": { "Authorization": f"token {github_token}" }, } ] }, }, ) Interface text: 18- 13:26
When you're ready or when you're comfortable with what your agent is doing locally, you can then push it up, and everything works as intended. The next question we often get from companies are, "Hey, I need my agent to be secure. What am I gonna do if my agent leaks a network credential?" And to do so, we've implemented a man-in-the-middle proxy. So what does this mean? A lot of times you need to make API calls to different providers, say the Gemini API, you might need to download private GitHub repositories, et cetera. We have a proxy that stands in the middle, and so
- 13:56
any call your model makes, we will look at the header, and we can replace items at will. What you see over here is a case where we'll transform every single outbound call to the GitHub API with... and inject in the token dynamically. And so even if the model is somehow prompt injected and leaks your GitHub API token, it's never actually gonna see the token. It's only gonna execute code that will run against the GitHub API, and your token is never exposed. Once you're comfortable with how the agent is performing, you've
Build your own managed agent
Two paths create a reusable named agent: define instructions and skill files from source, or prepare an environment through conversation and save its snapshot. The examples load GCS skills or install pandas before saving.
Text in image
Part 1: Interactions API & Agents Build your own managed agent Bundle instruction, skills and environment into a named agent From Source Define from Scratch agent = client.agents.create( name="data-analyst", base_agent="antigravity-preview-05-2026", instructions="You are a data analyst.", base_environment={ "config": { "sources": [ {"type": "gcs", "source": "gs://skills/", "target": "/.agents/skills/"} ] } }, ) From Environment Fork a Snapshot # Step 1: Iterate with base agent r = client.interactions.create( agent="antigravity-preview-05-2026", input="Install pandas. Create template.", environment="remote", ) # Step 2: Save environment as named agent agent = client.agents.create( name="my-data-analyst", base_agent="antigravity-preview-05-2026", instructions="You are a data analyst.", base_environment=r.environment_id, ) Interface text: ‹ 19 › ⋮- 14:26
set it up well, we give you two different ways for you to basically configure and set it up as a named agent, much like you've been using the Antigravity preview agent. The first one is to freeze it with a certain set of configured files and network sources. That's what you're seeing on the left. You... I have a data analyst agent that I'm creating from the Antigravity preview base agent with some base environment configurations, here just a GCS skill. The other way that we allow you to do so is by chatting iteratively with an agent to set up an environment as you like. For example,
- 14:56
you might want certain dependencies. You might want certain packages. And you can freeze an environment as a specific agent, and all your subsequent calls kinda go through that. Once you're happy with that, it makes it very easy for you to scale up your workloads. At this point, we support up to a thousand named agents, which means that you can have all these different configurations, and you don't pay for any of the storage. You don't pay for, you don't pay for the sandbox. You only pay for the model. We've optimized our infrastructure to be very performant to be able to handle a workload like this, and we're pretty excited to see what people are gonna
- 15:26
build.
The Gemini API CLI
The Gemini API CLI connects local testing to managed-agent deployment: run prompts, scaffold and test an agent, then create it. Ivan describes packaging the agent as a folder and uploading it when local iteration is done.
Text in image
Part 1: Interactions API & Agents The Gemini API CLI Structured, agent-friendly interactions from the command line Run prompts against any model Generate images and text-to-speech Scaffold, test, and deploy managed agents Gemini API CLI Skill Interactions API Skill # Run a prompt gemini-api run "What is the capital of France?" # Scaffold, test, and deploy an agent gemini-api agents init my-agent gemini-api agents test --prompt "Analyze Q1 data" gemini-api agents create # Run against a deployed agent gemini-api run "Summarize quarter" --agent my-agent Interface text: 20
- 15:29
Recently, we've also out- open sourced a new Gemini API CLI. This is a simple CLI that you can use to get started working with the Gemini models to test out the models locally using simple CLI commands. The other benefit of this is that you're gonna be able to iterate on your co- on your coding agent or your Antigravity agent locally. And once you're ready, it packages it as a single folder and uploads it straight and creates an agent for you.
Gemini API and Interactions Skills
The command installs the Interactions API skill for coding agents migrating to the API. Ivan says the team keeps model and method lists current and regularly evaluates the skills.
Text in image
google-gemini/gemini-skills Skills for the Gemini API, SDK and model/agent interactions 11 Contributors 4 Issues 4k Stars 356 Forks npx skills add google-gemini/gemini-skills --skill gemini-interactions-api -y Interface text: 21
- 15:55
Lastly, if you're thinking of switching over to the Interactions API, which is where all of our new models will be on, we've also created a new Interactions API skill that you can feed to your favorite coding agent, and that will then basically allow you to do that migration without any issues. I'm sure you guys have faced issues where you've asked Gemini to implement a specific kind of, uh, implementation, and it keeps using Gemini 2.5 Flash or Gemini 2.0. And so we... what we do is we make sure we keep up, up to date the list of all the different models. We keep up to date all the different
- 16:25
methods. We provide a good bunch of skills. And like Philip mentioned yesterday, we regularly evaluate these and make sure, make sure that they're actually doing what we want them to do, which is allowing you to use Gemini and build on Gemini well. So that's the end of my talk. Uh, thank you so much for coming by and staying around for the... and the, uh, conference, and yeah. I hope you guys get started. I'll be on the side if you guys have any questions about the new API.
AI Engineer World's Fair
Black end card with white AI Engineer World's Fair lettering inside a rectangular outline and the event website below.
Text in image
AI Engineer World’s Fair HTTPS://AI.ENGINEER