Why AI Agents Should Have Their Own Sandbox — Philipp Schmid, Google DeepMind

So hi, everyone. Uh, welcome to Why Agents Should Have Their Own Sandbox. Uh, I'm Philipp. I'm part of the Google DeepMind team, mostly working on AI Studio and Gemini API. Uh, we have a small group, so I try to keep the boring stuff short and, like, the exciting demo stuff long. And we can also ask some questions, I guess, at the end. Maybe you need to sit a little bit further at the front so I can understand you. Um, so I want to start with a little bit of a, a background. I'm not sure if you have seen that slide, but that's a very popular slide when you do,

like, Google presentations, um, to show a little bit, um, what we have achieved in the last two year with, like, the Gemini models, but also our open models with Gemma. And the slide is already outdated since yesterday. So yesterday we shipped NanoBanana Lite, uh, which is our new image generation model. Super fast.

In a box.

Uh, sorry. Super fast, uh, super cheap. Um, it's like four cent per image. And then also we shipped, uh, Omni Flash on the Interactions API, so you can now programmatically use Omni to create videos, to edit videos, to, yeah, build gen media applications. And to give a little bit of a brief history, I assume everyone has heard about agent, has heard about large language models, but to really quickly reflect to where we are coming from. So I think around two thousand and eighteen, nineteen with the

first BERT models and GPT models, we were super excited because we could continue sentences, right? We were able to generate poems without, uh, with providing, like, a first few words, and then the model coherently continued the text, which was nice, but not very useful or helpful for any one of us to build something. Then we, we moved into the, the instruction, uh, way. Before we had pre-trained models which could continue text, and then with instruction following, we basically trained models to answer questions more or less.

So we told a, uh, ask a question, and the model correctly responded. Um, and then with, with ChatGPT, um, everything changed a little bit. I think most of the people probably got started with that. And we went into this user model turn-based conversation where we had a user input, a model input, a user input, a model output. Then afterwards, we combined that with function calling because we wanted the model to take actions for us. And now we are in, like, this black box with goals, thinking, and functions where we no longer just prompt the model. We

basically give a goal to the model, tell them how to verify its goal or how to, like, deploy it and, like, hopefully have it running for multiple hours or even days and come back with the results we want to achieve. Um, so if you haven't used Gemini API or AI Studio before, you can go to ai.studio or to ai.dev. So very quick, very short URLs. Uh, get your own API key. So if you have your computer with you and you-- if you want to follow along a little bit later on, like, trying out a few things we have shipped recently,

you can do that. Um, no setup required. Can use your Gmail account. Um, can start, start experimenting. If you want to use bigger models, more usage, enter your credit card. And we improved that a lot, so you don't need to leave AI Studio anymore to go into Google Cloud. You can do everything inside of AI Studio. You can set up your own budgets that nothing gets out of control. And I want to start about or talk about the Interactions API, which is our new an- uh, API for Gemini models and Gemini agents. And

on the right side... sorry, left side, we have some code snippets. So if you never have heard about Interactions API, you can think a little bit about that's our answers to, like, the responses API from OpenAI. It is much more developer focused, much more simpler, much more web developer friendly. And you can already tell in, like, the two snippets here, uh, to call a model like Gemini or to call an agent here, like the anti-gravity agent, it's, like, literally the same interface. And this works for all of the modalities

Gemini offers. So you have a model or an agent. You have an input. The input can be a basic string, or it can be multimodal data. And then you have tools or environment which, uh, I'm very excited and why, uh, which we are going to talk about later why every model or agent should have its own box. Uh, new with the Interactions API is you have server side state management. So if you don't want to manage the state on the client, you can hand it over to u- the server. We have long-running operations, so async requests, basically you

start something. If we talk about Omni video generation takes quite a long time, and you might not want to keep your HTTP connection open. So you send a request, get back an ID, can set up streaming or polling to wait, uh, until the, the generation is completed. And then, um, it comes with a lot of simplifications for building agents and tool use. So you can combine built-in tools like Google Search, uh, with custom function calling and to look at some code examples. So for server side state, it looks very

similar if you have used the responses API. We call it Interactions because we just not get a response anymore. We interact with models and agents. You send an input, and if you want to do a follow-up turn, you provide a new input and basically just the ID from the previous turn. And then on the server, we, um, concatenate all of, like, the different inputs and make sure that the model really rec- uh, sees what has been done so far. And this goes for all modalities. So for audio generation, you have a model.

We have our Gemini TTS model. We have our input. And since we want audio back, not text, you define a response modality with audio and some con- uh, generation configuration which voice you, for example, want to use. The same pattern works for image generation. So our image generation model, uh, NanoBanana here, our input, some generation, uh, gen-generation config, and then you get back your image. And for tool use, uh, also much better, I would

say, than previously. You- Just define your tools with, like, Google Search, URL context, or your own custom function, and then you send your request, and the model decides on it o- on its own, do I first need to, like, run Google Search or call the function? And all of the code you see here is basically enough to create a very simple, small CVE agent to detect if there are, like, any CVE, um, for React in this case. And then, like, it uses Google Search, it uses URL context to visit those websites,

and then function calling to basically give structure to it so you can, like, save it in your database or look it up, for example. Um, and something new and maybe different to other model providers is that we moved away from this user model turn principle. So if you build an application with another LLM provider or with open models, you have always this, uh, user model, user model turn concept. So you as a user provide an input and

expect the model to re- return an output. This worked until, like, one or two years ago, I would say, where we just got back responses from those models. But with agents and with reasoning, all of that changed, right? Models now think for a very long time where they might expose some thoughts directly or might just expose a signature. Function calling is not very, uh, a user role, right? With normal or, like, with previous APIs, you would need to use the user turn to return some function output, which for

me always felt a little bit weird because the output or the new input is coming from the environment and not from the user. So that's why we moved away from, like, those turns into steps or, like, to a, um, like, synchronous timeline of steps where every action or generation is its own type, which makes it very easy to extend to new features and capabilities which might come in the future, but also makes it still, like, very simple to use because you really define what it has done. So if you

build a function calling agent, you return a function result and not, like, a, a user-- a role user with some content which might be a function result or not. And at Google I/O, we shipped the Antigravity agent, uh, which comes with its own sandbox. So the Antigravity remote agent inside the Gemini API is powered by the same agent harness which is used with the Antigravity product. So if you have used the IDE or the new Antigravity 2.0 or the CLI, you might have

used that harness already. Very important here, when we talk about harness, we don't mean the same agent. The harness is basically the, the back end of our agent, but it might be different between different products. So the Antigravity IDE is a coding agent, so it might have a little bit of different tools or a little bit of, of a different system instruction than the Antigravity agent we have in the Gemini API. But it's still the same harness, meaning if we improve Gemini to become better with that harness, we benefit inside Antigravity, but we

also benefit inside of the Gemini API. And with that new agent, we also introduced this, uh, environment field. And an environment here remote is basically for that API call, your agent gets a sandbox managed by us, meaning if you send a request, our back end starts a s- a cloud sandbox where the agent can run code, can create files, write files, um, install dependencies, install

your own tools, or maybe call your own APIs, and it's all managed and stateless on our side, meaning that you just send the API request. You don't need to manage the infrastructure, and those environments stay persistent. So if you have multi-turn conversation where the agent, um, you set new user input, I don't know, like, um, research the latest tech news and create a file, and then you want to do, like, a follow-up interaction, you can, like, use the same environment from your previous turn

with that new interaction. But also this means that you can share environments between different agents. So if you have, like, three agents working inside the same environment, they can share context through the file system, or you can, like, even use sub-agents to basically pick up a work, uh, another agent has been done. And then, of course, environments are very good, but the model needs the right context and the right information or the right tools to work inside those environments, so you can provide sources. Uh, sources can

be, um, GCS buckets, GitHub repositories, or inline files. It's very useful if you want to integrate skills. So the Antigravity agent, uh, is skills or file system native, I would say. So if you store skills inside the skills directory, it will automatically picked up and, uh, be pro- uh, included into the system instruction. But also if you have an agents.md file from your coding agent or from your existing agents, it will also be automatically appended to the system instruction. And then, um, you

can create an agent because, like, sending a request is nice and having all of the configuration stateless is also very good. But what if you want to share it? Or what if team one builds an agent team two wants to use? Um, therefore, we created the Agents API where you can define the base agent, so what's the underlying brain, so to speak. You can define instruction and then also the, uh, environment. Um, and the environment can be raw environment, like with sources, or you can, uh, use an existing

environment. So you can think about I have some kind of a scaffolding to do where I need to install, like, the GitHub CLI, uh, add some skills, and maybe do some, like, sanity checks. I can use the agent to create its own environment, and after the agent is done, I get back an environment ID. I can use that environment ID to create the new agent, and every time I call this new agent, it will start from that already pre-loaded, pre-configured

environment. And so it's, like, very easy for you to use agents to build their own environments and then, like, to create agents out of them which you can share with your users or with your customers without- Writing any terraform without thinking about Kubernetes, microservices, Firecracker, or anything else. And to make it easier for you and for your coding agents, we created a CLI for the Gemini API. So it's, it's not a coding agent. It's not Gemini CLI. It's really an, a

CLI that your agent can use to call the Gemini API, meaning it can call NanoBanana or Omni if you want to generate some assets. But you can also use it to create those agents, iterate on those agents, and improve those agents. The QR code should bring you to, like, the GitHub repository. And then also we have dedicated skills for working with the interactions API and the agents API, um, if you want to, to play around with it, uh, later.

And now let's, let's do some, some demos. So I'm in AI Studio inside the AI Talk Radio. If you have seen or attended the Google IO developer keynote, you might have seen, uh, this already. Um, that's just a simple UI on top of a managed agents. We, uh... managed agent. We are going to look, uh, at the agent in a little bit, but it's basically a talk show radio. And that talk show, um, agent has access to

all of the different Gemini models, so it can generate a thumbnail, it can generate, um, speech, it can generate music, it can do research, and it's basically a one-shot radio generator, and we give it a try. And I want a talk show radio on whether pineapples belong on a pizza or not. And it's free to use, so if you go to the applet later, uh, you can,

like, have free daily usage to cre- create your own, um, AI talk show radio. And what happens is, like, first step, we start our environment, and then the model start thinking on what it needs to do. And then the first step is, like, it installs our Python dependencies, and it reads our skills, and then it continues to work. Since this will take a little bit, I already ran a talk show radio on whether we should put in cereal first or milk first. And the very interesting

part about a talk show radio, it's not like a podcast you listen to. It's really, you have multiple, um, people, like, reasoning and, like, sharing their opinions on it, and we give it a quick listen.

Is there a correct physical order to preparing your breakfast bowl, or are we flirting with culinary chaos? Welcome to AI Talk Radio. I'm Paul, broadcasting from our London studio. Today, we are exploring the great breakfast debate: cereal first versus milk first. We will objectively break down the pros and cons of both methods. Let's go to our lines. We have David calling from Boston. David, you're on the air.

Yeah. Thanks, Paul. Look, i- it's gotta be cereal first. It's basic logic, right? Cereal is the independent variable. You pour it, you see the volume, then you add milk to complement it. Plus, like, if you pour dry cereal into a bowl of milk, it splashes everywhere. It's a total mess. And the mashed potato analogy, you wouldn't put gravy in a bowl and, uh, drop a lump of potatoes on top, right?

A striking analogy, David. Cleanliness and portion control are certainly strong arguments. Let's hear the counterpoint. We have Sarah on the line from Sydney. Sarah, welcome.

Hi, Paul. Okay. Look, David has it all wrong. Milk first is-- it's a total game changer for texture.

Okay, cool. So if you want to play around with it yourself, you can go to ai.studio. And on ai.studio, you have that build section on the left side, and there is a gallery. And gallery is basically pre-built applets, use cases for Gemini. And if you scroll down to it, uh, you have new build experience with, and there is the talk show radio. If you click on it, you will find the applet. You can, uh, remix it, meaning you can, like, adjust it. So if you don't like talk show radios, and you would

rather have, I don't know, some real podcast, uh, application, you can, like, very easily, together with Gemini, adjust it to your own needs. And since applets are all open, uh, in terms of code, we can also take a quick look on, like, how our agent really is, and our agent is just files. So we have an agent's MD file inside our environment. That file will be loaded on environment startup, which is then included into the system instruction. Very basic agent

MD file. It has all of, like, the instructions for our agent to do. So it should do some research, write some scripts for the different people, generate music, generate a speech, mix everything together. And to do all of those works, we have skills for it. So if we look into, like, the TTS generation, we have a skill on how to run it. We have a Python file. That Python file itself allows the Gemini agent to call other Gemini models, so it uses the interactions API to generate some speech, which we can

then use. And th- that's all it takes for our agent. And if you want to take that agent from the UI to, like, a back end, it's really, you just need to copy that file and create the agent and then use it. And if you want to play around with the, uh, agent itself, uh, inside the playground, we have, uh, inside the model pickers, a new agent section. So if you want to test deep research, go test deep research. But there's also the anti-gravity agent. And on the anti-gravity agent, we

have six different, uh, agent templates, uh, which you can, like, very easily test, select, and, and see how it works. We go into, like, the, the standard one, and what I really like to do is, like, explore the environment. Like, let's see what the agent finds out about where it is running. We send our prompt and now it starts the environment. Should take for the first request, like seven to 10 seconds, and then we see the agent like doing its work. So we have some reasoning. We have some batch commands. So it runs, okay, checks what, uh, version,

uh, what version I'm running on, what CPU memory I have. And then we get back a response from our agent that it runs on Ubuntu, um, has eight vCPU, 16 gigabyte of memory, Python installed, PIP installed, and then you can like really continue. Uh, let's see. Um, do you have... do we have Node installed? Um, so you can like very easily follow up. Uh, and it just continues based on like my prompt. And then also

you have the option on the right side to download the environment. So if your agent has created some files you want to extract out of the environment, share it, you can very easily download it or there's an API for it as well. And then, um, uh, what else? Uh, to quickly show you again how it works, I can make that a little bit bigger. So when you, when we send a request, um, we like start the sandbox and it

loads all of our files and then we have like this agent loop completely on our server. So when you build agents, normally a lot of times you need to manage the function calling on your client, right? You need to pass the function call, run it so- somewhere and then send back the function result. And with the managed agents it's really just a single API call and everything runs inside that remote sandbox. You can also connect remote MCP servers or l- run, uh, local MCP servers inside of that environment and, um,

yeah, really get started. So if you are interested about it, uh, we have great documentation on site at, uh, Gemini API u- under agents with a quick start which walks you through how to send your first request, build your first agent and get started. And then of course, uh, install the coding skills to have Claude, Codex or whatever model you use to, to help you get started. Uh, thank you for, for coming. If you have some questions, I'm like here.