Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI

Thanks for everyone for coming. So my name is Mikyo. Uh, I head up, uh, the open source group at Arize AI, and we work on a fully open source, uh, observability and evaluation, uh, framework, and we also work on our own agents. So today we want-- I wanna kind of walk you through what code mode is and our journey, uh, embracing code mode. So if you look back, uh, in the last thirty years of software development, you could kind of segment

software into two different categories. You could think of there's software built for software engineers and software for non-engineers. Take, for example, you have, um, you know, you're asked to get all the invoices out of your email inbox. If you're a programmer, you could write a script, you can go in there, you can dynamically, like see if there's an attachment, download it, put it into a folder. It might be a, you know, fifty lines of coding. You get this tool that you can use, versus if you were a non-technical, you'd have to go into these

inboxes, you have to manually download it. You're restricted by the interface of the software, and you had to kind of buy that software. And the thing that's really interesting about, uh, things like Claude Code is that that boundary is completely dissolving, right? Basically, anything you need, the coding agent can actually build that software for you dynamically. And the labs and the foundation model providers know this, right? If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do. And so

doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary? So before I get to, uh, started in code mode, let's look at kind of what an agent looks like. So hopefully you've built this, uh, uh, kind of agent before. You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools

that get pushed in the context window, it reasons about it, and then it calls another tool that get pushed in the context window. And, uh, hopefully most of you have built an agent like this. Uh, certainly if you have a certain set of tools, this can work pretty well. But we're now in that age where we want a lot more. We want, um, much more emergent behavior. We wanna put a lot more context into the agent. We wanna have longer horizons. And so this is where you kind of get this bloat inside of the context. You'll see there's a lot of

tools that get added into the context window that might not be relevant to the task. Um, maybe the tool call is not responding with a lot of junk, or maybe it needs to paginate through data. And so you deal with this kind of context rot. You also deal with multi-turn, a lot of turns as a result, and you're gonna have to deal with compaction. So luckily, code mode solves a lot of these things. It's kind of an alternative architecture. Um, instead of, let's say,

just giving it a huge index, uh, or, um, a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool. But you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flow. So they can do it sequentially, they can do them in parallel, um, can put conditionals around it. Um, and the nice thing about this is you can call multiple tools in one turn, and also

return only what's necessary to push back into the context window of the model. And so to kind of summarize it, put tho-- these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window bloat every time you call a tool. That's obviously gonna, you know, add to latency and cost, and it's also gonna increase, uh, increase turns. So the time your user takes to get to value increases, right? So has a lot of overhead.

Um, on, on the flip side, code mode potentially when done well can unlock a lot of abilities because it can call as many tools as it wants in a single turn, and it only has to put in exactly what's necessary into the context window to solve its problem, and you can also get a reduction in turn. So that's sort of the, the goal of, of code mode. And so let's just kind of go over some of the reasons why code mode works. Code mode works because, one,

you don't have to give it the giant catalog of tools up front. It can just pick and choose the tools that it needs and understand those tools, and then execute just those. So there's a progressive disclosure of tools. Uh, certainly there's things like tool search now also built into the foundation model providers. There's also the composability. There's a huge amount of benefit to not being able to-- being able to call tools in different sequences, either in sequence or parallel fan-out, chaining them together can create much more complicated and

efficient things. If you've heard of obviously Claude's dynamic workflows is built on this composability, but composability of sub-agents. I already mentioned sort of token efficiency. You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window. The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window. And the fourth one, which is the kind of the-- one of the major reasons we've embraced code mode,

is this idea of emergent behavior, being able to build its own tools, right? So for example, traditional-- with traditional tool calling, if I said, for example, like, "Take all these records and remove personally identifiable information," it won't be able to do that because it doesn't have a tool, or it might try to do it inside of its, its context window, but then you're leaking credit card numbers in the context window, and that's not great. So the emergent behavior of being able to create its own tools has a huge amount of, uh, benefit.

So here are some ways in which, uh, we've embraced code mode. Uh, so we build our own agent, uh, and harness inside of Phoenix, our open source, uh, our observability platform. And we started with Bash. You know, Bash is a great tool of being able to compose together things like, uh, you know, sed and, uh, grep to be able to, uh, compose together, you know, built-in Linux primitives with also our own CLI and be able to offload those into files for later use.

We also added, um, you know, a Python sandbox, and we hoisted all our REST CRUD endpoints. Those are like perform-- getting traces or, um, manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things. And we also added one additional one, which is the ability to write SQL and, you know, text-to-SQL. You've obviously, like, heard,

like, that probably many times before, but this also unlocks a lot of capabilities. The reason why is not everything fits into the tool category of just creating, updating, and deleting. Maybe you need to be able to do aggregate queries like, "Tell me all the LLM spans that have P95 latency of X, Y, and Z." You don't wanna have to paginate through all your traces to do that. You want the LLM to be able to pinpoint and find exactly what you're looking for. This is kind of a, a neat and creative way we've also

used code mode. So we've actually also created a sort of a programming paradigm in our UI that's bound to our UI. So we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end. And what's really cool about this is you kind of almost get this computer use, like emergent behavior, because we're able to put that programming paradigm in a Web Worker that has a two-way data binding directly into our browser, and it can kind of do complex

UI interactions for you, and it's really pretty fun to, to watch it, uh, cook. So code mode is not, uh, obviously for free. Um, you're dealing with LLM-generated code. If you've been at-- to the, the Anthropic talks, you've certainly probably seen some of these. You're dealing with resources. So if you're working with some hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. Networking is a huge one. Maybe you want it to be able

to go out to the web, but you never know what's out in the web, and so you have to carefully think about network egress. Uh, secrets is a hot topic. I think every sandbox provider is trying to come up with a different vaulting strategy. But let's say you want to use Jev inside of your sandbox, like Py, uh, Durable just shipped. You know, that's a crazy cool idea, uh, to be able to make fast decisions inside of a sandbox, but you have to make sure that those secrets don't come back into the context window.

And then the other one is permissions. Uh, there's some very interesting research around this as well, but typically, permissions are overbroad, and so maybe in the first five steps, the permissions are correct, but in the sixth operation it is not. Or maybe in composition, uh, the permissions are not, and so you have to kind of think of guardrailing some of these things if that is necessary. There's also operational complexity. Uh, certainly sandboxes don't boot, uh, all the time perfectly,

and so you have to, you know, be able to deal with failures. They're harder to debug because now you're debugging code, and you're not debug-debugging individual tools. And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code. Uh, so what if something fails in the middle? Uh, tool discovery is also still, uh, an issue. Previously, if you did have a tool, the LLM would upfront know it, but now it might have to go through a handshake to be able to understand which tools

are in its catalog of things to be able to do. And then this is sort of a, a big one. You know, you, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses, and now you have a double sandboxing, uh, problem. So sandboxing, you know, is gonna return stuff out as serialized JSON, and then it needs to be parsed and then put into another code mode, and so there's this, like, double duty that ends up happening that also can be, uh, an operational complexity.

These are sort of open problems. Uh, they're not like blockers, but durability is a problem. So let's say you build a giant piece of code that needs to execute, and it breaks in the middle. You can't resume from that step. Um, if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using. So JavaScript is something LLMs are really good at doing, but maybe not great for math. Um, for example,

uh, code mode can be a great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right? Because you don't wanna serialize the whole binary of the whole image that's gonna end up in the context window. So you might have to do some sort of signed image handoff. And then the other one, which is like the frontier models are amazing, uh, at code mode, but if you're trying to do cost-saving, you wanna swap out a smaller model, its capabilities might not just quite, um, get there.

So how do we at Arize, uh, evaluate code mode? Um, we use our own-- We dogfood our own platform, which is called Arize Phoenix. Um, and we run our agents through various permutations through something called the Harbor Framework, which basically puts the agents into, uh, sandbox environments and runs them on clear instructions with network access turned off and, uh- We run this multiple times in trials, so we eliminate a lot of the flake, and we build

evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing. And then Harbor is sort of rooted in this RL framework. We give it a, a one or a zero to indicate whether or not it performs the task, and we test it across many different permutations. So this is kind of why you wanna test like this, because as I mentioned, you can get a lot of flake. Um, sometimes it might work where you're dealing with an indeterminate system. It might take different trajectories,

uh, depending how you do it, and we, uh, track pass@k of each individual task that we think it should pass. So, you know, how does this work? Uh, this is kind of our results from, uh, running this Harbor run. You can see pre, uh, code mode and post code mode. We, you know, decreased the amount of tokens by forty-three percent. The cost went down by thirty-one percent. And you know, there was some overhead with the sandbox, but the latency stayed pretty constant. So we knew that across these

multiple surfaces, whether or not it was Claude Code or MCP or the CLI, we had accomplished something that, uh, was definitely better than it was before. How do we use this data? It's really, uh, a huge unlock if you, you both capture the trajectory and the evals, as you've seen with a lot of the other talks. This gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. So maybe, you know, first time we introduce a new tool, sometimes actually the

efficiency goes down, and so we can tell a coding agent why in these specific scenarios, what parts of the instruction are, are guiding it in the wrong direction, and we're able to hill climb these evals to be able to make our agents better.

So if you don't take anything other than this away, hopefully, uh, this will sum it up. We really like code mode at Arize because of the fewer turns you get out of code mode. Uh, we want you to be able to bring any model for us, uh, so we want it to be cheaper, and we're really excited about the emergent behavior and the long-horizon tasks that our agents can do. And so I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it.

And that's it. Thank you.