Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
5:08:48Great! Sorry, I'm good. Hey everyone, I'm Akshat, CTO of Modal, and I'm going to talk
5:08:56about really need from their infrastructure.
5:09:01I'm told this audience is very finance adjacent, so I thought I'd talk about what
5:09:10if every cell in a spreadsheet could be backed by an agent?
5:09:16And imagine you're a financial analyst at a fund.
5:09:20you want to run a complex analysis on every company in your portfolio.
5:09:28Normally in the past, this would take days.
5:09:31You'd have to connect every portfolio company,
5:09:34look at their data sources, run some Python for each of them.
5:09:39now obviously with agents, this should be much easier.
5:09:43Well, actually, one of our customers, Hebbia,
5:09:46built something that's exactly this.
5:09:48you can go in right now in their product,
5:09:51you have a spreadsheet,
5:09:54and you can run a query
5:09:57where literally every cell is
5:10:00backed by a different agent.
5:10:03And I wanted to talk about what does building this really look like.
5:10:10Well, first, every agent is backed by a modal sandbox.
5:10:17Sandbox is a secure environment for running untrusted code.
5:10:25It used to be that every sandbox was one-to-one with a container or micro VM.
5:10:32That's actually no longer the case.
5:10:34Today actually, modal has this feature where you can run multiple containers or micro VMs
5:10:41inside a sandbox.
5:10:42You actually need to do that for doing something like this, where in this architecture you
5:10:49run the harness.
5:10:51The harness is the agent loop that does things like running the, calling the model API for
5:10:57inference, running the tool calls, and then looping over it.
5:11:02want to separate that from the place where the tool calls actually run for security and
5:11:08isolation and a bunch of other things.
5:11:10So you have the sandbox that's running multiple containers, including a sidecar.
5:11:16You also want shared storage, because you want your agents to be able to talk to each
5:11:21other.
5:11:21And it's up to you how you stripe it.
5:11:24Like maybe you have every row gets the same volume, and agents in that row in the spreadsheet
5:11:30can collaborate.
5:11:32And you also want to share storage between your main container and the sidecar for some
5:11:37cases.
5:11:38And finally, you need all of this to spin up really, really fast, because your spreadsheet
5:11:43can have lots of cells.
5:11:45If someone wants to query, you want to launch thousands of agents, and then you want to
5:11:48spin them back down when it's over.
5:11:52So before I talk more about that, I just wanted to take a quick aside to talk about what is
5:11:58modal.
5:11:59Modal is a platform of primitives that is built for AI applications.
5:12:05We have built this platform from scratch, built our own file systems, scheduling and
5:12:12the runtime to give you great performance and devx for use cases like inference, including
5:12:18LLM inference, sandboxes, training and functions, running a lot of batch processing and what
5:12:25not.
5:12:26Today in particular I'm going to talk about sandboxes, what this primitive is and how
5:12:30it's evolving.
5:12:31So if you want to build something like this, or even further, and we'll talk more about
5:12:36what that means, what are the primitives you need for that?
5:12:40The first thing is, if you're building an agent, you need a workspace.
5:12:44And I have some examples from some of our current customers.
5:12:48You might have heard of Ramp Inspect.
5:12:50Ramp Inspect is Ramp's internal coding agent that I think at this point pushes almost every
5:12:57single PR at Ramp.
5:12:59If you're building a coding agent, you need your repo, your GitHub repo, your dependencies,
5:13:07you need your dev services, all in the same workspace for each agent.
5:13:14Another example is basis.
5:13:18it looks like is your documents might arrive, your agent is then triggered to work
5:13:24on them, but then it's waiting for a response from the accountant.
5:13:28You don't know how long this could take.
5:13:30It could take many days.
5:13:32When you want to persist the state over time, your usage is going to change.
5:13:38In the second chart, you can see usage for each sandbox.
5:13:43There's definitely some consistent trends, but it's all over the place.
5:13:47If you were provisioning this, it doesn't make sense.
5:13:51and then reliably get them when the sandbox needs them
5:13:54and only pay for them for that duration.
5:13:58The other interesting emerging thing is
5:14:01agents need a lot more cloud science
5:14:04where pre-processed some data on CPUs
5:14:08or spin-up GPUs for running pre-processed some data on CPUs
5:14:13or spin-up GPUs for running a scientific model,
5:14:17it will offload that to a modal sandbox.
5:14:19is a scientific model,
5:14:22it will offload that to a modal sandbox.
5:14:24And we're seeing this show up in RSI
5:14:26about what happens when you give an agent
5:14:29sandboxes to Hillclient.
5:14:34The next emerging thing is
5:14:36agents are collaborating a lot more,
5:14:39and that means that you share a file system
5:14:43with your sub-agent,
5:14:45you share file system.
5:14:47The next emerging thing is agents are collaborating a lot more, and that means that you share
5:14:56a file and files with your human where you're all collaborating with the same thing.
5:15:03This means file systems that are fast, have consistency, and devx you can reason about.
5:15:10Something we're working on with modal volumes.
5:15:13The related thing is we're seeing a lot more self-improvement.
5:15:17will save and update memory, they'll run background tasks, which is now called
5:15:23dreaming, to update this memory over time.
5:15:26Actually this also just looks like files in a file system.
5:15:31But again, you want really great performance from this file system.
5:15:38and the scary thing about agents is they can mess up and they don't really know when they might
5:15:48leak your sensitive data to a german forum or something so you need to have a lot of control
5:15:54over what they can and cannot do and you need things like controlling what ips can even talk
5:16:01to the agent, how do you authenticate it, what domains can it talk to, and also if you're
5:16:08using credentials for talking to services, basically you want to inject them in a way
5:16:14that the agent cannot actually see them and then exfiltrate them somewhere later.
5:16:20And the interesting evolution of this is we've realized that domain-related allow lists are
5:16:27actually not enough.
5:16:28a lot of our users actually want very specific control
5:16:32over even like what paths can I talk to
5:16:36on let's say Datadog or GitHub.
5:16:38Like maybe I want to reason about specific operations
5:16:42that the agent is doing.
5:16:43And going back to sidecars,
5:16:45we've seen this emerging as a pattern
5:16:47where users actually write their own lightweight proxy
5:16:51that has custom rules about what exactly you can do.
5:16:53And Ramp's another example of a company
5:16:57that's been using this successfully.
5:17:02And the final underrated thing is it's really important to have really great observability.
5:17:08Going back to the example of auto research and an agent doing a lot of training, what
5:17:14happens when the training run fails?
5:17:16You want your agent to be able to easily look at the logs of the failure, reason about what
5:17:21happened and do another iteration with a different set of parameters.
5:17:27While training is running, you want to look at the metrics, GPU utilization and make sure
5:17:33it's not stalled.
5:17:34Maybe you want to look at the budget and make sure you're not spending too much on your
5:17:38sandbox.
5:17:40And obviously all of this is really important for humans, but as agents are driving a lot
5:17:45more compute, we're spending a lot of time exposing all of this to the agents themselves
5:17:50so they can reason about all of this.
5:17:54All right.
5:17:55I think we know what we need.
5:17:57Why don't we have Astra build us an agent platform?
5:18:01Well, I don't think it's a good idea.
5:18:03I don't think you should vibe code a sandbox platform.
5:18:06And there's three main reasons for that.
5:18:08Number one is reliability.
5:18:10I think we're all painfully aware how hard it is to get right.
5:18:16Certain websites we use every day keep going down.
5:18:19and the reason for that is even with AI,
5:18:23it's actually really hard to reason about the feedback loop
5:18:26of what happens in the case of this tail event
5:18:30when the system breaks
5:18:31and this is why you need to be very careful in building systems.
5:18:35Security is another one.
5:18:36You could vibe code a sandbox platform
5:18:38but who is going to do all the work
5:18:40to continually reason about the security boundaries
5:18:43and make sure it's defensible
5:18:47in an age when everything is evolving very fast.
5:18:52And finally, scale.
5:18:53We have a blog post about how we scale to a million sandboxes,
5:18:57which involves a lot of very interesting engineering design
5:19:00and reasoning about systems.
5:19:02And this is, again, something that you kind of don't want to leave up
5:19:07to a Vibe-coded platform,
5:19:08because when you get to scale, you want it to work.
5:19:12And on that point, we're actually seeing massive scale
5:19:16for a lot of things I mentioned.
5:19:18So, for example, for the snapshots I mentioned,
5:19:21it's growing quite fast.
5:19:23We're seeing 20 million snapshots a week.
5:19:25We're processing about eight petabytes of data a day
5:19:30for each of these snapshots.
5:19:33And our users are creating almost 2.6 million volumes a week,
5:19:38and all of this is growing super fast.
5:19:42And just to take a customer workload for personal assistant,
5:19:47You need to create 10,000 sandboxes a second.
5:19:50You need to run almost a million sandboxes.
5:19:54And they're not going to last very long, they'll just last for like two or three minutes.
5:19:58But when they come up,
5:20:00you want them to be interactive in less than a second. That's what this workload really demands.
5:20:08And the other interesting thing on scale is you actually cannot get all of the scale from a single cloud platform. The world is actually getting crunched on CPUs as well, not just GPUs.
5:20:19So one thing we spend a lot of time doing is actually aggregating cloud capacity from
5:20:24a bunch of hyperscalers and neoclouds.
5:20:27So our users have access to CPU capacity across the world when they need it, but we cannot
5:20:32rely on one cloud provider to do that.
5:20:36Okay, so where is all this going?
5:20:38I hinted at this earlier, but the harnesses are really good at making use of dynamic environments.
5:20:46and what this is leading to is a lot of separation of the brains,
5:20:51which is the harness, from the hands,
5:20:53which is the sandbox or the runtime where stuff is running.
5:20:58And what this means, I have an example here,
5:21:01is the harnesses are going to be able to spin up environments
5:21:05based on what they need right now.
5:21:08They're good at snapshotting them and keeping them around
5:21:12as artifacts that they need later.
5:21:14And again, adjusting all of this as necessary.
5:21:18So I think this points at an interesting evolution
5:21:22of the pets to cattle analogy we're all familiar with.
5:21:28To recap, in the olden days,
5:21:31people thought of services like pets.
5:21:34Each one was bespoke.
5:21:35You had to manage it.
5:21:36You had to reason about this individual deployment specifically.
5:21:41And then over time, with containerization and Kubernetes, people started to think about
5:21:48it as homogenous templates.
5:21:53These servers are all the same, you can scale them up and down.
5:21:56If one of them fails, it's not an issue because our system can heal from that.
5:22:00And I think this is where a lot of people deploying agents are today.
5:22:05They're kind of using these homogenous environments and scaling up these agents.
5:22:08But I think that's really limiting because the models are so good at reasoning about
5:22:14dynamic environments, snapshotting, creating lots of environments when needed and working
5:22:19across them.
5:22:20So what does the new world look like?
5:22:25analogy here is it kind of looks like an ocean of microbes, of all these organisms
5:22:30that are self-evolving and working with each other.
5:22:34credits this fix for the analogy here.
5:22:37but one thing I'm really excited about
5:22:40is as you give agents these really rock-solid primitives
5:22:45for compute and storage and networking,
5:22:48they will be able to actually do a lot more
5:22:50by doing the stuff I talked about,
5:22:53like spinning up lots of environments,
5:22:54working with each other, and being very dynamic.
5:22:58So yeah, I and us at Modal really think
5:23:02that great primitives let your agents do more,
5:23:06and yeah if you're interested in talking about this more
5:23:08reach out to us
5:23:09thank you
