Zero Cold Starts: Serverless AI Agents on Akamai Functions — Thorsten Hans

Welcome, everybody. Thanks for joining my talk. In the upcoming twenty minutes, I want to give you an introduction on how you can build ultra-fast agents and MCPs with Spin and run them globally distributed. So my name is Thorsten. I work as a senior developer advocate with Akamai. And, um, I guess all of you have already built either MCPs or agents, and they are no different than any other kind of application, so they obviously demand super fast response times so that

we can deliver the best possible user experience. However, traditional region-bound architectures or infrastructures fall short when you wanna address a global audience. And they, they come with some frictions which lead to the inception of edge-native architectures. So obviously, latency is one of the biggest concerns. Depending on where you deploy your MCPs or your agents,

it might be fast for a subset of your global audience, but others might experience a way lower latency and resulting in a way worse user experience. Um, depending on your distributable unit, right, even-- So if you even, uh, either choose containers or VMs, you may experience a cold start delay that might range from several seconds up to multiple minutes, especially when you have to scale

dramatically fast to meet unexpected user demand. However, we can overcome that, obviously, uh, by overprovisioning the infrastructure. However, keeping around many, many containers or servers and actually not utilizing them results in obviously higher cloud spendings and is not sustainable from an environmental footprint. So there are higher level solutions, uh, that we can use to run

MCPs or agents, but they usually or often come with proprietary SDKs or, uh, deployable units or have their own way on how you should treat them in order to make them work in a production-grade environment. Once you figured out about those constraints, moving away or migrating your workloads, your agents to another hyperscaler, to another kinda infrastructure, it's often rather complex and

extremely costly. So... But there's a solution to that, and that solution involves a new way of how we as engineers could build back-end or server-side applications. And this is done by adopting WebAssembly on the server side. You can think of WebAssembly as, um, as containers relate to virtual machines. So they are binaries, super tiny in size, easy to distribute, super fast to run, and

portable. What I refer to as the edge-native AI stack is a combination of a CNCF project called Spin and Akamai Functions, one runtime where you can run those WebAssembly applications. So let's dive into Spin for a minute. Uh, Spin, as I said, is a CNCF project that is vendor agnostic, and it allows you to build WebAssembly applications that are meant to be executed on the server or in the cloud.

Spin itself consists of three major pillars. So it is a CLI that we as developers use to address all our day-to-day concerns, so everything, you know, in the realm of the inner loop. It is a runtime, so it's based on Wasmtime, um, that allows us to run these kinds of applications either locally or on microcontrollers or constrained environments. Last but not least, Spin gives us language-specific SDKs to boost the developer productivity

when it comes to day-to-day jobs like, for example, interacting with HTTP services, integra-- interacting with databases or different kind of things. Although we have language-specific SDKs, Spin itself is language agnostic, and you can choose from all the languages that are able to be compiled down to WebAssembly, including the WebAssembly System Interface, or short, WASI. And you can learn more about Spin on spinframework.dev, so

scan this QR code, and you will end up on the Spin documentation. So Spin is the developer tool that we use for building these kinds of applications. Akamai Functions is the first truly serverless computing platform that is globally distributed and has zero cold start at all. So every application that you deploy to Akamai Functions is cold started per request that hits your endpoint, and on average, we can

cold start in less than half a millisecond. It's fully managed, and it's, again, developer centric. Your developers deploy the workload, no knobs to turn, no Terraform to write, just a simple one-liner, and your app is globally distributed at no extra cost. Security is, um, is observed by WebAssembly's sandbox, so there's no way a workload could interact with external resources without us as developers explicitly

granting those permissions to our workloads. And to be a full-fledged serverless computing platform at the edge, we need some kind of persistence layer, so Akamai Functions comes with a built-in key value store that is, again, multi-tenant. You don't have to roll credentials. It is automatically associated to your app if your intent is to use it. Again, there is a QR code, scan that, and you can find out more about Akamai functions, sign up for

a trial or a free, a free account there and, you know, explore it. So these are the technical underpinnings that I wanna use in order to build first MCP servers. So instead of hand-rolling a new MCP server, I mean we can implement the HTTP protocol or the, uh, the specified API on top of HTTP to build an MCP. There is a smarter way of doing that, and the smarter way is WasmCP.

WasmCP, again, is a CLI that integrates seamlessly with Spin. So what is-- what this allows us to do is basically to focus just on the business logic. And in, in the, in the realm of an MCP, business logic basically means we implement our tools, we implement our resources, our prompts, and we are good to go. Everything else, all those non-functional requirements that we have to tackle, right? Like for example, authenticating users using OAuth2 and

OIDC, that's done by WasmCP, and we can take ready-to-use components off the shelf and compose a final MCP server component together and run that on Spin or on Akamai functions. And that's what we are going to do right now. So I have installed WasmCP on my machine. It's github.com/wasmcp/wasmcp, and it gives you a nice little

CLI. So we can do WasmCP New. We can choose a language, Rust, TypeScript, uh, Python, and we can choose from templates. The default template is me only caring about tools for now, and let's call this new MCP server Hello. We get instructions, "Hey, move into Hello." Uh, Hello. Then let's open up a new terminal on this side, CD into Hello over here as well, and let's run NPM

install because that takes a few seconds, and let's open this one in the editor, and let's do this full screen. So what do we get over here? Uh, we get a boilerplate that's, uh, ready to use, and those squirrel lines will go away once MP- NPM install is done. So all we have to do have-- is we have to express the scheme on which inputs we expect. We have, obviously, to provide the list of our

tools, so again, we have to be good citizen over here and have to provide precise descriptions so that humans and LLMs know how to handle these kind of tools. So we list them in a simple array, and then there's one ingestion function that's called callTool, and depending on, uh, the, the payload that we receive, we have to decide which tool we have to execute. If we jump into handleExampleTool over here, we can see, uh, all

incoming arguments are validated, and if everything's fine, then we basically, uh, concac- concatenate the i- received input with a prefix and return that back to the user. Nothing fancy. So let's close the editor over here and follow the instructions. Next on list is Make. Make goes on and takes the JavaScript code, compiles that down to WebAssembly. So we end up with a tools WebAssembly component,

which is not yet a full-fledged, uh, MCP server. Next in line would be WasmCP compose. Let's compose a full-fledged MCP by taking our tools and saying, "Hey, please give me an MCP. For the sake of this demo, I don't care about authentication." Okay. It pulls all the necessary, uh, other components and composes together an MCP server and gives me instructions on how to run it. I can do a spin up-f

server.wasm, and we should see that running on localhost three thousand. Okay, let's bring back the other terminal, and let's run NP- npx, the, uh, MCP inspector over here. That should fire up in the browser. There we go. Uh, it's already pointing to localhost three thousand right here, so let's connect. Let's list tools. There's our example tool. We provide foo as an input parameter. We run the tool, and we should get back foo.

Great. Nothing fancy. Cool. Works. But that's the local experience. So how can we take that to become a glo... or how can we turn that into a globally distributed MCP server? Well, instead of globally distributing, uh, the Hello World MCP server, I have a decision log MCP server, and let me open up this one first. So there is-- This one is implemented in Rust for the sake of implementing it in Rust. Um, so let's

go into lib.rs over here. There is an equivalent class or structure in Rust. As you can see, we're looking at the callTool function where we branch out and either use one of the crud APIs or tools that is provided by my MCP server. If we dive into the actual implementation, um, let me... No, this is wrong. I want this, and then I wanna do that. Uh, decision.rs. If we go to

the bottom of the file, you can see this is slightly different syntax, but the same meaning, right? I have to define my tools, and if we go to line fifteen or so, yes, list decisions, you can see over here that I use a store. So on my local machine, Spin recognizes, "Hey, Thorsten wants to use the key value store," so it automatically creates a key value store using SQLite on my machine for development time

purposes. And on Akamai functions, it's leveraging the globally distributed, um, key value store without me having to configure anything. From here, it's actually the same story, right? I have to use that make file and compose the server. However, I think it's smarter to update the application manifest and concatenate those tasks so that I can remain in the Spin developer experience and basically do a Spin

build to compile, compose and compile my MCP server. However, I can also combine this one with spin aka deploy. So this right now composes the server, asks me, "Hey, how do you wanna call that MCP server on Akamai functions?" I call it ADR MCP. I allow, um, my terminal to interact with the service registry.

Now it's taken my Wasm file and my application, uh, manifest, tr- generating OCI layers from those files, uploading them to Akamai functions. On the server side or on the service side, we receive those two files, and we distribute them across the globe through all the service regions. Every region has to confirm, A, that it received the workload, that it's capable of running the workload. Then we

instantiate the workload once and do a internal health check. And once that came back for all the service regions with an HTTP two hundred, then we roll a d- generic subdomain on your behalf so that you, that your MCP server will be on the public internet. Obviously, you can shield that with other products and services, uh, like the property manager, to shield it behind whatever you wanna enforce before, uh,

forwarding users to the MCP server. So we get that URL, so we are globally distributed in roughly sixty seconds. Let's move over to an, uh, to a client. I use Zed over here, but that works with any client that basically supports MCP. I add a server.

I go to remote over here, and let me replace this one with the URL that we received, add the server, and we can see I have five tools already. I can go here and say View Tools, and these are the crud tools that my MCP defined. So I can go back and chat with my IDE in the context of my project so that it could leverage, uh, my MCP server. Track, uh, decision to use OAuth two dot,

two dot O for Auth0. Okay, so it should now identify my intent and ask permission to run the insert decision tool. We can see the raw input. Looks good. We allow it. It now calls out to Akamai functions, have already, has already tracked my decision, and I can say, "Give me all the decisions, please." Now the intent is to use the list decisions tool, and again, we

are, uh, yes, it w- might wanna read, uh, details to give me everything that it tracked. Or, and, uh, with that, we have a globally distributed, um, MCP server that's routing all the requests to the closest service region, no matter where your users are. And again, you can write those tools or prompts or resources using TypeScript, Python or Rust templates. Cool. Uh, let's move

on with agents. Um, so... Oops, nope, this one. So for AI agents, the story is no different. However, instead of using another CLI, uh, we can simply pick any kind of SDK that's out there and that's popular. Um, I, I think the Vercel AI SDK is pretty nice to use. So this allows us basically to create a new Spin app using TypeScript, just bring in the Vercel AI dependencies

and to build custom agents. Again, let's move back to the terminal to see how that looks like. Uh, so let's move one up, and let's CD into a simple game agent over here. And

what's going on over here, obviously I have created a new Spin app using a template so that I can get started pretty quickly. I'm leveraging a large language model deployed to a Linode instance with a dedicated GPU. And all my Spin app does, it defines the tools. So this is a simple game agent for the sake of demonstration purposes that can flip co- uh, f- uh, flip coins, that can roll a dice, where you can specify how many sides that dice should have. And over here in the

actual HTTP handler, we load the config, and we define our agent, you know, pointing it to the Ollama instance running on Linode. We give our agent an identity and an instruction on what its intent actually is. We register our tools, and we give it some constraints o- of how many steps it might take. We generate a response and return it back to the callee. Okay, let's, uh, go,

let's do a quit all over here, and let's do another spin aka deploy, no confirm build. And it asks for the name. That's the game agent. So while it is uploading, let me explain what happens when we compile a lang- an interpreted language. So during compilation time, we load the interpreter, we load your custom JavaScript, and then we evaluate the global scope. This has one

advantage, that not just the runtime is hot, also all the dependencies are loaded. And then we take a memory snapshot, write that to Wasm, store that on disk, meaning that every WebAssembly application written in JavaScript is faster than a native JavaScript application when it comes to cold starts and runtime performance. The only trade-off though is that we have to ship the JavaScript runtime along with our, um, with our user code, but we use an

optimized JavaScript runtime called Starling Monkey, which is roughly ten megabytes in size. So we get back an URL. Let me replace this URL over here, and let's say, "Toss a coin."

And we see, um, we see a response back, and if we look at these headers, we can see, uh, the point of presence where we enter the Akamai network is Santa Clara, and the closest service region for our AI agent is LAX. Another interesting header is x-envoy-upstream-service-time, and this tells me, and you right now as well, that it took thirty-three, uh, thirty-four milliseconds to receive the request, to run

through the business logic, to reach out to the large language model, to generate a response, and to go all the way back to the user. Oh, there's an internal server error. Great. Awesome. Let's try this again. Oh yeah, that was an user error, sorry. So the error was staying thirty centimeters in front of the screen. Um, looking at it now, we are eight hundred sixty-six milliseconds. Um, that is more realistic because we have to go through the LLM to basically see that, uh, our to-- our

toss resulted in tails. Let's do that again. Tails, tails, tails, tails. Ah, there's the heads. Nice. So roughly eight hundred twenty milliseconds is what we could achieve with a full loop through the LLM by taking our agent and deploying it to Akamai functions. Okay, so the developer workflow is always the same, no matter if you build any kind of line of business application, if you build an MCP or if you build an agent, you

implement your application using any language that you could compile down to WebAssembly, Rust, TypeScript, Python, Go are the most prominent ones these days. You can test it locally with the Spin CLI. You deploy it using a single command called spin aka deploy, and finally, your, you point your clients to your MCP or to your agent. So final or key takeaways from my, from my point are portability. You

can take those Spin apps and run them on any WASI-compliant runtime. Akamai Functions is one runtime, and it's the globally distributed runtime. Everything is based on open standards, so Spin as CNCF, WASI, WebAssembly are governed by the Bytecode Alliance. But ultimately, you gain speed, right? You have fast response times. Even the time to market for your developers to build MCPs and to build agent goes down because you don't have to roll the infrastructure. You don't have to

implement non-functional requirements. That's all taken care of. And in terms of infrastructure, there's no ops. It's spin aka deploy is all you need. There are other commands in the box so that you can ge- gather metrics, that you can retrieve logs, that you can upgrade, and so on and so forth, but developer experience is our, our highest metric when we started Spin and why we continue in that direction. Um, we have a

new website that's called the Akamai Developer Hub, where we share stories and experiments and, and experiences like these. So go check out developers.akamai.com. And if you have any further question, find me at the Akamai booth. And with that, thank you very much.