Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer, Akamai Technologies

I mentioned before, my name is Du'an Lightfoot. I'm a developer ac- advocate at Akamai, and this is my colleague.

I'm Khaja Omer. I go by Omer. I'm a senior software engineer at Akamai.

And we're gonna be talking agents that own their own inference, building production AI agents on dedicated GPUs. Now, why are we here? Things are kind of changing. And for me, I started building agents before tool calling. Back in January twenty twenty-three is when I built my first agent. I would take a prompt, that context was about eight thousand tokens at the time, and I would parse that prompt and then make a tool

call with Python and then give the response back to an LLM, right? But as I was doing that, I found something interesting. It can get really expensive, right? And then I started using local models to figure out, okay, how can I do this? And then I learned something else. There's another challenge around that because if I'm using a smaller model, it may not be as capable. And then I may be compute bound. That

memory in that GPU may not handle the larger context, so that's the inputs and outputs. And then the multiple turns and the tool cost. How do we get the most out of the GPUs, the cost, the latency that we need in production or just for our own use cases, right? So the reason why we're here, we're gonna talk about agents that are often renting their intelligence, and then how you can take back that control and own it. When it comes to an agent, it's just a loop.

You invoke an LLM, it has access to tools, it has access to an environment, makes it a decision, a reasoning decision on what actions to take once you invoke it or give it a prompt, and then those actions are being performed by your tools, your Python code, your other programming languages, and then giving back to a model. The model evaluates that pu-- the actions, the results, and then updates the plan, takes additional actions, or the goal is achieved, right? But all

that

requires tokens to be generated, right? And depending on your application, can lead to latency challenges. So we're gonna talk about how you can actually mana- manage all of that on dedicated GPUs.

So what does it mean to own your inference? First, when it comes to owning, owning your inference, you actually own the GPU or rent the GPU. So when you do that, that means that you have access to the GPU, the hardware for that AI to be ran on, for the models to live on. Which means then you have access to control where it goes, so the data never leaves your cluster, never leaves your data center, never leaves your laptop, right? You now

own that intelligence and what's happening with the model. And if you're in the cloud, you can also do something like private VPC endpoints or have it in a VPC, so it never leaves your VPC, right? I know we all have different environments, have different ways to set up our infrastructure. We can talk about all of those. Now, what's the difference? Like, when do you own, when do you rent? Well, first, if your usage is high and

steady, why is that? Yes.

You have the slides.

Yes. At the end of the talk, I'll have a link to the slide and everything that we showed, code, everything. You can download it and take it with you. So thank you. Now, when do you own? If the usage is high, right, if you constantly have a workload running. Because if you're paying for a GPU, you're renting a GPU, what does that mean? That GPU should always be working. Otherwise, what are you paying for? So that's something you gotta think about. Then when it comes to the data,

if you need that data to stay private, if you need to own the data, you care about your intellectual property, or you have some governance and compliance reasons why you can't use a public provider when it comes to AI, that's another reason you wanna own.

And then

the hardware. That's something we gotta think about. Your hardware, if you can afford the capacity and you have the capacity, why not run those models, right, for your use cases? 'Cause then you have that full control. Now when do you rent? If your traffic is low, bursty, you just need intelligence sporadically, it might not be efficient for you to own some GPUs. Next, well, we start talking about intelligence. The more capable models,

they're gonna cost for those GPUs. And oftentimes those providers will provide that intelligence on the GPUs because they have the infrastructure for that, so it may make more sense to use those providers to get that intelligence, to get those benefits. And then the next part, which is often the challenge, is that your teams may not have the knowledge or the expertise to support dedicated or self-hosted AI.

Now, today's agenda, we're gonna start out discussing some of the foundation you need to understand, like the infrastructure stack we'll be using, units and memory budget. We'll look at prefill, decode, and KV cache and what that looks like. And then from there, we'll look at dis- dense models versus mixture of experts, so you understand what that actually looks like on the GPU and why you would pick one or the other. From there, Omer will discuss how to optimize your inference, right? Once you figure

out, okay, we want to do self-hosted, how can we optimize this to get the most out of our GPUs, the s- inference engine that we run, and just our entire workload? And then finally from there, we'll take that agent and deploy it into our own dedicated cluster. With that being said, I'm gonna pass it off to you, Omer.

All right. Thank you. So next, uh, we'll talk about, uh, the environment that y- each of you is going to get for this workshop. Um, we have a Kubernetes cluster with around 200 plus GPU nodes. Each of you gets a single GPU, um, and we have set it up in such a way that, uh, each of you gets sorta like your own namespace, and within that namespace you will have, like, your Jupyter Notebook running, and then you will have your vLLM

inference endpoint running as well that you'll be playing around with. Um, and then we're also gonna be deploying, uh, the agent on there as well. Uh, the cool thing here is that the vLLM endpoint that we have running in your namespace, you're going to be optimizing it, uh, as part of, uh, um, after the first four modules. And, um, I think that'll be pretty interesting. We're gonna look at different techniques that we're gonna be using there, uh, to do so, and you're gonna redeploy this vLLM endpoint over and over again until you

have sort of optimized it to increase the throughput, uh, reduce the latency, and, um, make sure, like, you're meeting all your SLAs.

And then the other thing I wanna talk about is, uh, GPU is on LKE. So I work on the Kubernetes side of things a- at Akamai, um, so this is sort of my wheelhouse, so if anyone has any Kubernetes related questions, uh, feel free to ask any time. But I wanna talk about what Akamai has. So we have the, uh, we're gonna be using the RTX 4000 ADA GPUs. Uh, we also have other GPUs available, uh, but these are dedicated GPUs. Each GPU has 200, uh, each

GPU has two gigabytes of, uh, VRAM with two 60 gigabytes per second memory bandwidth, so the decode will be pretty quick because of that. Uh, we have high performance in TFLOPS, uh, a large amount of CUDA cores, so the prefill will be pretty quick as well because of that. Uh, currently for your workshop we will be using a single GPU, but that doesn't mean when you wanna deploy this at the end on your own, you can have different plans that we offer as well, where you can have up to four GPUs in a node, so that goes up to 80

gigabytes of, uh, VRAM on a single node. So you can deploy, uh, decently sized, uh, models on there as well. Um, and, uh, everything is sort of orchestrated by LKE, so setting up the node, uh, deploying the drivers, uh, setting it all up for you so that way you don't have to do anything manually. Uh, everything is sorta like one click. You click a button, everything gets deployed, and you're off to the races. Um, yeah, for more information, we have a link here as well, um, and at the end you'll definitely get all these slides, so you'll

be able to get to the pages. And then, um, I'll give it back to Du'an.

All right, so for the inference engine, we're gonna be using vLLM. Now some of you may have heard of Ollama, which is great for local, but vLLM offers some benefits that are better for production. One of those be- being continuous batching out the box, the other being paged attention. Now, it also supports the OpenAI compatible API, so if you're using whatever SDK that supports OpenAI,

you can use it with vLLM. It also allows you to do some really interesting tweaks when you're setting up inference for your, for your endpoints and for your applications. Now, I mentioned continuous batching. For those of you that don't know what it is and how it actually works, this ha- happens at the prefill stage, right? The prompts come in, and normally with static batch, let's say you have a school bus full of kids. Well, in order for you to drop, get

new kids, with static batching you have to drop off every kid one at a time, and then once all the kids are dropped off, you go back and get the rest of the kids and then start the next batch. But with continuous batching, you can drop off a kid and pick up a kid all at the same time. This is way more efficient when it comes to the KV caching that happens during the prefill stage. Now, or when it comes to paged attention, paged attention is when you think about the KV

cache and you think about what's actually happening, there's memory space that happens. So the prompt comes in, and the longest prompt is gonna take up memory space. When you, when y- without paged attention, that memory space gets allocated even for smaller prompts, right? But with paged attention, now that is all managed through K- with the vLLM that ensures that you can continuously proch- process, generate, and batch those requests as they come

through vLLM. It has a scheduler that kinda manages the batching, but it makes this really efficient. Now, there are other service engines that are out there, and we'll talk about some of those in this workshop. With that being said, some of the metrics you're gonna wanna care about is gonna be throughput, right? To understand how your GPU is performing, that time to first token. Understand that, okay, you send a prompt, how long does it take to get the first token back? From there, that KV

cache utilization. We're gonna look at these metrics and show you how to get all of that out of vLLM and then the QDEP. So these are some of the metrics as we're going through this workshop you'll, you'll see and be able to better understand. And with that being said, we're going to access the environment, so if you can scan this QR code, pull out your laptops if you wanna follow along. If not, you can follow us as we go through the workshop. In this Jupyter Notebook, we have nine different labs that we'll cover, and all of

the resources and dependencies are loaded into this environment. Now- If you want to look at the code and try this on your own at home, I do have the repo public on Akamai's Developers. I'll have this link along with the presentation and everything that we share available with one QR code that you can download. And if you need me to make a short link, I will.

All right, with that being said, we're gonna start with the first module in the workshop if everyone is connected.

Which one is it?

The internet's not working. Yeah, it was kinda slow for me, too, which is why I went to my own link shortener, 'cause I kinda made that really fast with Russ. So, yeah. Hotspot isn't working. Ho- hopefully, like, if you do have a hotspot, I would use your hotspot. Um, but we'll try to make the best out of the experience.

Sound good?

Awesome. All right, so in the first module, module, we're gonna talk about the inference stack. We're going to connect to our vLLM endpoint, understand it, see how it generates tokens. We'll also look at building our first agent, some things that we need to know, look at some me- some metrics, and then understand the model that we'll connect to. The model that we'll use is gonna be Qwen3-4B, but we have several lo- models that sit on a PVC that you can actually load and unload for

this endpoint. Now, when we talk about vLLM, one of the things about it is when you serve an endpoint, it just has one model, right? So, if you wanna serve two models, you're gonna need another vLLM endpoint, right? Which means you may need some model router, a gateway in front of it to kinda handle that. Now, it does support authentication, but there's no authorization, so you may need a gateway around that, right?

So, Agent Gateway is often... KServe is another one. Um, any others?

There's llm-d. There's, there's all sorts of open source tooling out there that lets you do that. Uh, but the main, but the main one is, like, KA, KServe is what we use a lot, um, and it's, it's pretty good at doing the scaling, doing the routing, and the authentication as well.

Right. Another thing it doesn't have is rate limiting, right? If you're processing requests, you may want a rate limit. So again, you'll wanna find a gateway to put in front of it to manage those requests and also handle the routing.

I mentioned that when it comes to the runtime, there are other se- service endpoints that are service runtimes that you can actually use for the server. SGLang is one, TensorRT is another, and Llama CPP, which we do go over this later with a little bit more in depth so you can kinda understand it. Now, in this environment, you are connected to a Jupyter notebook that lives in a namespace. You also have the vLLM server endpoint that you will see here in

a moment, and all of this is in, in its own pod. And then you'll have another pod... Well, another pod that you can launch for the agent, and we'll see that, too. So, let's go ahead and install our dependencies.

So, we'll install our dependencies. I've kinda already ran through this, so some of this is already loaded into my runtime. I figured we don't know how the internet will be, right? But the good thing is once you send a request, it happens within a cluster. So, once we send our first request to understand wh- how, how we're configured in vLLM, you can notice the model that we have, the name of your namespace inside of the cluster. The kube

config is just for your namespace, so that way nobody can access anyone's else's namespace in the environment. And then we don't have a API key set, but you can set one with vLLM.

Now, from here, we're going to use the OpenAI API compatible endpoint to be able... Or API, to connect to the vLLM Q Qwen3-4B, four billion parameter model. I believe we're using that at FP8, meaning that it actually sits at, um, eight billion parameters. Is it eight billion parameters?

No, no.

Four billion?

Four billion parameters.

Four billion parameters.

We're using BF16.

Oh, BF. That's what it is.

Full precision.

Yeah, so we're using BF16 full precision, so you times the bits or bytes or parameters by two. So, it's four billion, which means it's eight gigs on the GPU.

Now, once we invoke, we got a response from the model, and we can see what it was actually served by. Now, this was all within our cluster. That was pretty fast, right? It lives within our cl- cluster. So hopefully with this experience is we're not hitting a public API. We're already connected to this HTTP. It's gonna happen right inside of the cluster. The response, it should come back a little faster. Now, we're gonna trace, send a trace in the end so we can actually understand how this actually looks like when it goes through the c- prefetch, the KV cache, and the

decode phase.

So here, we have our stream that we're going to configure. We're going to look at the chunks and do some calculations. We'll get that time to first token so we can understand that. We'll see how all of this is generated here in a moment. So, if we run this...

I ran that one again. We'll come down here and run that. This may take a little bit to go through, but we should see time to first tokens took thirty-seven milliseconds, and time output token, the decode phase, which means every token after the After the time of first token, right? That's gonna be that inter-interleave latency, right? That inner latency that happens when those tokens are processed. So we're calculating that, and we're getting about

forty tokens per second, which is probably gonna be our cap. But when we start talking throughput, that kind of changes because this is just one request. But imagine if we had hundreds of requests or sixteen requests, that throughput's gonna be a lot higher, but they're still gonna be capped at that forty tokens per second, right? Or it could go down, but we could still get more on that throughput. And I believe Omer will go deeper into that. So one of the things that I wanted to showcase is, the question

is around why don't you just run this on CPU? Um, I didn't wanna do CPU on environment because this, this, um, pod for this, or this container for the Jupyter notebook is only one gig, right? So trying to run a local model on this is gonna-- it's just gonna crash. So I kinda simulated what that would look like. But if you're trying to run a local model on CPU, it's gonna be a lot slower, right? It can't do the complex mu-multiplication in those matrices like a GPU can.

That's the purpose of the GPU. It does all those, all that math in parallel, which makes it a lot faster, right? So we simulate this on, on the CPU.

And then from there, we actually look at, look at this on the GPU to see how that looks.

We got forty-five tokens per second.

And one of the things about this, and which is interesting to me, is that I do a lot of, I would say, local models in running them, but forty-five tokens or thirty-nine tokens per second isn't that bad for an application if you're running it for a specific use case, right? If you have a specific use case and you don't care about the latency, you don't care about those... Like, if you have a chatbot, you want that kind of-- that first token to come fast. But for a standard application where

you just kick off a job in the background, well, that thirty-nine tokens per second may not be that bad, right? So sometimes a lower token per second or a lower time to first token may not be as important depending on the application and the latency that you care about, right? You may care more about the intelligence and the job getting done than you do actually how fast it actually happens, especially, like I said, if it's a background process.

Now, here we're gonna build our first agent. Now, for those of you that when we talk about agents, there are no tools in this agent, so it's actually just a request. But just the foundation of the agent, we're giving it a system prompt so it knows its role when we do give it to us later on in the course. But right now, we're just giving it a single prompt, and we're giving it a role of the system, uh, a solutions architect agent for Akamai.

And you can see that it was able to generate the response. Now, this response, pretty fast, it works. But then the question is, if we're trying to do this at scale, or we're trying to build an agent that does more than just take a, take a prompt and generate a response, it may need to make multiple tool calls, right? So that's when you start to think about things like the

KV cache. You start to think about how much context, 'cause when we start talking about that KV cache, that's actually how much context you can actually fit on that GPU, right? And so with continuous batching, what happens is you send a prompt to the model where you may have a hundred users trying to send prompts at the same time, right? Well, continuous batching are all gonna be different sizes. So a smaller GPU can actually handle a bunch of requests if you're using some-- with something like vLLM.

Now, if you're using something like Ollama, you'll see you'll get throttled a lot faster because it doesn't process those requests the same way. But vLLM actually handles that with their scheduler a lot better.

And so here we're going to set up our, our, our agent and invoke it, and we're gonna actually gonna look at the steps. Now, the code that we're using for this is inside a comment, and you can actually look at all the code that we actually are importing that is custom code. So here we have this foundation.py that allows us to do some measure- measurements from, with Prometheus from vLLM, right? We can look at those metrics. Now,

speak-speaking of metrics, you can use Prometheus, and you can use Grafana, right? At home, I always use Grafana. Also, when I'm building my agents, outside of just looking at the metrics, another thing that I always build, one of the first things that I build into my agents is observability, right? Langfuse, that's something I commonly use. How many of you have heard of Langfuse? Now, there are other tools out there, but it's open source. You can run that in your cluster as well, right beside your agents, and then have access. That, now your observability is private as well, right? And you can

see every step, every trace, every span in your agent with how many tokens, what was the latency, not just for the time of first token, but for every tool call, right? If you're trying to troubleshoot or you're trying to evaluate, now you can see all of the latency steps and the spans. But then you can see, okay, I'm using a smaller model, and it's making way more tool calls. It's taking way more time than it should. Why is that? Well, that's where evaluations come in. And Omer will talk more

about that. But it's important to have that visibility into your agents, so you can understand when you're using a smaller model how to optimize. And that could be something as simple as a prompt, removing tools, or making the agent just a little bit more specialized.

And so we'll run this.

And so we're looking at the steps. We're gonna look at the max tokens, and we should see the time to first token. So you can see as we increase the tokens, what happens to the time to first token in milli-millisecond as we increase the prompt, right? As the prompts get larger, that time to first token takes longer, right? So the question is, what do you optimize on? Do you optimize on the median or the largest, right? The one that took the most time during the latency. Well, you're going to optimize on the one that takes

the, the most time rather than the median, right? And then that way you can increase your median by doing that.

Now here we talk about renting versus owning. We're gonna look at tokens, how much are being generated. This is where I, I mentioned that if you're using bursty, where you're not using invoking a model as much in with a provider, versus if you have a agent or some agentic applications that are always working, you can kind of compare that cost.

So here we generate a response just to get those tokens, and now we're gonna actually look at the billing and compare which one would actually be more efficient. So if you're processing, let's say two hundred thousand tokens per day,

on a hosted bill, rent it, that's gonna be about fifteen hundred per month, right? But if you're renting a GPU, that's gonna be about a thousand per month. Now, it does matter the size of the GPU, the size of the model, the... if it actually performs the way you need. But these are some of the things you think about when you're trying to compare should I rent versus own, and you can actually do the math on that to break that down.

And we already talked about these with the data resiliency, you know, the compliance, the security, the controls, and all of those good things when it comes to running locally. Now, before we go, the runtime landscape, I mentioned that we're using vLLM, but you can also use SGLang, which is good if you're using-- if you have requests that share long prefixes, multi-turn chats, so for agents. A lot of organizations are using SGLang for that. When it come to Tensor, well, this

is about that peak throughput on NVIDIA hardware. It really specializes in providing you that. And then Llama CPP for that edge, like that really fast processing. It can help you with those heavily quantized models, right, on a ARM server.

So we're going to invoke one more time here. And we're looking at our client just to verify that we're still good. We're verifying that we can connect. We can see the model serve, and we can see the ID, and I believe I can print the entire... So we can see how that require-- that request actually looks when it comes back. So I'm gonna print this whole thing as well. So we'll... And now you can see... And actually, let's pretty print it so we can see it even better.

There we go. And so when that request comes back, you can see the ID, you can see the max model length, which the, that context length, you can control that, right? You can tweak those parameters, of course, when you work with a model, but we can see everything about the model itself.

Now we're gonna move on to the next session. Um, this will have some taking it further where you can explore the cluster, explore vLLM. You can feel free to do that in this notebook. So I'm gonna close that out. We'll discard that, and we'll move on to units and memory.

Now, the next section, and I'm gonna kinda go faster so Omer can have some time. Now, when it comes to units and memory, we're gonna talk about the ceiling when it comes to token per seconds on using a model. We're also gonna look at the model's real config and understanding how a model actually is, sits and how it actually looks, um, when it comes to a GPU. So here, like Omer mentioned, we are using... And I'm gonna make that a little smaller. Let's see if we can get more of that on there. All right. So when we talk about the,

the model actually sitting on the GPU, you have the model weights. For the four billion parameter model, that model weights at BF sixteen is gonna be eight gigs. So we have eight gigs, and the rest of the room is gonna be for KV cache. Meaning how much context can you fit on that GPU? 'Cause it all has to fit within there. Now, when your prompt comes in, when we talk about prefill, the model is gonna read that entire prompt and store it in a KV cache. And then every time it generates a new token,

it's gonna update that prefill so that prefill is gonna grow, right? So just because your, your, your prompt that goes in is one size, as it goes through and is processed through the model, it's gonna increase as it generates, right? And then imagine the tool calls, the system prompt. All of these go to the model, and all of that has to fit on the KV cache in order for it to be able to handle your request efficiently. And so we're also looking at, um, if we're using at FP8, which Omer is gonna talk about

quantization and what that actually looks like to s-shrink a model and to still get the efficiency that you're looking for. Now we're gonna install our dependencies.

And then from here we're going to connect to verify what we have loaded. So again, we have the Qwen three four billion parameter model, our workshop namespace, the cluster, API key Now, when it comes to models, we have the config.json. Like, what is actually that file? When you download a model, you get the config.json. Let's say you go to Hugging Face, you download a model, you're gonna get a collection of files. All of those files are needed, right?

You're gonna get the config.json, you're going to get the saved.tensors. Those tensor files are the model weights, right? But the configuration is what tells vLLM how to handle that model, right? So you can actually look at this and explore those files to understand what they actually look like. So here, we're showing what that configuration file looks like. So we have a hundred and twenty-eight layers. The size is one thousand twenty-four. The attention head, there are sixteen attention heads. The values are eight.

Vocabulary size, right? How does the model train? What is the vocabulary size of that mo- model, right? You can actually look at these configurations. Can you modify them?

No, right. These files cannot be modified, but you can fine-tune, right? You can use LoRA. You can do all these things to adapt the model, but to modify these files, if you try to do that, it's just gonna crash. It's not gonna load.

Now, here we're gonna look at the bytes per weight w- and where it comes from

and we calculate. Here, we're looking at a smaller model at BF sixteen for the point six billion parameter model and the parameters, you multiply by two, one point five gigs, and for the eight billion parameter models, what those parameters look like, and it's gonna be eight gigs.

And if we compare the BF sixteen versus FP eight. Now, when we talk about pre- precision, most models when they're trained, they're gonna be at sometimes FP thirty-two, but you'll often get it at FP sixteen, full precision, meaning that you multiply whatever the model size by what? Four? By four, right? So this four billion parameter would be sixteen gigs at full precision, right, on your GPU. When you quantize it down to,

let's say, FP eight, well, you kinda lose a little bit capabilities of the model, but not much, right? But that's when you start to evaluate, right? If you're compute constraint, you have this larger model, maybe seventy billion parameters that maybe you can do it at FP eight or maybe with int four, right? All the way down to where now that seventy billion parameter models is rather than thirty-two is sixteen

gigs, right? If that's possible, right? I don't know how much you're gonna lose, uh, for the precision, but you'll have to evaluate that. And so evaluation harnesses are important.

So we'll look at the GPU, where the weights live.

So here at BF sixteen, you multiply by two. FP eight, you multiply by one. We're looking at this for Qwen, and then we have another model that we're gonna use in this course where it say at eight billion parameters is five gigs on the GPU, which you can see how much you save with the still intelligence that works for the use cases that we're discussing if you decide to quantize down.

And so we're looking at this card, the gigabytes per second. You take the, what we're actually calculating, okay, we know the how, the speed of the GPU is three hundred and sixty, and we're showing how to calculate to get the tokens per second and where we're actually bound. So we're gonna be bound by the speed of the actual GPU on how fast it can go, and we're showing the math for the KV cache here, and I'm trying to go a little faster so we can get into the quantization and optimization. But if you go through these, you can read

through it and understand how the KV cache formula works. Oftentimes, we may not have to go through this and figure it out, but you will have to when you're trying to calculate, for example, you have a hundred users you want to serve, you have a smaller GPU. The question is, I got the smaller GPU. Let's say, I'll give you a perfect example. I had to do a workshop, and I had to figure out how to serve eighty students and perform inference for all the students with about three or four GPUs that were about twenty gigs, right? Do I need

that many? Is it gonna fit? Well, how big is my context for the workload? What if everybody is invoking at the same time? Will the GPU, will the cluster s- support that? Will continuous batching work? So you set up an evaluation to evaluate the entire workload and to determine, okay, is it gonna fit or do you hit that roof line, right? Do you have to start to have spill over the CPU? Does your application slow down?

Uh, the memory weights, memory budget versus KV pool. So we'll look at that, and then here's how you can actually get that, the math. So here's our... We can get thirty-seven thousand tokens on this GPU, right? On a single GPU, there's eight gigs for the model weights, and then the room that you have is gonna be thirty-seven thousand tokens, right? And so if we have multiple users sending requests, right, well, this will show how

many users and their context left. So if we have four users, they can all send eight thousand tokens, right? For input/output tool calls, it's gonna add up to eight thousand tokens at the same time on that GPU.

And now we're showing attacks as this increases the different requests and the generation of that request. And this kind of shows how to do that math when you're trying to understand your compute constraint, your GPU, and then the workloads that you're performing. So you can take all this code and apply it and be able to Take your use cases and do the math on it so you can see if it fits on your GPUs.

So we'll...

I will go to the prefill and decode. So going a little bit deeper into the prefill and the decode and the KV cache. Again, we're using the Qwen three model, but we're actually gonna dive in to see how this looks. So we have the prefill, which like I mentioned, is gonna be focused on that time to first token, right? From the time you press send, how long does the first token... If you're streaming, right? Because if you're not streaming, you're just gonna get the whole response back. But if you're streaming, how long does it take to get that first token back,

right? And that's gonna be that prefill stage. And then the decode is gonna be that token, every token after that. That's all measured. And if you're looking at the metrics inside of vLLM, you send that to Grafana, you can actually view that on the dashboard and you can watch, okay. As I increase my prompts, you can watch it increase, you can watch it decrease, right, right? And really understand what's actually happening on that GPU when you're sending requests. So

we'll go through here. We'll install our dependencies. Then from there, we'll look at our environment, make sure we're connected. Yes, and we're still using the Qwen three-four B. And we're gonna look at the self-attention, meaning how does that KV cache get processed? So this is the actual formula, so you can look at that softmax to really understand how that KV cache is being handled within a model. We're just doing it by hand just so you can see the math on

that. And so when it comes to the KV cache, there's the naive approach and there's the KV cache approach that I would say is handled. So the naive approach is you take a prompt, the model looks at the whole prompt, and then one by one stores that prompt in the KV cache and generates a response. What KV cache does, it says, okay, I'm gonna take the prompt, I'm gonna learn from it, and I'm gonna store it, right, in parallel. And so the KV cache allows you

to learn from the prompt to be able to reference the information rather than just doing the fixed sequence of the naive caching approach when it comes to handling that KV cache. And so if we generate this,

we can look how it computes

the reduction of how much if you use the KV cached approach

rather than the naive approach. And we're just showing all that by hand in code so you can really break it down and understand. And now we're gonna look, okay, using K- KV caching and we'll measure the time to first token. And so here we have fifty-seven milliseconds and then the decode was twenty-five milliseconds. So we'll continue on.

And we can see what vLLM is actually exposing, so we can actually look at those metrics and we're just showing some of the metrics that are actually being exposed and how you would view that in vLLM.

Now we're going to invoke the prompt and we're going to increase it and you can watch that KV cache fill.

And you can see this is gonna take a little while, so we'll give it a couple minutes while this process. We're increasing it so we can see the KV cache usage and we can look at the metrics. So we have peak usage running and then the KV's usage after it's finished. And this will be the math on how you would process that. Again, the worker that we're using for this code is in the foundation.py. So that's going to be

the helper for this code. And now we're gonna look at the warm, the cold, um, against the warm prefix caching. Now prefix caching is an interesting thing. Um, don't confuse it with prompt caching. For, and that's something I should have mentioned before. If you're using a provider and you're paying by the token, prompt caching, especially if you're, you have a continuous application that's running within every five minutes, prompt caching is a must, right? You can de- you can decrease the cost by like up to seventy-five

percent depending on who you're using. So there is some ways you can use a provider and save on cost. But when we're talking about prefix caching, this is pre- caching that prefix for the KV cache. And so when we do this, we can see how this actually is handled here. So the cold is gonna be two hundred and twenty milliseconds and is down to forty-one milliseconds. And this all shows you how to do that in the code. So you can go through that and I'm trying to go a little faster so Omer can get to his

part and hopefully we can deploy.

All right, so we'll go back and we'll go to the fourth section where we talk about dense model versus the mixture of experts on the GPU. Now, when it comes to a dense model, what is a dense model? Well, a dense model, when it sits on a GPU, that whole model is used to generate tokens, right? So if it's thirty billion parameters or let's, let's just use the model we're using now. So the Q- Qwen three-four B at BF sixteen, eight, eight

gigs on the actual GPU. Well, when that actually generates tokens, when it actually is working, that whole entire model is working, right? So it's gonna be a little slower than if you use a mixture of experts. Mixture of experts may be a larger model, maybe it's thirty gigs, but when it actually generates inside of the model, it takes a collection of experts and only that That section of experts does the generation. So even if it's 30 gigs, it may be

1.5 gigs just to do the generation, so it's gonna be a lot faster. So that's one of the benefits. Even though it's a larger model when it's a mixture of experts, it's actually a, can be, perform a lot better within an application or a lot faster within an application.

And so here we kind of talk about memory and why decode is memory bound, meaning that when we talk about the generation phase, after that time to first token, when that generation actually happens, that's actually gonna be bound by the memory. So when we looked at the speed of this GPU, that's bound. You can't go any faster than that, right? There's nothing we can do to tweak that to make it go any faster. But there are some things we can do to help this at scale. So we're gonna look at our roof line and plot

that so we can actually see it. So we'll make sure that we have our model loaded. So we have the Qwen three four billion parameter, right? And it's gonna be compute bound, so it's gonna hit that roof line. So we're gonna look at the flops. We're gonna look at this. So the ridge line is two hundred and ninety-seven flops per byte, right? And where this is bound by the memory.

Now, if we look at the mixture of experts, now we're just gonna look at this as code so we can actually understand it.

So at this thirty billion parameter models, right, a hundred and twenty-eight experts and only eight active per token. That's forty-eight layers, two thousand and forty-eight hidden. And so now the expert, mixture of expert, this is the weights, this is the share weight. So this is how much it c- how much is being used when it generates. That's a lot, that would be a lot more efficient than using a smaller model. It can actually be faster when it actually does the generation.

And so here we're looking at the, the dense model, right? It's nine times fewer bytes than a dense model, right? So it decodes a little bit faster. So that's kind of breaking down the two when you're actually trying to figure out, okay, do I want to use something like Mistral, right, or do I want to use like a Qwen model?

We compare the dense sizes.

S- so here we're gonna look at the weight, we'll look at the bandwidth, the GPU serve. So measure Qwen four B is about forty, forty tokens per second, right? The smaller model of the Qwen, the zero point six, now this isn't done on the CPU, but we're just showcasing three hundred tokens per second, right? But we're looking at the ceiling. This is actually looking at the bandwidth, and it's doing the math here to get that. The max that we can get, the ceiling that we have on this GPU is gonna be forty-five to-

tokens per second.

Now, if we enable reasoning

on that model, and we can look at that. Now, when it comes to these Qwen models, if you're wanting to do any type of agent and the model needs to reason, meaning that maybe the prompt comes in, you have tools, and it needs to make multiple tool calls, well, you're gonna en- enable thinking. So now when it makes the tool call, if it gets the results and it needs to make another tool call or verify the results to make it the accurate tool call, you're gonna have to enable thinking on these models, and that's just a parameter that you can

adjust in the model, right? But with thinking off, it just generated a response, it was thirty-five tokens. But with thinking on, now it had to think about it and it costs about six hundred tokens, right? Now, with some of the models, like provider models, you can actually set a token budget for like the thinking, so that's one of the things you can save more costs with a provider model. And maybe some of the open models you can do that too. I'm not sure.

Yeah, you can.

Yeah, you can also adjust that token budget on the thinking as well. So more tokens on thinking isn't always good, but sometimes it can help the model better troubleshoot and better get to that end result that you're looking for.

And so when we talk about inference, there's a lot of things to think about, but one of the things we want to think about is that memory bound limit for the actual card and then how to apply that for the agent. So we look at all of the modules, we talk about that KV cache, the prefix caching, the attention heads, as well as quantization, which, um, Omer's gonna cover here in a minute. And then from there, that continuous batching. We are also discussing that. So if we split the prefill from the decode, you have the prefill, GPU pool and the

decode, which is memory bound. So the prefill, compute bound, right? So it needs to go from your memory to the GPU, and it needs to be cached, right? There's... That, that phase you can, you can do a lot of optimizations, but when it comes to the decode phase, there's nothing you can really do because you're bounded by the GPU. And so we kind of show this in code rather than actually showing it because, again, we only have s-

so much limitation in this workshop. Now, with that being said, I am going to make sure that we are good to go here.

Um, I don't think I need to switch them out. I think you should be good, Omer. I'm gonna turn it over to you.

All right.

And then, yeah, we should be good to go. You can talk about quantization. Thank you.

All right. Thank you, Du'an.

Yep.

All right, so let me close that. Sure.

That, that's yours, actually. There you go.

Okay, cool. So I'll be talking about a whole different, um, uh, optimization techniques I think you can use to deploy these models on Kubernetes, um- Like I said, uh, my background is in Kubernetes. I develop a lot of Kubernetes stuff at Akamai, and I work on the managed Kubernetes platform there. So a lot of the stuff that we do is we wanna make sure that we're able to use GPUs on Kubernetes really well, uh, use all the different features that they have. Um, and, uh, we deploy a

lot of these models using some of these techniques that I'm gonna be talking about. Um, so there's, uh, different techniques. So we're gonna talk about quantization. I think this one's, like, the main big one. Um, it's super important. We're gonna talk about speculative decoding. Uh, this is a pretty cool technique. It can give you quite a big, uh, performance boost in your, uh, throughput. Um, we're gonna talk about, um, like, how do you, uh, you know, uh, tweak your LLM,

uh, VLLM config to make sure that you're getting max performance out of it and, um, you're meeting your SLAs. And then we're gonna actually do some tuning and evaluating. So this-- I'm gonna leave you guys with some sort of framework that you can use to figure out what exact configuration you need to use for your specific workloads and specific use cases. Um, so I'll dive into quantization. Um, let's see. And if we have time, we can definitely go, uh, we can have, like, a quick Q&A as well. But I

think we have thirty minutes. I'm gonna run through it, uh, at a good pace. So in quantization, um, there's a whole lot of different, uh, there's a lot of different types of quantizations you can do. Um, you know, there's the BF16, the, the one that the model is gonna start off with. That's a float precision model that, uh, Qwen provides you. And then there's a lot of other quantizations on top of that. People have created FP8. That's one of the models that we're gonna look at in this particular module. Um, there's INT4,

uh, there's AWQ, GPQ, uh, GPTQ. These are different types of, uh, ways you can quantize the model. Um, there's NVFP4, which is NVIDIA's proprietary quantization technique that they use. Uh, it's hardware accelerated on Blackwell GPUs and more. Uh, that one is pretty cool. Um, but let's see. Let's dive into it. Um, uh, make this... Hopefully we can fit that in

there. All right. So as you can see, you know, like, if you have the BF16 weights, it's going to take a lot of your memory. Uh, in this particular use case, we have twenty gigs of VRAM, and we sorta wanna maximize how many requests we can serve with these twenty gigs. Uh, so we wanna make sure we have enough space for KV cache. Uh, reason being, the more requests-- Uh, the more KV cache you can have on the memory, the more requests you can process in parallel, uh, thereby increasing your throughput. So that's super important. Uh,

so that's why we essentially wanna do quantization. We wanna reduce the size of the model, so that way you can fit more KV cache and essentially increase your throughput. Um, so that's one reason. The other big reason I think is also because you're shrinking the weights essentially by in half, you don't have to do that memory movement between the different, uh, memory hierarchies, resulting in essentially faster decode as well. So we will see you'll go from, you know, forty tokens per second, which is the base model, will probably go up

to, like, sixty tokens per second, if not more. Um, so let's look into this. Let's, um, install the dependencies, and then I'm just gonna make sure that the model is still available and everything is running. So yeah, we do have the model running, uh, the same four billion parameter model, and it's running in the namespace. Again, as I said, um, sorta like a high-level overview, there's a whole different sorts of different techniques you can do quantization with. Um,

uh, for example, like, um, uh, LlamaCPP is a pretty popular one for running a lot of these models at a lower precision, uh, that uses GGUF format. Uh, that format is pretty cool. You can have, um, uh, you know, five-bit, three-bit, two-bit, one-bit quantizations, which is pretty interesting, um, and pretty cool. But the, the downside of that is you lose a lot of, uh, you lose quality trying to get a lot of performance out of it. So there's a trade-off there. Um, but for the most part,

going from BF16 to FP8 is a very negligible drop in quality and performance. Um, and it will result in more performance while keeping the quality exactly the same. Uh, so that's what we wanna showcase. So that's one of the assumptions that we're making. Obviously, like, when you're doing this for your particular workload, depending on what you're using it for, I highly encourage people to have their own evals. Um, me-- So that way, if you're switching from, you know, BF16, your evals are saying, "Okay, everything is good. Everything's running well for our workload." You switch to

FP8, you might have issues where some things are breaking here, sometimes it's hallucinating, sometimes it's going in a loop. So a lot of those things can happen for very specific use cases, and in those scenarios, it's very important that you have your own eval set and you're running your own evals to make sure any changes you make, um, are, uh, okay to make, and also you don't have any degradation in performance.

Um, again, we'll run and make sure that we have the BF16 base, and then we're gonna measure the performance. So, uh, we have a little load test, uh, utility package that we use to do some, uh, testing. Essentially what we're doing, if you look at the levels where these are all, like, at dif- at different concurrencies, we're making this request. Each request is going to have about two hundred and, uh, fifty-six tokens, and the output is roughly going to be about one hundred and twenty-eight tokens.

So this is sorta trying to simulate how many requests you can process at one time, and we're sort of gonna use this as a baseline to see what the baseline performance is. Um, so for thirty-two, uh, we're getting aggregate performance of seven hundred fourteen tokens per second, which is pretty good. Uh, at single concurrency, we're getting thirty-seven point five tokens per second. Uh, so that's important to, uh, note down. And also we're getting, um-

61 milliseconds for time to first token. And then you'll notice the time to first token goes up, right? Like, the jump between thirty-two and sixty-four is quite large. Uh, we're going for 277 milliseconds to, uh, 1,300 milliseconds. That's quite a big jump, and you might not necessarily wanna make that trade-off, meaning you might wanna potentially stop at only 32, uh, concurrent requests that you're processing and not go higher than that. So that is an important thing to notice there.

Um, again, as I was saying, um, evals are extremely important. Uh, reason being, again, you can have performance degradation when you do quantization, and you can lose, uh, some intelligence. Reasoning can, uh, not work as it should. There, there's going to be a lot of issues, so you wanna essentially... I'm kinda simulating here, like a, a, at a very basic level, what kind of evals you might wanna think about. Um, you know, you wanna think about, like, if the JSON is being generated

correctly, if the JSON is being corrupted when you reduce the quantization. You know, at, uh, BF16, you might have good, uh, responses, but when you go to FP8, maybe potentially your function s- function calls are failing. Why is that? It's probably because it's not generating the JSON correctly. So there's a lot of these, um, eval cases that are already out there that you can kinda use on the op- open source. Hugging Face has a lot of evals. You can go on GitHub, you can find a lot of these. But you wanna come up with your own for your own use case.

Um, but then, so next thing what we're gonna do is we're gonna switch from BF16 to an FP8 model. So we are going to essentially reduce the size of the model in half, and once we do that, we'll kinda get to see, um, what the performance differences are and are they worth it, and are we losing any quality over it? Uh, so to do this, uh, you will need to be in the root and then click on Manifest. And in there, you'll see vllm.yaml. So,

um, I'll kinda go over this. So th- this is a typical manifest, uh, configuration file that you, uh, uh, have for Kubernetes deployments. Um, how many of you have used Kubernetes before or have knowledge of Kubernetes? Okay, quite a few. Okay, great. So this will be quick. I don't have to go over all, all the details. But essentially, we have, um, uh, a PVC, which is going... which is essentially our cache where all of our models are living.

And, uh, we have a simple deployment with vllm with a lot of different arguments and flags here. Um, so you can see we're using the Qwen, uh, three four billion parameter model. Um, and then we're gonna switch it to this FP8 version from Red Hat. Uh, let me...

One more quick thing I wanna quickly mention is you see the dynamic at the end. Uh, this is pretty important. This means that not everything in this model is quantized to FP8. In a lot of cases, in a lot of different quantization techniques, you might wanna quantize the model, uh, only specific parts of the model and not the others to make sure the performance stays the same and you don't have a lot of, uh, issues there. So that's one of the reasons why you will see dynamic. You will see all kinds of things. You'll even see quantizations where certain heads of the model are going to be four-bit, and then

some are going to be eight-bit, and some are gonna be full precision. And, you know, it's, it's sort of, uh, interesting that people are doing that and finding a lot of performance there. They're, they're finding the performance gain while making sure the quality stays the same. So once this is, uh, changed, we're gonna save it, and then, uh, we'll just quickly go through and run it and apply it. So this is just making sure that the diff is good, um, we're not causing any issues here. Uh, that should be fine.

And then we'll just go ahead and... Where is apply now? Right here. We'll, we'll just go ahead and apply and that'll roll out. Um, it's gonna take, uh, a little bit of time. Let me save that. But another thing you guys can also do is you can open up a terminal tab as well, like, if you just click on the plus sign, and then you'll see a terminal option. And then that'll also let you see sorta, like, what's in your namespace and what you can do here. So if you get... You

see the vllm endpoint is still coming up. Uh, this pod is essentially your Jupyter Notebook running within your namespace. Um, let's see. This will take a little bit of time to come up.

But yeah. While this is coming up, does anyone have any questions? We can probably have one question and-

That's pre-quantized, right?

Huh?

The model which we, uh, changed, it was pre-quantized already.

Yes, it was pre-quantized. Yeah. You can quantize your own models. There is LLM Compressor. Uh, it's, uh, it's provided by vllm, I believe. LLM Compressor.

So there, there are, there is tooling out there for you to do this on your own if you would like. Um, but yeah, the one that we're using currently is already pre-quantized that we downloaded off of Hugging Face.

All right. I think that should be up. Again, it's taking a little bit of time, mainly because it's taking all the weights out of the VRAM and loading the new weights in, so it's gonna take a little bit of time to do that. Um, anyone else have any question?

On your vllm YAML, your args, you're-- like you had two flags for your model.

The dash dash model dot- what's after dash model?

I'm sorry?

Look at your, your VL- your YAML file.

Mm-hmm.

In your args. Look... Oh, I meant... It looked like they had, uh, two flags on the model.

Oh, they did. Oh, that's a good call-out. Thank you for that. Did not see that. All right. We're gonna go through it again.

Let's rerun that.

Yeah, that's probably why it didn't come up as quickly as it should.

Oh, I can't do that.

Yeah. All right, any, any other questions while this is going through? Go ahead. How do you choose between FP8 and INT8? Uh, how do you choose between FP8 and INT8? Uh, that's a good question. So it depends, right? Like INT8 is going to be slightly less precision than FP8. That's essentially the reason. Uh, the, the difference is like FP8 is floating point, so you get a little bit more precision out of it. Uh, and then INT8 is integer, so you could use either or. Uh, again,

these are all the other options that you have. You can choose based on your workload and what your constraints are. Um, but yeah, you could totally go INT8 as well, and it should be fine. It says immature stacks are good support. Like, what does it mean by that?

Um, where does it say that?

Under good fit. Under a good fit. In the table it says, uh, it's a good fit if you're a mature stack. Mature stack. Oh, yeah. I mean, uh... Oh, that's the other good, good call. Um, it depends on the GPU as well. Some of the GPUs might not necessarily support it. Oh, GPU support. Yeah. So GPU support is important. That's another good call-out. So some of the, um, uh, GPU might support it, but it might not be accelerated by the GPU, meaning you might not get the full performance that you would be expecting out of it. Um, for example, the GPUs that

we're using, we're using RTX 4080s. They support FP8, and we're gonna get the full, uh, precision, uh, full performance out of it, and it's able to hardware accelerate all those calculations. But if we, let's just say, use FP4, it won't be able to do that, and we would actually not see a whole lot of performance gains, and we would ac-actually see performance go down instead of it going up. So that's another thing to keep in mind.

All right. Where is it?

All right. It has successfully rolled out, so we'll, uh, take a quick look. So now we can see it's actually using the FP8 model now. And then we'll also kinda quickly go through and do another load test. It's gonna be the exact same. We're gonna do it at, uh, concurrency one, uh, four, eight and, uh, eight, 16 and 32, I believe, over here. It's going to be exactly the same. So...

And then once that is up, we can compare and see the performance differences and why you would want to quantize.

Okay, the performance is in, and if we compare it before and after, you can see... Let me make this a little bit bigger. Uh, you can see even at concurrency one, you know, we're going for 37.5 tokens per second to 53.4 tokens per second. So we're seeing considerable jump in performance here. Um, and then at peak throughput, we're seeing, uh, at 64 concurrency, we're going from 800 to 1,140. So that's about, you know, 1.4, uh, 42

times, uh, which is quite a big performance jump. But again, keep in mind w- the time to first token is going up considerably as well, uh, compared to 32 and 64. Um, but then yeah, you can... A-and then again, you wanna run evals. Um, we can quickly run through this, and it probably won't have any differences. So yeah. Again, BF16 versus FP8, um, there's not a whole lot of, um, performance degradation you should expect. It's, i-it's going to work out of the box for the

most part, but, uh, for specific use cases it might not. But for simple eval cases that we had here, it's, it's working fine.

I think that is it for this module. Where we'll jump into next is speculative decoding, which I think is a pretty interesting one. Um, let's see. Let's jump in here. All right. So how many of you have heard of, uh, speculative decoding before? Okay, quite a few people. That is good. Uh, for those who don't know what speculative decoding is... Let me make this smaller. Uh, it's essentially we're using another draft model, which is going to be a much smaller model than the main model

that we're using, and it's going to generate multiple tokens ahead of time. And then our main model, which is the 4 billion parameter model, is going to look at those generated tokens, and it's going to either say it's g- uh, it looks good. That's... It's going to verify it and say it's gonna look good or it's not gonna look good. Uh, so in this like diagram, you kind of see like, um, the draft model, you know, generated four tokens ahead of time. And then the verifier model, which is our 4 billion parameter model, is going to say, "Oh, only the first and second token makes sense. The other

two don't," so it's going to discard it. And it's essentially gonna do this each time it's generating new tokens, so it's gonna go i-in a loop there. Uh, and, uh, for, for this, uh, workshop, we're gonna be using the 0.6 billion parameter model, uh, from Qwen as a drafter model and see if that is going to either help us or not help us. In a lot of cases, uh, it can also not help you and reduce the performance. Um, and there's multitude of

reasons for that, and we'll go into that shortly.

How are we doing on time? Good?

Uh, just make sure we are good here.

Okay, target model.

Okay. Before we get into it, I also wanna talk briefly about all of the different ways you can do, um, speculative decoding. Um, there's a lot of different ways, and there's more ways coming out, uh, every, every few months. Uh, but the, you know, the draft model is one way. Um, a lot of times... Uh, a lot of new, m- uh, ways of doing it is actually using a diffusion drafter model, which I think is really cool. Uh, I personally use that for one of my home lab machines, um- And, uh, that gives you a huge

performance. The difference between that and this is because it's a diffusion model, it's going to be generating a lot of tokens, uh, at once, uh, rather than doing what we do sequentially with LLMs. Um, and then the verifier model is able to verify X amount of tokens out of that. Um, but, but yeah, there's a, the, there's a lot of different types of, uh, methods you can use. You can use suffix decoding. Uh, there's a lot of times model would come with a built-in drafter model as well, so those are like the native models that are trained along with the main model.

Uh, so you don't have to, uh, import another model to do this. Uh, there's Medusa, Eagle, again, like diffusion. DFlash is a big one that, uh, has kinda been going viral, and it's working pretty well. So there's a configuration you set for figuring out how long... like how much it generates before the verifier comes in and verifies it. So you can set that. You can set it to two tokens, four tokens, and five tokens, and it's important that you set it correctly. Uh, because if there are g-

For example, let's just say you set it at four and, uh, the model verifier comes in, it's like, "Okay, I only accept two." That's 50% acceptance rate, and that is pretty bad. That's actually gonna have a negative impact because not only are you running... You're running two models, so you're spending compute to run two models, and then you're only getting 50% acceptance rate, and that's essentially going to result into a decrease, like, throughput. Uh, so that's not something you want. Um, so that's something you have to, like, kinda tweak and play around with and see how

many tokens do I wanna generate ahead of time. Um, so yeah, technically you could generate 15 tokens, but you probably don't wanna do that. You probably wanna do two, maybe three, maybe four. It also depends what type of, um... What's it called? Um, let me go back. Where is it? What type of method you're using. Again, like, uh, I use DFlash, uh, for a lot of my home labbing, and, uh, that works pretty well if I set it to eight. And if I set it more than eight, I actually get a lot m- uh, a lot less acceptance rate, and that causes my performance

to go down or be very flaky. Um,

but all right. So we will next, um-

You got one more question.

One more question.

Hey, uh, sorry. Uh, this is from the last module, but, uh, how does dynamic, uh, compression work, uh, i-in a sentence or two, I guess?

Um, so the way it works is if you... Do you understand, um, the different heads of the model? Uh, differ- 'Cause there's different layers within an LLM. So if you look at the model.json, you'll actually see there's a lot of different parts to an LLM, and there's a whole pipeline that sort of like pushes the tokens from one thing to another. And a lotta, a lot of times when you see a dynamic thing, it essentially is saying, "Okay, I'm gonna only quantize, uh, the attention heads to FP8, but I'm gonna leave everything else as is."

I see.

So, so that's the reasoning. And if you remember from previous modules, you'll, you saw that the FP8 version is not exactly half in size. It's not going four gigabytes. It's like 5.1 gigabytes. That's because not every part of the model has been quantized to FP8, so that's like sort of the dynamic part of it.

I see. Is the same math applicable to all the models, or it depends model to model?

Depends model to model. Every model has a different architecture, so architecturally speaking, it depends, you know, what parts of the model you quantize. Uh, there's people who quantize, you know, some parts of the model to, uh, INT4, but leave other parts to FP8, and then leave some parts to, like, BF16. So-

Thank you.

Yeah.

All right, we'll, we'll run through this. Again, I'm just confirming that our baseline is good. We're using FP8 model here. Um, and then, uh, we'll do another load test quickly, um, sort of to get another baseline of, um, what it would look like before we implement and add speculative decoding, uh, to this particular deployment in vLLM.

And then while that is running, I'm just gonna make sure, uh, to... Uh, let me actually copy this correctly.

No, not that one.

Oh, never mind. You can just, uh... So in here, um, we are setting the speculative config. This, so this is a particular flag you can pass in vLLM to set the draft model. So we're setting it to the, uh, 0.6 billion parameter model, and then we're also, uh, setting how many tokens do we wanna generate as draft tokens. So in this particular scenario, we're setting it to four. Again, this is something you have to do a lot of testing on. Uh, you have to try out, do I wanna set it to

four? Do I wanna set it to two? Do I wanna set it higher? It all depends on the acceptance rate and, uh, how much performance you're getting out of it. Um, and then, yeah, once you set this flag, and we will redeploy this. Um, if you go back in here, we got our baseline.

And then let me make sure that it's good. Okay. Things look good here. And then we'll go ahead and, um, apply and roll this out.

And this is probably gonna take another couple minutes, so, uh, does anyone have any, any questions? Go ahead.

I wonder if the

Oh, one second.

I, I noticed that you set the GPU memory utilization to 0.7-

Mm-hmm

... which leaves a lot of your, um... essentially leaves 30%-

Yeah

... of the VRAM unused.

Mm-hmm.

Is there any particular logic to that choice?

Uh, in this particular scenario, um- There could be a lot of reasons. Uh, sometimes, you know, you might have multiple deployments on the same GPU, and you wanna make sure that you have enough RAM for something el- VRAM for something else, and then you keep enough VRAM for this. But in this particular scenario, not really. We're gonna bump it up in a following module, actually. Um, uh, it's, it's one of those knobs, right? Like, you have to keep su- you have to make sure that you're setting the utilization correctly. If you set it too low, you're leaving performance on the

table. If you set it too high, um, you're gonna OOM kill your node, and that's not good.

Yeah. So, um, if you go to the one-

Uh, is this up?

You got a tab.

Yeah.

You don't have a number.

Okay, it's still coming up. So it's taking a little bit more time.

Go ahead.

The bump that we see from 32 to 64 parallelization, it's because of the key-value cache just-

Sorry, one second

... uh, the bump that we see from 60- 32 to 64 parallelization, it's because of the key-value cache just blowing out? Or I mean-

Yeah. Yeah. Essentially, the key-value cache is blowing out, and it sort of hit, like, its max limit, and it can't go above that. Uh, so there's a lot of reasons. So there's that part, and then there's also, uh, the continuous batching. And a lot of times what will happen if, if you don't... If you set the configuration to a level where the continuous batching is not optimal, it can also cause you to see, like, high tokens per sec, uh, uh, uh, you can see high latency there as well. Um, but yeah, in this, in this scenario, we're essentially hitting, like,

sort of like a ceiling on us, uh, uh, for us in terms of how much VRAM we can use there. That's why we're seeing that.

But in the following modules, we'll see how we can improve that scenario and hopefully, like, make it better. Uh, okay. All right. We're rolled out. So we now have speculative decoding, uh, set up correctly. So what we'll do is we'll do a quick load test again. It's the same exact test that we ran up here, where, you know, just keep in mind, we're getting 58.2 tokens per second for concurrency one, and then 42 milliseconds of latency. So this is our sort of

baseline before we do speculative decoding. Uh, but what I wanna showcase here is not a happy path where you will automatically get a crazy improvement in performance. You'll actually see the performance went down drastically. You're seeing 16%, uh, 16 tokens per second, whereas the baseline was 58. Um, and then even the time to first token is sort of going, uh, up a little bit as well. Uh, and, you know, we wanna figure out, like, why is this happening?

So there's a, there's a big difference, and there's a big, uh, decrease in performance. Like, even if you look at concurrency 16, we're going from 700 tokens per second to 240 tokens per second, so that's a considerable decrease in our performance. So the way to interpret this is there's a lot of different reasons. Um, one is, again, as I was talking about earlier, it's the low acceptance rate. Uh, if your acceptance rate is very low and the verifier model is just... It's gonna keep rejecting those tokens that are generating by draft

model. You're wasting a lot of compute to generate those draft tokens, and then the verifier's coming in and then saying, "Hey, I don't wanna take this work," like, "This is bad work." Um, so that's one. Um, it could be because we set the speculative tokens too high, meaning in, in our current config, we set it to four. Maybe we need to set it down, uh, and see if that actually helps inc- improve the acceptance criteria or acceptance rate. Uh, there's a lot of things with the scheduler as well with vLLM, uh, depending on how it's scheduling these requests

in. Uh, that could be leaving some, uh, performance out on the table, so it also depends on compatibility and things like that. Um, and then, yeah, again, same GPU contention. You're using the same GPU, uh, to do this work, so it could be, uh, resulting in, uh, the resources not being used correctly. Um, so in a lot of cases, usually this works, but in our scenario, it didn't work. Uh, and the main particular reason for that is, um, this particular

drafter model is not trained correctly for this particular model, so that's the reason we're seeing a, a very, very low acceptance rate. Um, so y- when you are doing this for your own models and for your own deployments, you have to make sure that the model, the draft model that you're using is going to be compatible with the model that you wanna use it for. If you don't do that, you will run into a scenario that I'm showcasing here, where, like, your performance is gonna tank and, um, there's not a whole lot you can do about it. I can tell you from the resource perspective, this GPU has enough

headroom to support all of this. But the reason for us seeing this decrease in performance is because the, the models are not trained correctly, and, uh, that is something you wanna keep in mind and test for as well. And you wanna make sure the speculative tokens, uh, are set to the right, uh, number as well.

Um, but yeah, uh, when you... In your own time, I would definitely implore everyone to play around with that number, meaning change this to from four to two to six to one and see, uh, what kind of performance gain you see, what kind of performance decrease you see, and why is that happening.

But yeah, that, that's, that, that is it for speculative decoding. It is actually not going to help us in this scenario, so I'm just gonna go ahead and, uh, comment that out, and then we'll redeploy this. So we are back to our baseline.

Hey, you got one question, Omer.

One question.

Hey, um, so I know there's probably some nuance depending on model architecture.

Yeah.

But in general, from what you've seen, is there a massive hit to acceptance rate if you use, like, mixed precision, essentially, like run the main model at FP8? And then quantize the draft model down to Q2 if I'm, like, really memory constrained.

Mm-hmm.

Is it often even worth doing that? Uh, or, you know, what have you seen in general? Like does-

Yeah

... have a be-

That's a good question. Um, a lot of times, uh, you... It doesn't matter as much. Um, most of the time, what you wanna do is you wanna keep the draft model and the main model at the same precision. But let's just say the draft model is at BF16, and then the main model is at FP8. The mismatch can cause the acceptance rate to be very low. So, and then the, the main part of it is you have to train the draft model correctly, so that way it's able to generate the right tokens. In our particular scenario,

the draft model was not trained correctly. Um, and then it was also quantized down to FP8. So there was probably some issues with that recipe when they trained this model and which resulted in us actually taking a massive perf- performance hit. Um, so a lot of times when you- when, uh, a lab releases a model, they release, uh, a drafter model along with it. So you wanna use those drafter models, uh, that come with the model. Uh, there's also third-party labs that do their own drafter models. So you can try out a lot of different options that

are out there and see which one works for you. And in some cases, depending on the use case, let's just say you're using for coding, the drafter model might work perfectly fine, but if you're switching that to maybe some English writing, very specific academic writing, and it might not necessarily work well, and you're gonna see, like, decrease in acceptance rates. So it's also, uh, it also depends a lot on your use case as well. So it's not universally applicable.

Gotcha. Cool. Makes sense. Thanks.

Yeah.

All right.

Let's jump onto the inference side of things. But before we make another jump, any other questions?

Oh, yeah, I can put the YAML on there. There we go.

Um, so the, the fact that we have speculative decoding, uh, presumably is because, um, for, for a model, it's faster to verify rather-

Mm-hmm

... than decode.

Yeah.

Can you just give a little bit of a mental model for, like, what it's-- what a model does underneath the hood, um, to verify such that it's faster than decoding? You know, it can't be decoding and then comparing against what the draft model produces.

Yeah, yeah.

So, like, what is it doing exactly so, so that's faster?

Yeah. So essentially, uh, when a typical LLM is, you know, decoding, you have a set of vocabulary. It's a huge array of a bunch of numbers, and it's representing, um, a confidence score for each of those items, right? So instead of, um, instead of doing that one token at a time, when you run with speculative decoding, you're running it for a lot of... You, you're running the model for parallel things. So with continuous batching as well, you're running a lot of different requests in parallel.

Uh, so you're doing the same, uh, you're doing the calculation for everything, uh, and you're doing it at once for a lot of different requests. In the same way, you're doing the same thing for speculative decoding. You're checking the confidence score for the s- uh, the speculated tokens and making sure that, oh, this is a high confidence, so we're gonna go for it. This is, you know, .95, and this one is, like, .9, and then you, you accept that. And then the other ones, you, you know, uh, you reject those. That's sort of roughly the mental model is, like, you are in parallel running,

um, through the layers and making sure that each of the tokens that the drafter model, uh, outputs is-- has a high confidence. Yeah. Hopefully, that answers your question.

Oh, one more question.

Um, do you have any examples of where that would excel?

Uh, yes, I do. That's a good call-out. Um, there's actually... I left a few links in here. Um, where did I leave those? Interpreting the results. Yeah, so there's a lot of different, uh, write-ups on speculative decoding. In, in general, if you do it right, you do see quite a bit of increase in performance. Uh, vLLM has actually done a big report on this, and they've seen about 1.5x increase in token generation if you set it up correctly with the right model and w- with the right drafter and the right

settings. Um, so yeah, this, this does excel. It's just a matter of getting it right and tuning it correctly and making sure you have the right, um, things in place for it. Um, but yeah, like, i- from my personal experience, um, for my home lab, I used a Qwen three point six thirty-five billion MoE model, and along with that, I use a DFlash, um, uh, drafter model. And I get roughly around two to three x token generation, uh, performance

improvement, uh, depending on the use case. So it, it does work. It's just a matter of getting the right models to work together.

How do you choose the model size for the... Is there some kind of, like, sweet spot for-

For the drafter?

For the drafter model.

Uh, you wanna-- You basically wanna figure out, uh, the smallest model that you can run that doesn't take as much GPU power because you-- The main model is the model that has all the knowledge. That's the model that you really care about, that has everything that you need. But you don't want draft model to take a lot of those resources away from the main model. So you want it to be small, but not so small that it's dumb, but big enough where, like, it's going to actually give you-

While you experimented with that, did you see, like, acceptance rates go up with, like, three B model maybe, like, bigger-

Three B is too big. Uh, the reason why is a lot of MoE models, like I was talking about Qwen three point six, uh, thirty-five billion. So it has m-- It's a mixture of experts model, and each expert's about three billion parameters. So the experts can be around that, uh, and you would get good performance out of it. But for the drafter model, you probably wanna be less than one billion parameter just to make sure that... Uh, it also depends on the main model you're using, I, I guess, right? Like, if you're using-- If your main model is like four hundred billion parameters, you could probably get away with using a bigger drafter model to make sure

that you get higher acceptance rate. So it's-- You gotta find the right balance, essentially. Yeah

Cool. Um, do we have any other questions? Over there, one more

before we jump in.

Yeah, I just had a question. How is the, uh, draft model trained? Like, is it distillation or something else?

It's essentially distillation. Uh, so you take the big model and, you know, you distill all of its capabilities. You create a dataset, and then you train the, uh, drafter model using that. And a lot of times, actually, what's-- There's different techniques to do this. Uh, the other techniques you'll see is the drafter model is trained alongside the main model. Uh, so that way, that's another way of doing it. And a lot of labs will actually give you a drafter model, uh, inside the same package, so you don't have to essentially download a second model and run it. Um, so there's

a couple of different ways of doing it. Uh, but probably the best way is train alongside it, and then that way you don't have a lot of issues and acceptance rate s-sorta stays high.

Cool. Um, we'll jump into module seven. So here I wanna talk about some of the different flags that you, uh, should care about when deploying using vLLM and what-- and how you wanna get the saturation to be as high as possible. Saturation meaning we wanna get the GPU to work as hard as possible, as long as possible, uh, and because you wanna get your money's worth. If you're renting a GPU or if you're buying a GPU, you wanna run it, uh, as much as possible. So,

uh, we will stop here. So, uh, one of the concepts I wanna talk about, uh, if you remember previously, we were looking at the thirty-four, uh, thir- thirty-two concurrency and sixty-four concurrency. We saw a steep increase in the latency, and the reason for that is essentially you-- The throughput at some point is going to flatten out and, but the th-- uh, but the latency can keep increasing if you keep increasing the, uh, concurrency. So you essentially

wanna find the sweet spot, and, uh, that is something that vLLM sorta lets you kinda tweak and make sure that you find that sweet spot. And once you do find it, you can set the right settings for it. Um, but, but essentially you wanna find the intersec- not, so not the intersection, but essentially right before where the throughput gets to, like, the flat state, but the latency is, like, still acceptable enough, is sorta like the point you wanna get to.

Um, we will do a setup, and then let's make sure our settings are all good still. Serving model is up. Um, let me just double-check if it's up.

Should be. Okay.

And then here we're again going to get, um, another-- We're gonna do another load test, so that way we know the data and then we can, based on that, figure out where that, uh, where this point is, essentially. The, we call it the knee, um, where things are flattening, but also the concurrent, uh, based on concurrency, uh, the time to first token is acceptable. Um, once that is up and... Oh, I think, ah, should be fine.

Da da da da da.

So, you know, the throughput is slowly increasing as concurrency is increasing. That is acceptable. Um, and then you also see the latency go up. That is also acceptable, but it's not going up drastically. Uh, but we should shortly see the big jumps, uh, with more, um, concurrency here. But while this is running, any, any other questions for anyone? Go ahead.

Oh, over there, sir.

So my question was again on the previous speculative decoding. Uh, just wanted to know, know based on your experience, um, with, like, drafter model, um, what kind of use cases you have seen it to work really bet-- Like for example, with agent loops-

Mm-hmm

... with tool call or with coding, like what kind of, like, use cases?

Uh, agent loops, it's really good at tool calling. It's really good at, um, browser use, is the one particular use case that I use it for. I wanna automate a lot of browser tasks, um, 'cause there's still... You know, you don't necessarily get APIs for everything, and so you wanna bridge that gap. Uh, and, uh, yeah, the drafter model helps you generate more tokens faster, so you see your browser agent just essentially run around and do things faster. Uh, so in a lot of ways it does help. Uh, I will say for browser use, I probably see around one point five x

increase, whereas for coding I see, like, closer to three X. So depending on your use case, you may see more or less and, and things like that.

Uh, and a follow-up on that one. Are there any benchmarks, like more trust me bro mentions, like by coding by ship-

There, there's definitely bri- benchmarks out there. I, um, I'm personally a little skeptical of benchmarks. Whenever I see them-

Same

... I never get the same results.

Same.

I don't know if people agree with that here. Um, so, um, yeah, for the, for the most part, like, uh, you know, you wanna do your own testing and see what works for you 'cause it very much depends on your use case. 'Cause the acceptance rate is going to be different depending on your use case, what you're using it for.

Thank you.

Um, yeah. Any other questions? Go ahead.

Mic check, mic check. Hey, how's it going?

Hey.

Um, do, like, big models like Codex or Claude, like, do they use this type of decoding? Like, have they published that they use these patterns-

I-

... or is this more, like, for open source if you're training your own stuff or running inference?

I don't know the question to if they do it or not. Um, I would imagine they do, sort of like free performance. But, uh, for, for the open source, yeah. This is particularly for the open source. All the open source big models that you wanna use, all the Qwen models, the GLM models, Kimi models, they all have speculative decoding, uh, drafter models available that you wanna use.

Is there, like, a main drafting model that they use, for example, for, like, DeepSeek or GLM?

DeepSeek actually came out with a new, uh, thing yesterday, I believe, or the day before. It's called DSpark. It's, like, their own technique for doing speculative decoding in a more dynamic way. Essentially, they're figuring out how... So here, our configuration's static, right? Like, we say we're gonna generate four tokens at a time, uh, but with DeepSeek's way of doing it, they are making it dynamic, meaning, you know, y- it, on, on the fly sorta decides, "Oh, I'm gonna generate only two tokens," or, "I'm gonna generate four tokens or five tokens," depending on the acceptance criteria, depending

on the GPU load, and all, all sorts of metrics. So yeah, every, every model, uh... every lab has, like, their own way of doing it as well, and you probably wanna stick to that, but you can also experiment. For example, the Qwen models, uh, Qwen Lab doesn't release the DFlash models for drafting, but these are released by, uh, a third-party lab. But they work quite well with them, so you sorta have to test it out and see.

Thank you.

Yeah. I think this is hanging, right?

Yeah, I would check the pod maybe.

Yeah.

Might have killed a cluster here. Uh-oh, I can't type.

Uh-oh.

This was working well.

Refresh.

Let me create the tab again.

Okay. Um, well, something is wrong.

It... Copy the URL and open it in a new tab, see what happens.

Let me open a new tab.

I wouldn't close that one. Yeah.

You know, in all the testing, that has not happened. But it happened today.

Can I at least scroll through and go through different things? Okay, at least I can scroll, scroll through.

Um, huh.

You know, since, uh, the cluster decided to take, uh, a vacation, uh, I'll sort of go through the content that I was going to cover. Um, the main point of running this benchmark here was to kinda show where that knee is, meaning where you see the max throughput. Uh, if we go back to this diagram, where you see the max throughput, while making sure you are seeing the lowest, uh, tokens, uh, lowest latency for token generation. Um, and that, that is, that

essentially limit was at 32 concurrency, um, because anything above 32 concurrency, you'll see, uh, the token generation is going to be kind of the same. It's not going to change much. Uh, the throughput is going to be the same, but, but the latency's gonna go drastically up, and you don't want that. Uh, so that's sort of, like, the limit you are hitting. Um, and that is important information that we need to keep in mind and look at when we're deploying these things. Uh, and if-

If you wanna jump on

... Yeah, you can go to 100. Go to look at the spreadsheet.

100?

Yeah, go to 100 and then just continue from-

Yeah

... seven to eight.

Where is it?

There you go.

All right. Well, this one's working. That's good.

All right, cool. So we'll jump into eight. Um, let me also-

He's looking at our service level agreement.

Open up.

Who's doing it?

It's me.

Okay. So for module eight, we're gonna talk quickly through, um, what sort of flags and knobs you can tweak to make sure that you can sorta get above that limiting point that we were talking about, at 32 concurrency. You're sorta seeing that limit, and how do we kinda get beyond that, and what things you need to keep in mind when thinking about this. Uh, so for this one, I'm sorta gonna give you guys, um, a thinking model, is essentially you wanna measure, and that's why with a lotta different cells, I was... I keep running load tests,

and it's super important to run those load tests to see what your performance is like and any changes you're making, how is that changing the performance. Is it making better? Is it making it worse? Um, and, and what particular metric it's, is it making worse, right? Like, if... Is it, if the throughput keeps going up or stays the same, but, like, the latency is changing, something is either correct or wrong there. So that's something we have to keep in mind. Um, and there's a bunch of different knobs you can sort of, like, uh, tweak in vllm to make it better. But the main four ones are, again, the

GPU memory utilization. You wanna make sure that you're setting that correctly depending on your use case. Again, if you are running multiple models on the same GPU, you might wanna keep it low, and that's a trade-off you're making. But if you're only running a single model and you wanna make sure that it's getting the best performance possible, you wanna bump that up to .9 or point, even .95. Uh, the other thing is you wanna make sure that, uh, the max num sequences, uh, is set to a good limit as well. This is something you have to figure out by testing.

Uh, in our particular deployment, if you look at the vllm.yaml,

um, we start with 32 sequences, right? So what that is doing is, uh, is telling vllm that it can only process 32 requests at a time, uh, and not more than that. And if you do more than that, you're gonna see some performance, uh, degradation there. Um, so there's a lot of different flags, but those are, like, the key ones that you wanna keep in mind. You wanna make sure that you're setting the model length to the correct length as well. Uh, and then, you know, you wanna make sure your, the batch tokens are also set to the correct length.

Um, you don't wanna... You wanna keep it bigger if you wanna do prefill faster if your GPU can handle it. You wanna do it smaller if your GPU is a little, on a little slower side and you can't handle it. Um, but yeah, the idea is you wanna change one flag at a time. You don't wanna change multiple things. Um, and when you change things, run the mi- uh, run the benchmarks and see, uh, what performance you're seeing, see- measure it, and then make assumptions on what you should do next. Um, I'll kind of go through it quickly here.

We have about, what, 10 minutes left.

Uh-oh.

I think we lost the P- PVC. I think that's what happened. I got-

Oh, it got the different one. Yeah, I need to actually change this. Rerun this, I think. Um,

let me actually just change this in here.

This was supposed to be, like, sequential, but other Jupyter Notebook died, so change that, and then that should work.

There you go.

Um, again, like I talked about, there's different knobs, uh, that you wanna tweak, um, and see what happens, but we'll get sort of the measure... We'll measure the baseline, and then we'll change a couple of flags and see what happens next. Um, does it work?

Okay, cool. There you go. It's working.

While this load test is running, any other questions anyone has?

Excuse me.

I noticed that there's a definite falloff when we're optimizing the model. Like here, the thirty-two to sixty-four, huge falloff. Uh, is that when you would s- decide that maybe we should scale the hardware, or is that something that you're confident you can still tune it further?

Mm-hmm.

Uh, essentially, i- is this, is this the ceiling?

Yeah. You figure out the ceiling by doing mathematics. I think in some of the previous modules, Du'an covered, uh, like, what the max ceiling can look like for tokens per second, what the throughput looks like, and, um, you know, what sort of expectations you should have for the deployment. And if you are, like, seeing those max ex- expectations, that's probably the maximum limit of your hardware, and then you probably have to get additional hardware or better hardware to do so. Uh, so yeah, there, there is sort of a limit. You know, there's only so much you can tune, but you're trying to squeeze as

much out of the GPU as possible.

You wanna share the slides before you continue? Oh.

You wanna share the slides real quick? On the server, how long this server is gonna stay up? This server? Yeah. I can leave it up for a little bit, couple hours. Okay. I'm not gonna leave it up just because you do have access to GPUs. Um, but what, what I am gonna give you, if you scan this QR code, you'll get access to the PowerPoint as well as the repo. If you wanna spin this up in your own Akamai Cloud account, or if you wanna run it on your

laptop, you can do that. So if you scan this QR code, it'll take you to a site that has all the resources. And if you don't have Akamai Cloud account, you can sign up, and you get three hundred dollars in credits with a free account. So that's everything you need. And if you all have questions, both of our contacts information is here. Omer, I'm Du'an Lightfoot.

Yeah.

And yeah, with that being said, if you all scan the QR code, I'll let Omer finish it up. I just wanna make sure you all get this before people started to leave. Thank you.

All right. We'll leave it up there for a couple more minutes as everyone gets it.

Question.

Yeah, go ahead.

So I'm a kinda, like, big nerd for a lot of different AI things.

Mm-hmm.

When, uh, would you say it's, uh, it's good to use vLLM instead of Ollama for, like, long, long GPU inference, like your personal assistant?

Personal assistant? Okay. Um, so-

Should I use vLLM or Ollama for those systems?

So you can use Llama S... It depends on your hardware. I think that's, like, the first thing I would look at. Um-

Can I still have the vLLM and the-

No problem. Thank you.

Oh, you have both? For the sake of the argument, let's just say you have the best GPU. Let's just say you have, um-

It's a six GPU.

Yeah. Yeah, you have the latest Blackwell GPU. Um, you know, I think for testing and doing, like, figuring out what model you wanna use for your use case, I think it's good to use Ollama and Llama CPP to quickly spin it up-

Thank you

... and try it out and see what happens. Uh, but once you have a model that you know I'm gonna use for sure for this use case and this w- this is working great, you probably wanna switch to, uh, vLLM, SGLang, or one of these other model, uh, runtimes. Uh, reason being, they allow you to run at higher concurrencies and give you more performance out of it. So if you have a single agent running, I think it's fine to use Ollama. It's not a big deal 'cause you only have one request going in at a time. But if you, let's just say you're running, like, twenty agents, and you wanna run it parallel,

uh-

I think Ollama is like performance for instance, but doesn't seem like decoding, so I can find some-

Yeah. Yeah. You can set up, uh, speculative decoding with vLLM. I don't think Ollama supports speculative decoding as far as I know, but I could be wrong. Um, but yeah, it... When you wanna get the max performance out of your hardware, you definitely wanna do that.

Well, I was asking if you want speculative, uh, uh, can you set up-

Yeah

... a local thing, uh, vLLM?

I actually don't know if there's a diffusion, uh, LLM. Are... Is there an open source diffusion model right now available?

Llama.

Yeah.

Llama.

Llama? Yeah. I mean, I think vLLM supports it, so I think you, you would be able to run it on vLLM. Uh, Ollama, if Ollama supports it, you can as well. Um, but yeah, that is definitely possible. Okay. We'll switch back to the module and kinda go through it. Um, again, like I was saying, at thirty-two you're seeing, you know, good performance, but at sixty-four, uh, we're going at... You know, the, the throughput is sort of staying the same if not going up, but we're seeing, um-

A significant increase in latency. And to kind of get around it, um, one of the things I wanna talk about is the max num sequence. Again, this is sort of, uh, how many requests vLLM is able to process at one time. It's sort of setting that flag. If you wanna have a lot of, uh, requests being processed in parallel, uh, then you wanna sort of increase this amount. So currently, I believe it's set up at 32, so we can bump this up to 128,

and then we can run this, deploy it, and see how that changes the performance.

It changed a lot of things here.

Yeah. Once this comes up, uh, we can do another test and, um, and you- you'll, you'll see the, the limitation that we're seeing here at 32 is going to go away, and you'll actually see the latency be manageable again, um, at higher concurrency levels as well. So this is sort of like the loop you wanna go in. You wanna find the right flags for wherever the bottleneck is. Find the bottleneck, figure out why that is happening, what flag you can tweak, uh, to make sure that you're kind of overcoming that bottleneck.

Um, it's come up.

That's gonna take a, a little bit of time. Um, any questions? One more over there.

Yeah. Let's say we've done all the tuning and everything, but you still have to contend with, uh, potentially bursty traffic-

Mm-hmm

... um, at times, um-

Oh, yeah

... is there such thing as, like, bursting to serverless? Is that the domain of m- you know, model routing layer on top? Like, h- what is a pattern for-

Yeah. Um, I mean, autoscaling is one of the patterns. Like, specifically, I'll tell you in LKE, uh, you can have, uh, dynamic, like, autoscaling. So it can scale up, uh, depending on your needs, depending on bursty traffic, it can scale up to, you know, maybe use a node that has four GPUs instead of one-

Zero

... 'cause maybe that handles it better, right?

Go to that one.

So you can automate a lot of those things-

That one right there

... uh, to make sure that you're doing well. And the other part of it is, like, you can also build, like, some statistical models to make sure that you kind of understand your traffic patterns over a long horizon.

Ah.

And then the, uh, the cluster can also dynamically, like, sort of predict your burstiness-

Mm-hmm

... in some ways, and then can also scale based on that too.

If you got your browser, I can show you where it's at.

Browser?

Yeah. If you open your browser, I can take you to it.

Okay. So this one is successfully rolled out.

Du'an.

Um, and then we'll do a quick load test, and

once that is run, we can see more.

All right. Uh, any other questions?

Uh-

Most folks are gone, but-

Yeah. Lightfoot. It's H.

Yeah.

Thirty-two, that's where we saw the limit last time.

Yes.

Once we get to 64, uh, we should see, uh, that it should improve. Okay. So there was a big, drastic improvement. The throughput actually went up quite a bit. Um, so it's at 1,100 tokens per second, whereas previously, if we look at it over here, at 64, we were at 800 tokens per second, and then the latency was, uh, 2,000 milliseconds. But now, after the change that we made, um, it is, uh, 1,100 tokens per second, and then the

latency is, uh, 800 milliseconds. So it's gone down. The latency has gone down quite drastically, and then we have also increased the throughput quite drastically. So these, these are the sort of the tuning you can do, uh, over time to make sure that things are running well, uh, once you have it up and running, and you can do this for your particular use case. Maybe you're not gonna have, uh, six, uh, you know, 64 concurrent requests coming in for a particular use case, but maybe you are, so, um, you have to decide based on your use cases.

Was your, was your, um, concurrency one higher than before? Like, was that a trade-off you just did?

Concurrency one? I mean, it can be a little higher or lower. It's not going to be the same all the time, but for example, before it was 37, but now it's 35. So it's a little bit of deviation that happens with running models. But it should be plus/minus two si- uh, two tokens there. Um, we're also running the full precision model now. We're not, we're no longer running the FP8 'cause we had to switch the notebook, so that's the other reason why you're seeing it go from, like, 50-something tokens to

30-something tokens. Uh, so that's the performance hit we're seeing there.

But, but yeah, that is all I have. That is one flag you can tweak. There's also all sorts of other flags that I implore everyone to take a look at. The workshop kind of goes through it as well. But I think we're on time, and I'll give it back to Du'an.

Yeah. Thank you all for attending our session today. If you have not scanned the QR code, the code, the repo, the PowerPoints are available on this site. Feel free to start a free Akamai cloud account. My name is Du'an Lightfoot, and I'm joined by my colleague Omer, and we thank you all for joining us today. Feel free to ask us questions after the session. Thank you.