Context Platform Engineering to Reduce Token Anxiety — Val Bercovici and Callan Fox, WEKA

This is Val Bercovici, WEKA's Chief AI Officer, and I am joined by

Callan Fox, head of the product management team here at WEKA.

And we're both thrilled to present Context Platform Engineering to you at the AI.Engineering Code Summit. Now, let's kick this off with, uh, an announcement we're making. We're actually open sourcing our Context Platform Engineering toolkit.

And this toolkit, uh, features a really cool load generator that Callan wrote that lets you configure agent swarms, uh, and agent subtasks with very specific SLOs, being able to cycle through deterministic and random prompt cycles, and engineer context platforms with all sorts of model parallelism options, disaggregated or aggregated prefill and decode options, and some really important memory

tiering options we're gonna be discussing here. So if we advance the next slide, we'll see that this is an open source toolkit that's already available to you on GitHub.

So Callan and I really encourage you to just get on GitHub, download this, play with it, and give us your feedback. Let us know what you need changed. Feel free to contribute and fork the project, uh, and advance the, the field of context platform engineering, which we're gonna be introducing to you later today.

So moving on, one of the key requirements for context platform engineering really relates to the context engineering, uh, insight that our friends at Manus shared with us earlier this summer in their c-- their pretty infamous now context engineering blog.

And they highlighted the fact that KV cache hit rate is the single most important metric for production grade AI agents. And the reason context platform engineering is so important is it dramatically simplifies reaching maximum KV cache hit rates, as we're about to show you.

On a more personal level, if we think about token anxiety, I know that each and every one of us, you know, feel that anxiety. The reason context platform engineering is so important is shared by the context engineering blog from Manus earlier this summer, where they particularly emphasize KV cache hit rates are the single most important metrics for

production grade AI agents. And context platform engineering quite simply maximizes KV cache hit rates in a very straightforward manner.

On a more personal note, if you think about t-the concept of token anxiety, as we all regularly hit token rate limits, context platform engineering helps you engineer platforms that eliminate token rate limits, uh, and help us be more productive with regards to developing our software.

Now, in the absence of context platform engineering, we often resort to context financial engineering, and that's fundamentally prompt cache arbitrage, where we balance the needs of pricing between the bookends of input and output tokens with these new token pricing categories that have appeared in the landscape over the past few months, focusing on cache writes and cache reads.

And we've got to be somewhat clairvoyant when we're doing this arbitrage to figure out how many cache writes we want to invest in for either five-minute time to live, in some cases, with Anthropic, for example, uh, we can do one hour time to live, and that's all against, balanced against the predictions we need to make on how

many cache reads and cache hits we think we're gonna have during those intervals. This becomes very, very tricky to be clairvoyant and predict the future, and I think it's much better to apply context prompt engineering techniques to overcome token anxiety and prompt cache arbitrage than to continue to, to do the arbitrage in context financial engineering.

And so one of the ways we're gonna be doing that

is looking at, and C- and Callan's gonna dive into this deeply, the cadence mismatch between the relatively slow human feedback loops for agents, and then the agent swarms and the agent subtasks themselves that iterate at much higher cadence, often in parallel, waiting on humans, but conducting a lot of really cool work in the background, consuming a lot

of tokens in the background, many of which are cacheable, but we just never know how the platform is able to respond. And that's one thing we're gonna be diving into here, is the fact that if we go to the next slide, we're looking at fundamentally a token storage problem.

And what we're gonna be doing is explaining how the service-level agreements we sign up to when we subscribe to our various, uh, you know, token tiers, or we actually commit in our instructions and our agentic instructions a specific token cache writes and cache reads, how those SLAs convert to service-level objectives delivered by the context platform itself.

And more particularly, one of the insights that Callan reached from his research at WEKA Labs is that what we're doing when we actually subscribe to our token tiers or we actually pay for particular token writes, is we're really purchasing cache, KV cache slots in token storage.

So there's definitely a whole science around the context platform engineering to how context platforms take those SLA requirements, optimize infrastructure, optimize KV caching and memory tiers, and deliver specific SLOs to try and meet those SLAs as much as possible.

So with that, let me actually hand it over to Callan for, uh, actual research findings and lab and, and test results from WEKA Labs.

Thanks, Val. So look, what I wanna do is just go back to one of the slides that Val showed earlier. And what I'm gonna do from now on is I'm gonna focus on that right-hand loop, and the first thing I'm gonna do is I'm gonna start by visualizing what that loop actually looks like, and then we're gonna

go into a little bit more detail So if you, if you think about that loop as a column, and I've got a graph here that shows a very, very common, uh, pattern that happens in agents.

So the, the salmon color is showing new tokens that the system's being exposed to. The gray is something that could be cached, again, within a limited amount of cache.

We'll get into that shortly. The blue is the output tokens, and these blue dots down the bottom are showing when the user is actually giving responses in this particular case.

This is a really common example you get, where basically you start off, you consume context all the way up until you hit a, um, a high, a high watermark set by either the model maximum length or by the inference provider itself.

There's a summarization, um, phase, and then you start a new cycle. And everybody knows that summarization phase where sometimes, you know, the, the agent loses a little bit of its fidelity, a bit of its intelligence, and, uh, and that's why we're trying to, you know, uh, get more context engineering to a larger set of platforms, and we

can, we can raise that watermark. So if we go into this in a little bit more detail, the question I often get is, "Okay, well, the... what is that?

If that's all gray, what's, what's that made out of?" So here I'm able to, um, get the data and actually look at individual prompts and what actually makes them up.

So when you look at agentic data, especially agentic coding, the actual user input is only a really small part of it. And you can kind of see it here just visually, that if you just scan across the, the lighter watercolors, uh, the, um, the system prompt and the user text itself, and the rest of it is tool

use and tool responses. So, uh, this is, this one in particular is from Claude Code, where you're spending a lot of time, um, where the, the system is, you know, doing like, for example, a, a Bash command.

It's grabbing something, it's getting a result, and then it's doing something else. So where, where, where this really shows out in the data is if you actually look at the median time between requests, it may be something for, for a conversation that looks like that.

We have data for billions and billions and billions and billions of tokens. Um, the median time is ten seconds, fifteen seconds maybe. Um, that heavily depends on whether a human's involved in checking every single, uh, tool use.

But the meantime is in the minutes because the human... or even hours, because the human time to respond is much, much, much higher. And that's what we were showing before of the two sides of the loop.

So the other thing that's interesting and, and something that's very common today is, is, uh, is multi-agent. So you might have a core agent, which I've shown here as the orchestrator, and then you've got these sub-agents that are like spun up to do individual tasks.

And depending on the type of agentic, uh, coding, um, or just any agentic software in general, these agents or these sub-agents may be short-lived, as in their context does not endure between one wake-up and the next, or there are some, some when they do endure.

And it's really important to use our agents because it allows us to create, to effectively target more context at very particular parts of what the problem you're trying to solve.

But as a result, you do actually end up using more context, and I'll explain that very shortly. But if you visualize this gray section a different way and I show you the colors, you can kind of see how there's this common relationship of the common context between all of them.

Again, this is, varies a little bit depending on Codex versus Claude Code versus, versus others. But you can see how it changes over time and how the agents, um, relate to each other and have this common understanding, and then back to the orchestrator to, to wake up the next agent.

The, the, the, the thing that we're here to talk about today, though, mainly is that like while there's a lot of gray that could be cached, the reality is very different.

So if you send this to an inference provider, what ends up happening is you don't actually get one hundred percent of the cache hits that you could, um, that you could get.

Now, why does this matter? Well, there's two ways to look at this. If you're paying for API tokens, uh, you're literally... it's literally costing you more money because every time you see a yellow here, and this is just a simple example, you're paying input token costs.

So you're re, you're refreshing your cache and you're paying a full hit for that, so potentially ten times more than, than what you were if it was cached. If you're a subscription user and you're thinking, "Well, I don't care about the cost.

I don't pay for that, I pay a flat rate," that is true, but you're still, like we said before, you're paying for a subscription and that subscription is rate limited due, due to your cache usage, and, um, you may actually hit rate limits further, uh, or quicker.

So that's something that we want to be able to do, and we work with a lot of providers today to, to remove as much of this as possible. That's good for the user experience and it's also good for the provider.

So why does this happen? Well, I mean, it, it... If you think about the last graph where I show the columns, they're, they're not, they don't take into account time.

They're just one after the other after the other. But there's obviously, um, a, a, a temporal, uh, way to look at this. So this is the way that I like to think about it, and I know this is a little bit more of a complex graph to look at, but bear with me for a second.

So on the left-hand side, I'm talking about working set. So that's the number of tokens that the cache system is holding in its memory based on different time to lives of the co- of the actual cache itself.

And then the, the bit at the top, the dotted lines based on the right-hand secondary axis is showing the cache hit rate as a result. So the red is showing one minute time to live, and what you can see is there's prompts here at the start on the left where the, um, it's thrashing up and down.

And the reason it's doing that is the time between requests at that period is, is longer than one minute. So you're getting a period where you might, uh, take the cache, get a hit or two, and then drop the cache, and then you get another one, you've got to refresh it.

So it, it just, it doesn't really make sense, right? You go to five minutes, which is the blue, and you can now ride out more and more of those cache hits.

And as a result, you get a higher cache hit rate. You can see it at that very start, um, up there, uh, comparing the two. But then you're still missing many others.

There's still many times where the, the time between a request is even larger. So the next one up is showing one hour. And while that requires the cache system to hold, uh, you know, a little bit more tokens in cache and eventually quite a fair bit more tokens in cache, it's got to hold it for a longer

period of time. But the result to the end user is a better, um, actual experience and to the enterpr- to the, uh, inference provider, which I'll show very shortly, it's a much better experience for them as well.

The problem, though, is to do that, you need to be able to k- hold a lot of tokens in cache, and you need good memory tiers to support that.

Um, so the next thing I want to go into is that a lot of people think of... cache hit rate isn't really something that a human's able to really internalize well.

So another way that I can visualize it is by thinking about it in terms of the number of times on average that a chunk of, of tokens, which is a group of tokens, is refreshed.

So in this particular conversation that we're looking at here, you can see that there's this, this is showing the relationship of as I increase the time to live, how that affects my cache hit rate.

But it also shows, based on the secondary axis, that at one minute I'm literally rep- re- uh, prefilling like fifteen, sixteen times the same tokens. And over time we can get that all the way down to approaching one, um, and, um, make significant differences to, again, the experience of both the user and the inference provider.

So with that, what I'd like to do now is go into the, the context engineering side of it, some of the lessons we learned and, um, just sort of really drive this home.

So now I want you to think about, uh, what I think will, will be common in twenty twenty-six and onwards of people s- hosting their own or having their own dedicated systems hosting for them.

So imagine you being an inference provider now. Okay, so now what I want you to think of is think of yourself as an inference provider. Uh, maybe you've, um, you've, you know, worked with us or one of our partners to build our own, your own self-hosted instance.

Um, and, uh, you want to get the most out of it. What this graph is showing you is, uh, a relationship between a certain context length and the cache hit rate and how many output tokens you get as a result of that cache hit rate.

Now, the first thing you'll see is it's not linear, and, uh, it, it... and the, the shape of this curve will change based on the context length, based on the accelerators you use.

Ba- there's lots of things that come into it, how you do pis- uh, disagg and prefill. Uh, th- there's a lot of stuff that comes into it, but the c- the, the curve is more or less the same.

And if I asked you as an inference provider, where do you want to be? You'd obviously say C, and if you're in A or B, you're def- you're not making money or you're not getting enough value out of the system.

And inference providers that we work with, that they, that they have the same answer obviously. So the question is, well, how do they keep in C? And this is where it goes back to a, a slide that, um, Val showed earlier, where what they're doing is they're incentivizing users to stay within C.

And this is where we c- we came to the realization that a lot of the times, because of how much cache hit rate imp- uh, impacts your actual output, that's why s- you're buying cache allotments in storage when you're actually buying subscription services, because it is so important to them that you stay in a certain cache hit

rate band, especially for agentic workflows. Otherwise they literally... you'll just melt the GPU clusters that they have. Um, and I, and I think it's a really powerful thing to, to have in your head about how that works.

So what we're going to do now is go through and think about, okay, well, what, what makes up this token storage?

So when you think about the token storage, there's lots of aspects that, uh, the memory tiers that support the token storage need to be able to do. But to really make it really, really simple, it's literally as, as, a- as simple as you need enough capacity in these memory tiers so that you can hold a optimal amount

of cache. Uh, if you think back to the, the slides I just showed, there's this point where having more cache helps you a little bit, but it kind of gets to a point of diminishing returns.

Um, w- you need to get at least to that point, and you need to be able to store extremely fast into it, because if you can't, you're going to be dropping KVs before they're in the memory tier or you're going to be blocking GPUs, which is probably even worse.

And then the other way you need to do it is you need to be able to fetch from that token storage very, very rapidly so that you can, again, not block the GPUs.

They're the primary first-class citizen of this whole system.

So what does it look like? So there's a few different types of memory tiers. The most common obviously is HBM and, uh, Val and I would love it if all our sessions were in HBM at all times.

It's just not reasonable. Um, there's many reasons for this around how the batch works, but which we're not going to go into today. But the point is, is that the, the, the main common way that this is done today is DRAM.

And there's nothing really wrong with DRAM as such. It, it, it's sort of a means to an end, but it's quite limited in size. It's, it's okay in terms of performance.

But the other thing is, is it's tightly coupled with the compute. So if you want to expand your DRAM, there's not really many good ways to do that. There are some technologies out there that kind of do this, but the way they're implemented, they kind of just hurt your performance.

And that's what I'm showing with pooled DRAM. You could pool more together, but it's, you know, it's kind of a, uh, uh, it, it doesn't help that much. So what we at WEKA, um, did is we took all the durable advantages of our product, which has been, you know, tried and tested in AI training and HVC environments,

and Augmented Memory Grid is basically a, a supported, um, optimized connector between the inference systems and our, um, existing product. And because we're backed by NVMe, we- we're, we're much denser.

We're like a thousand times, depending on how you look at it, denser. It's quite significant. And then I show another example of, uh, a storage at the top there where, you know, not, not something sluggish, something that can still get fifty, sixty gigabytes a second, but, uh, and it has the capacity, but still relative to what we're

talking about is, is still quite slow. Okay. So then moving on to how do we test this? So again, um, we, we talked about how we're, we've open sourced this.

Um, basically, um, Val already covered the, the main part of it and that the fact that it, it, it acts like it's an inference provider. It's trying to keep the load within two SLOs if you enable them.

You actually don't have to enable them, and it'll just go as hard as it can regardless of, of an SLO being time to first token or output tokens per request.

But the main thing that it can do is you can either set a static number of coding agent users, or you can, um, increase the number of those users over time so that you can slowly utilize more of the memory tiers and be able to compare different configurations.

So there's two ways that it works. Um, I'll just be quick through these sections 'cause you can read about this. I have a blog that explains how I do with the testing that goes through all of this in detail, and there's obviously the GitHub as well.

But basically it can do the initial working set and then sequentially go through those prompts. So this will be very, very, very deterministic 'cause as soon as you over- overflow the memory tier even the slightest bit, you'll see a massive drop-off in performance.

But the other way that it can be done, and realistically the more fairer way that it can be done, is you can ex- increase the size over time, so the amount of concurrent users that you're accessing out of a pool, and you can randomly sample where in that sample set you'll get that, uh, prompt from.

So sometimes you might be hitting HBM, sometimes you might be hitting your, your memory tier two, let's say it's that, let's say that's DRAM, and you get a really nice blended number.

So with that, let's go in and tell you, show you some results and just sort of explain and, and show why we're so excited about what we're talking about today.

So this is showing three comparisons. Comparison number one is HBM with WEKA, that's the purple. Uh, there's orange, which is HBM and DRAM. And there's the, you know, orangey pinky color with, uh, HBM plus DRAM plus that, uh, other, uh, POSIX system that I talked about earlier.

The dotted line is showing, uh, concurrent users, so the amount, the amount of users that are in a pool, and that's increasing over time. So in the initial shaded area, you can see that all three of them get an advantage of HBM.

The primary, uh, hit out of, uh, cache hit rate is coming out of HBM. But then over time, as we increase the users more and more and more, you're overflowing what the DRAM system, what the DRAM memory tier can do, and both orange and the pinky color start to drop off quite dramatically.

Um, we also, from a WEKA perspective, also drop off 'cause we get less and less advantage from HBM, so we have to, uh, pull back our concurrency a little bit.

The system does automatically, the, uh, the benchmarking tool. But then once we've sort of got down to the steady state, all three start to like, um, level out a little bit.

But the main difference is, is that once you get down to that steady state, we can maintain that at a much higher amount of users at a much higher amount of output tokens.

The other way that you look at this is, um, that was a decode-focused role. Um, if you look at a prefill-focused role, if you're doing disagg prefill, um, then the prefill's actually even better result for us because the systems, the GPUs are so much more efficient when you're doing large, um, batches of prefill tokens with a single

decode. Um, then we, we can basically saturate things more fairly and, um, and it continues. Now the main difference between pink and orange is that we, uh, sorry, purple and orange, is that we have a lot more cache, so we can hit a lot more.

The interesting thing about the orangey pinky color is that it also has the ability to hit every single thing that it's possible, but it's not fast enough to get it into the GPU for it to make a difference.

And that's why we're sort of showing the difference between these three, 'cause with purple you're getting the advantage of capacity but at DRAM speed, so you can maintain that benefit for longer periods of time.

And then maybe, Val, I'll hand back to you.

Absolutely. No, that was a great walkthrough, Callan, of all of your research and benchmark results in WEKA Labs. So once again, we're thrilled to be announcing the open sourcing of this context platform engineering toolkit today.

Please do download it, use it, give us your feedback. Again, feel free to fork it and improve it yourself. And we look forward together to just contributing to less token anxiety overall, less prompt cache arbitrage, and more context and context platform engineering in the future.

A nice QR code for you to find out even more information. And at the end of this video in, um, in the actual transcript section and so forth, there'll be links to all the blogs we referenced here.

So thank you for joining us today, and we look forward to carrying on the context platform engineering conversation with you in the future.