The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

Uh, thank you for everyone for coming here. Uh, not too sure whether to be happy or sad that so many people are suffering from their token bill. It's a full workshop. Uh, yeah. The, uh, we are gonna talk about tokenomics. So a little bit of back-background with kickstarting it, right. I'm Eugene from Featherless AI. I'm CEO and founder of, of it. Uh, and I'm part of the team behind the RWKV open source project, the first AI model under the Linux Foundation. And, uh, and we work on next generation AI architecture. Uh, that's a little bit of my academic background, and my Twitter handle is

picocreator. But, but more importantly, right, uh, uh, a little bit of background for Featherless AI is that our mission is to make AI accessible for everyone, every language, every dialect, every country. And we do this, we do this by creating the world's largest platform of open models of over forty thousand open models. Th-this includes specialized fine-tune, like the Sunflower model, which is Uganda's first language model specifically for their language, and a lot of other specialized models. So we are just not talking about the biggest open source model, but specialized language

models or specialized use cases. Uh, some of it which I'll touch as well. And when we-- Where we can go on scale in terms of variety of model, Featherless AI also we serve customers at scale. So some of our largest customers are literally serving two trillion tokens a day, and, and that... And, and they're serving it at around zero point three per million tokens or sixty k a day. And they're doing it on around five hundred and twelve AMD GPUs, uh, MI325. Uh, and for this customer in particular, it's a

private four hundred to eight hundred BMoE model. Uh, and they are serving at around Opus 4.8 class performance. So it's lots of concurrent requests, lots of token volume. That's what we mean by scale. And if you look at it, zero point zero three per million token at Opus 4.8 class is not something that, is not something that you can get off the public cloud market. This is specially-- very specialized setups that is meant to run at the scale that we are talking about. So that kind of, uh,

leads in into two major disclaimers that I'm gonna start off. One, we sell inference. So everything I cover here will cover the... Will, will touch on any provider. Nothing here is specific to Featherless. But to, to... For clarity, we sell inference. So, so this is from our data points, from our experience respectively as well. Okay? The other thing is that this workshop is gonna cover a lot of optimization at scale, and it probably will not fit you if you're like spending less

than ten k a month in your organization. Uh, my word of advice for those who are below this, this, uh, metric mark, and this is an arbitrary m-metric, is that if you're spending less than this, you're probably better off focusing on, like, trying to get strong use cases within your organization rather than trying to optimize the token spending bill. Because a strong use case can bring your organization way more than ten k a month. And, and that's, and that's should be your main area of focus, and that's not my area of focus for this talk.

So what we're gonna cover in this workshop, uh, is four moves in, in, in four major order: measuring, right-sizing, uh, how do you buy your capacity, and how you check your process.

So measuring. Yeah, you'll be surprised at how essential this is. Uh, like, how many of you know your total token spend? Can I have a raise of hands? Wow. Only half the room raise hand. Okay, how many of you can measure at the department level? Those, those who have their hands raised up, leave you up. And so I see about another half went down. Yeah. And it gets lower and lower down to the team level, to the use case level. Um, we... Like, easily

thirty, forty percent of our customers that reach out to us, the first thing they had was a single shared key that every o-one in the organization used. And that was like, uh, uh, how do we even measure that? How do we start from there? So as... So this is not an, a vendor specific. Like, you need to start splitting up your keys, start splitting them by departments, start splitting up by major use cases so that you know where all your token spends are. Um, and this is... Then from then on, it can inform your workload. So yeah,

API keys shouldn't be that hard to issue. Um, once you know that, right, you can start right-sizing things, uh, respectively. Like, like which teams, uh, and workloads spend on it. What was the person waiting for? Like, some teams deserve tokens per second. Some background agents running at midnight probably can-- don't deserve the tokens per second speed that you're paying for it. Uh, yeah, don't pay for Ultra on a background agent. That's just dumb. Um, and, and from there also, you can also look at your usage to better advise like, uh, dedicated

capacity that you may purchase, be it for open models or even closed model. Uh, yeah. And also, you can also check which teams spend the most token and whether they are productive. Uh, I'll touch on that as well.

One of the things that you should know in AI workloads, right, uh, and you all may already experience, right, is there are two major tracks with very opposite economics, uh, respectively. So there is the agentic workload, and obviously there's things in between, where it tends to be like long-running multi-turn agents. Coding agents are a particular example of this. Some, some, some may have some devs like run coding agents like literally overnight over a long period of time with, uh, with multiple compaction. And, and these things maximize the context, uh,

usage. Process automation is the other extreme. Uh, think call centers, think email routing, think your meeting notes transcribing, and things like that. It's all... It's much shorter in context in general, though there are a lot of examples that defy this statement. Uh, but it's very process-driven, very task isolated, and it's, it's a volume game essentially.

So, well, what do I mean by more usage? Uh, and this is probably one of the first major explosion a lot of organization face, w- uh, in particular, when they jump onto the OpenClaw hype or one of the Claw clones. Uh, so y- like, like from, from, from various developer teams that we, we support, like we, for example, we sampled one of the team, and it was only five devs, and they were averaging around continuously eight running agents, uh, for one of the companies we work with. And, and so they were previously running Opus 4.8, uh, off the API, and they were essentially

spending a hundred fifty K a month continuously running these agents. Uh, some of it is the agents that they used to code, some of it is the agents that kick in at night. They, they were slightly more advanced. They already knew that n- they, they, they should kick things off in the night, uh, when no one's, uh, working. But essentially, they were spending a hundred sixteen billion tokens a month. And we literally helped them switch to a GLM-5.3 instance. Uh, previously it was five point two, they already upgraded to five point three, and that was like a single node that was less than ten K a month. And they were-- and they didn't see any disruption in their impact. So

knowing your team workload allows you to make these decisions respectively. And like, like I highlighted, like knowing processes workload also allows you to split things off. So like for example, previously they were running everything throughout the day, but they, they started scheduling some of the background agents when the team was asleep, and it runs through night. So this allows you to utilize their hardware respectively. Uh, the, as I, like, touched on briefly, uh, the other side of process automation is purely a

volume game. Um, extremely predefined. Uh, I, uh, I can give you more specific example off the, off the record, but you think of it as like millions per day or even tens of millions per day, decisions in call centers, lines of code scanned, emails and text message filtered. Like one of the telcos we work with, like they easily hit peak ten million messages in a single minute, and this shouldn't come as a surprise. Happy New Year is the worst. Yeah, so, so, so this-- uh, and they, and they help... And the

reason why they filter messages is because they provide a feature about like providing safe phone for kids. So they needed an AI to scan, like, basically is it profanity, sexual content, et cetera, et cetera. So, so this, th- this is really not a long context, but a volume game. Other things like-- Other things is like maker checker processes, approval processes, document processing. These are all volume game for multiple organizations. So I suspect for the financial institutes here, since we are in New York, a lot of OCRs, a lot of

documents, a lot of approval, a lot of transaction risk, the, is, is under this category. Okay. So yeah, if you're not yet tracking your token, token spend and token bill split right, we are open sourcing a, a tokenomics tracking tool, which is just a very fancy way of like splitting up your API key and tracking which API key spends how, how much, uh, uh, which is separate from this talk. Yeah. So if you, if you want to again update, feel free to, to scan this QR code and rest to it. Also, if you want the slides for

this talk and when the recording is released from AI engineering to be notified about this talk, you can also get, uh, fill up the form there. Okay. Okay. Okay. So yeah, that was about measuring. Um, yeah, if you are not measuring, you can pretty much ignore everything else I say and start with measuring first because ev- all your biggest ROI is just by measuring. But once you have things measured right, the question is how do you rightsize it? And this is where things gets more complicated.

Uh, in, in general, right, one of the things that we start to observe as, as AI gets deployed at scale within organization is that there tends to be a he- pattern of like, because AI is the shiny new hammer, we try to apply it to everything, especially within the innovation labs. We just try to apply to every damn thing. And we tend... And, and I've seen organizations literally apply Fable to really the silliest of things. So, so you, you-- there's a

lot of models, and a lot of models are really expensive and, and, and, and, and figuring out what to do and where it fits, it's, it's, it's important, it's an important step of the process.

I know open models. I mean, a lot of you probably heard about open models, and that's our line of business. Uh, yeah. Uh, and, and so a l- when a lot of companies come to us, for example, to like find open models that fit their use cases, uh, and drop-in replacements, this is, uh, we generally guide them along their use cases. So this, uh, this is a general rule of thumb that is probably gonna be outdated next month because open models release every few months, a- and same thing with closed source models. Things change rapidly. But, but there, but the thing is what we find very exciting

actually within this year is things have started to change. Like in the previous year, the best closed source model is without better than the best open source, almost. But as this year approach, right, we are starting to see the gap close. We are starting to see situations where, where some of the open models are even on par or even surpassing. So like for example, before the Opus 5.5 update came out, Kimi K3 was beating Opus in UI design in

some of the UI benchmarks. And, and this is like very public benchmarks. Within customer benchmarks, right, we were already starting to see open source models beat very specialized benchmarks in specialized use cases. So like, uh, um, particular one to highlight is like, for example, Step Fun- Five model. They... It was able to achieve near Opus and Astra quality despite being like a ten X smaller model in financial benchmarks, uh, financial use cases. And this, is it the best coding model? No. But in that use

case, it's actually extremely strong and competent. So, so, so the landscape is slowly changing where, where, where you can have drop-in replacement for a lot of use cases that are sometimes even just ten X cheaper, uh, respectively for, uh, if, if you know your workload, and you can drop it in respectively. Special shout is also DeepSeek-V4.1-Flash. For some reason, and this model is extremely popular across all workloads respectively because it's easy to prompt steer, uh, for a lot of companies.

So while I-- while the rest of the models I have listed there, they are, they are like default for good in their categories. DeepSeek-V4.1-Flash seems to like be quite popular regardless the case. Yeah. Um, though benchmark score doesn't seem to justify why is it so popular. Yeah. Yeah. Uh, another thing to highlight, right, is that recently, like indeed just like for a few weeks ago, uh, m-m-- the a new class of, uh, AI models have been introduced called, uh, classifiers, uh, one-shot cl-cl-classifier, most

popularly known as Jev. Open source projects have already started, uh, taking off like Simple Jev and these are models that are entirely focused on just making decisions and classification on, on tasks. Uh, the main advantage is that because on the technical level they're prefill only, they are commonly ten x cheaper than existing models or even faster at the same time. Uh, the downside is it cannot give a chat response. So, so, so these are things that are ha- changing and happening that you all may want to start looking into on like,

on like segments of how you want to improve your, your token economy. Yeah. But, uh, but back to the, the, the, the, uh, the token economy part, there is two major tracks, and it is important to separate out and track this, uh, track them respectively. A- on the agentic side, we're talking about long running, long context, as I said earlier. And to un-- and you have to understand this workload tends to be extremely cache heavy, extremely, uh, input

prefill heavy and cache heavy, and there's only a small percentage of output. We see across like agentic use, use cases in deployment, only one percent of the request is to the output tokens. And, and, and this is, this is very logical because it takes multiple turns, rinse and repeat. But this also decides your token cost because like if you've seen token cost, right, uh, deployment cost, right, output is may have the biggest number, but even your input where it's much smaller or the cache is much smaller, it,

it gets repeated heavily in, in this use case. Yeah. This is not a coding agent workshop, uh, but one of the most common patterns that we, we see in a lot of organizations, right, is that, is that typically, uh, typically in terms of like trying to scale up their AI agent deployments within, within the coding teams, right? They tend to have the best agent do pre-planning and a faster, uh, a faster, cheaper agent do, do the implementation, and then maybe the

best agent do the review again. Uh, there's a good reason why this split is very common across all the agent frameworks, uh, is, is that you get the best of both worlds because like the implementation phase is tends to be, tends to be very token heavy in process. A lot of processes repeated back and forth, like keep modifying the file, testing, checking for results, rinse and repeating. And, and you-- and most, and most of these agents are, especially the s-, uh, especially like GLM-5.3-Flash, is more than capable of doing a task at the

same quality level as a much more stronger agent, like let's say Astra or Opus, uh, uh, respectively, assuming those agents do the planning. We have worked with some of the organizations where literally the planning and review agent is the best frontier closed source model, and then the workers that implement it is the GLM-5.3-Flash as an example, or DeepSeek-V4.1-Flash. Yeah. One of the observations that we made, uh, from, from this, uh, from, from this kind of split pattern, right, is that

unfortunately, as much as we wish everyone does so, nobody really switches their model to save the company money. Um, this, uh, people are lazy to switch the models. Uh, in general, you, you want to give an incentive to switch or, or alternatively, if your agent harness is good enough, so like for example, some harness like Amp, uh, they automate the process. Like, like it uses a, the weaker model as a baseline, and then it uses the big... the smarter model to automate switching when it needs a smarter, uh,

consultation. Um, this, this helps reduce the friction respectively and helps improve your, your token economics, like, uh, user scale. Like a ma- a major rule of thumb that we, we, we now give our clients after going through a few times is that you want the smarter model, uh, and since it's already so expensive, uh, as a baseline, but y- uh, but you want the s- the faster model to, the cheaper model to be faster. And that naturally sets the incentive. People want

to get their work done fast. If the model is s-smart enough to get it done and it's cheaper, they will just use it because it's faster, not because it's cheaper. Um, we had teams where, where, where, where both models were the smarter one and the cheaper one was maintained at the same tokens per second speed and it, and as long as they had the token budget in excess, right, no one switched. And it, and on the flip side, we have seen teams, which is the other extreme, where they set an extremely low budget, and so that if they use a smarter model, right, they just burn it

in one day. Yeah. Then everyone just switch to the s- to the dumber, uh, in, into the faster model, a dumber and faster model, because that's their budget. It's like almost as if the smarter model didn't, didn't exist. So, like be aware about your team dynamics on how they switch. The, the best trade-off is still speed because people are naturally incentivized for it. So faster, cheaper, they'll just switch to it respectively. Yeah.

Once again, reminder, measure, and you measure through separate token keys. Uh, and, uh, but this also goes beyond just measuring. For example, like CI/CD is a very common use case right now of things running in the background. Uh, we, we have helped set up for teams like faster tokens per second, uh, that, that is running for the, the users like the humans, but much slower tokens per second, um, speed, but much cheaper at scale for the CI/CD pipeline. Uh,

and w- and this is just the nature of, like, what's your priority respectively, uh, respectively across the task. Yeah. And you can control that at a key level. So, another major finding is, uh, that we had experienced with, uh, deploying for, for teams, right? Uh, uh, so, like, for example, for some companies, um, how many of you had the fortunate or unfortunate experience of the CEO saying, "Everyone must implement AI now, and everyone must

have a second brain now"? Yeah. Multiple organizations have started this. Uh, in some cases, like even in, like, in the UAE, it became literally mandated by, by, by, by the law of the country. So, so, like, a lot of departments are not prepared to know what is the trade-offs respectively. So, a good rule of thumb, right? Well, developers tend to know, okay, all these smart models, what's the trade-off, uh, between

DeepSeek Flash and GLM-5.3, for example. Most, uh, your, your finance department have absolutely no idea. Your, your support department has absolutely no idea. They, they'll just pick whatever's the first model and probably just go with it. So, so our general rule of thumb is that give department models labels, but not the model names, and this also gives you a lot more control as you switch it out. Uh, like, like I advised earlier, fast versus smart, but in some cases also, like finance agent or Excel spreadsheet agents. Um, these are

hyper-specialized agents that will fine-tune to those tasks respectively. And, and this also gives you a lot more control. It allows you to swap out the model. It allows you to split, split the traffic and test in between with- without, without telling, "Hey, users, please try this." The downside is also users may sometimes complain when you do that and you screw that up, but having that control is a good thing. Uh, another pattern that we see even though you do this is maybe someone will add a dash preview version, so that's a variant of

it. But it gives you control without, without taxing the communication burden across the organization. And this is extremely important because, like,

like some of the teams that we work with, right, we literally saw members running Llama 2 models when, when we are talking, when, when we're talking about like in this year, because no one switched the option. Yeah. So you really, really need to, need to, uh, need to set this in place. Yeah. Okay. Process track. Process track, a lot of things that I mentioned earlier don't really apply as much to process track because process track tends to be a lot integrated into the

applications, into the workflows, uh, respectively, and, and that's just by the sheer volume of the traffic. Um, yeah. And, and i- for a lot of teams in the process track, right, uh, they tend to be a lot more focused on, focused on the right sizing already by default because, because if you are running like trillions of tokens, you-- most departments will not approve a like $10 per token, uh, million token. It's just not possible. And, and so, so, so they

start, they start early with the evals, the benchmark, and then subsequently finding the model that right size fits it. Uh, yeah. So it-- So the thing is like the common question is: How do I know, uh, the smaller model is good enough? You don't need to. That's the wrong question to say. You just need to make sure you set up your test cases, your evals and your examples, and then subsequently just do the search process from there. Uh, companies like us will be more than happy to help you because we have over 40,000 models. But then, but then if you-

it's also not that hard to just download 10 models from Hugging Face and just try. Uh, so, uh, so from there, you, you just do the search, find the cheapest model that kind of like pass most of it, and then you just see how you iterate from there, fine-tune from there. Uh, we also have a more specific process that I don't think is unique to us that we'll cover as well also. So for example, like in, for one of the major telcos that we work with, right, uh, we actually helped their team lower their inference cost 250k to 7.6k a month switching from an existing

closed source model just by go- going through this process of like, okay, they're running the best model first. Let's see what, what workloads we can split off and then subsequently what's the models that we can fit in, and then what can we fine-tune? And this was, this was like to, uh, build their customer support use cases and, and so on and so forth. Yeah. And to re-highlight back Jev, uh, and, and Simple Jev , which is an open source project that we did, uh, that replicates it, you'll be surprised at how many percentage

of your AI traffic is classification traffic. Uh, uh, at, at the end of the day, right, like yeah, sure, answering the customer support is text. But even then, right, even then, right, a lot of teams right, already have predefined text output that they put into the context for the model to output for most cases and, and you could use a classifier there. And there's a lot... This is a new, new, new shift within AI workloads and that this is one of the big levers for these

kind of processes in lowering your, your token bill. Yeah.

So one, one, one example of, uh, such a model when it comes to fine-tuning, so a lot of models and processes, right, uh, do benefit at this scale where you start to fine-tune your own model. Um, on the flip side, for coding models, well, there are organizations that will fine-tune models for you, like Devin, for example. Uh, a lot of developers generally just want the best model. So it's kind of y- unless you are in this AI space, it's kind of hard to keep pace in

fine-tuning the best coding model. But on the flip side, for your domain, it might be worth it to fine-tune a domain model for your use case, especially if the model is much smaller. So for AT&T, for example, when they fine-tune their models, they, they, they went from be Opus class to like- 27B to even 7B models, and they have published this. So, so we were one of their launch partners, uh, and you can actually try their hotel line of models, uh, uh, respectively. And, and, and this is

basically what they use behind the scene. And, and it's all about volume here. Like I, I think, I think most of you understand the scale of AT&T and the call centers and things like that. Um, yeah, but it's a good example of how you fine-tune models to, to specialize use cases and once you hit reach level. Other organizations that are major example are like Shopify has started doing this and, and things like... And, and it doesn't necessarily need to be a large scale fine-tune. So for example, in the Shopify use case, they did it because

Shopify web store had a proprietary l- programming language for how to configure the, the web store, and the existing models just weren't that great at it, so they went to fine-tune it. Uh, but-- and AT&T also they had examples of that, uh, in particular like how to arrange the telco towers and things like that. Yeah. Uh, very telco specific tasks. I might not be the right expert to answer that question, uh, but they, they had those use cases, so, so they fine-tune respectively. You too may have the

those use cases. And how do you bootstrap such a model, right, is that the good news is that, uh, you can easily bootstrap things these days because if let's say you're willing, uh, if the best model that is too expensive to run at scale can do the task, you can already get started. So, so, so an easy way to do this, right, uh, that we see very commonly is like either you generate your own data or prepare own data, which is hard. No one likes to do that

for some reason. But the-- but another common path is that they start with an expensive model, and then they deploy in a small scale, and then they generate the, the, their dataset. I understand that some labs it's against their terms of service to do that. So you may want to check with your lawyers, but there are labs that openly embrace it. So for example, Google Gemini, um, their, their frontier models, they literally have a dedicated API service for this. Uh, so, so, so this allows you to build data

sets respectively on your use case on a strong model, which you can then, which, which you can then reduce dramatically to, let's say, a 7B scale. Uh, alternatively, open source models are more than happy for you to, to, to run and distill on, so you can run the best, uh, open source models respectively as well. Once you collect them, you can then fine-tune the data, a-and then you can fine-tune and respectively like different sizes to see what fits best. So once again, like in the AT&T example, they have the

27B and 7B for different levels of the call center. Um, like your routine, your routine calls may come in at a... Wait, I don't think I can say which size goes where. But your, your routine calls can come in for the much smaller model, and then it escalates up due to some classifier triggers, as an example. Then, then so, so, so that's how you get the traffic to flow, g-get the traffic split, and then you get it at an efficient level. Uh, one of the things that we also tend to work with teams on, on when doing this is that even,

even when they go through the stages and they completely split over their traffic to a fine-tune model, because the best op- the best models still progress, uh, they get, they, uh, we typically advise them to cut over, uh, cut a part of their traffic as a shadow traffic to the best model to keep building the dataset. And, and, and this allows you to ride the wave and progress of the best open models respectively. Uh, and the unit economics is actually very important here because, like,

one thing that I would like to remind people, right, is people severely underestimate the, the, the speed that AI is going, especially for models that you can run much cheaper at scale. Like you take this laptop as an example. I like to measure, right, what is the best model that I can buy in the most expensive MacBook Pro period. And today's best models, be it MoE models at 31B, 32B, 27B size, is already much better than

GPT-4o. So essentially, this model that can run on a laptop is already better than the best model that you have last year. Give and take. People like to argue on the specifics, but let's just say broadly. And when you fine-tune it to your task, that means if you can run that model on a laptop chip, uh, respectively, you can run it on, on, on your GPU servers at scale much more efficiently. Yeah. So yeah. So that brings

in to the next topic, buying GPUs, which, uh, which, which is apparently increasingly a challenge. So yeah, that's where companies like us come in. One of the things that s- we r- we realize through a lot of customer conversations, people are rather surprised that things don't scale linearly. Uh, what I mean by that is like there is a huge trade-off between tokens per second on an individual request to, to the cost, to the cost of the request itself. So for

example, if you do not care about, about the cost, you could run the best-- the, the model, let's say a fixed model, on the best hardware on all eight GPUs, batch size one, and you could get, let's say, 100 tokens per second, 200 tokens per second. If you like, if you really want, and, and, and this is not our specialization, like you really want like thousands of tokens per second, that's where companies like Cerebras is more than happy to offer that as well, and it costs even more. So there are definitely use cases where this makes sense,

but it's-- but a lot of people fall into the trap of thinking, "Hey, so if I have the speed, maybe it's half the cost." It's not even close. Like, like a rule of thumb that I sometimes say, right, for every two x of speed is three x of cost, and it compounds. Uh, in this specific example for GLM 5.3, uh, it's like one agent on the node is like you can see the same server, 112 tokens per second for running individual requests. But if you run 16 requests in parallel You get about half the speed for all the requests,

but you literally 10X the output, total output. You... And, and that c- that gives you like literally n- around nine to 10X lower in cost in just like changing your s- changing your target speed profile. And so when a lot of customers ask us like, "How do we deploy things at scale?" Sometimes the first question is, "What's the model quality you want, and what's the tokens per second speed that you want?" And then we start recommending the amount of GPUs, the amount of deployment, and this, this target tokens per second is an important

trade-off, essentially. Um, in more extreme cases, right, if you really, really can run things at 10 tokens per second, rarely anyone does, but there are some weird use cases that allows it, you can get even more incredible token, total throughput volumes respectively. Typically, we see customers choosing between 20 and 30, which is kind of like, uh, your ChatGPT speed without, without too much thinking, all the way until like 60, which is like the faster speed, and if they want 120 token, like it tends to be this, this range.

Most stick between 20, 30 to 60, depending on how, how long the chain of task is. Yeah. So yeah, uh, I, I kind of mentioned it, you generally only want to pay for the tokens per second speed, uh, when, when it's in front of someone, uh, whe-whether it's in front of your developers or whether is it in front of your customers. Those are the trade-offs that you might make. And even within then, right, we have seen cases where even just a slight tweak down, right, can easily save

two times the cost but has near minimal impact for the cust- uh, customer respectively. Yeah. And once again, background jobs, please don't run it at 120 tokens per second. It makes no sense when you do that.

Yeah. So o- once you have previously, I said measure, right, you're able to see your, your, your traffic pattern. So a lot of traffic patterns tend to be bursty in some nature with some base load because someone's always running something at some time, uh, especially background agents. And, and this is where you can, you can also plan your deployments respectively. Uh, I know i- since we are dealing with financial institute, a lot of financial institute may not have this as an option, but because they are only in dedicated

instance deployment. But if for some cases where people allow mix, where they have dedicated instances with some burst to the public cloud, this is where you have some flexibility respectively. So like the biggest cost savings you have right now in the market is deploying a dedicated instance if you know your base workload. So as long as you fully use it, you kind of have a savings there. But it's, it's a bit more complicated depending on your traffic pattern, and a lot of these things can be tested and validated, uh, respectively. And this is like essentially what any inference company will work with you.

It's not unique to us. We'll be like, "Hey, this, uh, do you know your traffic workload? Can you share some data?" And then subsequently, we'll be like, "Okay, we propose this much. You can try switching to this," and then we, we generate, uh, we can re-generate, show you your savings respectively. And in some extreme cases where you have a lot of spikes, you may want to ha-have limit your dedicated capacity to a certain amount and let the spikes spill over to the public cloud respectively. Different use cases have d- uh, have different right sizing, but this is something that

by measuring your token workload, once again, at the beginning, you can start exploring these options, and the savings can be huge. Like once again, like for the customer, they reserved two, 2 trillion token, and that was like, uh, 4.8 Opus class model. They are paying less than $0.03 per million token. Like you can't get that price anywhere. Yeah. So yeah, simple disclaimer is the spikes really matter. Uh,

it's very rare that someone has a short spike at the extreme end where they like run a workload every month at that scale and then nothing else matter. So, so usually some base workload exists. It's just how big of a base workload. Yeah. Yeah. So yeah. Then the last one is check your processes. Uh, this has nothing to do with inference optimization from here onwards. Um, some slides may sound that very critical of

AI, but I'm generally an optimist. Uh, just, just w- I just want to lay that out because things can be quoted out of context. So one of the, uh, Pandora's box question or problematic question that we are starting to see is that a lot of organizations are starting to ask, "Hey, I'm s- 10X or 100X-ing my token spend, but I'm not seeing results at that same scale." Um, respect... Like, like we are starting to see reports from, from various

organizations like, uh, AI, where AI has limited productivity, only has 6% productivity or 7%, which is not the scale that a lot of us were promised or we envisioned. And, and unfortunately, this is the case because we are at a stage where everyone's just taking a giant hammer and hammering everything. Um, it's not that AI is not useful, it's just that we tend to just spam it everywhere. And yeah, if, uh, there's a s-

there are times where I just want to like, "Can you please not do this as another copilot because it's just too much of it." You need to think of how you use it respectively. Uh, um, yeah. And, and another major issue that we start to realize is that while A- while AIs are, um, uh, being deployed at a larger scale and more frequently, and people get faster in creating stuff, the organization processes may not go as fast. The first

bottleneck for, for coding, coding environments would be the CI/CDs, and you all may heard that CPU becomes a bottleneck respectively. Uh, another pro- another thing is like, it's just the processes in place, uh, respectively. How do you trust? How do you review? I think the previous speaker spoke at length at trust. Yeah. The lack of trust is what slows things down, and this could be rightfully so. Maybe the agent, the, the AI agent was badly designed and shouldn't be trusted. But it also flip and reverse. What if, hypothetically, the, that agent was

doing everything perfectly, but people weren't trusting it because it didn't have the test in place, it didn't have the validation, and it didn't build that trust? That slows things down, and these things ma- uh, these things have a much bigger impact.

S- so once again, be very wary about how the hammer is being used because you need to make your adjustments accordingly. So, and this goes back into saying, like, tokens are not work done. You have, uh, um... and, and one major example that I will highlight, right, is, is this happens more often than, than I would like, is that sometimes spamming more agents and more tokens sometimes means you get the same task done multiple times. So

one of the anti-patterns that we started seeing, right, uh, we are-- Well, in overall, the engineering organizations improve productivity by clearing more issues quickly, automating clearing of issues, for example, like when some- like when a QA and, and things like QA teams reporting an issue and the agent automatically fixing it without even an engineer. Those are great. But one of the is- one of the common issues that we start facing is that people get ambitious really fast, and they, instead of doing small changes, they start

doing huge changes, like changing like half the code bases or one quarter of the code bases. Sometimes when this is planned with proper testing, it's great. But a lot of times, right, the change is happening at such a large scale, right, the bottleneck becomes the, on the reviewer to merge the code. And because, because everything is done on such a large scale where a human cannot process it, one of the common anti-patterns that we start seeing is that, okay, someone has implemented the, the, uh, there's issues being done. And because

there's so much code being generated, issues that do not have code generated do not get review- uh, did not get picked up by the reviewer. So someone needs to generate the code using an agent. And then when the agent generate that giant pile of code, which should be a lot less if they were to like properly review, prompt it, and stuff, the reviewer look at it and it's like, "Huh, this is too many code. Can I ask my agent to highlight what changes are relevant and then re-reimplement it?" And, and, and in the end, they started

rebuilding hand by hand with the, with their agent. So they literally discard the first agent's job. There's multiple good reasons why this happened. For example, like reviewers want to maintain control and quality of the main code base. But this is a very common example of a bottleneck where you, where, where if you didn't think the process through, you're just throwing more and more things respectively, and the speed is not increasing. So review your processes. A good way around this is naturally

to, to, to instill better testing practices, and so that it can automate that and filter out. Also reminding teams to like, "Hey, split up your, your, your merge or, or make it more modular to make it easier for the review process." Like things like that. These are human processes. It has nothing to do with the AI. And yeah, we tend to get too excited about AI sometimes, so...

This is not just a code problem. Uh, we are starting to see, like because, uh, AI agents are being deployed to make decks. This deck was entirely design-wise was generated by AI. I went through multiple iterations. It's not a zero shot. I wen- I went through every page like at least 30 times to, to change the text, but the design was done by agents. And, and sometimes within your organization, you're starting to notice people that just one shot prompt the entire

slides, things like that. And, and they start passing it around. They start doing meetings, and then it's on everyone else to filter what's the inconsistency, what's, uh, what's the misalignment, why is these two data points completely different. This is also another example of like same as the previous example, people generating code or people generating slides or people generating report, and the burden shifts onto some human down the road to verify things. So be very aware about this organization change. Um,

why am I still an optimist this way? I don't think this is necessarily a bad thing in itself because the problem is everyone needs to learn, and this is where we just start adapting our processes accordingly. Uh, and the other major thing is that, well, there are a lot of common examples on how to use AI. You all know your subject domain for your organization better, and there will be use cases where an AI could be used in place to automate a process, for example, that could save your organization, uh,

millions of dollars or increase your productivity at a grand scale that, that only you will know and not me as an AI expert will come in. I like to point to the most, uh, famous example that Walmart had where... Was it Walmart? I believe it's Walmart. Where they literally started putting a stool, a stepping stool inside every single truck. And that was like the simplest change, but it lowered their medical bill by the millions because it lowered back injuries across for heavy goods,

respectively. It, it's some- something specific like that with the use of AI, a- and it may not even necessarily be, uh, the best AI model that actually can have the biggest impact. Yeah. Yeah. So, uh, doubling down on my previous point, uh, one of the term that, that was coined by, uh, that, that's been trending is called the slop grenade. So generating lots of things with AI and then re- leaving it to someone else. Do not let slop grenades happen. Uh, and this is a culture thing and

a process thing. Yep. So the some guideline rules that I would set, and this is more subjective, uh, people, people will disagree. Um, to prevent a slop grenade from happening, enforce that people should own what they hand over or generate. So for example, these slides, I own every text that is generated. If there's an issue, that's on me. It's not my AI agent who, who, uh, who has an issue. I do not get to hand it off to an agent. I must own the

responsibility. Same thing for code, same thing for anything that is generated. And you also have to measure outcomes, not usage. Uh, and so s- this gets more complicated because I, uh, I don't want to explain too, too long into this, but like there, there is like this simple formula that's, that I think people take as a rule of thumb, tokens times price divided by outcomes someone actually use. Um, and, and this help guides a lot of things. Smaller, more efficient models, faster model, bigger model,

yada, yada. But what I don't like about this is that another analogy that I'll use about AI is I think of it more as like supplements or steroids that enhances a team. Think of it as, uh, like if, let's say I use construction work as an example. A, a, let's say I can use it to increase the muscle mass for the workers. Example. Not that it's ethically to do so, I, I know that's not the case, but let's just say as an example, the train of thought. Some teams will have it and they improve productivity, some teams don't.

But even without that, it's quite clear some teams are much better than others. And, and so the AI being a supplement here, right, doesn't necessarily change everything. So you need to be aware that outcomes, outcomes are hard to measure, uh, respectively. But it's something that you can start m- measuring and observing. Most importantly, back to the first step, measure y- you know, the token spend per department, per team, and you can start seeing which team is performing better, and you

can start seeing how much tokens they use. You can start seeing what works. Yep. Yeah. As I said, there's flawed measurement. So yeah. Uh, that's, uh, that's the high level o- overview, uh, respectively. And if, i- if you want, uh, to, to like kickstart your process internally within your organization, you can, you can either talk to me and my team or like you have six questions for you to start asking like, "Who team spend what? What's, uh, what, what must you test to use a cheaper model? What's your base load? What's the default model for each team? And

which workload has a person waiting that you need something faster?" Yeah. "And how much of what we generate is useful?" These are like guideline questions for you to start asking to refine your tokenomics. Yep. So Q&A.

Yes.

Yeah. So when you mentioned, uh, fine-tuning models, is that-

Test. Test

... um-

Yeah. I'm trying

... like SFT sort of fine-tuning or is there like other sort of, uh, fine-tuning methods that you have depending on what specific use case there is or what data there is?

Uh, okay. So in general, it tends to be SFT, but I don't think, uh, uh, um...

Well, we do provide fine-tuning services, and it tends to be SFT. I- there is a lot of methods that are, are happening right now, uh, currently. I know RL is very sexy right now, especially in the coding environment, because it's easy to generate the data in the loop. But what we realize for a lot of organizations, especially like in New York, for example, financial institute, things like that, there's not really an RL environment for financial tasks, for example. And, and so it tends to be more

SFT based. Uh, um, and, and there are also other organizations like Adapted- Adaption AI, um, uh, where, where, where they are starting to also automate the, the, the refinement of your datasets for you to train the model, be it SFT based or thinking- thinker machine based. Uh, focusing on this might be the wrong question. It-- to me, generally, what people should focus on is the evals respectively. So

here's a good example. Uh, for Jev that came... When the Jev model came out, and then, uh, we did the open source project called Simple Jev, uh, which replicates the Jev API for o- existing open source model. A lot of people quickly realized that the Qwen 27B was really good at this task. And, and there were multiple teams that went to fine-tune the Qwen 27B. I, uh, on, on our side, because we are trying to take the approach like our company, like we want to accept any model, and then we just build the API on top of it. We started taking the approach of like, "Hey, we

see the benchmark, we see the existing model. Why don't we prompt tune it instead?" It's not even fine-tune. And then, and then, uh, two weeks later, as Hugging Face went to do all the benchmarks at scale, like over 10,000 questions and things like that, somehow my prompt tune Qwen 27B beat most of the 27B fine-tune. So yeah, this, this is not a flex because I'm quite sure now my model is beat by the recent Cloudflare fi- fine-tune model that was done on a more proper basis. But it's more of like

the tuning method might, uh, the tuning method might not be the right question. It's more like what's your evals and what's the results that lead to it.

And then a follow-up for that, do you need, do you need to collect golden traces for the, um, set of data you need if you're doing SFT or whatever it is? Or can you do... Let's just say like your, your, for your task, the frontier models get like 80% accurate, right? So maybe 20% of the time it makes some sort of mistake. If you fine-tune on that, won't the fine-tune model also get like 20% of it wrong? Are you supposed to filter that out somehow and get the, only the golden traces for training? Or, uh, can you just throw all the traces in, um, and

train?

So this is m- more a personal opinion technique because, uh, so, um, the... Generally, I try to split the dataset because people are lazy filter. Um, so I try to at least get a golden trace dataset. And the golden trace dataset maybe is much smaller, uh, respectively, and we just, uh, and we then we fine-tune that with a much heavier weightage, uh- As well. But there have been multiple papers, uh, especially, uh, uh, es- uh,

especially, um, with thinking tokens that even the f- non, uh, the failed task, right, the mo- a much smaller model when you fine-tune tends to benefit from the training because it actually improved the thinking process even if it got the wrong answer. Um, this is something-- this is really a good case of like try and then measure, and it works seventy percent of the time. Sometimes you just-- you need to increase the golden trace ra-ratio, but it's a general rule of thumb. I

think for a lot of customers when they're trying to, to SFT down a smaller model, the general rule of thumb they're trying to keep is, can I keep to the near same level of performance? And maybe the eval is down one percent or even in some cases it improved by ten percent because the base model wasn't specialized for these tasks. And more importantly, does it hundred X reduce my bill? So that's usually the motivation, uh, respectively. Um, yeah, golden samples are critical if you want to do beyond the

baseline, and sometimes that requires humans to actually look at the data which as much as I love my customers to look at their data more, people don't like to do that.

Um, hi. Um, how would you deal with, uh, code spend that would mostly come from Claude Code and all the integrations there? Because then your model is fully locked into the Claude, uh, ecosystem, I, uh, s-suppose. Do you have any advice on how to deal, deal with those type of situations?

Uh, yeah. Unfortunately, Claude Code is the example of fully locked in. So, uh, sometimes, sometimes the first movement is actually getting, um, developers to switch to Harness. Um, Harness, it seems to be a lot more flexible thing right now. Like we have seen organizations that are like fully locked in, like I'm only gonna use Devin only or Amp only. We have seen organizations that, um, that are more flexible, like literally they are running your-- they are creating your own agent Harness.

And I'm-- my response is like, "Why? Why are you creating your own agent Harness from scratch?" Uh, but like a common one that we see people switching to is, for example, Pi. And this is also because there's a lot of excitement around it. People have shown that even with the same model, it gets a much cheaper cost per token because of the way it's simple and efficient. Um, and from there als- the-- I think the beauty for the Pi coding agent Harness is that it's easy to adapt because it's so s- so lightweight. We've seen a lot of teams that just start prompting their agents to make

modifications. And once, once they have that, then, then they start getting the cycle of experimenting and changing. Uh, a common way to get them to be very incentivized to change is to actually introduce a s- a good enough model that is really fast, like a hundred tokens per second, and then suddenly they start giving that a try. And some people start, uh, like some of the first shift will be they still use their Claude Code as their planner, and then they instruct the Claude Code, "Use this Pi agent with this

model to do the implementation," hence the planner implementation split. It's more... Yeah, and that's where the transition start to happen. It's not easy to change people's agents, but yeah.

Um, yeah. So an example of this could be perhaps to expose this, this smaller, uh, agent as an MCP in inside cloud or, or having these tools where you can offload, I, I suppose. Is that something that you would, you, you would say could work?

Uh, MCP is more tricky because MCP agents, unless done right, don't have access to your file system. So typically it's just something way simpler like Pi is installed and they run the batch command Pi this. Yeah. That, that's usually the case.

Thanks.

Yeah.

Can you share, if possible, any case studies about, uh, caching solutions being very thoughtful about caching and the outcomes that have been generated for specific customers, even if it's anonymized?

What's the specific challenge you're talking about? You do want caching... You-- Okay. Is your question more like you want anonymized... You do-- Wait, you do not want customer private information to save in the cache. Is it-

No. So, so, so like with customers that you've worked with, have you got any case studies you can share about the outcomes that they've been able to generate because of being thoughtful about how they cache and different sort of caching methodologies?

Uh, I think this one we can take offline, but in general, for our larger scale customers, we deploy them with dedicated cache. So it kind of solves a lot of PI issues because it's the c- the cache is only for that customer. Um, yeah. And obviously there's prompt formatting, but I may need to dig in more specifically like what you're trying to ask here. The short answer is yes, but there's a lot of way to interpret this. Yeah.

Cool.

Sorry about that.

No worries.

Uh, thoughts on turn or task-based auto routing.

Oh, um, I'm a-- I'm generally more towards task-based, um, mostly because it's much easier to control expectation at a department organization level. So this, this is back to like, hey, a department may have a labeled model, and I kind of auto routed that model for them.

Yeah.

Uh, I know there's a lot of, uh, hype right now for turn-based auto routing. One of the issues with turn-based auto routing is that you need to lock down your, your turn-based auto routing to, let's say, at most two to three models. If not, your cache will be extremely inefficient to be used. So, um, so for example, we have seen turn-based for GLM-5.3 and GLM-5.3 Flash. Makes sense. But if you put in thirty models and it keeps changing your model every turn, your, your l- low cache hit rate will start to wreck your,

your optimization.

Yeah. That's what I've seen before. I said, I started like request level and it just didn't make sense. And then-

Yeah. But two models tends to be okay.

Mm.

Uh, task and organization makes the most sense. Yeah.

Yeah. Yeah, I agree.

All right.

Okay. Uh, thank you very much. And, uh, if anyone wants to find me specifically for more questions, feel free to just come up front and ask me or find me by the side once the next speaker's here. Yeah. Thank you very much.