AI Engineer World's Fair 2026
Stop Renting Your Cognitive Infrastructure
Read the talk
Stop Renting Your Cognitive Infrastructure
Inference that looks cheap per call can become expensive at scale, while enterprise workloads introduce requirements that a lower token price cannot solve.
From a talk by Thiyagarajan Maruthavanan
When inexpensive calls become an expensive dependency
Thiyagarajan Maruthavanan opens with an unnamed major retailer that, he reports, spent nearly $200 million on Anthropic inference before building its own infrastructure. He follows with the Uber CTO’s budget-overrun account: in Maruthavanan’s telling, an annual token budget was exhausted in month four. These are accounts of inference spending, not measurements of what ownership would have saved.
The difficulty is that an inexpensive individual request gives little intuition for the accumulated bill. A monthly phone bill supplies a recurring number to anchor expectations. Prepaid inference credits encourage a different rhythm: load credits, consume them, replenish them. Maruthavanan compares that cycle to buying casino credits—each replenishment feels manageable until the total crosses the threshold you had in mind. His own application, Ultasuno, brought that problem home.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reverse music-prompt application meets scale
Suno turns a text prompt into music. Ultasuno asks the reverse question: given a song, what prompt could have generated it? The output is a plausible description for generation, rather than a recovery of some known original prompt. Maruthavanan built it for himself, enjoyed using it, and shared it with friends; distribution then extended beyond that initial circle. He reports hundreds of thousands of users and hundreds of thousands of dollars in inference spending.
Adoption explains why requests multiply, but it does not explain how much work each request needs. Maruthavanan identifies three places where applications waste inference:
- Context management: carrying more context than the current task requires.
- Input compression: sending material without reducing its token footprint.
- Agent loops: making repeated calls that consume tokens without enough useful progress.
His diagnosis is that the inference endpoint does not understand the shape of the consuming workload. It can serve the request it receives, but the application has the context needed to decide whether that request, its input, or another loop iteration is necessary. Workload efficiency starts before the endpoint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A stolen key creates a second spending problem
Not all consumption comes from legitimate users or inefficient agents. Maruthavanan says his API key was stolen three weeks before the talk, and attributes the unauthorized use to someone in China. He watched the cost rise through $7,000, $7,500, and $8,000 before his co-founder, who leads research and technology, stopped it at $10,000. His suggestion that it could have reached $100,000 describes a feared outcome, not an incurred expense.
This is a different failure mode from a successful application becoming expensive: possession of a credential allowed someone else to consume the endpoint against his account. He does not describe the containment procedure. The incident leads into a broader question about dependence on rented inference—and the alternative commonly offered in response, a token factory.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a cloud token factory to local inference
A token factory changes where inference comes from. Instead of paying Anthropic or OpenAI for their models, a customer uses open-source models deployed by a neo cloud or inference provider, with capacity provisioned in tokens per second. The local version takes the same idea into a garage or basement: buy GPUs, assemble a machine, and run the models yourself. Maruthavanan encountered that approach through people promoting local inference on AI Twitter.
He bought DGX Sparks and moved Ultasuno off Anthropic onto that setup. The application initially worked well, but memory became a bottleneck. He then expanded beyond the application to agents for his research lab, describing growth from two agents to six, eight, and twelve. The setup was useful enough to support that experimentation.
The next step exposed a different standard. Three enterprises asked him to replicate the setup, and a local installation that worked for his own applications did not automatically settle their reliability requirements. Serving tokens locally and operating an enterprise inference service are different responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control, audit approval, and reproducibility
The three organizations were an investment fund, a hospital, and a tax practice. Each had encountered a constraint that cheaper inference alone would not remove.
| Organization | Workload or situation | Blocking requirement |
|---|---|---|
| Investment fund | Investment analyst on a neo cloud | Control over rate limits |
| Hospital | Working use case undergoing audit | Acceptable vendor dependencies |
| Tax practice | Generated tax recommendations | Ability to recreate a recommendation |
The fund did not want an outside provider to dictate its consumption limits. The hospital had a working use case, but an audit red-lined a third-party vendor dependency and prevented it from proceeding. For the tax practice, producing a recommendation was insufficient: it needed to recreate that recommendation later. Maruthavanan attributes that obstacle to insufficient access to model internals; the example establishes a reproducibility requirement, not that access to internals alone guarantees reproducibility.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When does renting stop fitting the workload?
Maruthavanan places the decision boundary at product-market fit. Before demand is established, renting lets a founder test whether a use case deserves to exist. After product-market fit, he argues for building owned infrastructure. He applies the same reasoning to an enterprise that has already budgeted a project: in his framing, committing that budget assumes that the use case has sufficient demand to justify an enduring deployment. This is his proposed ownership threshold, rather than a calculated cost break-even point.
His analogy is moving to a new city. An Airbnb or rented home lets you experience the city and its neighborhoods before committing to a house. Eventually, however, the temporary arrangement no longer fits the life being built around it: “You cannot raise a family in an Airbnb.” That was the conclusion he reached for his applications, agents, and the infrastructure needed to scale them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ownership and token optimization operate at different layers
Maruthavanan calls his infrastructure effort jusInfer. Building it also reinforced that optimization is available before ownership: input cost, token management, and context management remain relevant while renting or leasing inference. The local deployment described here is his historical account; the current jusInfer site describes provider routing and prepaid billing, rather than documenting that local architecture.
He packaged some of those optimization experiences into the open-source project jusTokenMax, positioning it as an alternative to Headroom. Although he calls the comparator “Headroom from Netflix,” contemporary reporting identifies it as Tejas Chopra’s project, created while he was a Netflix engineer, not an official Netflix project.
Maruthavanan claims that jusTokenMax outperforms Headroom on many parameters, but supplies no numerical results or benchmark conditions in the talk. The project’s published results report input-token reductions without a Headroom baseline or an answer-quality comparison, so they do not establish that comparative claim. The useful architectural distinction remains: reducing the work sent to inference and choosing who operates inference are separate decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose against your workload, not a market prediction
Maruthavanan also introduces his book, Peak Inference: Infraeconomics of AI Inference, as a treatment of when to build inference infrastructure. He describes an AI market whose rules change every three to six months, leaving buyers surrounded by competing predictions.
He associates Jensen Huang with a future of token factories, Satya Nadella with local, unmetered intelligence, and neo clouds and inference providers with the expectation that endpoint providers will capture the value. These are the market narratives as he presents them. Their disagreement leaves the application builder with the practical task of deciding which arrangement fits the workload. His own path—from an application to agents, then scaling and infrastructure—led him toward ownership.
“Rent to learn, own to earn.” That is Maruthavanan’s closing rule, paired with a qualification: listeners must come to their own answers. Renting supported his exploration; established usage, accumulating costs, and enterprise requirements changed what he needed from the infrastructure beneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source toolkit for reducing coding-agent context through file, log, structured-data and attachment compression, with installation instructions.
Context-compression software for tool outputs, files and retrieved material, with library, proxy and MCP integrations.
The project's current public overview of workload-aware inference routing, supported integrations and billing.
Further reading
- jusTokenMax benchmark resultsDocumentation
Project-reported token counts and assumptions for PDF conversion, logs, structured data and incremental file reads.
Read the complete timestamped transcript
- 0:00
One of the largest retailers in the country spent close to two hundred million dollars on inference with Anthropic and decided that things got way out of hand and built their own infrastructure.
- 0:09
I'm pretty sure most of you have read the news from Uber CTO on how they had planned a budget of their tokens for an entire year, and it got over in month four.
- 0:19
I'm also confident that half of you in this room have come to a very similar conclusion that as time goes by, the cost of intelligence really builds. Using inference feels like, you know, it's one of the most inexpensive thing.
- 0:30
But then this is very different from using a phone where you get a bill once every month, and then you have like a specific set of amount that you can actually anchor your mind to.
- 0:40
But in case of using these rented intelligence platform, they are like prepaid. You load credits. It's almost as if you're loading credits inside a casino. You put some, and then you pull it, and then you are so addicted to it, then you end up doing more and more of it.
- 0:54
And by some time, you realize that you've blown past the threshold that you had mentally kept in mind. And I had this experience myself. I built an app called Ultasono, and I experienced the inference cost ballooning here.
- 1:07
Sono.com, has anybody heard about sono.com? Yeah, sono.com is this application that allows a user to turn a text prompt into music. What I was interested in is, is doing the reverse, which is given a particular song, what prompt could have actually generated it?
- 1:23
This is something that I wanted, so I built this, and I was having a lot of fun using this application, shared it with a few friends, and spread w-wide around.
- 1:32
I had hundreds of thousands of users, but then the cost ballooned way more than what I had anticipated. Hundreds of thousands of dollars had to spend on inference. Now, this happens for many reason.
- 1:43
There are many talks that are there at AI itself, where people talk about how you need to manage your context better,
- 1:50
and many people forget about doing compression of their input token. And when there are agent loops, then there are many of these calls that are happening which are very, very wasteful.
- 2:00
The inference endpoint that is consuming this is completely unaware of the shape of the workload and which is why this happens. And I have this other issue that had happened.
- 2:08
Three weeks ago, my key got stolen. Someone in China got hold of it and then was sucking my endpoint dry. I could see the cost rise up from seven thousand to seven thousand five hundred dollars to eight thousand go on and so forth.
- 2:23
Thanks to my co-founder, who heads the research and technology, we were able to arrest it at ten thousand dollars. Otherwise, it could have been a hundred thousand dollars. Now, many people suggest that the alternative to rented intelligence platform is is to use token factory.
- 2:35
Token factory is, is basically saying that why are you paying money to Anthropic and OpenAI? Instead, go open source, have these open-source models that are already deployed somewhere on the cloud, and then they are provisioned as tokens per second.
- 2:48
There are neo clouds, and then there are inference endpoint providers who actually do this. In fact, there is also an argument saying that, you know, you can build this token factory locally.
- 2:56
There are AI Twitter influencers who actually talk about building inference in your garage, in your basement. Buy GPU cards, rig them up together, and then you could actually run a local token factory.
- 3:08
In fact, I was in-inspired by that a little bit. I bought my own DGX Sparks, and then I first moved Ultasono from Anthropic to DGX Sparks. It worked well.
- 3:19
I ran into this one issue of memory being the bottleneck, and then it was good enough that I started building my next applications. I started having agents. I have some agents that I need for running my research lab, so these agents started shaping up inside the DGX Sparks, you know, two, six, eight, 12, and it worked all
- 3:36
right. The issue, though, is, is that, you know, it may not be reliable for enterprise, which is what I exactly faced. Three enterprises reached out to me to replicate the same setup for them.
- 3:47
But for enterprises, renting and leasing don't cut it. Bill is a problem, but then there are secondary set of problems that makes it extremely ineffective approach. The enterprises that I reached out to me, one was a fund, another a hospital, and the third a tax practice, and each of them had different wall that they had hit.
- 4:06
The fund, it was an investment fund. They were running an investment analyst on a NeMo cloud architecture, and they didn't want somebody else to dictate as to what the rate limit that they could consume.
- 4:16
So control became a big issue for them to actually go with token factories. Hospital had a different issue. They used the use case, it worked well, but later when they went through an audit, a third-party vendor dependency was red-lined, and then they couldn't go forward.
- 4:32
The tax practice was a completely different issue. In a tax practice, what is happening is, is when intelligence generates a recommendation, you want to be able to recreate it.
- 4:42
And when you don't have access to the in-depth of the model, you will not be able to do this, and that became the issue. So that brings me to the most important point of this presentation.
- 4:52
Where do you sit? When do you stop renting infrastructure? If you're a startup, if you're a founder who is doing pre-product market fit work, so you're still figuring out that the use case that you have, if there is demand for it, you can get by, by renting.
- 5:05
But if you're post-product market fit, you cannot afford to do. And if you're an enterprise who's already budgeted a project, which means you're telling that you are assuming that this particular use case has product market fit, then again, you cannot ignore to build your own infrastructure.
- 5:21
Which is what I realized, and I said that this, this situation is, is like if you're going to a new city, you may initially start with saying that, "I don't want to buy a house.
- 5:31
Let me actually rent and see." Sometimes you might even Airbnb. You experience the environment, you experience the city, the neighborhood, but then eventually you have to buy the house.
- 5:40
You cannot raise a family in an Airbnb. As I went through this experience, I decided, I came to the conclusion that I need to build my own inference infrastructure for the apps, the agents, and the scaling of the apps that I'm building.
- 5:53
And I call this as JustInfer. And while I went through this exercise, I realized that there is optimization to be done at multiple layers. Even at the renting and the lease layer, you can do optimization around input cost, token management, and then, you know, context management and so on and so forth.
- 6:09
Some of those experiences that I've had in the last couple of months combined it into an open source project and published it as JustTokenMax. If you have used Headroom from Netflix, then this is an alternative to it.
- 6:21
We have benchmarked against Headroom, and on many parameters, JustTokenMax is, is far superior. If this is a thing that is of interest to you, give it a try, maybe a GitHub star if you like it.
- 6:31
And I also wrote the book called Peak Inference: Infraeconomics of AI Inference, when you have to think about building your own inference infrastructure. The AI market is, is very different compared to the rest of the technology market that used to exist because here the rules of the game change every three to six months, which means it becomes
- 6:50
a very noisy marketplace. You talk to someone like Jensen, he would say, "Token factory is the future." You hear someone like a Satya Nadella, he will say, "Unmetered intelligence is the future.
- 6:59
It is going to be local." And then when you hear neo clouds and inference endpoint providers, they'll say, "Hey, inference endpoint providers are the ones that are going to capture the value in the marketplace."
- 7:08
Now, my experience working from application to agents to scaling them and then building my own inference infrastructure taught me that if you want to learn, you can rent, but if you want to earn, then you have to own.
- 7:22
And if there was the one sentence that you were to take away from this entire presentation, it is that: Rent to learn, own to earn. But then you have to come to your own answers.
- 7:32
Thank you. And if any of these topics are of interest to you, then I'm happy to talk to you about renting, about JustTokenMax, about how to build your own inference infrastructure.
- 7:44
I'm here [REDACTED:username] the AI Engineers Conference for the next three days. Hit me up on, uh, Twitter [REDACTED:username] or through my site, mtrajan.com.