The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

A single shared API key can hide an entire organization’s AI bill. Eugene Cheah, founder and CEO of Featherless AI, says 30–40% of the customers who approached his company began that way. Split keys by department and major use case, and the bill starts answering a useful question: which work generated the spend?

That distinction informs more than chargeback. A developer waiting for a response may need high tokens per second; a background agent running overnight may not. Usage by key also helps estimate dedicated capacity and compare spending with a team’s results. Cheah sells inference, so his examples come from that business. His advice is aimed at scale: below an arbitrary $10,000 a month, he recommends finding valuable uses before spending much effort optimizing the bill.

The scale can be considerable. Cheah describes a customer serving two trillion tokens a day on 512 AMD MI325 GPUs in a specialized deployment. The recording gives conflicting per-million-token prices for this example, so the volume and hardware illustrate scale without establishing a reliable unit price.

Two workload extremes have different economics, with plenty of cases in between:

• Agentic work: Coding agents can run for many turns, repeatedly using long context and compacting it as work continues.
• Process automation: Email routing, call-center decisions and document processing tend to be shorter, isolated tasks. Their challenge is often the number of decisions rather than the length of one conversation.

One team of five developers averaged eight running agents and used 116 billion tokens a month. Cheah reports that its API bill was about $150,000 monthly; after moving to a single-node open-model deployment costing under $10,000 a month, the team noticed no disruption in its impact. That is the team’s reported experience, not a general equivalence test. Scheduling some background agents overnight also helped use the hardware outside the team’s working hours.

At the other extreme, a telecom company scans messages for content such as profanity or sexual material as part of a safer-phone feature for children. Cheah says it can encounter a peak of ten million messages in one minute, especially around New Year. Each decision has relatively little context, but a small cost per decision is multiplied by enormous volume.

A model does not have to be the best coding model to be useful in finance. Cheah describes a specialized financial workload where a model roughly ten times smaller approached the quality of stronger general models. His practical point is to test candidates against the actual task; model rankings and the available choices change quickly.

Classification is another opportunity to change the shape of the work. Cheah describes one-shot classifiers as prefill-only: they process the input to make a decision without generating a chat response. He reports that they can be around ten times cheaper and potentially faster than generative models, while emphasizing their limitation: they cannot write the response. In customer support, for example, a classifier may be sufficient when the system chooses among predefined replies. Featherless’s Simple Jev tutorial is a practical starting point for that approach.

For agents, the token mix matters as much as the advertised output price. Cheah says output accounts for only about 1% of tokens in the agentic deployments he sees; long, repeated inputs and cached context make up much of the rest. Even a low input or cache rate can matter when it is paid repeatedly across many turns.

One coding pattern assigns planning and review to a strong model and implementation to a faster, cheaper worker. Implementation can consume many tokens as the worker edits files, runs tests, inspects results and tries again. Cheah reports organizations using this split successfully, with the quality of the plan an important condition of the result.

The human incentive matters. Cheah saw teams leave a cheaper model unused when both options ran at the same speed and users had ample budget. An extremely low budget produced the opposite problem: people exhausted the stronger model’s allowance in a day and effectively lost access to it. His preferred incentive is a cheaper model that is also fast enough and capable enough for the task. Some harnesses can instead start with a cheaper model and consult a stronger one when needed.

Keys can also encode latency priorities: serve people quickly while giving a CI/CD pipeline a slower, cheaper service. For departments that do not want to follow model releases, labels such as “fast,” “smart” or a task-specific finance option provide a stable choice. The operator can change the underlying model or split traffic for a test. That control has a cost: a bad swap can still disrupt users.

High-volume process workloads are often already sensitive to price. Instead of guessing whether a smaller model is good enough, assemble examples and evaluations, try candidates, and find the cheapest one that meets the task’s requirements. Cheah describes one telecom customer whose monthly customer-support inference cost fell from $250,000 to $7,600 after the team split workloads, selected models and fine-tuned where appropriate. He does not provide the evaluation results behind that reported saving.

Fine-tuning becomes more attractive when a domain is stable and the volume is large. Cheah describes AT&T using 27B and 7B models for different call-center needs, and Shopify fine-tuning for its proprietary store configuration language. A small model can handle routine cases while a classifier triggers escalation for harder ones; Cheah presents that routing as an example rather than specifying AT&T’s exact assignment of model sizes.

A practical bootstrap is to run a strong model on a limited amount of the real task, collect examples, and use them to fine-tune candidates at different sizes. Check the model provider’s terms before using its outputs for training. After a smaller model takes production traffic, Cheah recommends sending a portion as shadow traffic to a stronger model to keep building a dataset as capabilities improve.

Serving speed and cost do not scale linearly. A fixed model running alone across a server’s GPUs can generate quickly for one request. Running more requests together can slow each one while increasing total output from the hardware. Cheah’s rule of thumb is that doubling speed can cost three times as much, although the actual tradeoff depends on the deployment.

How much capacity does that trade buy? In Cheah’s example, one request gets 112 tokens per second; with 16 requests in parallel, each gets roughly half that speed. The relationship matters: roughly 16 × 56 is 896 tokens per second in aggregate, about eight times the single-request rate. Cheah describes the total output and cost improvement as roughly nine to ten times; the rounded speeds alone do not reproduce that figure.

Cheah sees customers commonly choose between roughly 20–30 and 60 tokens per second, depending on the task and whether someone is waiting. A background job has more room to trade speed for throughput. Even for interactive work, a modest speed reduction may save money without materially changing the experience.

Traffic shape determines what to reserve. A well-utilized dedicated instance can serve a predictable base load, while bursts spill into public cloud capacity where policy permits. If the workload is spiky, reserving for the peak can leave expensive hardware idle; if it is steady, relying entirely on burst capacity may miss the savings of a dedicated deployment. Financial institutions or other customers restricted to dedicated instances may not have the mixed option.

A company can multiply token spend without multiplying useful output. Faster generation moves the bottleneck elsewhere: CI/CD may take longer, or reviewers may lack the tests and context needed to trust a change. That distrust can be justified when an agent is poorly designed; even a correct change can wait if nobody has a way to validate it.

Cheah describes a particularly wasteful coding pattern. An agent produces a huge change so an issue will get attention. The reviewer cannot comfortably inspect the pile, asks another agent to identify relevant changes, and reimplements them with that agent. The first implementation is discarded. The reviewer’s concern for control and quality is understandable, but more generated code has created duplicate work.

Better tests can filter changes, and smaller, modular merges make them easier to review. The same handoff problem appears outside code: a one-shot deck or report can leave colleagues to reconcile inconsistent claims and data. Cheah says the design of his own deck was agent-generated, but he went through each page repeatedly and took responsibility for its text.

Cheah calls a mass of AI output handed to someone else to sort out a “slop grenade.” His proposed rule is simple: the person handing over generated code, slides or a report owns it. A recipient should not have to absorb the verification work merely because generation was cheap.

A rough cost measure is tokens multiplied by price, divided by outcomes someone actually uses. It directs attention toward useful results, but Cheah cautions that outcomes are difficult to measure: teams differ in capability before AI is introduced, and AI may amplify those differences. Spend by department and team is a starting point for investigation, not a clean causal productivity score.

The operational questions follow from that uncertainty: who spends what, which tests would justify a cheaper model, what is the base load, what model does each team receive by default, where is a person waiting, and how much generated work is actually useful?

Cheah says his company’s fine-tuning work generally uses supervised fine-tuning (SFT). Reinforcement learning can be attractive for coding, where a feedback environment can generate data in a loop; he says comparable environments are not always available for financial tasks. His advice is to start with the evaluation rather than assume a particular method will win.

For the classifier project, Cheah’s team prompt-tuned a Qwen 27B model instead of fine-tuning it. He reports that, in a benchmark of more than 10,000 questions, it beat most of the 27B fine-tunes compared at that time; he also says a later fine-tune surpassed his result. The useful comparison is prompt tuning versus weight updates on a particular evaluation, not a universal ranking of methods.

What if the strong model produces flawed training traces? Cheah’s personal approach is to keep a smaller set of checked, high-quality “golden” traces and give it more weight. He says failed traces can sometimes help a smaller model learn a reasoning process, but recommends testing the mix and increasing the golden proportion when needed. Human inspection may be necessary to build that trusted set. The goal is to compare task performance and cost together, rather than assume every teacher answer is correct.

When a coding workflow is tied to one model ecosystem, changing the harness can be the first hurdle. Cheah describes teams trying the lightweight Pi harness and making a gradual transition: keep Claude Code as planner, then have it invoke Pi with a fast, good-enough model for implementation. A visibly faster worker gives developers a reason to try the change.

Asked whether to expose the worker through MCP, Cheah notes that an MCP agent may lack file-system access unless configured appropriately. In the pattern he describes, invoking an installed Pi batch command is often simpler. That choice depends on what access the worker needs.

On caching, Cheah says Featherless gives larger customers dedicated caches, isolating one customer’s cache from another’s; he does not provide the requested quantitative case study. For routing, he prefers task- or department-based choices because expectations are easier to manage. Turn-by-turn routing can work with two or three models, he says, but frequently switching among many models can reduce cache hits and erase some of the intended savings.