Why No One Agrees on What Their AI Tokens Cost - Pratyusha Duvvuri, Cisco

Pratyusha Duvvuri gave the room seven seconds to think about what tokenomics meant. Her team had put the question to people at about 100 companies, from engineers and product managers to CFOs. One person wanted to know what a token cost. Another wanted a financial breakdown of the return on investment for every team. Duvvuri had come to the problem through agent observability and distributed tracing at Galileo. Even with that background, counting usage would get her only partway to the second answer.

The meaning of a large bill was changing, too. Duvvuri recalled an executive suggesting that an employee earning $500,000 ought to spend around $250,000 on tokens. High consumption was a sign that someone was making use of the tools. Then came an account of a company spending $1.5 million in a week and the alarm that figure caused. Once efficiency enters the conversation, the interesting decision is which calls merit the expense.

Galileo had an internal task on which a cheaper model scored 96% on the team’s evaluation. Running the evaluation with a more expensive model raised the score to 99%. The team kept the cheaper one. Those three percentage points had too little value on that internal task to justify switching. Duvvuri would weigh the same improvement differently for an agent talking to customers, where a better response could improve someone’s experience. That was a judgment about where she would spend more. The measured gain was on the internal evaluation.

Even the price side of that decision depends on who signed the contract. Customers described agreements at four cents per credit and others at three cents. Another agreement charged nothing up to a $30 threshold, then applied a rate of ten cents per credit after the threshold was crossed. In that last case, knowing how many credits a call used still leaves a question: where was the customer relative to the threshold? A single published rate cannot reproduce all three bills.

The customer interviews gave the team three audiences to design for. Each had a decision to make while the meter was still running.

• Engineer. With a $10,000 budget for the month, how much have I used? Seeing which models and skills account for consumption helps an engineer decide where a cheaper model might do the job.
• Engineering manager. The observability team is delivering results and overspending. The product team is spending less and missing its promised KPIs. Ranking the two by cost would put the cheaper team first and leave the manager’s question unanswered. The manager wants to see how usage and adoption are changing in each team, then consider those trends alongside the results.
• Finance. A CFO needs budgets and alerts early enough to act, and an account of the return that will make sense to the CEO or board. Duvvuri likened metered usage to a taxi ride. You know the meter is running, but you learn the fare at the end. That makes waiting for the invoice an uncomfortable way to manage a budget.

A provider portal could already show Duvvuri costs, skills used and sometimes a projection. As an engineer using several services, she still had to move between portals to piece together her usage. Adding their totals introduces another problem. In her example, a provider initially reports a monthly charge of $100,000. Its reconciliation process later puts the charge at $120,000. A dashboard that preserved only the first figure would now understate that same month’s cost by $20,000.

Even the corrected total leaves finance looking for an owner. A workforce system such as Workday traces a person to a manager, that manager to another manager, and so on up to the CEO. Finance organizes costs according to its own reporting hierarchy, taking into account categories such as operating expenses (OPEX) and cost of goods sold (COGS). Knowing who manages the person who incurred a charge answers only one of those ownership questions.

Duvvuri’s working definition of tokenomics is “the observability and interpretability of token usage and cost.” Observability gets the usage and charges into view across providers. Interpretability lets someone ask which models contributed to a charge, how to categorize the work for a particular use case and what return might justify it. The distinction matters when designing a dashboard. A total can be accurate and still give a manager too little information to make a decision.

How does one person’s usage become both a team total and a finance report? Onboarding brings together three inputs. A workforce hierarchy from Workday, ADP or an equivalent system supplies reporting relationships. Provider API keys or account identifiers let the product fetch usage data every day, giving the dashboard a one-day refresh cadence. The customer also supplies its finance hierarchy, since the accounting structure differs from company to company.

The workforce import could be a CSV with six required fields. Customers were willing to share information such as names, IDs and job functions while keeping details such as tenure private. A limited export made it possible to supply the reporting structure without handing over an employee’s entire history. Customers who supplied more attributes could break down usage in more ways.

The demo opens in the leader view, designed for engineers and engineering managers and also accessible to finance. Actual company figures are blurred. Selecting a week or month to date sets the period for reading the cost and adoption trends, with a fiscal calendar available for financial reporting. The cost dashboard tracks token consumption and money spent. The adoption dashboard answers a different question about the same period, such as whether the team’s use of a coding tool is growing or declining.

Will the budget last a month, or could it run out in a week? That question from managers and finance teams prompted a forecast preview. Look for the part of the graph that opens out as it projects usage forward. Its width matters: the chart presents a range of possible usage, and Duvvuri tied the wide spread to changes in models and costs. Forecasting was still awaiting public release at the time of the demo.

Whose usage is included in an executive’s total? Below the daily cost trends, the leader view starts with the CEO’s direct reports. Duvvuri’s usage contributes to the chief product officer’s branch because she sits somewhere in that reporting chain. Expanding the branch reveals the people who report to that leader. Continuing down reaches Duvvuri and her own reports. At that level, a reader can distinguish the team’s usage from the amount attributable to Duvvuri individually.

Changing the grouping to job family answers a question that would take several trips down the tree. A leader responsible for product, sales and engineering can compare consumption across those functions. At one customer, sales was using more than engineering, a surprise that the leader’s combined total would have hidden. For a closer look, the product breaks cost down by model and by low, medium or high thinking level, and lets the reader filter by person, job function and total cost.

Where does the same charge land for a financial controller or analyst? The finance view opens a cost map organized around the company’s financial reporting structure. Follow a branch to its end and you reach a department. Everyone from an engineer to the CEO belongs to a department, so each person’s cost rolls up through a node in that tree. This gives finance a way to inspect the spending behind a total using the structure it already uses to report OPEX and COGS.

Role-based access control limits what any one reader can inspect. A manager responsible for team A can see that team’s usage. Looking across to team B requires administrator access in Duvvuri’s example. The product can therefore aggregate costs across the company while restricting which underlying records a manager can open.

The team went from its first customer conversations to general availability in 10 weeks and had paying customers. Those conversations also surfaced a request that went beyond allocating cost. Could a team tie its token usage to a customer it landed, a migration it completed or interest in a new product? Duvvuri described this as ROI inference. The dashboard could identify who incurred the cost. Establishing how much that spend contributed to a business outcome remained an open question.

New requests could change the data model after people had already come to rely on it. Duvvuri illustrated the scale with a hypothetical organization of 90,000 people and 10,000 users accessing the platform. A change to accommodate a new use case still has to serve the people using the existing views. And it has to leave an engineer checking a model choice, a manager comparing teams and a controller tracking a budget able to find their own answers.

Reconciliation makes even a backward-looking question tricky. When the provider revises last month’s charge, the next fetch can change a figure finance has already used. Meanwhile, an employee can leave or move to a new manager. Applying today’s reporting tree to last month’s usage can change which manager appears to have incurred the cost, even if the provider leaves the charge alone. That leaves a concrete question for anyone reading a historical total: does a change reflect a revised bill, a change in who owns the spend, or both?