FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft

An AI bill arrives, and you cannot trace it back to the agent workflow that produced it. That is the opening problem for Tisha Chawla and Susheem Koul’s FinOps talk. Exploration can justify spending heavily—Chawla makes room for the enthusiasm around becoming a token millionaire—but eventually the question changes: what useful work did those tokens buy?

Software’s controls have usually followed its billing model. SaaS products use seat limits, usage caps, and tier policies. Pay-as-you-go cloud infrastructure adds provisioning and autoscaling policies. Agent workflows create spending through code that repeatedly calls a model. The proposed missing control sits close enough to that execution to understand why another call is happening and whether the workflow should change.

A gateway can impose a hard cap or route a request to a cheaper model. A runaway agent loop presents a different problem: individually acceptable calls can accumulate into an expensive, unproductive run. Controlling that behavior requires a view of the work unfolding across calls, rather than waiting for the final bill.

Cost begins at the model-call boundary, so that is where accounting must connect consumption to an agent and a run. An aggregate total can tell you that spending rose; attribution identifies the execution responsible. That identification makes a targeted policy possible: change the costly part of the workflow instead of treating every request alike.

The run supplies the context needed to choose an intervention. Repeated agent–tool exchanges, growing context, and the spawning of multiple sub-agents can all increase cumulative consumption. Chawla’s design calls for a shared budget across attributed work and enforcement in the call path. If context grows excessively, for example, the workflow should be able to compact it in place; a budget halt becomes the last resort after useful corrective policies have been tried.

This is the motivation for TokenOps, a run-aware token-governance system. The talk contrasts its execution-level interventions with request-level caps and routing in gateways such as LiteLLM, Portkey, and Cloudflare. The distinction concerns the control being proposed: a gateway can change or reject a model request, while run-aware governance can also act on the loop, retrieval output, or context that produces future requests.

The architecture separates three jobs:

• Instrumentation: Capture telemetry, including OpenTelemetry data, and enrich it with attribution so consumption has an identifiable cause.
• Accounting: Accumulate the attributed activity in a ledger rather than leaving each call as an isolated event.
• Enforcement: Apply policies that steer execution, with halting available when the budget is exhausted.

The control plane is deliberately out of band, but its effects still enter the execution path. Those two choices fit together: policy and accounting live separately from the agent’s application logic, while an instrumented boundary carries observations upward and authorized actions back down. Keeping the plane separate does not make enforcement passive.

Koul develops the implementation as three connected layers: the existing agent runtime, a bridge, and the control plane. The bridge associates runs with developer-supplied usage dimensions. Its central mechanism is a boundary annotation applied to existing methods, allowing the integration to work around method inputs and outputs without rewriting the workflow around a new agent framework.

The annotation has two directions of traffic. It records method inputs and outputs as ledger entries, attaching the agent run ID and other attributes. It also receives actions from the control plane. Observability supplies the information for a decision; the return channel lets that decision change the ongoing run.

Consider the talk’s hypothetical retrieval-augmented generation example. A retrieval tool returns twenty chunks per call, sorted by relevance. Suppose the model only uses the first five because the remaining chunks are irrelevant. The extra material still contributes to the budget problem. The control plane observes the situation and proposes limiting the tool output to five chunks.

The proposal travels through the boundary to a governor. The governor knows which actions the developer has allowed and how to apply them. If output reduction is permitted, the tool’s returned context changes from twenty chunks to five, retaining the relevant leading chunks in this example. The intended saving comes from reducing the material passed onward, rather than ending the retrieval workflow. This depends on the example’s assumption that the omitted chunks are unused; the talk does not demonstrate a general method for establishing that relevance.

How does a cost observation become a change to retrieval without giving the control plane unrestricted access to the agent? The diagram follows the twenty-chunk example through observation, proposal, and developer-authorized application. The governor is the point where a requested action must fit the agent’s allowed capabilities.

For model integrations exposed as objects rather than methods, a completion-wrapper helper applies the same boundary idea. It is an integration convenience: capture the operation and provide an action channel without requiring every provider interface to have the same shape.

Attribution dimensions become useful when the control plane groups them into segments. A preview agent shared with a conference audience can emit a cohort tag; a segment selects the users carrying that tag, and a budget applies to their combined usage. This supports both fine-grained limits on individual runs and coarser limits across a group. The grouping follows the dimensions the application emits.

The ledger gathers a run’s traces in one place. A budget adds a static threshold over a time window for a segment or run. A policy then connects that budget and its scope to an action. These are separate decisions: whose consumption counts together, how much they may spend during a period, and how execution should respond.

Actions have two broad behaviors:

• Halt: Stop the agent when it exceeds a budget. This is the circuit breaker that limits further work by terminating the run.
• Steer: Change the agent or one of its components so the run has a chance to finish within the allotted budget. Retrieval-output reduction is one example; changing instructions is another.

Integration therefore involves more than attaching telemetry. Existing methods receive the boundary annotation, and a governor instance receives configuration declaring allowed actions. Koul describes hosting the control plane in the application owner’s own tenant. That hosting choice keeps the described plane under the owner’s control; it does not by itself establish that captured method data cannot leak.

The demonstration uses a two-agent research workflow. A research agent receives a question and can repeatedly use a web-search tool. Once it considers the findings sufficient, it hands them to a summarizer that creates a research report. This gives governance something meaningful to preserve: completing the report, despite a research phase that can keep making calls.

The first scenario runs TokenOps in preview mode. Policies evaluate normally, but their associated actions cannot execute. The research workflow completes, and the dashboard displays policy activity—including the cost budget and cost guard—while governance remains off. Preview mode separates learning what the policies would do from allowing them to change production behavior.

That separation creates a practical introduction sequence: install the instrumentation, observe policy decisions, adjust guardrails and thresholds, then enable enforcement. The second scenario turns governance on. The dashboard reports that the pre-call cost cap has been exceeded, and the agent is killed immediately. The cap works as a circuit breaker, but the work stops with it.

The steering scenario uses another prompt with a slightly higher budget that Koul describes as still insufficient for the unmodified run. The cost guard considers two signals: how much of the budget has already been consumed and the velocity of token consumption. If those signals predict exhaustion before the run ends, the guard injects a system instruction asking for more succinct or summarized model outputs.

The causal difference is when and how the response acts. The hard cap stops execution at its threshold. The cost guard tries to change future generation before exhaustion, reducing the verbosity of subsequent outputs while leaving the workflow running. The talk explains the intervention but does not specify the prediction formula or establish the final report’s quality in this scenario; instruction injection is an attempt to influence consumption, rather than a guaranteed spending limit.

Where does preview mode differ from enforcement, and where does steering differ from a stop? The diagram makes the three paths visible. Preview ends with a recorded decision; enforcement can deliver either a halt or an instruction intended to reduce future output.

The benchmark extends beyond the small research demonstration to Browser Use and MetaGPT. Koul reports repeated runs across stress tests and simpler and harder scenarios. In the recording, enabling the full TokenOps policy suite lowers average spending by almost 78%, without an explicit spend baseline. He then compares completion with simple throttling, reporting an increase from 67% to roughly 96%.

The authors’ Microsoft companion article gives a more specific account with a different comparison: both results are measured against no governance. It reports 27 scored trials across Browser Use and MetaGPT scenarios, a 78.9% reduction in mean cost per completed run—from $0.068 to $0.014—and within-cap success rising from 18/27 to 26/27, rounded to 67% and 96%. The recording’s throttling comparison and the article’s no-governance comparison conflict; these figures cannot establish a measured advantage over throttling without resolving that baseline. The article’s completion measure means finishing under the cap, and cost governance does not determine whether the resulting answer is correct.

Completion is nevertheless the consequential companion metric. A control can lower consumption simply by killing runs. Steering instead tries to retain the useful result while spending less. Evaluating those controls together asks whether the budget buys completed work, rather than rewarding a low bill in isolation.

The benchmark uses a suite aimed at several failure modes:

• Spend management: Budget policies and the cost guard respond to consumption and its trajectory.
• Context management: Context compaction and tool-output reduction reduce material carried through the workflow.
• Loop and progress detection: Policies look for repeated activity and inadequate progress, extending governance beyond the size of an individual request.

The available action vocabulary includes allowing, mutating, or injecting for steering, and killing for halting. Different policies can therefore share an action mechanism while detecting different reasons for waste.

The ending turns the ledger into a proposed learning input. A self-learning module would examine the continuously updated records to find failure modes that current policies still miss. It could then generate new policies for remaining runaway costs or refine the parameters of existing policies. This is the envisioned next stage, rather than a demonstrated capability of the system presented.

That proposal closes the loop between accounting and control: record what happened, identify spending that escaped intervention, and change the rules used on future runs. The closing invitation points to a public wiki. That historical wiki front page is now archived and directs readers to the main TokenOps repository, where the SDK, control plane, dashboard, documentation, and runnable examples live. The useful starting decision remains which actions your agent may accept while preserving the work it exists to finish.