How to Actually Tokenmaxx and Save Millions — Alvin Pane, OutRival

A customer-intent classifier accumulates twenty edge cases, pedantic definitions and a paragraph pleading with the model not to hallucinate. Each addition may have answered a real failure. Together, they become text the application sends again on every request. Multiply that by millions of requests, and prompt maintenance becomes a recurring compute bill.

Alvin Pane, introduced as head of agent intelligence at OutRival, uses this fictional classifier as a synthetic proxy for production prompt bloat. OutRival’s digital workers handle tasks over phone, text and email; Pane reports application-layer usage of billions of tokens per day. At that scale, redundant calls and repeated instructions consume money and latency that could support additional capabilities. The synthetic example teaches the optimization procedure; it does not demonstrate realized million-dollar savings.

That makes token-consumption leaderboards a peculiar success metric. As Pane puts it, “Tokens aren't the work, they're what you pay to get the work done.” His version of tokenmaxxing asks how much useful behavior an application can retain while reducing the payload and compute needed to produce it.

Start at the shared model client. Instrument each attempt so a task’s model calls, tool calls and retries belong to one trace. Record the prompt version, model settings, token usage, latency and cost. Pane names OpenTelemetry as a framework for this instrumentation. The trace connects the bill to the work that generated it, while the recorded production examples provide a benchmark for testing changes.

The tempting first experiment is straightforward: give the existing prompt to a cheaper model and compare its answers with historical production answers. In Pane’s account, a candidate disagreed with the expensive baseline about 18% of the time across a few thousand historical examples. The team rejected it. That decision treated agreement with one production answer as the quality standard.

Messy customer inputs complicate that standard. A production answer is itself one probabilistic decision; another run of the unchanged system may choose differently. The 18% disagreement therefore combines behavioral differences between systems with variation already present inside production. It cannot, by itself, tell the team how much quality the cheaper model lost.

The correction was an A/A test: run the expensive production model three separate times on the same inputs and compare its answers. Production disagreed with itself roughly 7% of the time, yielding about 93% self-agreement. Pane calls this the “ceiling test.” It calibrates expectations against the system’s observed repeatability; it does not establish an accuracy ceiling or turn the remaining difference into a measured error rate.

With a measured baseline, the first cost lever is prefix caching. Structure prompts to maximize reuse of their repeated beginning, reducing the cost of processing text shared across requests. This comes before rewriting the task instructions: first reduce avoidable repeated work, then reduce the amount of instruction text itself.

The fictional classifier’s prompt is almost 5,000 tokens long. It contains accumulated rules such as “do not confuse enthusiasm with authorization,” plus long hard-coded transcripts serving as worked examples. The maintenance trap is familiar: nobody wants to delete the line that prevents a production failure. Manual shortening leaves each deletion dependent on someone’s confidence about why the text was added.

DSPy makes that editing problem searchable. Define the inputs, required output schema and scoring function—the application behavior that must remain stable—then give the instructions to an optimizer such as GEPA. Candidate instructions run against the benchmark’s historical labels, and feedback from failures guides revisions. In the illustration, the optimizer replaces almost all the long worked examples with compact rules, reducing instruction tokens by 51% while retaining the same application interface.

The useful comparison is between the information carried by long examples and the compact rules that replace them. A shorter prompt has removed input cost, but matching the interface only means the application can still consume its answers. Whether those answers preserve quality is the next test.

A tighter prompt and a benchmark make model downsizing easier to investigate. Pane describes two distinct strategies:

• Split the work. Break a large prompt into smaller components. A cheaper model handles initial routing or data extraction; expensive frontier models handle the heavy reasoning. The saving comes from assigning simpler work to cheaper compute.
• Optimize for the smaller model. Run the DSPy search with the smaller model, using answers generated by the large production model as benchmark targets. GEPA searches for instructions that help this particular model reproduce the task performance needed on that dataset.

The second strategy changes the prompt rather than the model’s weights: the smaller model searches its own instruction space. But success on the benchmark can conceal overfitting, and decomposing work can introduce additional failure points. Pane reports that many downsizing experiments produced prompts that were not good enough to release. A cheaper model remains a candidate until the broader quality comparison holds.

After cutting the fictional prompt roughly in half, shadow evaluation asks both systems to answer the same inputs. Repetition matters on both sides: production varies, and the cheaper candidate has its own internal variation. The evaluation therefore needs three comparisons:

• Production against production: how much the unchanged baseline moves between runs.
• Candidate against candidate: how much the proposed replacement moves between runs.
• Candidate against production: how much their answers differ across versions.

Pane describes a disagreement adjustment: take the cross-version disagreement and subtract the average within-version disagreement. This uses the two self-comparisons to account for answers moving around even without a system change. The remaining quantity measures separation between the systems’ decision probabilities. The recording does not specify the estimator or its assumptions, so this is a conceptual recipe rather than a complete statistical implementation.

That adjustment answers how behavior changed, not whether the change was good. Production may already contain mistakes; faithfully reproducing them would preserve agreement while preventing improvement. Inspect the disagreements and distinguish three outcomes: the candidate did better, production did better, or both answers were acceptable responses to an ambiguous input.

The fictional review makes the distinction concrete: suppose reviewers find 443 regressions and 2,463 improvements. That is almost six improvements per regression, with 2,020 more improvements than regressions, from a prompt roughly half the size. These are illustrative review counts, not reported production outcomes. Their purpose is to show why a raw disagreement percentage can hide a favorable change.

Those counts still describe one batch. To assess whether its net gain is convincing, calculate a confidence interval while keeping the comparison paired: both versions must answer the same requests. Keep repeated runs of the same conversations grouped together, too, rather than treating repeated observations as unrelated new evidence. The review counts answer which system won on this batch; the interval addresses uncertainty around the estimated gain.

The release criterion requires the lower confidence bound to exceed zero. If the interval crosses zero, gather more evidence before claiming the candidate is better. A positive point estimate alone is insufficient. The talk does not supply an interval value or confidence level for the fictional counts, so the example establishes the decision rule rather than a completed statistical result.

Run the analysis separately for critical customer categories. An overall gain can conceal deterioration in a vital intent, so a favorable global interval cannot carry the release by itself. Pane’s rule combines the two requirements: critical categories must hold up, and the lower confidence bound on net gain must be above zero. Then ship—and keep measuring.

The closing assignment is deliberately bounded: choose the most expensive task, measure its baseline variation and put its instructions through a search loop. Cache repeated work, shorten the prompt and investigate a smaller model, then test whether quality holds. The objective is to discover how much bloat the application is paying for while preserving the work it needs to perform. That gives “making every single token count” an operational meaning—and a release rule.