AI Cost and Performance Engineering

Useful completion and representative demand

A record-update service retrieves evidence and the current record, proposes a change, validates it, obtains review when required, and confirms the write. An acknowledgement means the request arrived. A proposal means a candidate change exists. Accepted completion requires the intended, authorized update and its required checks. These are different events with different clocks. [2]

An acceptance criterion states an observable requirement: correct fields, supporting evidence, required approval, and confirmed persistence. Parseable output alone is insufficient. Evals: Evidence for a Bounded Claim develops these criteria; here they establish which outcomes an optimization must preserve. [3]

A service-level indicator, or SLI, measures a defined service property. A service-level objective, or SLO, sets its target over a stated population and period. Specify response and completion targets separately, plus quality, coverage, and spending limits. A target without its measurement boundary, eligible tasks, and operating conditions is incomplete. [4]

A token is a model vocabulary unit; text length and token count are not interchangeable. Tokenization explains the representation. Extend the serving workload distribution to whole tasks: retain which input lengths, output lengths, and execution paths occurred together instead of combining unrelated averages. [5]

Work amplification is the internal work generated by one incoming task. A difficult case might require additional retrieval, correction, and review, but those relationships must be observed rather than assumed. Preserve complete task records so a change in traffic mix cannot masquerade as an efficiency improvement. [9]

Task, attempt, and shared-cost attribution

A task represents user intent; an attempt is an execution toward that intent; a service call is one operation within an attempt. Preserve their relationship across failures and escalation. Identity across services and execution attempts supplies the tracing convention. For accounting, record usage per call before aggregating by task. [10]

Include material model, retrieval, tool, environment, storage, transfer, evaluation, and required review expenses. Direct costs belong to identifiable work. Shared expenses need a documented allocation rule, such as reserved capacity or measured consumption. Do not add both a shared pool's full charge and its allocated portions to the same total. [12]

Provider usage and billing are separate evidence streams. Anthropic's documented reports, for example, have different coverage for token categories, tools, and service tiers; reports can arrive later and require pagination. Reconcile estimates against covered billing totals while retaining unmatched charges. Provider aggregates cannot reconstruct missing task-to-attempt relationships or external review expense. [11]

Failed, abandoned, and speculative work remains attributable even when no answer is selected. Telemetry also consumes resources: collection, processing, and storage belong in operating costs. Task metrics and usage accounting defines the reporting boundary. Raw prompts are usually unnecessary for a cost ledger and introduce additional storage and exposure. [13][10]

Unit economics of accepted outcomes

Assume two routes each process the same 100 tasks under identical acceptance rules, with all outcomes resolved. These example charges include one initial model call per task, subsequent attempts, required review, and allocated capacity. The ledger makes the cheaper initial call's downstream consequences visible. [14]

Report acceptance rate, timely acceptance, review burden, and task-cost spread alongside the ratio. A lower average can conceal expensive exceptions. Marginal cost is the extra expense of another task; average allocated cost includes its share of fixed expense. Total cost of ownership also includes acquisition, operations, maintenance, and labor. Paid idle capacity does not disappear because it produced no calls. [16]

If 20 tasks remain unresolved at the reporting cutoff, report them separately and mark both outcomes and incomplete charges as provisional. Compare cohorts after the same follow-up horizon. An unresolved eventual outcome is not success; missing a specified completion deadline can already be a definite failure. Delayed outcomes explains this distinction. [17]

Feedback from only the easiest or most satisfied users does not establish the whole cohort's acceptance rate. Preserve known successes, known failures, and unresolved cases. Better observation coverage can change the measured ratio without changing system behavior. [18]

Latency budgets and dependency paths

A latency budget allocates an end-to-end allowance across required work and waiting. The critical path is the longest dependency chain under the represented constraints. Independent operations can overlap, but a dependent operation must wait for every prerequisite. Accelerating a shorter parallel branch leaves completion unchanged while another unchanged branch still determines the join. [19]

Represent queueing and resource constraints explicitly. Taking the latest predecessor finish already accounts for prerequisite waiting; adding that same wait again double-counts it. Shared equipment can serialize otherwise independent work. Shortening an apparently noncritical operation may therefore help by releasing a resource, even when a fixed dependency-only analysis predicts no change. [20]

Span parentage identifies execution structure; temporal overlap identifies concurrency. Neither alone proves the complete dependency graph. Reuse spans and completion boundaries, and include network, admission, tool, validation, delivery, and review waits wherever the task contract requires them. Historical waits explain a trace but need not remain constant after an optimization. [20]

Time to first token, or TTFT, needs an explicit observation boundary. AIPerf measures request start to the first nonempty content response. That fragment need not be useful or complete. Streaming can improve first useful output without advancing verified completion; the user may still be waiting for validation or a confirmed write. [21]

Prefill processes the supplied prompt; decode generates its continuation iteratively. Their resource demands differ, as LLM Inference explains. Separate these engine intervals from client-observed latency. Faster inference cannot remove an unchanged tool wait or network round trip. [22]

Tail latency and deadline boundaries

Tail latency describes the slow end of a latency distribution. P50 is the median; P95 and P99 locate the 95th and 99th percentiles. P95 is not the maximum. State the population and percentile convention, and retain separate targets for typical and slow experiences. [2][4]

Component percentiles do not generally add. Assume 100 tasks each execute A then B, using nearest-rank percentiles. The paired durations below place the slow A and slow B operations on different tasks. [23]

Each component's P95 is 1 ms, but task P95 is 101 ms. For parallel dependencies, the task instead waits for the required maximum. More dependencies create more opportunities for a slow operation to determine completion. Measure the combined workflow rather than constructing its percentile from component summaries. [23]

Investigate the tail by execution path: admission waits, cold starts, cache misses, long generation, tool delays, and recovery. Keep timeout and unfinished counts beside completed-task latency. Excluding slow unfinished work can improve the reported median while deadline attainment deteriorates. [22][17]

A deadline states when the caller stops waiting; a timeout is an allowed duration. Propagate the remaining allowance rather than granting each downstream call a fresh full timeout. Reserve time for validation and delivery. gRPC deadlines distinguish cancellation of the call from the application's responsibility to stop spawned work. Neither proves immediate billing termination. [24]

Cache validity and reuse boundaries

A cache hit means finding an entry eligible for reuse under the current request's conditions. A cache key identifies candidate reusable work; a matching key is insufficient if authorization, freshness, or other validity requirements no longer hold. The cached object determines which work can be skipped. [25]

For a cached record answer, identity can include request parameters, verified tenant and permission scope, personalization, source revision, and relevant model and instruction versions. Tenant identity must come from trusted authentication state, not model-proposed fields. Recheck access at reuse; a previously valid result can become forbidden after permissions change. Authorization boundaries owns those controls. [29][30]

Semantic caching retrieves answers through approximate meaning similarity. A similar sentence can still differ in negation, entity, date, or conversation context. MeanCache, for example, checks conversation context as well as query similarity. Evaluate false reuse on decisive differences; similarity does not establish freshness, permission, or suitability for a particular user. [31]

Prefix caching reuses numerical representations derived from a compatible shared token prefix, not a stored answer. Semantic resemblance is insufficient, and a valid prefix hit does not refresh external facts. Compatible prefix reuse explains the mechanics. Its prefill savings may matter little when most elapsed time is spent generating a long continuation. [28]

A cached confirmation cannot perform a newly requested update. Read reuse and action execution have different contracts. If a previous update's effect is uncertain, reconcile that operation's identity and state before repeating it; this is safe retry handling, not ordinary answer caching. [25]

Cache lifecycle and net savings

Request hit rate is hits divided by eligible lookups. It does not weight avoided expense. Assume two caches each hit on 50 of 100 requests: avoiding $0.01 per hit saves $0.50 before overhead; avoiding $1 per hit saves $50. Equal hit rates can therefore have very different economic value. [33]

Time to live, or TTL, limits an entry's retention or freshness interval. Longer retention can bridge request gaps but occupies capacity longer. Eviction removes entries under capacity policy; invalidation removes reuse eligibility after a relevant change. Deployment warm-up starts with missing entries. None of these states makes cache lifetime equivalent to source age. [27]

Forecast reuse gaps, not just request frequency. Human review can interrupt otherwise rapid request sequences. Also measure refill work and retrieval time: retained context is useful only if it reaches execution quickly enough. A larger cache can hold more reusable state without delivering the expected latency improvement. [7][34][35]

A cache stampede occurs when many callers independently refill the same missing entry. Request coalescing shares one eligible in-flight fill. Matching authority and validity conditions remain necessary. A failed fill must leave waiters with an explicit failure or bounded recovery; it must not create a reusable success entry. [33]

Coalescing trades duplicate computation for waiting. If the result cannot satisfy every waiting request, additional forwarding may be needed and some callers wait longer. Capacity tests should therefore include invalidation bursts and incompatible requests, not only steady traffic against a warm cache. [25]

Routing and cumulative cascade costs

Routing chooses an execution path using information available at that decision. Explicit task rules can select deterministic code, a specialist model, or a general model. A fixed chain prescribes operations; a router chooses a branch. Neither requires an open-ended agent loop. [36]

Pre-generation routing sees the query but not its future answer. RouteLLM illustrates this boundary by predicting model preference before generation. A cascade instead generates an answer, checks it, and escalates selected cases. Its additional decision evidence has already incurred expense and elapsed time. [37][38]

Models have uneven subtask strengths. Evaluate each route on the cases it actually receives, including accepted quality, rejection, escalation, elapsed time, and total task expense. Fallback cases may be disproportionately difficult. A smaller model's lower token price can be overwhelmed by additional tool calls and repeated unsuccessful attempts. [39][15]

Selective prediction accepts some outputs and abstains on others. Coverage is the accepted fraction; selective risk is error among accepted cases. Report both, together with the expense and outcome of rejected cases. A low-risk accepted subset can still leave an unaffordable review queue. [40]

Calibration asks whether assigned correctness probabilities match observed frequencies. A preference score, a useful ranking, and a verbal confidence statement are different quantities. Validate the event and routed population before interpreting a threshold as assurance; probability calibration and selective automation explains the distinction. [41]

Recovery within shared resource limits

Recovery should consume one task-wide allowance for attempts, spending, and elapsed time. A retry budget limits repeated attempts; it does not bound money when operations have different prices. A fallback changes execution path. Independently resetting allowances at every layer can turn a small failure into substantial extra work. [42][43]

Assume three layers each permit four total attempts: one incoming operation can cause 4 × 4 × 4 = 64 deepest-layer attempts. Backoff progressively delays retries; jitter randomizes the delay to reduce synchronized arrivals. Combine per-task limits with a service-wide retry budget, and do not retry permanent failures. Shared dependencies can make apparently different fallback providers fail together. [44]

Hedging launches a delayed duplicate and accepts the first suitable response. It may reduce read latency while adding work; cancellation can race with execution. Do not transfer this pattern blindly to generated answers or side-effecting tools, where the first response may be unsuitable or duplicate execution harmful. [23]

Before concurrent work starts, reserve allowance for outstanding operations rather than comparing each against the same unspent balance. Include possible late charges. Without bounded operation costs and enforceable termination, an estimated remaining balance is not a precise monetary ceiling. Harness Engineering owns runtime enforcement; safe retries owns uncertain effects. [43]

Stopping can produce an explicit failure, unresolved outcome, or permitted deferral. A degraded mode must identify reduced completeness and preserve authorization; it cannot silently lower the normal acceptance criterion and count the result as ordinary success. [45]

Sustainable capacity and useful throughput

Capacity is the work rate sustainable under a stated workload and service constraints. A bottleneck is the resource limiting that rate; headroom is capacity retained for variation or failures. High arrival counts, long tasks, and bursts compete for finite resources, so a configuration that succeeds at light load can miss its latency targets under sustained demand. [46]

Utilization needs a named resource, denominator, and interval. NVML GPU utilization measures the fraction of a sampling interval during which at least one kernel executes. It is not a fraction of peak arithmetic performance or successful tasks. Its memory-utilization field measures read/write activity time, not allocated memory capacity. [47]

Busy hardware can perform ineffective or duplicated work. Krea's training discussion uses tensor-core activity as an additional compute proxy, while treating workload changes as relevant to interpretation. Such diagnostics help locate constraints; they still require end-to-end throughput and quality measurements. [48]

Serving goodput counts completions meeting declared latency criteria per benchmark second. It does not establish answer correctness. Track raw completions, latency-qualified completions, and accepted timely tasks separately, alongside offered, rejected, and unfinished work. Token throughput cannot substitute for task throughput when tasks generate different amounts of internal work. [21][6]

Suppose each admitted task generates two model calls and three tool reads, while one quarter require review. At an arrival rate of $\lambda$, those stages receive $2\lambda$, $3\lambda$, and $0.25\lambda$ operations per unit time before retries. Increasing model capacity cannot eliminate an overloaded review stage. Memory and batching mechanics remain separate from this workflow accounting. [9]

Provisioning, warm capacity, and admission policy

Usage-priced service charges track consumption; provisioned capacity incurs expense while available, including idle periods. Compare them at equivalent quality, demand, latency, and resilience. Allocate hardware cost over its useful life with an explicit residual value and workload share; do not count both purchase price and depreciation in the same period. Include operations, energy, and remaining service fees. [16]

Autoscaling adjusts capacity as demand changes, but requested workers are not immediately ready workers. Model loading and initialization can delay service. Warm workers retain prepared capacity at an idle-cost premium. Maximum-worker settings limit expansion; they do not independently establish billing limits or acceptable behavior for queued work. [53][54]

Admission control limits accepted work to what can responsibly be served. Backpressure communicates limited capacity upstream so callers slow or defer submission. Bound waiting and expose rejection, deferral, expiry, and completion separately. A growing queue can postpone overload symptoms without creating capacity. [45]

Separating interactive and deferred work changes which waits are acceptable. Batching executes multiple requests together and may improve resource use, but must respect deadlines and memory limits. Batching independent requests explains execution; the policy decision is which workload may wait and which capacity it may consume. [57]

On-device execution removes remote inference calls for that work but transfers computation and energy to user hardware. Device capability and sustained performance still constrain usefulness. Local and On-Device AI covers deployment details; eliminating a provider charge is not eliminating the underlying resource cost. [58]

Interventions with end-to-end impact

Amdahl's law expresses the limit imposed by unchanged work. In the fixed trace below, halving proposal generation saves 200 ms, not half the task duration. Even eliminating it leaves 800 ms. The inference holds only while the workload, remaining durations, and resource interactions stay unchanged. [51]

Quantization uses lower-precision numerical representations and can change serving cost and latency. Its quality effects require evaluation. Quantization owns the method; context selection owns choosing necessary input. Combined changes need joint testing: retrieval can add useful evidence and delay while a serving optimization changes a different resource constraint. [64]

Implementation expense also needs recovery. Assume a change costs $600 to build and saves $0.02 per accepted task after recurring maintenance and service costs. It requires 30,000 accepted tasks to recover that expense. Lower volume, changed acceptance, or added maintenance moves the break-even point. [16]

Comparable optimization experiments

Specify the hypothesis, baseline, candidate, acceptance gates, workload, and decision thresholds before comparison. Use matched tasks, not necessarily identical trajectories, and apply uncertainty analysis to repeated measurements. Bind each result to versions, resource guarantees and hard limits, dependency state, price assumptions, and measurement boundaries. [56][65]

A closed-loop test issues replacement work after replies; an open-loop test schedules arrivals independently. Slower replies therefore reduce the closed-loop client's offered rate. Use the serving experiment contract appropriate to the demand claim. Report actual arrivals and unfinished work so self-throttling does not look like stable capacity. [5]

Controlled fixtures and live workflows answer different questions. MLPerf's end-to-end retrieval benchmark fixes intermediate inputs and hop counts for repeatable performance runs, evaluates accuracy separately, and initially measures offline throughput. That controls variation but does not measure unrestricted live trajectories or interactive tail latency. Pair repeatable comparisons with tests of actual dependency behavior. [6]

A test harness can itself limit confidence. Web-data infrastructure work reported difficulty generating realistic traffic beyond its tested range; an apparent ceiling did not identify whether the generator, service, or both were limiting. Record unavailable coverage rather than extrapolating capacity from the last successful load. [67]

Preserve targeted failures alongside aggregate scores. In a prompt-optimization demonstration, the selected relation-extraction error persisted after the optimized instructions were actually connected to the application. Better aggregate validation did not establish repair of that case. Verify both the deployed configuration and the behavior the change is meant to improve. [68]

Feasible operating points and reassessment

Feasible configurations satisfy every hard requirement. Among them, one dominates another when no objective is worse and at least one improves. The Pareto frontier contains the nondominated evaluated choices. [1]

Choose an operating point for the application's required quality and latency, then compare expense and operating burden. Do not hide hard requirements inside an unexplained weighted score. Small measured differences may remain unresolved under uncertainty; a benchmark frontier describes evaluated configurations, not every possible design. [69]

A canary is a partial, time-limited deployment evaluated against a control. Bound exposure, distinguish candidate and control records, and connect results to expansion, pause, or rollback. Check task quality, deadline attainment, coverage, and total cost—not transport errors alone. A small canary can miss rare failures. Release decisions supplies the evidence gate. [70]

Use task indicators to distinguish real deterioration from changing observation coverage. A recent release has had less time to accumulate downstream corrections or approvals. Compare equal follow-up horizons and keep unresolved work visible before concluding that its economics improved. [17]

Open questions

1. 01

   Routing thresholds under changing task populations remain difficult to maintain. A score that ranked familiar cases well can become overconfident after a domain shift. Progress requires held-out shifted-task evidence showing acceptable route-specific risk and coverage without transferring an unsustainable burden to escalation. [73]

2. 02

   Semantic-cache validity remains harder than semantic matching. Similarity can miss a decisive entity, date, or contextual change, while stricter checks consume some of the avoided work. Progress would demonstrate lower total expense with bounded false reuse on deliberately contrasting, authorized requests. [31]

3. 03

   Joint cache, routing, and capacity control can change the workload it is trying to optimize. Routing changes request lengths; cache misses change prompt-processing demand; queues change deadlines. Progress requires controlled interaction tests that remain feasible after a traffic shift, rather than separate wins under fixed conditions. [46][71]

4. 04

   Exact task-level spending limits remain difficult when remote work and charges outlive local waiting. Delayed reports and incomplete provider coverage obstruct immediate settlement. Progress would provide enforceable operation bounds and reconciled accounting for outstanding work, clearly separating estimated limits from guaranteed ceilings. [11]