Model Routing and LLM Gateways

Gateways and routing decisions

A large language model generates responses from supplied input. A provider operates an endpoint that serves a model. An LLM gateway mediates the application’s requests to those endpoints; a router chooses a destination. One service can implement both responsibilities, but forwarding a request does not imply that it predicts which model will answer well. [1][2]

The distinction becomes practical when something changes. Initial selection chooses before generation. Quality escalation invokes another model after assessing an inadequate answer. Operational fallback changes destination because an attempt failed or could not proceed. A request policy specifies which of these actions are permitted and under what conditions. Each additional attempt must preserve the caller’s requirements. [3][4][5]

Central mediation can keep provider secrets outside the application’s agent environment. In What if the network was the sandbox?, Aperture supplies provider credentials while identifying the calling workload through its network identity. This simplifies shared control, but introduces another request-path dependency and concentrates sensitive credentials in the proxy. Its availability and custody controls therefore matter alongside those of the providers. [6][7]

The gateway owns the request-management boundary. Inference Engineering explains execution inside a serving system; AI Platform Engineering covers the broader shared services and their ownership. Here, the central obligation is narrower: determine where this request may go and preserve its contract throughout the resulting work.

Origins of mediation and selection

Request mediation predates model APIs. The January 1997 HTTP/1.1 specification distinguished a proxy acting on behalf of clients from a gateway receiving requests as though it were the origin server and mediating access to another server. Forwarding and representation translation were already established intermediary responsibilities. This is historical terminology, not an invention date or the current HTTP specification. [8]

Managed API services brought common operational controls into that boundary. AWS’s July 9, 2015 announcement of Amazon API Gateway combined traffic management, authorization, monitoring, and API version management in front of backend applications. Those responsibilities remain useful when the backend generates language, even though successful delivery no longer establishes a useful answer. [9]

A parallel problem concerned choosing the right algorithm for an individual input. SATzilla’s June 2008 paper described selecting solvers for Boolean satisfiability: deciding whether Boolean constraints can all hold. It learned performance predictions from representative problems and candidate solvers. Computing input features itself cost time, so the system included short preliminary runs and a backup when feature computation timed out. Selection had both overhead and a failure path. [10]

LLM selection added a consequential choice about when to gather evidence. FrugalGPT, submitted in May 2023, assessed generated answers before continuing through a model cascade. RouteLLM’s 2024 work selected from the query before generating an answer. The former purchases answer-specific evidence; the latter tries to avoid unnecessary generation. These approaches address complementary problems rather than marking a universal replacement of one method by another. [3][2]

The gateway interface

A gateway interface should express what the caller needs independently of the provider’s field names. An adapter is translation code between those representations. A model alias names a logical service whose deployment can change; it does not identify the endpoint that actually served a request. Keep required behavior separate from preferences that the router may trade away. [11][12]

A familiar request shape can hide lost guarantees. Claude’s documented OpenAI SDK compatibility layer ignores response_format and strict function-calling settings: requests to constrain the answer’s format or tool arguments do not enforce those constraints through this layer. It also combines system/developer messages at the start. An adapter must reject unsupported requirements or offer an explicitly agreed alternative. [12]

Keep refusal, incomplete generation, and invalid output distinct. Claude, for example, distinguishes natural completion from reaching an output cap or exhausting context capacity, including within successful API responses. A complete response can still fail application validation. Recognize a complete response develops these acceptance boundaries. [18]

Compatible destinations

Provider capability means support for an operation and its relevant semantics. Eligibility requires all mandatory capabilities in the same endpoint configuration. Record the model, provider, endpoint, version, region, and configuration being assessed, together with the evidence date. A model’s general ability cannot compensate for a serving interface that drops required input or ignores an output constraint. [12][13]

Capacity is another hard constraint. Tokens are the units a model processes and generates, not necessarily words. The context window bounds a complete invocation: input and generated output, including applicable reasoning tokens, share capacity. Reserve output space and respect separate output limits before accepting input. Counting complete requests explains the accounting. Quietly truncating input to fit a replacement can change the task. [19]

A schema describes permitted machine-readable values; a tool request is generated data proposing an operation. Neither term guarantees that every endpoint supports the same schema subset or tool behavior; see Data, requests and effects. OpenRouter’s require_parameters setting filters for supplied-parameter support, whereas default routing can permit unsupported parameters to be ignored. Advertised support should lead to contract tests, especially after endpoint or adapter changes. [13]

Authority and information recipients

Authentication establishes identity; authorization determines permitted access. A tenant is an independently authorized customer or organizational partition. Derive that scope from trusted application state. A caller’s model preference, generated routing instruction, or possession of a gateway credential must not grant arbitrary upstream access. Attach provider credentials only after validating the destination and the caller’s permission. [14][5]

Aperture obtains identity through a tailnet, Tailscale’s identity-bearing private network, rather than trusting a request field. It resolves the requested model to a configured provider and injects authentication headers. A shared external-access bridge represents its callers under the bridge’s identity, so individual attribution is lost unless another trusted mechanism preserves it. [1]

Effective permission depends on how rules combine. Aperture’s grants are additive: their union supplies access. A narrow grant cannot remove permission supplied by a broad one, and its shipped configuration grants all users access to all models. To revoke access, narrow or remove every grant that supplies that access. Separately, constrain credential scope and lifetime and provide rotation and revocation procedures; Credentials and communication explains the custody boundary. [20][21]

Permission also concerns information recipients. A selector reading the prompt and a judge reading the answer receive data, just as the generator does. Check approved recipients, processing location, retention, and training use before each disclosure, including fallback. If no destination qualifies, deny the request rather than relax its conditions. [22][23]

Zero data retention, or ZDR, is a commitment not to retain request and response data after processing, within the service’s stated scope. OpenRouter’s ZDR control tracks policies by endpoint, treats unknown policies conservatively, and excludes enabled plugins and tools. Its interpretation permits implicit in-memory prompt caching. Check these boundaries separately from processing location and training use: approval of an inference endpoint does not cover every recipient. Changed destinations or terms require approval under the changed conditions. [24]

Admission and shared allowances

Permission to perform work does not establish capacity to start it. Admission decides whether a request starts, waits within a bound, or is rejected. A rate limit bounds consumption over time; a quota allocates an allowance within a stated scope and period; a concurrency limit bounds simultaneous work. Always specify units and reset behavior because provider terminology varies. [25][26]

One request can encounter application, tenant, credential, model, and provider limits. Claude distinguishes requests, input tokens, and output tokens per minute from monthly spending allowances. Its token bucket replenishes capacity continuously, allowing bursts only within available capacity. Organization limits still apply alongside workspace restrictions. Passing one check does not establish that every other allowance remains available. [25][27]

For a shared hard allowance, gateway instances need coordinated admission. Independently allowing each instance the full quota multiplies the effective allowance. Envoy distinguishes local limiting from coordinated global limiting and can use local token buckets to absorb bursts before global checks. Request-rate coordination alone does not reserve model tokens or money. [28]

Completed spend also omits work already underway. Consider an allowance of ten units and two simultaneous requests estimated at six units each. Both can pass an unchanged balance check; reserving six atomically for the first leaves only four for the second. LiteLLM documents estimated maximum-cost reservation before execution and replacement with priced consumption afterward. This requires a usable estimate; its documentation identifies routes that cannot be fully priced in advance. [29]

Aperture documents a different contract: applicable quota buckets must have positive balances, then estimated cost is deducted after completion. That is not a strict reservation guarantee for concurrent work. Distinguish the configured behavior from the desired spending bound. [30]

Bound waiting by queue capacity and the remaining deadline, and prevent one tenant from exhausting shared service. If authoritative allowance state is unavailable, define whether affected work must stop; silently assuming unlimited capacity defeats the control. These gateway decisions are separate from the engine scheduling covered in Admission and fair service. [26][29]

Choosing an eligible endpoint

Model selection chooses behavior; provider selection chooses where it is served. They can be implemented jointly, but hard constraints still come first. Selection then orders the eligible candidates using the evidence the policy actually has. [11][13]

LiteLLM groups configured deployments behind aliases. Its latency routing uses an observation window and can distribute traffic within a buffer around the fastest deployment. These are operational placement signals. Keep their age visible, and do not equate two endpoint names with independent capacity: Claude, for example, shares a rate-limit pool across inference_geo values. [11][25]

Measure the interaction that matters. In Voice In, Visuals Out, a smaller, cheaper model still missed the application’s responsiveness needs; the serving platform mattered too. The comparison was workload-specific, not a lasting provider ranking. AI Cost and Performance Engineering explains how to set and measure those targets. [32]

Suitability by task

Task segmentation groups work whose requirements or observed outcomes differ meaningfully. First choose the routing unit: one request, an entire conversation, or an application-declared step. A conversation-level assignment favors continuity; a step-level policy can distinguish extraction from explanation. Such distinctions are design choices to evaluate, not evidence that one model necessarily excels at either task. [31][33]

A workload slice is a subset defined by relevant conditions, such as language, input length, schema requirements, domain, or error consequences. These conditions guide testing; none is a universal difficulty score. Prefer task labels supplied by trusted application logic when available. If a classifier infers the label, measure its mistakes separately from the models it selects. Coverage of the intended workload explains how to choose representative cases. [34][35]

Keep information available before dispatch separate from later outcomes. Input length is available before generation; an answer’s factual error is not. Suitability evidence comes from evaluating completed work under specified conditions. RouterBench compares routing policies on shared tasks. Its learned routers beat a simple mixture based on aggregate model performance on some task collections, but lost to it on others. A router therefore needs evidence for the task slices it will actually handle, not just a favorable overall result. [36]

Lessons from building GenAI based applications emphasizes evaluating replacements inside the application. Record both supported task slices and unmeasured conditions. An empty cell in that evidence is a reason to limit the routing claim, not an invitation to fill it with a leaderboard rank. [31][37]

Predicting model suitability

An explicit task rule uses a known label to select a model. A learned selector instead predicts a selection-relevant outcome from request features—information describing the input—using examples of model outcomes. It earns its complexity only if it assigns requests better than a fixed choice or a simple rule after its own overhead is included. [38]

RouteLLM estimates from the query how likely the stronger model’s answer is to be preferred to the weaker model’s. It selects the stronger model when that estimate exceeds a threshold, before seeing either answer. Lowering the threshold sends more requests to the stronger model. The useful tradeoff depends on which requests change destination and how their answers perform, not just the traffic shares. [2]

Preference probability and task-correctness probability describe different events. Calibration asks whether a claimed probability matches observed frequencies for its specified event. A useful ranking score need not be calibrated, and calibration on familiar inputs need not transfer to changed conditions. What confidence predicts explains the distinction. [39][2]

Define behavior outside the selector’s validated scope. An application can use a conservative fixed assignment or decline unsupported work, but must test how that scope is recognized. SATzilla’s backup for failed feature computation illustrates the older principle: the selection mechanism itself needs a bounded cost and failure path. [10][34]

Escalation after an answer

An escalation cascade first generates an answer, then assesses whether to accept it or ask another model to try. Unlike selection before generation, this decision can use the answer itself, but it incurs the cost of both generation and checking. FrugalGPT scores each query-answer pair and stops when the score exceeds that stage’s acceptance threshold; otherwise it continues to another model. Its average API-cost constraint is not a per-request spending ceiling. [3]

The checker introduces its own mistakes. False acceptance lets an inadequate answer through. Unnecessary escalation spends more on an answer that was already acceptable. A structural validator may miss factual errors; a model judge adds another inference call and can misclassify the result. In selective image processing, unnecessary intervention can even degrade an already good image. Additional computation needs a task-specific reason. [40][35]

Choose the next model for the unmet requirement, not simply for greater size. Decide whether it receives only the original task or also the first answer and feedback; these are different inputs and need separate evaluation. Every stage must satisfy capability, recipient, authority, remaining-time, and spending conditions. Bound the number of attempts. If continuation is impossible or no answer passes assessment, the cascade must be able to abstain: stop without presenting an answer as accepted. [41][5][27]

Later stages receive a selected population: cases rejected earlier. Measure the checker and subsequent model on that population, not only on all requests. Coverage is the fraction accepted; selective risk is error among accepted results. Report both, with the treatment of deferred work, as described in Deferral, risk and review capacity. Broader verification and search belong in Reasoning and Test-Time Compute. [42]

Recovery by failure cause

A retry is another attempt. Operational fallback switches to an approved alternative because an attempt failed or could not proceed. These decisions need the cause, not merely a generic failure flag. LiteLLM, for example, documents fallback to another model group after configured retries; that mechanism alone does not establish that the replacement preserves the application contract. [4]

Even one status code can hide different causes. Claude documents 429 for temporary rate limiting and for exhausted monthly allowance; the latter lacks a retry-after hint. Refusal and inadequate content require separate interpretation. Switching providers must not become a way to evade the request’s authorization or handling policy. [43][23]

Runtime routing controls can change future destination choices without an emergency deployment, as Agents Need Feature Flags describes. A changed flag is not evidence that an executing request stopped. When no permitted recovery remains, return the specific exhaustion or failure state. [33][45]

Bounded retries and recovery probes

Retries consume capacity precisely when a dependency may be struggling. Independent retry layers multiply attempts: Amazon’s five-layer example with three attempts per layer produces 243 downstream attempts. Coordinate ownership across application, gateway, and client library. Use bounded attempts, capped exponential backoff—progressively longer waits—and jitter, which varies those waits to avoid synchronized retries. [46]

A timeout is an allowed duration; a deadline is when waiting ends. Earlier attempts and backoff consume part of that allowance, so pass only the remaining time downstream. gRPC, a remote procedure call framework, documents this propagation principle. An HTTP gateway must implement it through its own adapters. When the deadline expires, notifying downstream code of cancellation is only part of stopping: that code must also stop any work it started. [15]

Circuit states

A breaker limits exposure; it neither repairs the dependency nor authorizes a fallback. Probe limits protect a recovering service. Recovery modes also need tests: Amazon cautions that breakers can complicate operation and delay recovery. Keep overall attempt and deadline limits even when a breaker changes state. Cancellation and state release explains why requesting a stop is separate from reclaiming execution resources. [47][46]

Partial output and commitment

Streaming delivers output incrementally, often through server-sent events: messages carried over a continuing HTTP response. Upstream generation, gateway receipt, caller-visible content, and confirmed completion are separate boundaries. From tokens to a stream explains the delivery mechanics. Buffering content before release preserves an opportunity to assess or replace it, but delays the caller’s first useful output. [16][48]

Response commitment is the protocol or application boundary after which recovery cannot remain invisible. gRPC stops transparent retries after response headers arrive. A gateway application may additionally care about the first visible content. These are different boundaries. Once a prefix is visible, appending another model’s replacement as though it were the same answer violates the delivery contract. [48]

OpenRouter documents ordinary HTTP errors before streaming and terminating error events after streaming has begun, when HTTP status remains 200. Its usage frame is accounting, not another completion. Preserve incomplete delivery and offer an explicit restart only if the application supports one. Cancellation stops processing and billing only for supported provider/streaming combinations. [16]

A disconnect does not establish that upstream work stopped. OpenAI also documents that interrupted streamed Chat Completions may omit the final usage chunk. Missing consumption evidence remains unknown. If generated output proposes a tool operation, remember that a proposal and an executed effect are different records; Data, requests and effects explains that boundary. Cancellation cannot undo completed external effects. [49][44]

State that binds a route

A later turn may depend on more than messages. An opaque identifier names state held elsewhere; it is not the contents of that state or proof that the caller may access it. Distinguish application-held content from provider-bound references before promising that a replacement endpoint can continue. [50]

Claude’s Files API illustrates the access boundary: files are workspace-scoped, so applications must maintain their own user-to-file mapping rather than accept arbitrary file IDs. The API is ineligible for ZDR. A migration therefore needs both content access and handling approval, not just a destination that accepts the same field name. [50]

Session affinity keeps related requests on a compatible destination. Workers AI’s documented affinity header seeks an instance with reusable prefix state while the application still supplies conversation input. Affinity can improve reuse, but binding required state to one destination limits recovery choices. When reconstruction is unavailable, refuse the switch explicitly. Reconstruct the next input covers input reconstruction; Agent Runtimes and Harness Engineering covers persistent execution. [51]

One request, several attempts

A logical request is the caller’s unit of requested work. It can own several separately identified attempts: selector inference, generation, checking, escalation, and recovery. Count the logical request once, but attribute all its work, including failed or cancelled attempts. Otherwise a route that recovers frequently can look inexpensive because only its final answer remains visible. [27][52]

Estimates, provider usage, and billing are different evidence streams. Anthropic’s usage and cost APIs have different category coverage, require pagination, and can report after request completion. They do not supply the application’s task-to-attempt mapping. Preserve unknown usage after interrupted delivery rather than interpreting an absent report as zero. [54][49]

Categories can overlap: OpenTelemetry’s conventions include cached input within input totals and reasoning output within output totals. Adding those subcategories again double-counts consumption. Settlement should also apply each completion event once and survive a crash between consumption and reconciliation. Those are implementation requirements to verify, not consequences of merely having a budget setting. See Usage and cost attribution and AI Cost and Performance Engineering. [17][29]

Explaining the actual route

A trace connects execution records. For routing, preserve the relationships between policy evaluation, destination selection, attempted execution, delivery, and later assessment. Identity across boundaries explains the instrumentation mechanics. The gateway’s responsibility is to make its decisions reconstructable, including rejected requests that never reached a provider. [55]

Open Policy Agent, or OPA, evaluates rules against supplied input and returns a policy decision. Its decision logs can connect that decision to a trace, the input, the result, and the policy revision. The gateway must still enforce the result. Masking can remove sensitive fields, while filters and rate limits can drop entire records. Check both enforcement and record coverage before treating the logs as an audit history. [55]

Identity fields also have limits. OpenTelemetry warns that compatible APIs and proxies can make the instrumented provider label differ from the actual provider. Requested model, response model, and server address add evidence, not independent attestation of served weights. Likewise, gateway-visible tool requests in Aperture’s demonstration do not prove execution or success. [17][59]

Start with decision metadata. Capture sensitive payloads only for a defined diagnostic purpose, with restricted access and retention; never include credentials. Auditability requires sufficient evidence, not unrestricted copies of every prompt and answer. [22]

Testing the enforced contract

An invariant is a property that must survive every permitted transition. Derive tests from those properties rather than waiting for production to encounter each failure. Deterministic policy tests examine decisions; adapter fixtures supply controlled dependency responses; concurrency tests exercise shared state; live checks establish behavior of actual configured endpoints. None substitutes for the others. [60][61]

Test controls have their own interface contracts. LiteLLM supports mock fallback triggers in direct Router tests, but strips corresponding flags from incoming Proxy requests starting with version 1.85.0. Triggering fallback establishes that a branch ran; assertions must still establish what it preserved. Test each guarantee separately develops this separation. [4]

Evaluating routing usefulness

Passing enforcement tests makes a policy eligible for use; it does not show that routing helps. Compare complete policies on shared representative cases: eligible fixed models, simple task rules, predictive selection, and bounded cascades. Specify required outcomes and constraints first. Count selector and checker overhead, retries, and failed attempts alongside the final answer. Baselines, budgets and repeated attempts explains the comparison design. [38][63]

Coverage is accepted cases divided by all cases. With zero-one error, selective risk is wrong accepted answers divided by accepted answers; it is undefined when none are accepted. Lower accepted-case error can accompany more deferral. Report both and inspect the deferred workflow through Deferral, risk and review capacity. [42]

RouterBench’s Zero router mixes models using aggregate cost-quality information without predicting from each query. It is a useful simple baseline. Its cascade experiments used known answer scores with simulated scoring errors, so those results do not establish a deployed checker’s reliability. Evaluate the checker actually used by the application. [36]

Ordinary route logs reveal the selected model’s outcome, not what unselected models would have produced. Authorized paired calls can fill that gap, but send information to additional recipients and consume additional resources. Mirrored execution is real secondary processing, not merely recording a proposed choice. [64][65]

Finally, protect assessment from threshold tuning. Repeatedly selecting the best development score can select favorable noise. Cawley and Talbot demonstrated this mechanism in model selection: the selection criterion improved while independent performance deteriorated. Use development evidence to choose the policy, then independent evidence to assess the chosen policy. [66]

Maintaining a valid policy

Review routing rules, selector versions, model aliases, adapters, capability evidence, and recipient approvals together. Drift is a change in the conditions those choices were tested for: new tasks can undermine model suitability, while endpoint changes can invalidate compatibility. Production outcomes help identify what needs revision. In Uber’s vision workflow, humans label sampled production data under the original guidelines. Mismatches trigger diagnosis and configuration tuning, and a candidate must pass a benchmark before the new version is registered. This connects observed errors to tested changes rather than promoting every proposed correction. [67][68]

Distinguish exposure modes. Offline tests exercise retained cases. Decision-only shadowing records a proposed route without invoking it, so it cannot reveal that model’s answer. Shadow execution sends an additional request and therefore requires recipient approval and resource allowance. A canary serves a bounded portion of live traffic with the candidate. Choose exposure according to consequences: rollback cannot retract an answer already delivered. [65][69][70]

A service can remain available while enforcing an outdated policy. OPA distributes policy and associated data in bundles. Updates are eventually consistent, so instances can temporarily use different revisions. With persistence enabled, an instance can restart from its last activated bundle while the bundle server is unavailable. Define how old a policy may be and when the application must fail closed—reject work because current permission cannot be verified. Restoring service with an older configuration must not restore revoked access. [62]

Agents Need Feature Flags emphasizes runtime controls, change records, and regular shutdown drills. Give temporary rollout flags owners and removal dates; keep emergency controls maintained. Record who changed what and when. Use Make and maintain the decision for ongoing quality evidence, and AI Platform Engineering for responsibility beyond this service. [71][68]

Open questions

1. 01

   Routing evidence remains incomplete when only selected destinations receive traffic. Additional comparisons cost money and disclose data, while statistical correction cannot recover outcomes for choices never observed. Progress would provide useful policy comparisons with explicit coverage, uncertainty, and bounded additional execution. [64][65]

2. 02

   Reliable settlement after interrupted work requires more than a usage counter. Missing reports, duplicate notifications, and crashes can separate consumption from recorded completion. Progress would be a versioned, fault-tested contract that preserves unresolved liabilities and prevents both duplicate charging and premature release. [49][29]

3. 03

   Provider-held state limits interchangeable service. File identifiers, access scope, and retained processing state can bind a request to one endpoint. Progress would require explicit export and reconstruction contracts preserving content, authority, and handling conditions, with refusal when those conditions cannot survive transfer. [50]

4. 04

   A selector or checker can stop being useful when the workload changes. Detecting that change is difficult when trustworthy outcome labels arrive slowly. Progress would connect drift signals to fresh task evidence and conservative operating limits without mistaking a score or a successful response for correctness. [67][39]