Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

A follow-up conversation turn can become cheaper simply by returning to the engine that served it before. That engine may already hold useful context in its KV cache, avoiding recomputation and reducing latency. A destination choice therefore affects more than the distribution of incoming work: it determines whether previously computed context remains useful. OpenAI inference engineers Lu Zhang and Qianru Lao explain how their routing system evolved to handle performance, reliability, geography, and cache locality together. 0:12

The inference load balancer, or ILB, runs in front-end CPU clusters. These clusters receive user requests and prepare inference requests for engines, which usually run in GPU clusters containing multiple engines. ILB selects an engine and processes the request; this discussion concentrates on selection. A request typically targets a model backed by several engines, potentially spread across regions or continents. That distribution provides resilience against localized degradation and cluster failures.

Several signals describe different parts of the serving experience:

• Time to first token (TTFT): The delay before generation produces its first token.
• Time between output tokens (TBOT): The spacing between tokens once output is underway.
• Health and utilization: Whether an engine is functioning and how busy it is.
• KV cache locality: Whether the selected engine can reuse context from an earlier turn.

These considerations can pull routing in different directions. Returning to a cached conversation avoids work, while the engine's current health and load still affect how quickly it can serve that conversation. 2:41

The early routing design first filtered the engine set. Capabilities, compute restrictions, and data residency could prevent a request from using particular engines. Among the eligible destinations, ILB used weighted consistent hashing to select an engine for a request or user. The important policy question was upstream of selection: where did those weights come from? Zhang notes that early days in this industry sounds more historic than it really is. 3:41

A periodic controller turned engine signals into weights. It smoothed the reported signals, computed a performance score, and compared each engine's score with the fleet average. Better relative performance increased an engine's weight; worse relative performance decreased it. Those weights changed incoming traffic, which changed the signals observed in the next cycle. The design borrowed the proportional part of a PID controller. Zhang's aside: this PID will not help you kill a Linux process.

The loop had useful properties. Many signals could influence a single routing decision, and uneven eligibility could partly balance itself. An engine capable of serving a wider range of request types might become busier. Its signals would then discourage additional traffic that could go elsewhere, moving less constrained requests toward other eligible engines. This reduced the need for manual intervention.

The cost was coupled behavior. Combining many signals into one score made it difficult to explain why an engine received a particular weight. Tuning one aspect could move another. Engines serving the same model could also have different characteristics because they used different GPU SKUs, making even load distribution harder to reason about.

The most concrete failure was oscillation. Consider a busy engine holding useful conversation context. The controller moves traffic away, so the engine cools. Its improved signals then look like permission to send more traffic back. Repeating that sequence bounces requests between engines and disrupts KV cache utilization—the reuse that made returning to an engine valuable. 6:41

Why does the controller reverse its own action? The cycle below makes the relationship visible: moving traffic changes the engine's observed condition, and that changed condition drives the next routing adjustment. An improvement caused by removing work can become the reason to put that work back.

Lao begins the replacement architecture with one question: for a request arriving at a CPU cluster, which engine should serve it? Round robin ignores differences in hardware, capacity, health, and network distance. It can also scatter related requests across engines and lose cache locality. Letting each CPU cluster independently choose its locally preferred engine creates another problem: several clusters can converge on the same destination, overloading it while other engines remain underused. 7:38

The newer design gives a control plane a global view of CPU clusters and GPU engines. It computes routing weights for the fleet. A data plane in each CPU cluster uses those weights to make individual engine choices quickly. Its engine selector reads locally stored candidate engines and their routing weights; both refresh asynchronously in the background. Selecting an engine therefore does not require a synchronous call to the control plane.

Engine signals still matter, but their jobs become more explicit. The data plane uses live information such as ready replica counts and engine health as fast local guardrails. The control plane's data loader combines live signals with network overhead and offline regressions of capacity, TTFT, and TBOT. An optimizer turns these inputs into the next routing-weight snapshot, publishes it, and lets data planes pull the update into their local caches. 9:18

The architecture separates three paths:

• Request path: A request arrives at a CPU cluster, the data plane selects an engine from local routing state, and the request is forwarded to that engine.
• Signal path: Engine measurements feed both global optimization and local guardrails.
• Weight path: The control plane publishes weights, and data planes pull them into local caches.

Only request-time selection is synchronous. Signal collection and weight updates improve future decisions asynchronously. Each request uses the latest snapshot already installed locally, while live engine signals provide a faster check on changing conditions.

Where does global coordination enter without delaying each engine choice? The diagram separates the request flow from the update flows. The optimizer affects selection through cached weights, rather than sitting between every incoming request and its destination. Engine signals also reach the data plane directly, allowing global planning and fast local protection to operate on different timescales.

Before opening the optimizer, Lao tests a tempting shortcut: send each request to its nearest engine. This works when demand and nearby capacity line up. In region one, CPU cluster A sends 90 requests per second (RPS), and nearby engine A can serve 100 RPS. Keeping that traffic local fits within the engine's capacity. 12:10

Region two changes the outcome. CPU cluster B sends 120 RPS toward nearby engine B, which can serve only 100 RPS. Nearest-only routing sends more work to B than it can serve, creating engine-side waiting. A farther engine C has unused capacity. Sending some traffic there adds network latency but can avoid a larger delay at B.

Follow the change for the diverted traffic: its destination changes from B to C, its network journey becomes longer, and it avoids waiting at the overloaded local engine. The farther route can finish sooner when the added travel time is smaller than the engine-side delay it avoids. How much traffic can move depends on C's spare capacity. The example teaches the comparison without establishing a measured latency saving. 13:10

The optimizer receives four kinds of input:

• Demand: Requests arriving from each CPU cluster.
• Network latency: The cost of reaching each engine from each cluster.
• Available capacity and health: How much traffic engines can effectively serve.
• Latency profiles: TTFT and TBOT profiles describing how engine-side latency changes as load increases.

Together, these inputs let the policy compare the extra travel to engine C with the waiting created by keeping too much traffic at engine B. 13:35

The output is a set of routing weights: for each CPU cluster, what fraction of its traffic should go to each GPU engine? The objective is to minimize expected end-to-end latency across all routed traffic, counting both network latency and engine-side latency. A nearby engine is attractive while it has room; a farther engine can become preferable as nearby engines approach full utilization.

Three hard constraints keep the allocation meaningful:

• Route all demand: Every cluster's traffic must receive an allocation.
• Respect effective capacity: The traffic allocated to an engine must stay within what it can serve.
• Keep weights non-negative: Traffic fractions cannot be negative.

In the region-two example, the capacity constraint prevents the optimizer from solving B's overload by creating another overload at C. Latency determines which feasible allocation is preferable.

This is the move from engine signals to policy. Signals continue to inform routing, but the control plane uses them within an explicit optimization problem and publishes weights for local selection. The solver and the way cache affinity enters the newer optimization are not specified here. Cache locality remains a routing concern, without a demonstrated cache term in the objective described in this talk.

A globally planned allocation still runs on a production fleet where clusters fail, GPUs and nodes degrade, and networking develops mysterious problems. Zhang closes with protections for three different failure mechanisms. Routing can redistribute work, but faulty hardware and insufficient total capacity require additional controls. 15:33

• Outlier penalties: Detect an anomalous engine and reduce its routing weight. Less incoming traffic gives a transient problem room to recover or allows an operator to rotate the engine out and replace faulty hardware.
• Dynamic retry budgets: Cap retries so failure recovery does not add unlimited work. Under heavy utilization, failures trigger retries, retries raise load, and the added load produces more failures. A budget interrupts that retry storm. It must tighten as utilization rises because the fleet can tolerate more retries during normal operation than near overload.
• Load shedding: When production capacity cannot meet demand, proactively shed a portion of traffic as a last resort. This allows graceful degradation rather than letting excess work cause the whole system to fail.

Load shedding gives a practical limit to the optimizer's requirement to route all demand. An allocation that routes everything while keeping every engine within capacity requires enough available capacity. When that condition fails, the system must reduce the work it attempts to serve. Penalties reduce traffic to unhealthy engines, retry budgets constrain extra attempts, and load shedding removes a portion of incoming traffic. These controls address separate sources of pressure so the fleet can continue providing useful service under stress. 17:03