# Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Lu Zhang, OpenAI · Qianru Lao, OpenAI

[Talk page](https://ai.engineer/talks/sOB3HSiG8vo-routing-llm-inference-in-production-from-engine) · [Video](https://www.youtube.com/watch?v=sOB3HSiG8vo)

## Routing LLM Inference in Production: From Engine Signals to Policy

Qianru Lao and Lu Zhang explain how OpenAI moved from feedback-driven routing weights to a global latency optimizer, while keeping engine selection local and protecting the fleet against failures, retry storms, and excess demand.

### An engine choice changes latency and future cache reuse

[0:12](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=12s)

A follow-up conversation turn can become cheaper simply by returning to the engine that served it before. That engine may already hold useful context in its KV cache, avoiding recomputation and reducing latency. A destination choice therefore affects more than the distribution of incoming work: it determines whether previously computed context remains useful. OpenAI inference engineers [Lu Zhang](https://www.linkedin.com/in/luzhang1/) and [Qianru Lao](https://linkedin.com/in/qianru-lao) explain how their routing system evolved to handle performance, reliability, geography, and cache locality together. [0:12](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=12s)

The **inference load balancer**, or **ILB**, runs in front-end CPU clusters. These clusters receive user requests and prepare inference requests for engines, which usually run in GPU clusters containing multiple engines. ILB selects an engine and processes the request; this discussion concentrates on selection. A request typically targets a model backed by several engines, potentially spread across regions or continents. That distribution provides resilience against localized degradation and cluster failures.

Several signals describe different parts of the serving experience:

- **Time to first token (TTFT):** The delay before generation produces its first token.
- **Time between output tokens (TBOT):** The spacing between tokens once output is underway.
- **Health and utilization:** Whether an engine is functioning and how busy it is.
- **KV cache locality:** Whether the selected engine can reuse context from an earlier turn.

These considerations can pull routing in different directions. Returning to a cached conversation avoids work, while the engine's current health and load still affect how quickly it can serve that conversation. [2:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=161s)

### Weighted hashing worked—until the weights became hard to explain

[3:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=221s)

The early routing design first filtered the engine set. Capabilities, compute restrictions, and data residency could prevent a request from using particular engines. Among the eligible destinations, ILB used **weighted consistent hashing** to select an engine for a request or user. The important policy question was upstream of selection: where did those weights come from? Zhang notes that early days in this industry sounds more historic than it really is. [3:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=221s)

A periodic controller turned engine signals into weights. It smoothed the reported signals, computed a performance score, and compared each engine's score with the fleet average. Better relative performance increased an engine's weight; worse relative performance decreased it. Those weights changed incoming traffic, which changed the signals observed in the next cycle. The design borrowed the proportional part of a PID controller. Zhang's aside: this PID will not help you kill a Linux process.

The loop had useful properties. Many signals could influence a single routing decision, and uneven eligibility could partly balance itself. An engine capable of serving a wider range of request types might become busier. Its signals would then discourage additional traffic that could go elsewhere, moving less constrained requests toward other eligible engines. This reduced the need for manual intervention.

The cost was coupled behavior. Combining many signals into one score made it difficult to explain why an engine received a particular weight. Tuning one aspect could move another. Engines serving the same model could also have different characteristics because they used different GPU SKUs, making even load distribution harder to reason about.

The most concrete failure was oscillation. Consider a busy engine holding useful conversation context. The controller moves traffic away, so the engine cools. Its improved signals then look like permission to send more traffic back. Repeating that sequence bounces requests between engines and disrupts KV cache utilization—the reuse that made returning to an engine valuable. [6:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=401s)

Why does the controller reverse its own action? The cycle below makes the relationship visible: moving traffic changes the engine's observed condition, and that changed condition drives the next routing adjustment. An improvement caused by removing work can become the reason to put that work back.

Routing changes the signal that changes routing
Moving traffic away cools the engine; improved signals encourage traffic to return. Repeated movement between engines disrupts cache reuse.
- Engine is busy — Reported performance influences its routing weight.
- Traffic moves away — The engine cools after receiving less traffic.
- Cooler signals return — The controller interprets improvement as room for more traffic.
- Traffic moves back — Back-and-forth routing disrupts KV cache utilization.
- Engine is busy → Traffic moves away (Reduce traffic)
- Traffic moves away → Cooler signals return (Lower load improves signals)
- Cooler signals return → Traffic moves back (Send more traffic)
- Traffic moves back → Engine is busy (Load rises again)

### Compute policy globally, select engines locally

[7:38](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=458s)

Lao begins the replacement architecture with one question: for a request arriving at a CPU cluster, which engine should serve it? Round robin ignores differences in hardware, capacity, health, and network distance. It can also scatter related requests across engines and lose cache locality. Letting each CPU cluster independently choose its locally preferred engine creates another problem: several clusters can converge on the same destination, overloading it while other engines remain underused. [7:38](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=458s)

The newer design gives a **control plane** a global view of CPU clusters and GPU engines. It computes routing weights for the fleet. A **data plane** in each CPU cluster uses those weights to make individual engine choices quickly. Its engine selector reads locally stored candidate engines and their routing weights; both refresh asynchronously in the background. Selecting an engine therefore does not require a synchronous call to the control plane.

Engine signals still matter, but their jobs become more explicit. The data plane uses live information such as ready replica counts and engine health as fast local guardrails. The control plane's data loader combines live signals with network overhead and offline regressions of capacity, TTFT, and TBOT. An optimizer turns these inputs into the next routing-weight snapshot, publishes it, and lets data planes pull the update into their local caches. [9:18](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=558s)

The architecture separates three paths:

- **Request path:** A request arrives at a CPU cluster, the data plane selects an engine from local routing state, and the request is forwarded to that engine.
- **Signal path:** Engine measurements feed both global optimization and local guardrails.
- **Weight path:** The control plane publishes weights, and data planes pull them into local caches.

Only request-time selection is synchronous. Signal collection and weight updates improve future decisions asynchronously. Each request uses the latest snapshot already installed locally, while live engine signals provide a faster check on changing conditions.

Where does global coordination enter without delaying each engine choice? The diagram separates the request flow from the update flows. The optimizer affects selection through cached weights, rather than sitting between every incoming request and its destination. Engine signals also reach the data plane directly, allowing global planning and fast local protection to operate on different timescales.

Three paths, one local engine choice
Request-time selection reads local state. Engine signals and routing-weight snapshots travel asynchronously to improve subsequent choices.
- Incoming request — Arrives at a front-end CPU cluster.
- Data-plane engine selector — Selects locally and uses fast engine guardrails.
- Local routing state — Contains candidate engines and installed routing weights.
- GPU engine — Serves the forwarded inference request.
- Engine signals — TTFT, TBOT, ready replica counts, and health.
- Control-plane loader and optimizer — Combines signals, network overhead, and offline regressions to compute global weights.
- Published weight snapshot — Available for each data plane to pull.
- Incoming request → Data-plane engine selector (Request-time selection)
- Local routing state → Data-plane engine selector (Read installed state)
- Data-plane engine selector → GPU engine (Forward request)
- GPU engine → Engine signals (Report measurements)
- Engine signals → Data-plane engine selector (Fast local guardrails)
- Engine signals → Control-plane loader and optimizer (Asynchronous global inputs)
- Control-plane loader and optimizer → Published weight snapshot (Compute and publish)
- Published weight snapshot → Local routing state (Pull asynchronously)

### A farther engine can finish sooner

[11:59](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=719s)

Before opening the optimizer, Lao tests a tempting shortcut: send each request to its nearest engine. This works when demand and nearby capacity line up. In region one, CPU cluster A sends 90 requests per second (RPS), and nearby engine A can serve 100 RPS. Keeping that traffic local fits within the engine's capacity. [12:10](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=730s)

Region two changes the outcome. CPU cluster B sends 120 RPS toward nearby engine B, which can serve only 100 RPS. Nearest-only routing sends more work to B than it can serve, creating engine-side waiting. A farther engine C has unused capacity. Sending some traffic there adds network latency but can avoid a larger delay at B.

Follow the change for the diverted traffic: its destination changes from B to C, its network journey becomes longer, and it avoids waiting at the overloaded local engine. The farther route can finish sooner when the added travel time is smaller than the engine-side delay it avoids. How much traffic can move depends on C's spare capacity. The example teaches the comparison without establishing a measured latency saving. [13:10](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=790s)

### Make the objective and capacity limits explicit

[13:35](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=815s)

The optimizer receives four kinds of input:

- **Demand:** Requests arriving from each CPU cluster.
- **Network latency:** The cost of reaching each engine from each cluster.
- **Available capacity and health:** How much traffic engines can effectively serve.
- **Latency profiles:** TTFT and TBOT profiles describing how engine-side latency changes as load increases.

Together, these inputs let the policy compare the extra travel to engine C with the waiting created by keeping too much traffic at engine B. [13:35](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=815s)

The output is a set of routing weights: for each CPU cluster, what fraction of its traffic should go to each GPU engine? The objective is to minimize expected end-to-end latency across all routed traffic, counting both network latency and engine-side latency. A nearby engine is attractive while it has room; a farther engine can become preferable as nearby engines approach full utilization.

Three hard constraints keep the allocation meaningful:

- **Route all demand:** Every cluster's traffic must receive an allocation.
- **Respect effective capacity:** The traffic allocated to an engine must stay within what it can serve.
- **Keep weights non-negative:** Traffic fractions cannot be negative.

In the region-two example, the capacity constraint prevents the optimizer from solving B's overload by creating another overload at C. Latency determines which feasible allocation is preferable.

This is the move from engine signals to policy. Signals continue to inform routing, but the control plane uses them within an explicit optimization problem and publishes weights for local selection. The solver and the way cache affinity enters the newer optimization are not specified here. Cache locality remains a routing concern, without a demonstrated cache term in the objective described in this talk.

### Protect the fleet when routing cannot fix the problem

[15:33](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=933s)

A globally planned allocation still runs on a production fleet where clusters fail, GPUs and nodes degrade, and networking develops mysterious problems. Zhang closes with protections for three different failure mechanisms. Routing can redistribute work, but faulty hardware and insufficient total capacity require additional controls. [15:33](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=933s)

- **Outlier penalties:** Detect an anomalous engine and reduce its routing weight. Less incoming traffic gives a transient problem room to recover or allows an operator to rotate the engine out and replace faulty hardware.
- **Dynamic retry budgets:** Cap retries so failure recovery does not add unlimited work. Under heavy utilization, failures trigger retries, retries raise load, and the added load produces more failures. A budget interrupts that retry storm. It must tighten as utilization rises because the fleet can tolerate more retries during normal operation than near overload.
- **Load shedding:** When production capacity cannot meet demand, proactively shed a portion of traffic as a last resort. This allows graceful degradation rather than letting excess work cause the whole system to fail.

Load shedding gives a practical limit to the optimizer's requirement to route all demand. An allocation that routes everything while keeping every engine within capacity requires enough available capacity. When that condition fails, the system must reduce the work it attempts to serve. Penalties reduce traffic to unhealthy engines, retry budgets constrain extra attempts, and load shedding removes a portion of incoming traffic. These controls address separate sources of pressure so the fleet can continue providing useful service under stress. [17:03](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=1023s)

## Key takeaways

- Inference routing must account for reusable conversation context as well as performance, health, and geography. Bouncing traffic between engines can undermine KV cache reuse.
- Global coordination can stay outside request-time selection: the control plane computes weights asynchronously, and data planes select engines from locally cached snapshots with live guardrails.
- Compare network latency plus engine-side delay when choosing a destination. A farther engine can serve a request sooner, provided the allocation respects its effective capacity.
- Production needs separate controls for unhealthy engines, retry amplification, and insufficient total capacity: penalties, dynamic retry budgets, and last-resort load shedding.

## Related talks

- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://ai.engineer/talks/YXowceUKYJI-kv-cache-aware-routing-p-d-disaggregation) — Continue with cache-aware routing and the placement of inference work in Kubernetes.
- [How fast are LLM inference engines anyway?](https://ai.engineer/talks/DeFF3J8T5Pk-fast-are-llm-inference-engines-anyway) — Explore the engine performance that supplies routing policy with latency and capacity inputs.

## Slides

### Routing LLM inference in production

Qianru Lao and Lu Zhang explain how OpenAI’s inference routing evolved from engine-signal feedback loops to a more explicit and predictable policy still informed by those signals.

[0:27](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=27s) [Title slide for a talk about routing LLM inference from engine signals to policy, presented by Qianru Lao and Lu Zhang.](https://assets.ai.engineer/sites/aiecode2025/sha256/885c98cd2f3c446ee8e7e94e4b78318dc0da9177c994b2e3c1dbee043d1d8f95.webp)

OpenAI

Routing LLM inference in production
From engine signals to policy

Qianru Lao
Inference

Lu Zhang
Inference

### Agenda

The talk follows the inference load balancer’s evolution through its architecture, a global networking case study, and safeguards for production stress.

[0:42](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=42s) [Agenda covering the inference load balancer, architecture, a networking case study, and protection mechanisms.](https://assets.ai.engineer/sites/aiecode2025/sha256/c01ffb15e64f3496a221890fc0fdd057e69de1eb431575e8d0cf1e6a969db024.webp)

Agenda

01 Inference Load Balancer
Introduction and its evolution

02 Architecture
Control plane and data plane

03 Case study
Reducing global networking overhead

04 Protection mechanisms
Error penalty, retry with cap and load shedding

OpenAI

### What is Inference Load Balancer

Frontend CPU clusters prepare inference requests. The load balancer selects an engine using performance, load, network, error and KV-cache signals, then proxies the request to GPU engine clusters.

[1:13](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=73s) [Diagram showing requests moving from frontend clusters through engine selection and request proxying to engine clusters.](https://assets.ai.engineer/sites/aiecode2025/sha256/a83d0d42a31ba7fc86171445bf6135a5c16405d19fc1021ec0acd197453cab38.webp)

What is Inference Load Balancer

Incoming request

Frontend clusters
CPU clusters
Processes user request
Forms into inference request
Send to backend engines through Inference LB

Inference Load Balancer
Select an Inference engine
A model may have multiple engines behind
Select the best one, what to consider?
TBoT, TTFT, load, network overhead, error rate
But also KV match

Proxy the request

Engine clusters
GPU clusters
Each hosts various inference engines

OpenAI

### Load balancing in the early days: A periodic feedback loop

Requests pass eligibility filtering before weighted consistent hashing selects an engine. A periodic feedback loop smooths engine signals and adjusts weights against fleet-average performance.

[3:40](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=220s) [Diagram of eligibility filtering and weighted consistent hashing, with periodic engine signal updates feeding back into selection.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4a65e06e9606250dd0fd3d4e9cfe2e09c785e4b80c2498ccc41f02ff5203ba4.webp)

Load Balancing in the “early days”:
A periodic feedback loop

PERIODIC SIGNAL UPDATES
→ EWMA / smoothing
→ PID like weight update

INFERENCE REQUESTS

ELIGIBILITY FILTERING
(e.g., capability)

WEIGHTED CONSISTENT HASHING SELECTION

INFERENCE ENGINES

OpenAI

### Which GPU engine should serve each request from a CPU cluster?

A control plane uses a global view to optimize routing across CPU clusters and GPU engines. Each cluster’s local data plane uses that guidance to select an eligible engine.

[7:37](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=457s) [A global control plane connects to local data planes in CPU clusters, which route requests to GPU engines.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa68908a4ae3e10e7bc786f0a80f51c8abf2cb41f461f040ef46d985c0f3d1e9.webp)

Which GPU engine should serve each request from a CPU cluster?

Input: CPU cluster + model. Output: one eligible engine in a GPU cluster

Control plane
Global view for routing optimizations

CPU cluster 1
Local data plane

CPU cluster 2
Local data plane

CPU cluster N
Local data plane

GPU engines across GPU clusters
engine 1
engine 2
engine N

OpenAI

### Separated control plane and data plane

The data plane selects an engine using local routing state. The control plane computes and publishes routing weights asynchronously, so each request can be routed without consulting it.

[9:16](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=556s) [Diagram showing request routing through the data plane and asynchronous signals and routing weights connecting it to the control plane.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e652770ade11f414e5bca22d07813a3496ae098224cb52eaa7705235abb94ac.webp)

Separated control plane and data plane

Hot request path
Async feedback
Async routing weights pull

Control plane
Network overhead
Capacity + TTFT/TBoT regressions
Engine signals → Data loader → Optimizer → Publish routing weights

Data plane
Async refresh candidate engines
Local routing state: candidate engines + routing weights
Requests from CPU clusters → Engine selector → GPU fleet → Engine signals

No request waits on the data plane.

OpenAI

### Three paths through the system

Each request is routed synchronously from a CPU cluster to a GPU engine. Two asynchronous paths collect engine signals and update routing weights to inform future routing decisions.

[10:39](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=639s) [Three rows show the synchronous inference request path and asynchronous engine signal and routing weight paths.](https://assets.ai.engineer/sites/aiecode2025/sha256/c67dea94814c0c8f58c209dd55693f7834879552b6d3c2fc3062c17faa463417.webp)

Three paths through the system

One synchronous request path; two asynchronous control paths

01.
Inference request path
synchronous, per request
Request from CPU cluster → Engine selection + route → GPU engine serve request

02.
Engine signal path
async feedback
GPU engine signals → Aggregate signals → Control/data plane ingest

03.
Routing weight path
async pull
Control plane computed routing weight → Publish routing weights → Data plane weights updated

OpenAI

### Why nearest-only fails

Cluster B sends 20 RPS beyond Engine B’s capacity to Engine C, trading extra network distance for potentially less engine-side waiting time.

[12:08](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=728s) [Three regions illustrate an overloaded engine in region 2 sending 20 requests per second to spare capacity in region 3.](https://assets.ai.engineer/sites/aiecode2025/sha256/cabd1c49eae928db8be7930528c7d7224c34465a356c39fcb8f1b4af8b229719.webp)

Why nearest-only fails
Demand and GPU capacity are not geographically balanced

Region 1
Engine A
100 RPS cap
CPU Cluster A
90 RPS

Region 2
Engine B
100 RPS cap
At capacity
20 RPS spillover
CPU Cluster B
120 RPS

Region 3
Engine C
80 RPS cap
40 RPS spare
CPU Cluster C
40 RPS

OpenAI

### Optimizer: From signals to weights

The optimizer assigns each CPU cluster’s traffic across engines, balancing network and engine-side latency while routing all demand within capacity and keeping weights non-negative.

[13:33](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=813s) [Optimizer takes demand, network latency, engine capacity and health, and latency profiles as inputs and produces routing weights under capacity and other constraints.](https://assets.ai.engineer/sites/aiecode2025/sha256/11e8dcf40ed2f188a6f97eb4b3da7c410715aff5855c5705dc24bce0b860db4b.webp)

Optimizer: From signals to weights
Compute how each CPU cluster should route traffic across engines

Request from CPU cluster (demand)
Network latency
Engine capacity + health
TTFT / TBoT latency profiles

OPTIMIZER
Traffic allocation

Routing weights

Optimization goal
Minimize expected end-to-end latency across all routed traffic.
Balance network distance with engine-side latency.

Hard constraints
Route all demand
Stay within effective engine capacity
Keep routing weights non-negative

OpenAI

### Protection mechanisms

The router lowers weights for anomalous engines, caps retries to prevent retry storms, and sheds traffic as a last resort when demand exceeds capacity.

[15:28](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=928s) [Three cards identify penalties, retry with cap, and load shedding.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f6c215f4524b678032fffc58981364d317527b268d6775dd8a8746793d44f72.webp)

Protection mechanisms

Penalties
Retry with cap
Load shedding

OpenAI

### AI Engineer World's Fair

[18:06](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=1086s) [AI Engineer World's Fair title card on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/35a015b35c58391151718a3241624b4b4a6577077d84bae5e4c245fe333d5009.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:12](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=12s) Hi everyone. Thanks for joining our talk. I'm Lu, and this is my colleague, Qianru. So we- today we're gonna talk about the in, uh, we both work on the inference team at OpenAI, and today we are going to talk about routing LLM inference in production, specifically how our system evolved from routing based on feedback loops driven by engine signals to a more explicit and predictable policy, which is still informed by engine signals, however, it's more like, uh, the way we use it is different.

[0:43](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=43s) So, uh, for the agenda today, we're gonna begin by introducing the inference load balancer, what it is, what it does, and how it has evolved. And then, Qianru will walk us through the newer control plane and the data plane-driven architecture, uh, what are the responsibilities of each, and followed by a concrete case study of how we reduce the global network overhead. And in the end, I return to discuss the protection mechanisms that help keep the system stable under production level

[1:12](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=72s) stress. So to begin with, what is the inference load balancer and where does it sit? So this is a very high level, uh, diagram of the system we are talking about. On the left-hand side are the front-end clusters. Those are the GPU clusters, uh, sorry, those are the CPU clusters that act as gateways into our system, and they receive user requests, then prepare them into the inference request that can be

[1:42](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=102s) processed by the inference engines. And on the right-hand side are the engine clusters, uh, which are usually GPU clusters, and each hosting multiple inference engines. So that's why we, they got the name of engine clusters. And as you may already heard, nowadays, GPUs are pretty popular and expensive. So, um, sitting in the middle, it is the ILB or inference load balancer. It actually runs on the front-end clusters, but is also a

[2:12](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=132s) bridge into our inference stack. It has two main responsibilities, select an engine and processing the request. For this talk, we are going to focus on the engine selection part. So in some ways, ILB resembles a very traditional load balancer because a request usually targets a model, and a model is backed by multiple engines. They may live on different clusters in different regions or even across the continents, because that

[2:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=161s) gives us a good resiliency towards localized degradation or cluster failures. However, the inference, uh, stack or the uniqueness of the inference introduces a lot of nuances, like, uh, it have to consider a bunch of signals reported in real time, like, uh, the well-known time to first token  time between output tokens, also known as token throughput or time between tokens, and other healthiness and utilization signals.

[3:11](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=191s) Besides, there's an important concept of a KV cache, which is also well known. But for example, when the conversation already has a lot of the useful context cached in one engine, sending the follow-up turns of the same conversation back to the same engine will avoid recomputation, improve efficiency and, uh, reduce latency. So the combination of performance, reliability, locality, cache awareness is what makes it such an interesting problem. Uh, so how we

[3:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=221s) attempted in the problem, let's take a look at the early days. And to be honest, early days in, in this industry sounds a lot more historic than it really is. And the routing process at that time began with a filter of like, uh, each request may not be served by all the engines because of, uh, constraints such as capabilities or due, uh, restrictions due to compute or data residency. And among the remaining engines,

[4:11](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=251s) ILB used a weighted consistent hashing to select the best destination engine for a request or, or for, for certain user. Then the important question becomes where are the weights come from? So they were generated by a periodic feedback loop. The inference engines, as mentioned earlier, uh, reports all kind of the signals we care about. And the controller will periodically smooth out those signals and compute a performance score.

[4:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=281s) The performance score then will be compared against the fleet average. Then the weight will be adjusted basically for each engine as the weight goes up if the performance is better, or it goes down when the performance is worse than the fleet average. And this generated weight will impact the routing, and then it's basically a, uh, control loop. Um, conceptually, it's, it's very similar to the PID controller. And no, this

[5:11](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=311s) PID controller will not help you kill a Linux process, but instead it's a classical control theory technique that continuously steering the system towards its desired state. And we just borrowed this important concept, the proportional part of it, and, uh, applied into our, uh, contr- our load balancer. So it has a lot of nice properties. For example, it could combine the useful signals we care about into the single routing decision. And because of the, it

[5:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=341s) adapt to the ob- observed performance, as what me- we mentioned earlier, there's a lot of constraints, and those constraints might have the, some engines busier because they can serve more requests, more kind of requests than the remaining. But those busier signal will be fed into the next loop and resulting in the less constrained sig- less constrained request can go to more of those kind of engines. So basically, they self-balance it out. And to some extent, this just means we don't need to- Do a lot to manual

[6:11](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=371s) intervention, and it, it just works. However, that kind of adaptability comes with big trade-offs. Because of the same reason that it combines so many signals, it's also very hard to reason about a particular routing decision, or, like, why search engine gets a higher weight than we expect. And every time we want to fine-tune towards some aspect, it's alm-almost impossible to not impacting something else. And the load is not always w- uh, where, where

[6:41](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=401s) evenly distributed because, uh, sometimes a model is served by engines on different GPU SKUs, and they have different characteristics. Then the problem becomes a lot more trickier. And the f- the feedback loop sometimes creates bad oscillations because when you're shifting an engine away some traffic, the engine turns a bit cooler, and this signal get fed to the controller. The controller now thinks like, "Hey, this

[7:10](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=430s) engine can take a lot more traffic." Then the, some traffic gonna be shifted back and forth between a few engines and disrupting the KV cache utilization. So all those limitations motivated us to rethink about the architecture and, uh, see if we have new ways to address the problem. So I'm going to hand over to Qianru to, uh, deep dive into the new architecture we tried out.

[7:38](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=458s) Yeah. Thank you, Lu. So I'm going to talk about the architecture of the load balancer and how do we reduce the overall overhead with our routing algorithm. The load balancer answers one question. Uh, for each request from a CPU cluster, which engine should serve it? One most naive baseline might be round robin, which send requests across engines evenly. But if you think a little bit more, that doesn't

[8:08](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=488s) make sense. Because engines are not homogeneous, they can have different hardware and capacity, different health, and also different distance from CPU cluster. Also, round robin could break cache locality. S- related requests that could reuse the same engine cache might be sent to different engines. A probably better solution might be for each CPU cluster, it choose the best engine from its own local view.

[8:39](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=519s) But that's not enough either. Think about one extreme case. Multiple CPU cluster route traffic to the same engines independently, which could overload that engine while leave other engines underutilized. So what we need is a globally optimized solution, a control plane that has a global view for all the CPU cluster and GPU engines and could compute a globally optimized routing answers. And the data plane can make a

[9:09](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=549s) routing decision quickly based on the answer pulled from the control plane.

[9:18](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=558s) Now let's look inside the control plane and data plane. In the data plane, there is an engine selector which select engine for each request. It reads the local routing state, which includes the candidate engines and the routing weights for each candidate engines. Both of them are refreshed asynchronously in the background, so we don't need to ask the control plane before we make a routing decision for each request.

[9:48](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=588s) Also, the data plane collects real-time engine signal, such as number of ready replica, engine health, et cetera, to serve as fast local guardrail. In the control plane, the data loader combines those live engine signals and network overhead. And with offline regressions of capacity, TTFT, and TBOT, the optimizer could turn those data into routing weights, and the control plane will publish the

[10:18](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=618s) routing weight for each data plane to pull. In this way, no request need to wait on the data plane. The control plane continuously compute the next globally optimized routing weight snapshot while the data plane make a routing decision based on the latest snapshot already installed locally.

[10:42](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=642s) In summary, there are three important paths through the system. The first path is the inference request path. The request arrive to the CPU cluster, and the data plane inside that CPU cluster will select engine for that request based on the local routing state and forward the request to the selected engines. The second path is the engine signal path. The system continuously collects real-time engine signal, such as TTFT,

[11:12](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=672s) TBOT, number of ready replica, and engine health, et cetera. Both planes need those real-time engine signals. The control plane need them to compute a globally optimized routing weight, while the data plane need them to serve as fast local guardrail. And the third path is the routing weight path. The control plane compute and publish the routing weight, and the data plane pull the updates to its local cache. So only

[11:42](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=702s) the first path is synchronous, but it's fast and only local inside the data plane of the CPU cluster. The other two loops are asynchronous loop, and they are to improve future routing decision.

[11:59](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=719s) So that's pretty much of the architecture part, but that still leaves one question. How do we compute those routing weights?

[12:10](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=730s) But before answer that question, let's answer another question first. Why not just send the request to the nearest engine? That's because the traffic demand and GPU capacity are not geographically balanced. For example, in region one, CPU cluster A send ninety RPS, and the nearby engine A can serve one hundred RPS. So in this case, nearest-only is fine.

[12:40](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=760s) While in region two, CPU cluster B send one hundred and twenty RPS, and the nearby engine B could only serve one hundred RPS. So in this case, if we insist on keeping everything local, the extra twenty RPS need to wait on an overloaded engine B. While in region three, we are only using four RPS of an eight RPS engine C. That still leaves four RPS spare. So if we send the extra twenty

[13:10](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=790s) RPS from cluster B to engine C, that will add network distance. But it could also avoid a probably much larger engine-side waiting time. So in this case, a further engine might be faster end-to-end. That's why we need something better than the nearest-only routing.

[13:35](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=815s) Now, let's open the black box of the optimizer. The optimizer accepts four types of input: the request from each CPU cluster, the network latency to each engine, the available engine capacity and health, and also the TTFT, TBOT latency profiles that tell us how's the engine-side latency change as the load increases. And with those

[14:05](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=845s) input, the optimizer turn the input to the output routing weights. The routing weights say, for each CPU cluster, what fraction of its traffic should go to each GPU engine. And the optimization goal is straightforward. It's to minimize the expected end-to-end latency across all routed traffic. The important part is that the end-to-end latency includes both the network distance

[14:35](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=875s) and the engine-side latency. That means a nearby engine might be attractive when it still has room to serve traffic, while a further engine might be better if all the nearby engines are close to full. And the optimizer also need to respect several hard constraints. First, it need to route all the traffic demand. Second, it need to ensure all the engines stay within the effective capacity.

[15:05](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=905s) Third, it need to keep the routing weights non-negative. With this, the controller, control plane get the routing weight from the optimizer and publish them, and the data plane pull them and use them to make a globally optimized routing decision. And that's pretty much of my part, and Lu will continue to talk about the protection mechanisms in the system.

[15:33](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=933s) Thanks, Qianru. So as AI engineers, we all kind of know that production, in many cases, are not behaving in the most ideal case. So clusters can fail, GPUs or individual nodes can degre-degrade, and networking can just get to all kind of mysterious issues. So how do we keep our production system, uh, healthy as much as p-possible under the heavy load? The first design we

[16:03](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=963s) have is, uh, penalties. Basically, when an engine is an outlier, we detect the anomaly and try to reduce the routing weight to that engine. In that way, we give it a chance to either recover by themselves if there's a, uh, s- if, if some transient issue, or we can have a human intervene to rotate it out or replace the faulty hardware. And secondly, the retries, which is a very common technique used to mitigate

[16:33](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=993s) problems. However, during some cases, it actually could make sense even worse, like when the system is very close to, like, a tip over or very heavily utilized. Retries will send more load, and these more loads will, uh, cause more failures and cause more retries, which is a infamous retry storm. So we incru- implemented caps or budget to constrain retries into a acceptable region. And this is actually even need to be

[17:03](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=1023s) dynamic because in the happy time or in the normal time, we can tolerate a lot more retries than when the system are heavily utilized. And finally, we have the load shedding, which is our last resort when the production capac- uh, capacity couldn't meet the increasing amount of, uh, inference demands. So we instead, uh, we try to have all the system fail. We basically proactively load shed a portion of the traffic

[17:33](https://www.youtube.com/watch?v=sOB3HSiG8vo&t=1053s) to have the system degrade gracefully. So that pretty much concludes our talk today, and, uh, thanks for joining us. Uh, both of us will be around in our, uh, booth area this afternoon. So if you have further questions, feel free to walk, uh, to the area and ta- and chat with us. Thank you.
