Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Inference demand becomes harder to predict when one user action starts many model calls. Nishant Gupta and Naman Ahuja, who work on efficiency, training and inference infrastructure at Meta, begin with the scale of that change. Gupta describes inference as the fastest-growing workload they have seen, with traffic already exceeding the largest microservices. Keeping that workload affordable and reliable requires planning for the work each interaction creates. 0:12

The historical comparison starts around 2008. Cloud engineering initially centered on virtual machines and virtualization. Complexity then accumulated in schedulers such as Borg, Kubernetes and Mesos, followed by service meshes, autoscalers and platforms. Gupta sees inference following a compressed version of that progression: models on GPUs led to serving frameworks such as vLLM, TorchServe and Triton; an emerging orchestration layer now coordinates routing, cache management and multiple models. Better kernels remain useful, but operating the whole system becomes a distinct engineering problem. 0:54

Two terms explain much of what follows. Prefill processes the input prompt and builds the state needed for generation; decode generates the response token by token. The KV cache, or key-value cache, holds reusable attention state from tokens already processed, so generation can use that earlier work. Prefill/decode disaggregation separates those execution phases. That creates additional placement and coordination decisions around the expensive state a request needs. 1:24

Traditional web capacity could often be approximated from users and queries per second: double the users, roughly double the requests and fleet, before optimizations. Agentic demand adds two multiplying factors: calls per user and tokens per call. Gupta’s examples range from one model call per chatbot turn to ten to twenty for copilots, fifty for research agents and thousands for autonomous workloads. These are workload examples rather than fixed sizing rules; model choice, hardware and optimizations also change the capacity required. Elasticity, workload-aware scheduling and admission control must account for what users cause the system to do. 2:06

The microservice comparison becomes concrete at the request level. Gupta describes LLM requests in their workloads ranging from fifty to a hundred thousand tokens, with different compute profiles during prefill and decode. Counting requests consequently hides substantial differences in the work entering the fleet: processing a long prompt and producing a streamed response place different demands on serving resources. 3:09

Several serving assumptions change together:

• Batching: Continuous in-flight batching groups work from requests that are already running. Gupta describes an order-of-magnitude-or-greater throughput loss without it, though the workload conditions for that comparison are unspecified.
• Request state: The KV cache preserves expensive work for each request. Discarding it loses that investment.
• Capacity: GPUs are costly and slower to acquire than ordinary CPU capacity, making casual overprovisioning expensive.
• Recovery: A replacement instance takes time to become warm. Losing a GPU during decode can discard thousands of in-flight tokens and leave requests accumulating in queues.

A prompt also hides a sequence of infrastructure decisions: authentication, model selection, region selection, admission, cache lookup, GPU placement and batching. Model execution—prefill and decode—is one part of that path. Whether those phases run together or separately, the surrounding infrastructure determines when work starts, what resources it consumes and whether the user receives a usable response. That gives platform decisions a direct influence on product quality. 4:58

Routing, caching and scheduling are familiar layers. Their coupling makes inference difficult to operate. A routing decision changes which cache state a request encounters. A changed cache hit rate changes the work entering batches; batch composition changes GPU utilization; utilization changes the autoscaler’s next decision. An apparent capacity regression may therefore begin several layers away from the GPU. 5:50

How can one routing decision affect fleet capacity? The diagram follows that chain. Each arrow represents a possible effect, rather than a guaranteed improvement or regression. The relationship matters because changing where work goes can also change how much work remains and how the fleet interprets its load.

The request itself travels through a gateway, router, cache lookup, scheduler and serving runtime. The scheduler chooses a GPU cluster and hardware; a runtime such as vLLM or SGLang executes the work. The response streams back under service-level objectives for time to first token, time between tokens and throughput. Every network hop can time out, retry, fall back or fail. Gupta calls this a distributed transaction to emphasize the coordinated failure problem across the path. 6:54

Streaming makes partial failure visible to the user. Suppose two hundred tokens have already arrived when maintenance preempts the GPU host. A simple retry does not resolve what should happen to the output already delivered. Reliability needs a control plane that sees the whole workflow and can coordinate recovery across its steps; the talk identifies that requirement without prescribing a continuation protocol. 7:54

Traditional bin packing might consider CPU, memory and fault domains. Inference placement must consider at least seven axes, spanning both the machine’s readiness and the request’s importance. 8:31

• GPU type: Hardware generations and network topologies differ.
• HBM headroom: Available high-bandwidth memory constrains placement.
• KV cache state: Existing request state affects the work needed.
• Model-weight readiness: Loaded, warm weights avoid a cold start.
• Tenant priority: Tenants share capacity but have different service objectives.
• Workflow context: Earlier completed steps may already represent substantial spending.
• Latency budget: The application determines how much delay remains acceptable.

Request R provides the concrete example. It is at step three of a five-step workflow. Step one spent X dollars and step two spent Y dollars. If step three fails and terminates the workflow, the system loses the useful value of that earlier X + Y investment. Looking only at the next request’s resource demand misses the cost of losing the larger task. 10:01

Making R’s position visible changes the scheduler’s decision inputs. Admission can account for completed work; priority can reflect the consequences of interruption; retry policy can consider whether a failed step ends the workflow. The objective is placement that helps the task finish quickly and cheaply. This example explains why workflow context matters, without establishing a universal rule that later steps always outrank new work.

Gupta organizes serving optimizations by what they do to the work. The framework helps compare techniques across runtimes and asks how they combine, rather than treating every optimization as an independent win. 10:25

• Avoid: Prefix caching, response caching and semantic caching can reuse work and skip computation.
• Share: Batching lets multiple requests share compute. Chunked prefill and speculative decoding also appear in this part of the discussion, though their algorithms are not developed here.
• Move: Routing can send a request to a smaller model, a cheaper region or somewhere closer to the user.
• Delay: Admission control and queues can wait for a better execution opportunity, using priority classes and deadline-aware scheduling.

The comparison needs a product-level denominator. GPU execution is only part of the bill: retries, storage, failures, networking and operational work also cost money. Cost per token or cost per request can improve while a workflow becomes less likely to finish. Cost per successful task measures spending against the outcome the user cares about: obtaining a useful completed result. 11:47

For R, the distinction is immediate. A cheap step-three attempt that terminates the workflow leaves the earlier X + Y spending without a successful task. Evaluating the complete task keeps that lost work in view. It also connects optimization to reliability: avoiding retries and preventing workflow failure can matter as much as reducing the expense of one model call.

A degraded GPU begins a larger failure story. Latency rises, clients retry, queues deepen and healthy GPUs saturate. Saturation produces further retries, potentially spreading the incident into a regional failure. The dangerous part is the feedback loop: attempts to recover individual requests add load to a system already falling behind. 12:36

Rerouting cannot necessarily supply immediate relief. A cold pool must warm before it can absorb traffic, while the hot pool carries the requests during that interval. KV cache state makes restarting or moving work consequential. Available machines and immediately usable serving capacity are different operational quantities.

Where can the system interrupt the amplification? The diagram shows the retry cycle and its points of intervention. Circuit breakers limit routing into trouble; admission control can reject work instead of adding it to a queue; load shedding responds to queue depth; retry budgets limit extra attempts. Queue depth matters because CPU or memory utilization alone does not describe how much unfinished work is waiting. These controls also limit the additional spending a cascade can create. 13:06

Naman Ahuja takes over with the operating question: how does the platform know what to do next? Production inference inherits queues, scheduling, autoscaling and fault isolation from distributed systems, while adding constraints around HBM, KV cache and task economics. Observability supplies inputs to a control loop. Telemetry feeds analysis; analysis drives decisions; decisions change routing and scheduling; the changed system produces new telemetry. 14:56

Different measurements answer different operating questions:

• Time to first token: How long does the user wait for the first response?
• Utilization ratio: Is memory or compute constraining execution?
• Success per dollar: Is spending producing successful tasks efficiently?
• End-to-end trace latency: Where does time go across the full request path?

Those signals help choose among latency, cost and throughput tradeoffs. Larger batches can improve throughput and cost efficiency while hurting tail latency. Speculative decoding can reduce latency while consuming extra compute. A smaller model can reduce immediate latency and cost, but insufficient response quality can require retries and bring the cost back up. Ahuja compares this tension to CAP as an analogy; these are serving tradeoffs rather than a formal impossibility theorem. 16:19

The useful setting depends on the task. A fast, inexpensive response that requires another attempt changes the economics of completion. A high-throughput batch that delays the slowest requests changes the user experience. The control loop therefore needs both resource measurements and measures of successful delivery to choose a serving policy.

Routing, batching, caching, scheduling and reliability converge into what Ahuja calls an inference control plane. It decides which model serves a request and how that request is batched, while coordinating GPUs, cache, token latency and cost. The cloud comparison returns: virtual-machine operations developed schedulers, autoscaling and control planes; inference needs a layer that owns its interacting decisions. Whether built internally, adopted from open source or bought from a vendor, that layer belongs in the system design. 17:03

The closing operating lessons describe what that layer must accomplish:

• Find infrastructure bottlenecks early: Production failures may come from placement, routing or capacity breakdown before model execution becomes the limiting factor.
• Build elasticity: Adding GPUs can conceal a scheduling problem without resolving it.
• Improve decisions within the fleet: The same hardware can deliver different results depending on scheduling and batching.
• Close the loop: The platform should sense problems and adapt automatically, rather than depend entirely on manual intervention.

Ahuja’s forecast places orchestration after two earlier priorities: making models smarter, then making inference faster through latency, batching and GPU-utilization improvements. In the next phase, GPU capacity, memory and cache become resources to schedule and control together. The closing phrase—“an orchestration problem”—names the practical shift: completing a task depends on decisions across the system, including the work already spent, the state already built and the failures still possible. 18:49