# Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Nishant Gupta, Meta · Naman Ahuja, Meta

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/7c9FSUVcXR0-operating-distributed-inference-systems-at-scale) · [Video](https://www.youtube.com/watch?v=7c9FSUVcXR0)

## Operating Distributed Inference Systems at Scale

Nishant Gupta and Naman Ahuja explain how routing, caching, scheduling and reliability interact in distributed inference—and why efficiency should be measured by cost per successful task.

### The cloud’s orchestration shift, compressed

[0:12](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=12s)

Inference demand becomes harder to predict when one user action starts many model calls. [Nishant Gupta](https://www.linkedin.com/in/nishantgupta-ai/) and [Naman Ahuja](https://www.linkedin.com/in/namanahuja/), who work on efficiency, training and inference infrastructure at Meta, begin with the scale of that change. Gupta describes inference as the fastest-growing workload they have seen, with traffic already exceeding the largest microservices. Keeping that workload affordable and reliable requires planning for the work each interaction creates. [0:12](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=12s)

The historical comparison starts around 2008. Cloud engineering initially centered on virtual machines and virtualization. Complexity then accumulated in schedulers such as Borg, Kubernetes and Mesos, followed by service meshes, autoscalers and platforms. Gupta sees inference following a compressed version of that progression: models on GPUs led to serving frameworks such as **vLLM**, TorchServe and Triton; an emerging orchestration layer now coordinates routing, cache management and multiple models. Better kernels remain useful, but operating the whole system becomes a distinct engineering problem. [0:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=54s)

Two terms explain much of what follows. **Prefill** processes the input prompt and builds the state needed for generation; **decode** generates the response token by token. The **KV cache**, or key-value cache, holds reusable attention state from tokens already processed, so generation can use that earlier work. Prefill/decode disaggregation separates those execution phases. That creates additional placement and coordination decisions around the expensive state a request needs. [1:24](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=84s)

Traditional web capacity could often be approximated from users and queries per second: double the users, roughly double the requests and fleet, before optimizations. Agentic demand adds two multiplying factors: calls per user and tokens per call. Gupta’s examples range from one model call per chatbot turn to ten to twenty for copilots, fifty for research agents and thousands for autonomous workloads. These are workload examples rather than fixed sizing rules; model choice, hardware and optimizations also change the capacity required. Elasticity, workload-aware scheduling and admission control must account for what users cause the system to do. [2:06](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=126s)

### Requests carry uneven work and expensive state

[3:09](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=189s)

The microservice comparison becomes concrete at the request level. Gupta describes LLM requests in their workloads ranging from fifty to a hundred thousand tokens, with different compute profiles during prefill and decode. Counting requests consequently hides substantial differences in the work entering the fleet: processing a long prompt and producing a streamed response place different demands on serving resources. [3:09](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=189s)

Several serving assumptions change together:

- **Batching:** Continuous in-flight batching groups work from requests that are already running. Gupta describes an order-of-magnitude-or-greater throughput loss without it, though the workload conditions for that comparison are unspecified.
- **Request state:** The KV cache preserves expensive work for each request. Discarding it loses that investment.
- **Capacity:** GPUs are costly and slower to acquire than ordinary CPU capacity, making casual overprovisioning expensive.
- **Recovery:** A replacement instance takes time to become warm. Losing a GPU during decode can discard thousands of in-flight tokens and leave requests accumulating in queues.

A prompt also hides a sequence of infrastructure decisions: authentication, model selection, region selection, admission, cache lookup, GPU placement and batching. Model execution—prefill and decode—is one part of that path. Whether those phases run together or separately, the surrounding infrastructure determines when work starts, what resources it consumes and whether the user receives a usable response. That gives platform decisions a direct influence on product quality. [4:58](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=298s)

### A routing change can reach the autoscaler

[5:50](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=350s)

Routing, caching and scheduling are familiar layers. Their coupling makes inference difficult to operate. A routing decision changes which cache state a request encounters. A changed cache hit rate changes the work entering batches; batch composition changes GPU utilization; utilization changes the autoscaler’s next decision. An apparent capacity regression may therefore begin several layers away from the GPU. [5:50](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=350s)

How can one routing decision affect fleet capacity? The diagram follows that chain. Each arrow represents a possible effect, rather than a guaranteed improvement or regression. The relationship matters because changing where work goes can also change how much work remains and how the fleet interprets its load.

The request itself travels through a gateway, router, cache lookup, scheduler and serving runtime. The scheduler chooses a GPU cluster and hardware; a runtime such as vLLM or **SGLang** executes the work. The response streams back under service-level objectives for time to first token, time between tokens and throughput. Every network hop can time out, retry, fall back or fail. Gupta calls this a distributed transaction to emphasize the coordinated failure problem across the path. [6:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=414s)

Streaming makes partial failure visible to the user. Suppose two hundred tokens have already arrived when maintenance preempts the GPU host. A simple retry does not resolve what should happen to the output already delivered. Reliability needs a control plane that sees the whole workflow and can coordinate recovery across its steps; the talk identifies that requirement without prescribing a continuation protocol. [7:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=474s)

How routing affects the autoscaler
A routing change can alter the work reaching GPUs and the capacity decision that follows.
- Routing decision — Changes where requests go.
- Cache hit rate — Changes how much earlier work can be reused.
- Batch composition — Changes the work grouped for execution.
- GPU utilization — Changes as batch composition changes.
- Autoscaling decision — Responds to changed utilization.
- Routing decision → Cache hit rate (Changes cache hits)
- Cache hit rate → Batch composition (Changes batch composition)
- Batch composition → GPU utilization (Changes utilization)
- GPU utilization → Autoscaling decision (Changes capacity decisions)

### Schedule the workflow, not just the next request

[8:31](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=511s)

Traditional bin packing might consider CPU, memory and fault domains. Inference placement must consider at least seven axes, spanning both the machine’s readiness and the request’s importance. [8:31](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=511s)

- **GPU type:** Hardware generations and network topologies differ.
- **HBM headroom:** Available high-bandwidth memory constrains placement.
- **KV cache state:** Existing request state affects the work needed.
- **Model-weight readiness:** Loaded, warm weights avoid a cold start.
- **Tenant priority:** Tenants share capacity but have different service objectives.
- **Workflow context:** Earlier completed steps may already represent substantial spending.
- **Latency budget:** The application determines how much delay remains acceptable.

Request `R` provides the concrete example. It is at step three of a five-step workflow. Step one spent `X` dollars and step two spent `Y` dollars. If step three fails and terminates the workflow, the system loses the useful value of that earlier `X + Y` investment. Looking only at the next request’s resource demand misses the cost of losing the larger task. [10:01](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=601s)

Making `R`’s position visible changes the scheduler’s decision inputs. Admission can account for completed work; priority can reflect the consequences of interruption; retry policy can consider whether a failed step ends the workflow. The objective is placement that helps the task finish quickly and cheaply. This example explains why workflow context matters, without establishing a universal rule that later steps always outrank new work.

### Avoid, share, move or delay the work

[10:25](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=625s)

Gupta organizes serving optimizations by what they do to the work. The framework helps compare techniques across runtimes and asks how they combine, rather than treating every optimization as an independent win. [10:25](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=625s)

- **Avoid:** Prefix caching, response caching and semantic caching can reuse work and skip computation.
- **Share:** Batching lets multiple requests share compute. Chunked prefill and speculative decoding also appear in this part of the discussion, though their algorithms are not developed here.
- **Move:** Routing can send a request to a smaller model, a cheaper region or somewhere closer to the user.
- **Delay:** Admission control and queues can wait for a better execution opportunity, using priority classes and deadline-aware scheduling.

The comparison needs a product-level denominator. GPU execution is only part of the bill: retries, storage, failures, networking and operational work also cost money. Cost per token or cost per request can improve while a workflow becomes less likely to finish. **Cost per successful task** measures spending against the outcome the user cares about: obtaining a useful completed result. [11:47](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=707s)

For `R`, the distinction is immediate. A cheap step-three attempt that terminates the workflow leaves the earlier `X + Y` spending without a successful task. Evaluating the complete task keeps that lost work in view. It also connects optimization to reliability: avoiding retries and preventing workflow failure can matter as much as reducing the expense of one model call.

### Retries amplify failure while cold capacity warms

[12:36](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=756s)

A degraded GPU begins a larger failure story. Latency rises, clients retry, queues deepen and healthy GPUs saturate. Saturation produces further retries, potentially spreading the incident into a regional failure. The dangerous part is the feedback loop: attempts to recover individual requests add load to a system already falling behind. [12:36](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=756s)

Rerouting cannot necessarily supply immediate relief. A cold pool must warm before it can absorb traffic, while the hot pool carries the requests during that interval. KV cache state makes restarting or moving work consequential. Available machines and immediately usable serving capacity are different operational quantities.

Where can the system interrupt the amplification? The diagram shows the retry cycle and its points of intervention. Circuit breakers limit routing into trouble; admission control can reject work instead of adding it to a queue; load shedding responds to queue depth; retry budgets limit extra attempts. Queue depth matters because CPU or memory utilization alone does not describe how much unfinished work is waiting. These controls also limit the additional spending a cascade can create. [13:06](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=786s)

The retry cascade and its loop breakers
Retries feed saturation back into further retries. Recovery capacity needs time to warm, so the platform must also limit amplification.
- GPU degradation — Starts the latency increase.
- Rising latency — Prompts client retries.
- Client retries — Add more requests.
- Deeper queues — Increase pressure on healthy GPUs.
- Healthy GPUs saturate — Produces further retries and wider failure.
- Retry budgets — Limit extra attempts.
- Admission and load shedding — Reject work and respond to queue depth.
- Routing circuit breakers — Interrupt routing into failure.
- GPU degradation → Rising latency (Slows responses)
- Rising latency → Client retries (Triggers retries)
- Client retries → Deeper queues (Adds load)
- Deeper queues → Healthy GPUs saturate (Overloads healthy capacity)
- Healthy GPUs saturate → Client retries (Causes further retries)
- Retry budgets → Client retries (Limits attempts)
- Admission and load shedding → Deeper queues (Limits queued work)
- Routing circuit breakers → Healthy GPUs saturate (Limits routing into trouble)

### Telemetry tells the platform what to change

[14:56](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=896s)

Naman Ahuja takes over with the operating question: how does the platform know what to do next? Production inference inherits queues, scheduling, autoscaling and fault isolation from distributed systems, while adding constraints around HBM, KV cache and task economics. Observability supplies inputs to a control loop. Telemetry feeds analysis; analysis drives decisions; decisions change routing and scheduling; the changed system produces new telemetry. [14:56](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=896s)

Different measurements answer different operating questions:

- **Time to first token:** How long does the user wait for the first response?
- **Utilization ratio:** Is memory or compute constraining execution?
- **Success per dollar:** Is spending producing successful tasks efficiently?
- **End-to-end trace latency:** Where does time go across the full request path?

Those signals help choose among latency, cost and throughput tradeoffs. Larger batches can improve throughput and cost efficiency while hurting tail latency. Speculative decoding can reduce latency while consuming extra compute. A smaller model can reduce immediate latency and cost, but insufficient response quality can require retries and bring the cost back up. Ahuja compares this tension to CAP as an analogy; these are serving tradeoffs rather than a formal impossibility theorem. [16:19](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=979s)

The useful setting depends on the task. A fast, inexpensive response that requires another attempt changes the economics of completion. A high-throughput batch that delays the slowest requests changes the user experience. The control loop therefore needs both resource measurements and measures of successful delivery to choose a serving policy.

### Make orchestration a first-class layer

[17:03](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1023s)

Routing, batching, caching, scheduling and reliability converge into what Ahuja calls an inference control plane. It decides which model serves a request and how that request is batched, while coordinating GPUs, cache, token latency and cost. The cloud comparison returns: virtual-machine operations developed schedulers, autoscaling and control planes; inference needs a layer that owns its interacting decisions. Whether built internally, adopted from open source or bought from a vendor, that layer belongs in the system design. [17:03](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1023s)

The closing operating lessons describe what that layer must accomplish:

- **Find infrastructure bottlenecks early:** Production failures may come from placement, routing or capacity breakdown before model execution becomes the limiting factor.
- **Build elasticity:** Adding GPUs can conceal a scheduling problem without resolving it.
- **Improve decisions within the fleet:** The same hardware can deliver different results depending on scheduling and batching.
- **Close the loop:** The platform should sense problems and adapt automatically, rather than depend entirely on manual intervention.

Ahuja’s forecast places orchestration after two earlier priorities: making models smarter, then making inference faster through latency, batching and GPU-utilization improvements. In the next phase, GPU capacity, memory and cache become resources to schedule and control together. The closing phrase—“an orchestration problem”—names the practical shift: completing a task depends on decisions across the system, including the work already spent, the state already built and the failures still possible. [18:49](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1129s)

## Key takeaways

- Agent capacity planning must account for calls per user and tokens per call, along with hardware and model choices. User count alone misses the demand multiplier.
- Routing changes can propagate through cache hits, batch composition and GPU utilization into autoscaling decisions. Diagnose regressions across the stack.
- Workflow-aware scheduling considers hardware readiness, request state, tenant objectives and completed work. Losing step three can waste the spending on steps one and two.
- Retry budgets, routing circuit breakers, admission control and queue-based load shedding interrupt cascades. Cold capacity needs time to become useful.
- Use telemetry to coordinate serving decisions and optimize cost per successful task. Lower token cost can lose its value when failures, retries or poor responses prevent completion.

## Related talks

- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://ai.engineer/talks/YXowceUKYJI-kv-cache-aware-routing-p-d-disaggregation) — A focused next recording on the cache-aware routing and prefill/decode disaggregation named in this talk.
- [Hacking the Inference Pareto Frontier](https://ai.engineer/talks/Y2qc0UhDSnc-hacking-inference-pareto-frontier) — Continue with inference optimization tradeoffs after the latency, cost and throughput discussion.

## Slides

### Operating Distributed Inference Systems at Scale

Nishant Gupta and Naman Ahuja of Meta introduce the infrastructure behind modern AI systems and operating distributed inference at scale.

[0:18](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=18s) [Title slide introducing the infrastructure behind modern AI systems, presented by Nishant Gupta and Naman Ahuja of Meta.](https://assets.ai.engineer/sites/aiecode2025/sha256/05c82cfe3ef94a23faceb2a33f73f3b2fa9ef452412963bd3cda1df9a1e0be39.webp)

OPERATING DISTRIBUTED INFERENCE
SYSTEMS AT SCALE

The infrastructure behind modern AI systems

Nishant Gupta & Naman Ahuja
Meta

### Every computing era creates an orchestrator

The slide pairs cloud’s path to a control plane with an emerging AI orchestration layer. The speakers say it must handle challenges such as routing and KV cache management.

[0:52](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=52s) [Cloud-era and AI-era stacks progress from physical servers and GPUs respectively to their control planes.](https://assets.ai.engineer/sites/aiecode2025/sha256/476636dd4dfc2b7efea8d4bb89f1659cb50a0db1bd83c17484e1bf54804999ce.webp)

ACT 1 · WHY INFERENCE CHANGED

Every computing era creates an orchestrator

CLOUD ERA
Physical Servers
Virtual Machines
Containers
Kubernetes
Cloud Control Plane

AI ERA
GPUs
Models
Inference
Agents
Inference Control Plane

Cloud didn't stop at VMs. It became orchestration. AI is following the same path.

### The agentic demand explosion

The slide traces model calls from ~1 per chatbot turn to ~1000+ per autonomous job. The speaker says capacity planning must also account for users and tokens.

[2:04](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=124s) [A progression from chatbot to copilot, research agent, and autonomous workflow shows increasing model calls per activity.](https://assets.ai.engineer/sites/aiecode2025/sha256/b480ab8231195b73aeec66fca45bce3f85ca01a66895bf3ff572f6182c6c471a.webp)

ACT 1 · WHY INFERENCE CHANGED

The agentic demand explosion

1
Chatbot
~1 call / turn

2
Copilot
~5 calls / task

3
Research Agent
~50 calls / query

4
Autonomous Workflow
~1000+ calls / job

### The great inference mismatch

Modern inference combines streaming requests and continuous batching with costly GPUs and large KV-cache state. A failure can disrupt an entire batch mid-generation.

[3:06](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=186s) [A table compares traditional serving and modern inference across request shape, compute, batching, state, scaling, and failure impact.](https://assets.ai.engineer/sites/aiecode2025/sha256/e581be05d42bdd41fea33bbaa96d77d1ffa751cfa5ef6266200a6f28602f607c.webp)

The great inference mismatch

ACT 2 · WHY TRADITIONAL SERVING BREAKS

Dimension | Traditional serving | Modern inference
Request shape | Stateless, ms-scale | Streaming, seconds to minutes
Compute unit | CPU cores | GPU + HBM + KV cache
Batching | Per-request | Continuous / in-flight
State | Cache-friendly, small | KV cache, model weights, huge
Scaling unit | Pods (fast) | GPUs (slow, scarce, costly)
Failure blast radius | One request | Entire batch mid-generation

The bottleneck is no longer the model. The bottleneck is orchestration.

### Hidden decisions behind every prompt

The slide marks generation and safety as ML. The other ten steps, from authentication and model selection to billing and response, are infrastructure.

[4:56](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=296s) [Twelve prompt-processing steps are labeled infrastructure or ML, from authentication and model selection through generation, safety, billing, and response.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea2a6da9f74fd14f84b9bc499d2a1baf03bc5e9cbe90eadfa0288a06e8aca606.webp)

Hidden decisions behind every prompt

ACT 2 · WHY TRADITIONAL SERVING BREAKS

01 INFRA — Authenticate
02 INFRA — Choose Model
03 INFRA — Select Region
04 INFRA — Admission Control
05 INFRA — Cache Lookup
06 INFRA — GPU Reservation
07 INFRA — Batch
08 ML — Generate
09 ML — Safety
10 INFRA — Logging
11 INFRA — Billing
12 INFRA — Response

Which of these are actually ML? Almost none. They're infrastructure.

### The modern inference stack

The stack runs from applications to GPU fleets. A routing decision can change cache hit rates, batch composition, and GPU utilization, so diagnosing a bottleneck requires looking across layers.

[5:47](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=347s) [Nine inference stack layers run from application and agent runtime through routing, caching, scheduling, serving runtime, and the GPU fleet.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a7ec472276e8294e88605052935db09c6c940227a289d278ff247cd0e0c3593.webp)

ACT 2 · WHY TRADITIONAL SERVING BREAKS

The modern inference stack

Application — Product surface: chat, agents, batch pipelines
Agent Runtime — Planning, tool use, memory, orchestration of multi-step reasoning
Inference Gateway — Auth, quotas, safety, protocol translation, tenant isolation
Routing Layer — Model / region / capacity selection based on SLA + cost
Caching Layer — Prompt, prefix, KV, and response caches
Admission Control — Prioritize, defer, or reject under load
Scheduler — Batching, GPU placement, workflow-aware decisions
Serving Runtime — vLLM, TensorRT-LLM, SGLang - continuous batching, paged attention
GPU Fleet — Heterogeneous accelerators, HBM, interconnect topology

### One prompt becomes a distributed workflow

A prompt passes through a gateway, router, cache, scheduler and serving runtime. Each network hop can retry, time out, fall back or fail; retries become harder once tokens have streamed to the user.

[6:51](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=411s) [Prompt-to-response workflow alongside retries, timeouts, fallbacks, failures, and observability concerns.](https://assets.ai.engineer/sites/aiecode2025/sha256/e89fd139dc56b6b93e62e4aac0d06facf0f0e6973fd685aae6d59d0b6a16dfd8.webp)

ACT 2 · WHY TRADITIONAL SERVING BREAKS

One prompt becomes a distributed workflow

Prompt
Gateway
Router
Cache
Scheduler
Serving Runtime
Response

Retries
Idempotency, backoff, budget

Timeouts
Per hop and end-to-end

Fallbacks
Smaller / colder / degraded model

Failures
Batch peer poisoning, mid-stream drops

Observability
Trace across every hop

Inference behaves like a distributed transaction.

### Scheduling GPUs is no longer enough

An inference scheduler weighs model, GPU memory, latency, tenant, context, cost and workflow. Workflow-aware placement aims to finish the whole workflow faster and cheaper.

[8:29](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=509s) [An old CPU-and-memory scheduler compared with a modern scheduler considering model, memory, latency, tenant, context, cost, and workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/adc3306d84df18e7f20f14e8ba59ed84e731a1cb23e2aa1cbafd964946dace2a.webp)

ACT 3 · OPERATING INFERENCE AT SCALE

Scheduling GPUs is no longer enough

OLD SCHEDULER
CPU
Memory

Two axes. Bin-packing.

MODERN SCHEDULER
Model | GPU Memory
Latency SLA | Tenant
Context Size | Cost
Reasoning Workflow

Agent-aware scheduling: place work where the whole workflow will finish fastest and cheapest - not where one GPU is free.

### Optimization is decision making

The framework organizes inference optimizations by what they do to work: caching avoids it, batching shares compute, routing moves it, and queues delay it until a better moment.

[10:22](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=622s) [Four optimization choices: avoid work through caching, share it through batching, move it through routing, or delay it through queuing.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2ef680f95699d033e480ace0b00c76863f524869f2741b31b4dae7de441cd36.webp)

ACT 3 · OPERATING INFERENCE AT SCALE

Optimization is decision making

Can we AVOID work?
→ Cache (prompt · prefix · KV · response)

Can we SHARE work?
→ Batch (continuous / in-flight batching)

Can we MOVE work?
→ Route (region · model tier · accelerator)

Can we DELAY work?
→ Queue (priority · admission · defer)

Every optimization is really a question about the work itself.

### The economics of tokens

Inference cost includes memory, network, retries, failures and storage alongside GPUs. The speakers recommend optimizing cost per successful task: it captures what users care about beyond the cost of a token or request.

[11:44](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=704s) [Inference cost components and four metrics, highlighting cost per successful task.](https://assets.ai.engineer/sites/aiecode2025/sha256/f2460903dd858fb50219db7e3d6e4cfbae36cc413f48a3e448b17bae32db6a92.webp)

ACT 3 · OPERATING INFERENCE AT SCALE

The economics of tokens

Cost = GPU + Memory + Network + Retries + Failures + Storage

Cost / Token
Serving-runtime efficiency

Cost / Request
What the product actually charges

Cost / Reasoning Chain
True cost of an agent turn

Cost / Successful Task
What the user actually paid for

### Reliability means preventing cascades

A GPU failure can trigger retries, queue growth and timeouts, amplifying load until the system collapses. Circuit breakers, admission control, load shedding and retry budgets can interrupt the cascade.

[12:34](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=754s) [A cascade from GPU failure through retries, queue growth, and timeouts to system collapse.](https://assets.ai.engineer/sites/aiecode2025/sha256/4948e60c94e0edf9394e3dcd554bfbf98ed2ad170594a7b0c299ef1ca31d54e4.webp)

Reliability means preventing cascades

ACT 3 · OPERATING INFERENCE AT SCALE

GPU failure → Retry storm → Queue growth → Timeouts → More retries → System collapse

Feedback loop - small failure, big blast

### Observability enables intelligence

Telemetry feeds analysis and decisions that change scheduling; measurement closes the loop. Signals such as time-to-first-token, GPU HBM utilization and success-per-dollar help guide the next decision.

[13:41](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=821s) [A continuous control loop from telemetry through analysis, decision, scheduling, and measurement, with key inference metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/0469fb2566f10d7872bad707a259925c67c6f6e69a171a5a3a7e78f0c63dc6c9.webp)

ACT 3 · OPERATING INFERENCE AT SCALE

Observability enables intelligence

Telemetry → Analysis → Decision → Scheduling → Measurement
Continuous control loop

Signals that matter
• TTFT · time-to-first-token
• TPOT · time-per-output-token
• Batch fill rate
• KV cache hit rate
• GPU HBM utilization
• Queue depth per tenant
• Success-per-dollar
• End-to-end trace latency

### The impossible triangle

Larger batches can improve throughput and cost efficiency but may hurt tail latency. Each scheduling decision moves the operating point within the latency, cost, and throughput triangle.

[16:19](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=979s) [A triangle connects latency, cost, and throughput, with a point inside representing the current operating position.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5b86c181b4335f6b68a1bc02179dbb666bc9ed752b93c1ea10959ffc8348d05.webp)

ACT 3 · OPERATING INFERENCE AT SCALE

The impossible triangle

LATENCY

you are here

COST
THROUGHPUT

Optimization is continuous, not static. Every scheduling decision moves the system somewhere inside this triangle.

### The inference control plane

The proposed control plane brings routing, scheduling, and reliability into one layer. It decides which models serve requests and how requests are batched.

[17:01](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1021s) [A diagram connects control-plane responsibilities to an inference control plane, which manages the GPU fleet, models, and applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/728227a4a5e497ff3a4666113386476fd662088d87e08fea86964384d7439c84.webp)

The inference control plane

ACT 4 · THE FUTURE

RESPONSIBILITIES
Routing
Scheduling
Autoscaling
Cost Optimization
Observability
Policy
Reliability
Admission Control
Multi-Tenancy

INFERENCE CONTROL PLANE

GPU Fleet
Models
Applications

Models become resources. The control plane decides how they are used.

### Lessons from operating AI infrastructure

Production failures can stem from routing, scheduling, or capacity; adding GPUs can hide the problem. Scheduling and batching shape fleet efficiency, while control loops let the platform detect and adapt automatically.

[17:48](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1068s) [Five lessons cover infrastructure bottlenecks, elasticity, scheduling, visibility, and automated control loops.](https://assets.ai.engineer/sites/aiecode2025/sha256/6431fa1735fb3da6a7c741389da0a9ce1e2ddfde7703271b7d77e2abc04ade61.webp)

ACT 4 · THE FUTURE

Lessons from operating AI infrastructure

1. Infrastructure bottlenecks appear before model bottlenecks
2. Elasticity beats overprovisioning
3. Scheduling decisions dominate efficiency
4. Visibility precedes optimization
5. Control loops outperform manual operations

### What’s next for AI infra

The speaker frames AI infrastructure as shifting from better models and serving to orchestration, where GPUs, memory, and cache are scheduled and controlled as resources.

[18:46](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1126s) [A sequence moves from better models in the past to better serving in the present and better orchestration in the future.](https://assets.ai.engineer/sites/aiecode2025/sha256/abcf8959d5176b02a770b24623b9d5ce6419cb429af11201afcde5775a069f09.webp)

ACT 4 · THE FUTURE

What’s next for AI infra

PAST
Better Models

PRESENT
Better Serving

FUTURE
Better Orchestration

Future will be about making infrastructure smarter

### Thank You

[19:27](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1167s) [Closing slide reading “THANK YOU” on a dark background.](https://assets.ai.engineer/sites/aiecode2025/sha256/92b2ed748b3cf3d0213db7e6243103627112c25aaad75c7ea13e8ba82da4ee9e.webp)

THANK YOU

### AI Engineer World's Fair

[19:46](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1186s) [Black title card with white AI Engineer World's Fair text and the event website.](https://assets.ai.engineer/sites/aiecode2025/sha256/32a4795fe9d390f8d4ad8baf91ed5c6df08e2e3a140c553e60d293e49fe831d8.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:12](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=12s) Good morning, everyone. Welcome to the first inference talk on the last day of AI Engineer World Fair. My name is Nishant Gupta, and I'm joined today by my co-speaker, Naman Ahuja. We work on building the efficiency, training, and inference infrastructure at Meta. Today, we're going to be talking about how do you operate distributed inference systems at scale. As we all know, inference is no longer just a research artifact boiled to a product. It's a foundational hyperscale infrastructure workload, which is growing at a

[0:42](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=42s) tremendous rate. The inference traffic already outpaces the largest microservices in the world, and the rate of growth is fastest of any workload we have ever seen.

[0:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=54s) Let's rewind back to around two thousand and eight and try to compare the AI era with the cloud era. In around two thousand and eight, the cloud started as virtual machine offerings. The interesting engineering was virtualization. Then over time, the value moved up to the stack to the schedulers. Think Borg, Kubernetes, Mesos. Then to service meshes, then to autoscalers, and then to various platforms that were built on top of it. The orchestration layer is what actually captured the value and the complexity.

[1:24](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=84s) AI is on the exact same trajectory, but just compressed into the last few years instead of a decade. We started with simple models running on GPUs. Then we saw the evolution of model serving frameworks like vLLM, TorchServe, Triton, and now we are watching the orchestration layer emerging in real-time, addressing the complex challenges of routing, KV cache management, prefill decode disaggregation, and multi-model multiplexing. In this next phase of AI era, it's not just about the best

[1:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=114s) models or the kernels or the optimizations, it's about the whole ecosystem. It's about the control plane and the orchestration, and this is what we're gonna be focusing in this today's talk.

[2:06](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=126s) So let's talk a little bit about agentic demand explosion. In the classical web serving, before the AI inference workloads kicked off, the capacity scaled roughly linearly with users, depending upon the workload type. Double the users, mostly double the QPS, double the infrastructure fleet if there are no optimizations, and capacity planning was mostly a spreadsheet exercise. In this new agentic serving, capacity is scaling with number of users, time, number of calls per users, times number of tokens, which varies

[2:36](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=156s) depending on the model, the optimizations, the hardware SKU you have. A, a chatbot can have one model call per turn, which is evolving now with ten to twenty for copilots and fifty for research agents, and now it's thou- thousands of such calls for autonomous workloads with no human in the loop. The key to co-- takeaway that you cannot plan capacity for the agents the same way we did for microservices. We need to think about elasticity and implement workload-aware scheduling and admission control.

[3:09](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=189s) Now, let's try to dive a little bit deep into the differences, pros and con-- differences between the traditional microservice in serving versus the modern inference serving across these key dimensions. The request shape. Micro services assumes short, uniform request, wherever the LLM request we are seeing in our workloads, they can vary from fifty tokens to a hundred thousand tokens with vastly different compute profiles between prefill and decode stage. For batching, classical stacks for microservices was doing stacking, batching mostly at the load balance layer,

[3:39](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=219s) if at all. However, LLM serving requires continuous in-flight batching. How-- otherwise, the throughput collapses by an order of magnitude or more. State. Most of the classical microservices were stateless when we talk... not talking about storage layer. However, LLM serving does require a huge per request state, the KV cache, which require... which is very expensive to build and even more expensive to throw away. Scaling units. When we think about traditional microservices, we could run them in cheap

[4:09](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=249s) CPUs, in pods. However, for model inferencing, we're required to run on GPUs, which are a hundred times more expensive, which are ten times slower to acquire, and we cannot overprovision them casually, otherwise it will lead to a huge wastage. Failure mode. When you think about traditional microservices, like most of us have built over the last couple of years, we could, even if a micro-- e-even if a host crash or a pod crash, we could restart it. We could rebuild the state if needed be. However, for the model inferencing, it takes a huge

[4:39](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=279s) amount of time to go from cold to hot startup. And these, if any for GPUs mid-decode, it can drop thousands of in-flight tokens, and it can lead to a queue buildup. The takeaway is that the bottleneck is not just the model, it's the orchestration itself.

[4:58](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=298s) Now, as we can see in these hidden decisions behind the prompt, when we go to an agentic application, it requires a bunch of steps which are behind the scenes. We have to authenticate. We have to choose a model depending on the request type. We have to select the region where it goes. We have to do admission control. We have to do caching lookup. We have to do the... We have to run it on a GPU. We have to do batching. And there's a bunch of other steps involved. And as we can see, out of all these steps, only one step requires the model, which is the prefill decode if it's disaggregated inference or just

[5:28](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=328s) if it's not disaggregated inference. Other steps require infrastructure. The intelligence might lie in the model, but the economics, the reliability, and the user experience are all in the infrastructure. And this is why a lot of platform teams across a lot of companies are having much more impact on the product quality and the success rat-- uh, b- much more than before.

[5:50](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=350s) Now, most of us in this room have a deep expertise across one or two or three layers. We might own kernels or kernel optimizations. We might own routing or the product itself, or we might be operating the GPU infrastructure or the cluster itself. But very few of us have operated the whole stack or see-- thought about it end-to-end. As you can see in these layers, n- these layers are not new. They have been around for twenty years or more. What is new is the combination and the coupling between them. A decision at the routing layer

[6:20](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=380s) can change the cache hit rate at the model layer, which can change the batch composition, which can change the GPU utilization, which can change the autoscaling decision because of the change in GPU utilization. So everything is entangled. When we have any regressions in our inference workloads, it's not just about understanding what happened at the caching layer or admission control. We need to think about the stack top to bottom. And when we see any bottleneck, it's very, very im- important to understand at which layer is that bottleneck so that we can invest

[6:49](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=409s) properly.

[6:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=414s) Now, diving a little bit deep into a, how a prompt works. When we have a prompt for any application, be it just if you want to generate an image, or if you have a research task, or if you have complex multi-agent orchestration, more or less, it involves a bunch of these steps. The prompt goes to the gateway, then it goes to the router, after which it does the cache lookup if the request was already seen before. Then it goes to the schedulers, which decides on which GPU cluster it should run, dep-- on which hardware. It can be on NVIDIA or AMD or your

[7:24](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=444s) in-house silicon chip. Which then goes to the appropriate serving runtime, vLLM, SGLang, or whatever we have-- we are working on. And then we stream the response back to the user according to the SLO profiles of time to first token and time between each token, and while making sure the throughput is what the user desired. Now, as we can see, this inference behaves like a distributed transaction. Each arrow in this diagram is a network hop. Every of-- every one of these hops can retry, it can time out, it can fall back, it can even fail.

[7:54](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=474s) And each of these hop will have an SLOs, and it will-- it, it is streaming back to the user. So if it fails, the partial fail semantics are much more harder to deal with than if for a regular RPC call. Think about what happens if we have already streamed two hundred tokens back to the user, and suddenly a GPU host is preempted due to a scheduled or a planned or unplanned maintenance event. We cannot just retry. We have to think about it holistically. This is where-- this is why reliability-- we cannot build reliability at the edge. It has to be a property of the control

[8:24](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=504s) plane. Because the control plane is the one which sees the whole workflow.

[8:31](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=511s) Now let's talk about schedulers and some of the optimizations, uh, and how we can think about it. So for the traditional microservices, we used to think about bin packing of traditional microservices across three, four dimensions. Could be across resource utilization, CPU, memory, or could be across fault domains, depending upon if you're using AWS, if you're using your own in-house cloud providers. But for inference, the scheduler needs to be aware a- of-- across at least seven axes when we schedule a particular request. It has to be aware of the GPU type. There can be N

[9:01](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=541s) number of heterogeneous hardwares in your cluster, H100 versus A100 versus B200, with different network topologies. It has to be aware of the HBM headroom, KV cache state, the model weights, whether they are already loaded, whether they are cold, or we have to-- whether they have already warmed up, we have to cold start it. It has to be aware of the tenant priority. There can be N number of tenants running on that multi-tenant cluster with different SLO profiles. We have to also be aware of the workflow context. Are we in the third step of

[9:31](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=571s) reasoning that has already spent X dollar, or are we at the initial stages, and we can terminate the workflow if it-- if we are over-provisioned? We have to also think about latency budget, depending upon the type of agentic application we are building. So this brings us to the idea that we have to make sure that we implement agentic-aware scheduling. We have to make sure that we place the work which will, which will finish the, in the fastest and the cheapest time, as opposed to just placing it on a random GPU. A concrete example might be that the scheduler

[10:01](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=601s) needs to be aware that request R is at step three of a five in a workflow, and it has already st-- and step one and two has already spent X plus Y dollar. So if a step three fails, the whole workflow will be terminated, and we have wasted all that compute resources. That's why workflow-aware orchestration is very, very important, because it, it will change the admission decisions, the priority, and how we retry.

[10:25](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=625s) Now let's talk about optimizations. I will not go too deep into a lot of optimizations. There's a lot of research which has already been done outside. But I would like to share a framework which at least I like to use when it comes to it. And, uh, we can place it into four quadrants. First, can we avoid the work? Meaning, can we skip it entirely through caching, through techniques like prefix caching, response caching, semantic caching. The second, can we share the work? Can multiple requests share compute through batching? Think

[10:55](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=655s) contentious batching, prefill decode, chunk prefill, speculative decoding. Third, can we move away the work? Can we send it somewhere to a cheaper model or closer to the user through the-- mostly routing. Can we route it to a smaller model or a cheaper region or some other techniques? And lastly, can we delay the work? Can we wait for a better moment through admission control and queuing, which requires us to understand the priority classes of these requests and implement deadl-deadline-aware scheduling.

[11:25](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=685s) Now, this framework is very powerful because it transfers to various stacks. You might be using vLLM or SGLang or TensorRT, but every technique fits into one of these quadrants. So whenever we think about any optimization to our model, we have to apply-- we have to do a comparison and contrast with the previous techniques and see how all these stack with each other.

[11:47](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=707s) Now, whenever we think about scale, it's not just important to think about the performance of the model. We have to think about the cost economics as well. This is where it's important to understand what metric we are trying to optimize. Because the cost is not just the cost of the GPU or the model, it is the cost of all these parameters, retries, storage, failures, network, and of course the operational cost of development and all that stuff. What is important is to understand what is the key per-- key performance indicator for your product, which will add value to the users.

[12:17](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=737s) So it's not important to optimize just cost per token or cost per request. We have to optimize cost per successful task because this is what actually users care about. And if you're able to optimize that, the cost for the overall product decreases and the users are much more happier.

[12:36](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=756s) Now let's talk about reliability, a little bit about reliability and, uh, what it means to prevent cascading failures. Now, the failure story is never a GPU is preempted or a GPU has died. The interesting story is, is the feedback loop that follows. So a GPU can degrade, the latency can rise, the client retries, the queue depth increases, the healthy GPUs will saturate, which will follow more retries, much more, more full regional failures. This is the classic cascading failures, but

[13:06](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=786s) for... with a twist for agentic application, the KV cache. We cannot just casually restart or re-reroute to a different cluster. A cold pool has to warm up before it can absorb traffic, during which the hot pool has to take on all that, all those requests. This is why it's important to design the loop breakers con-- very deliberately. The circuit breakers at the routing layer, the admission control, which ejects rather than just queuing, the load shedding tied to queue depth, not just CPU or memory utilization. And we have to also think about retry budgets because if you

[13:36](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=816s) don't think about all these things, the cost can scale much, much more quickly.

[13:43](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=823s) Now I'll hand it over to my co-speaker, Naman, to talk about the remaining talk.

[14:56](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=896s) Okay, this works, I guess. Sorry for the hiccup. So once inference reaches production scale, it starts looking much more like a distributed system. We are no longer call-- just calling a model. It's more like a classic distributed, uh, system problem. So in distributed system, we talk about queues, scheduling, uh, autoscaling, fault isolation. These are some of the dimensions. Inference has all these problems, but there are new constraints now. Instead of CPU memory alone, we have CPU, HBM, KV cache, and cost per

[15:26](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=926s) successful task. So the operating question becomes how the platform know what to do next? And that's where observability comes into play. It's not just about dashboards. It's about, uh, how to provide input signal to the control loop. Telemetry feeds analysis, analysis drive decisions, decisions change the scheduling and routing, and finally, we just rinse and repeat. Um, I'll just give an overview of what are some important metrics. First one is time to first token, which tells me about, uh, how much time does it

[15:56](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=956s) really take to get the first response. Then we have a utilization ratio, which tells me whether memory or compute is the bottleneck. We have success per dollar that tests us whether platform is actually delivering and working efficiently. And finally, we have end-to-end trace latency that tells us what's the time being spent across the full request path.

[16:19](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=979s) There's a code trade-off between latency, cost, and throughput. You cannot just get all of them. It's pretty analogous to CAP theorem. If I increase the batch size, um, I improve the throughput and cost efficiency, but I may hurt tail latency. If I use speculative decoding, I may improve latency, but there are some extra compute. Ultimately, you know, we increase the cost per token. And finally, I can use a simple model, smaller model. I can reduce the latency and cost, but the response will be of low quality. Ultimately, I'll do

[16:49](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1009s) failure analysis, do retries, which brings the cost back up. So every serving decision moves system somewhere in this triangle, and our job is to find the perfect setting. It's just an optimization problem now.

[17:03](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1023s) This is where the industry is heading right now. Inference needs its own control plane. Everything we discussed, routing, batching, caching, scheduling, reliability, they cannot be a separate kno-knob now. They're converging into a logical layer. Let's call it inference control plane. We used to manage VM before in distributed system. We have autoscaling schedulers, and we had Kubernetes, which turned these into control plane. Inference is going through the same transition now. Uh, models are becoming resources. GPU, KV cache,

[17:33](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1053s) token latency, cost are now scheduled around. The control plane decides which models serve which request and how it is batched. So whether we build this layer internally, use open source or from a vendor, the key design is to assume layer will exist. Now let's discuss about what are some of the operating lessons we have experienced in AI infra and how they are applicable here. The first lesson is that infrastructure bottlenecks usually show up before model bottlenecks. In production, many

[18:03](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1083s) failures can come, but they can be just about scheduling and routing or capacity breakdown. So these are not related to inference. It's about infrastructure problems. Then we have elasticity. We need elasticity in a system. We can have more GPUs, but this will not really solve the problem. We are just hiding the problem. Then our solu-solution to schedule the decision overpowered efficiency. The same fleet can deliver very optimally depending on how you're scheduling it or how you're batching it. Then we have control

[18:33](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1113s) loops, which beats the manual process. The platform has to sense, detect, and automatically adapt to the system. So main takeaways, do not optimize for tokens. Optimize for successful task.

[18:49](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1129s) So this is a broader shift I want to leave you with. The first phase of AI infrastructure was about better models. We invested a lot of time in improving our models, smarting-- making the models smarter, and coming with better benchmarks. The current phase right now is faster inference, lower latency, better batching, and better GPU utilization. But the next phase is about orchestration. That means GPU, memory, cache, and everything. These are just resources, and they need to be scheduled and controlled. The teams that understand this early on

[19:19](https://www.youtube.com/watch?v=7c9FSUVcXR0&t=1159s) will build infrastructure for the future. So the closing idea is infrastructure is no longer a server problem, it's an orchestration problem. Thank you.
