Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

An inference platform must serve customers who want an endpoint and customers who want control over their GPUs. Sitanshu Gupta, about four months into leading inference at CoreWeave, begins with those consumption choices. They determine which responsibilities the platform must absorb and which decisions it must expose, while supporting small and large models without repeatedly redesigning the stack. 0:13

The choices divide responsibility in three useful ways:

• Serverless: Customers use an API or UI and pay per token. CoreWeave manages hardware, clusters, and orchestration; the model catalog determines which models customers can use.
• Dedicated inference: Customers know which hardware they use and control their model deployments. CoreWeave supplies orchestration and the platform features needed to tune performance, while deployment and performance choices remain with the customer.
• Provisioned throughput within serverless: Customers supply a known traffic profile, and CoreWeave carves out capacity behind the scenes to meet throughput and service-level agreements. Hardware remains hidden, and billing remains per token. 1:42

Provisioned throughput addresses a concrete serverless failure mode: everyone requests the same model, the capacity behind it becomes crowded, and requests time out. A customer with predictable traffic can reserve serving capacity without taking on GPU operations. Dedicated inference becomes useful when choosing the hardware and deployment itself matters. 2:13

Input and output lengths alone do not describe an inference workload. Agentic and chat requests often both have long inputs and short outputs, yet their next turns arrive differently. A person reads an answer before asking another question. An agent can continue immediately. That difference changes how conversation state must remain available between requests. 3:20

Four workload shapes put different demands on the same infrastructure:

• Agentic: Long inputs, typically short outputs, and little delay between turns make repeated input processing and cache reuse especially important.
• Chat: Similar sequence lengths, with human reading time between turns. Cache management must account for those pauses.
• Voice and video: Steady streaming makes latency a central requirement. Delayed processing interrupts the stream.
• Batch: Loose completion windows permit work to wait. Gupta describes customers offering ten to twelve hours of work and allowing the platform to process it whenever capacity becomes available. 3:50

The short gap between agent turns is distinct from the overall serving requirement: Gupta characterizes agentic and chat demand as largely throughput-oriented, while voice and video are especially latency-sensitive. Batch adds scheduling freedom. Fitting these workloads together over time becomes a game of “Tetris”: flexible work can occupy capacity that urgent work does not need. 4:50

A request begins in an application, notebook, or agent and reaches a gateway. The shared platform handles authentication and authorization, rate limits, usage tracking, and observability. Usage supports billing; observability checks whether serving behavior meets the signed service-level agreements. 5:12

Below those services, inference engines such as vLLM, SGLang, and TensorRT-LLM run across different generations of NVIDIA GPUs. The platform must distribute work across that heterogeneous hardware rather than assume every serving deployment has the same capabilities. 5:42

For serverless requests, billing monitors token usage rather than token contents, which Gupta connects to CoreWeave’s zero data retention policy. The router distinguishes provisioned-throughput customers from shared customers: reserved customers target their explicit deployments, while multitenant requests use separate deployments. Within the eligible deployments, cache-aware routing seeks useful computation from earlier requests. 6:42

The expensive repetition sits in prefill: processing the input before generating the response. Gupta estimates that, depending on the customer, 80 to 90 percent of the input in agentic requests repeats across requests. Recomputing that shared portion spends compute on work already done. A KV cache preserves reusable input-processing state, allowing a cache hit to avoid repeating the corresponding prefill. 7:42

That reuse has an economic consequence. Prefill is compute-bound and expensive, so cached input tokens can be priced much lower than fresh input tokens. The router’s destination affects whether the request can reuse earlier work: finding available compute is only part of the job; finding the relevant cache matters too.

Separating prefill from decode—the phase that generates output—is another deployment option. CoreWeave supports prefill-decode disaggregation but does not make it mandatory: Gupta cautions that it is not cheap for every use case. The platform exposes the choice so the deployment can match the workload. 8:12

Dedicated requests enter through an isolated private gateway and are billed per GPU per hour. They retain the same cache-aware routing logic, while customers choose the inference engine, whether to separate prefill and decode, and how to divide reserved capacity. One model deployment can scale across the cluster, or the capacity can host several models and deployments. Owning the hardware allocation does not remove the value of reusing computation. 8:42

Routing becomes harder when capacity spans different GPU types, zones, and regions. CoreWeave’s stated priority is KV cache locality first, with the least-loaded destination as the fallback. This ordering expresses the value of avoiding repeated input work: a deployment’s current load alone does not tell the router how much computation a request will require there. 9:42

The Tetris analogy becomes concrete with a dedicated customer serving real-time traffic during US daytime and running batch jobs in the evening and overnight. The GPU capacity remains the same; its assignment changes. The API lets the customer specify when serving deployments should scale up and down. At the scheduled scale-down, capacity becomes available for batch processing through the night. 10:12

The mechanism depends on batch work tolerating delay. A streaming request needs attention when it arrives; a batch queue can wait for the scheduled opening. That flexibility lets one allocation support two traffic patterns and increases the time its GPUs spend doing useful work.

Return to the chat example. The first question has been processed and answered; now the user pauses to read. If the system completely evicts that conversation’s KV cache during the pause, the next question must pay for the repeated prefill again. The observable consequence is a longer wait when the conversation resumes. 11:35

CoreWeave instead describes offloading the cache to high-bandwidth storage. When another request arrives for that conversation, the stored state can return to GPU high-bandwidth memory, or HBM. Storage preserves the earlier computation outside GPU memory during the gap between turns. Gupta describes an internal implementation and mentions LMCache among external approaches to this problem. 12:05

What changes when the user asks the next question? The diagram follows the same conversation through the pause and contrasts eviction with offloading. Both paths resume inference, but the offload path preserves the earlier prefill for reuse. The relationship separates two cache problems: routing finds reusable state, while offloading keeps it available across time.

Cache reuse avoids input work. The closing performance discussion adds other deployment levers:

• Four-bit quantization: Gupta identifies this as one of the team’s two biggest recent levers.
• Speculative decoding: A speculator proposes output that the serving model checks. Acceptance length describes how much proposed output survives that check; longer accepted proposals let more output advance together, improving throughput.
• Parallelization choices: The degrees and strategies used to distribute inference work require careful selection alongside the engine and prefill-decode arrangement. 12:36

Customers can supply datasets for training custom speculators. The process is asynchronous: CoreWeave receives the data, trains the speculators, and deploys them into the customer’s serving deployments if requested. Training happens outside the live request path. The intended causal chain is to adapt the speculator to the customer’s data, improve acceptance lengths, and increase output throughput. 13:06

The results discussion distinguishes benchmark traffic from production traffic. Gupta describes strong Kimi leaderboard results on Artificial Analysis, then uses GLM results from OpenRouter to bring actual user traffic into the comparison. He identifies the Weights & Biases-branded deployment as CoreWeave’s and describes its speed as close to Fireworks Fast. These are workload-specific comparisons: numerical results and matching test conditions are not supplied here, so they do not establish a general speed or price advantage. 13:35

The closing objective is price performance. One platform supports serverless and dedicated consumption, including pay-as-you-go and provisioned throughput within serverless. Its improvements compound through different mechanisms: reuse repeated input computation, preserve cache between turns, schedule flexible work into available capacity, and tune generation performance. The customer’s workload determines which combination earns its cost. 14:37