AI Engineer Code 2025
What Makes Open Models Fast in Production — Sujee Maniyam, Nebius
Loading slides…
Read the talk
What Makes Open Models Fast in Production
Dylan Bristot and Sujee Maniyam explain how Nebius Token Factory connects production data to model improvement, then walk through the hardware, routing, caching and decoding choices behind fast inference.
From a talk by Dylan Bristot and Sujee Maniyam
At a glance
Ideas worth remembering
A production model needs a recurring path from inference logs through dataset preparation and post-training back to deployment.
Serving the same model on different hardware and engines can produce different performance and cost; engine selection belongs to the model-specific optimization work.
Cache-aware routing improves reuse by sending requests toward GPUs that already hold useful cached state.
Speculative decoding pairs a fast draft model with a large verifier; custom training uses application traffic to make the draft more useful.
Cache offloading preserves reusable work outside GPU memory, while prefill/decode separation assigns different resource demands to separate GPU groups.
Quantization requires experiments to balance serving efficiency against model quality loss.
The model is only the beginning of the production job
An easy model API can become a difficult production dependency. Dylan Bristot, who leads product marketing for Nebius Token Factory, opens with the choice facing teams that want to control model behavior and operating cost: use a closed API, or take responsibility for serving an open model themselves. Sujee Maniyam, a Nebius developer advocate, joins him to explain the engineering underneath that choice.
- Closed APIs: They make the first integration easy. Bristot’s concern is the ceiling afterward: limited ability to tune the model for a specific task, little visibility into shared infrastructure, and costs that grow with usage.
- Self-hosting: It gives the team control over the model and serving setup. That control also brings a substantial engineering project and an ongoing responsibility to keep it running.
- Managed open-model inference: Token Factory’s proposed middle path is to handle deployment and serving optimization while customers concentrate on their products and the behavior they need.
The infrastructure matters to this proposition. Nebius describes a stack extending from its own data centers and NVIDIA systems through bare-metal capacity, managed cloud and model serving. Owning those layers gives the inference team room to change how a model runs, rather than treating the hardware underneath an API as someone else’s fixed constraint.
Turn production completions into the next deployed model
Token Factory connects four activities that teams otherwise have to stitch together. At the time of the recording, Bristot describes more than 60 hosted models, including GLM, Kimi, DeepSeek and Qwen, with dedicated endpoints, structured outputs, function calling and batch APIs. Those endpoints start the cycle by producing real application traffic.
- Data Lab: Import inference logs and completions, then filter, version, export and batch datasets. This turns accumulated responses into material a team can inspect and use.
- Post-training: Use production logs or synthetic datasets for LoRA or full fine-tuning, distillation, custom speculative decoding, and model-specific quantization and calibration. The goal is to make the model better suited to the application’s expected behavior.
- Deployment: Push the resulting model onto the owned infrastructure so the improvement reaches the product and produces the next round of experience.
How does an application response become a model update? The cycle below shows the handoffs. Completions enter the data layer; selected data feeds post-training; the trained model returns to production. The useful relationship is the return path: deployment creates new experience to inspect, so serving a model becomes an ongoing improvement process.
Keeping that cycle working also requires less glamorous operations. Token Factory groups its responsibilities into engine-level optimization of latency, throughput and cost; operation of the underlying infrastructure; and enterprise requirements such as security, compliance and reliability. Customers still care where the model runs, which chips serve it and how much it has been quantized. A managed API hides operational work without making those choices irrelevant.
Production experience feeds the next deployment
Read the diagram as text
- Inference. Run models and produce application completions.
- Data Lab. Import, filter and version production datasets.
- Post-training. Customize models using logs and synthetic data.
- Deployment. Push the improved model into production.
- Inference → Data Lab: Logs and completions.
- Data Lab → Post-training: Prepared datasets.
- Post-training → Deployment: Updated model.
- Deployment → Inference: New production experience.
Match the model to the hardware and serving engine
Maniyam begins the technical half with model quality. An Artificial Analysis chart compares open and proprietary models, and he uses it to show that open models can be competitive, sometimes ahead of proprietary alternatives. That comparison supports considering open models for intelligence as well as cost; it does not establish parity for every application. Their portability also gives customers more choice about who serves them.
The hosted models have different roles. Large, capable models can act as teachers for smaller student models. That gives post-training a route to a smaller serving target: use the large model’s capabilities to help train a model that is less expensive to run. Later, a different pairing of small and large models will accelerate generation itself.
Serving performance then depends on matching several layers. Nebius optimizes kernels and runtimes for its hardware, including adoption of NVFP4, NVIDIA’s four-bit floating-point format. Above that, Token Factory uses open-source serving engines and an internal fork that it continually modifies. Models do not perform equally well on every engine, so engine selection happens per model rather than through one universal choice.
The customer sees an API, but the provider chooses the GPU, runtime and engine combination behind it. This helps explain Bristot’s earlier observation that the same open model can have different operating costs on different providers: identical model weights do not imply identical serving behavior.
Route requests toward the work already cached
A random load balancer misses something important about LLM workloads. Early requests might have had a short prompt and a short answer. Coding agents now send large codebases for refactoring, summarization or analysis, and a large input can lead to either a short or a long output. Request count alone does not describe the work being assigned.
Consider the coding workload Maniyam introduces. A request reaches a GPU that may already have useful cached context—or may have none of it. With random routing, requests scatter across GPUs without regard to that reusable work. The result is fragmented caches and missed opportunities to reuse them.
Cache-aware routing changes the destination decision. The router knows where cached material is stored and sends requests accordingly. In Maniyam’s comparison, scattered cache colors become grouped together: related work reaches a location where it can benefit from existing state. The observable change is better cache locality, which supports more cache hits and faster inference.
Let a small model draft and a large model verify
Speculative decoding targets the cost of generating tokens one after another with a large model. A smaller, faster model proposes tokens, and the large model verifies the proposed work. Maniyam compares it to a senior engineer handing work to a junior engineer and then checking the result.
The speedup depends on how much of that draft the large model can use. Accepted work avoids having the large model generate everything itself. Rejected work requires regeneration, reducing the benefit of drafting. The small model therefore helps most when its proposals suit the large model and the application’s traffic.
That creates a reason to train custom draft models. Maniyam reports up to 30% improvement from draft models trained with generic data and says application-specific data can improve the result further. The recording does not define the improvement metric or evaluation conditions, so the figure is a reported experimental result rather than a general speed guarantee. He describes a launching feature that captures production data and trains a draft model with nearly one-click setup, connecting the earlier data loop directly to serving optimization.
Keep reusable context, even when it leaves GPU memory
KV caching addresses repeated work during generation. Maniyam’s example starts with the prompt “The cat sat on” and then generates the next token. As the sequence grows, its existing context remains useful for later steps. The cache retains reusable state associated with that context; the next step looks up retained work rather than rebuilding everything already processed.
The example makes the distinction useful: caching supports the next generation step, rather than choosing all future words in advance. Maniyam reports speedups of five to ten times from caching, but does not specify the workload or baseline for that range. Long inputs and large context windows also make the retained cache consume substantial memory.
GPU memory is valuable, so retaining every cache there is costly. Token Factory automatically offloads cached state to regular memory and brings it back when needed. In the cat-prompt example, the important change is where the retained state lives: leaving GPU memory need not mean discarding the work used to build it.
The next optimization separates two phases of inference. Prefill processes the input context and calls for compute-efficient execution. Decode generates the continuation and places heavy demands on memory. Running both on the same GPUs makes them compete for resources; Token Factory assigns them to separate GPU sets and transfers the KV cache between them.
What connects the two GPU groups once their jobs are separated? The diagram shows the KV cache as the handoff. Prefill prepares context state; decode receives that state and continues generation. Separation lets each group suit its phase, while making cache transfer part of the serving path.
Separate GPU groups share context through the KV cache
Read the diagram as text
- Prefill GPU group. Process the input context with compute-efficient execution.
- KV cache. Carry retained context state between phases.
- Decode GPU group. Continue generation with memory-efficient execution.
- Prefill GPU group → KV cache: Context state.
- KV cache → Decode GPU group: Transfer for decoding.
Find the precision that preserves useful quality
Quantization reduces numerical precision to make a model run more efficiently. Reducing it too far can degrade model quality. Token Factory experiments to find a sweet spot: an efficient representation whose quality loss remains acceptable. This is a model-serving decision that needs attention alongside engine choice, caching and decoding.
The closing recap brings the mechanisms together. Speculative decoding reduces the generation work assigned to the large model. KV caching retains useful work, cache-aware routing sends requests toward it, and separate prefill and decode groups divide competing resource demands. The API can conceal all of these operations, even as they determine how well very large models serve production traffic.
Maniyam ends by returning to coding as a practical use case. He describes using hosted open models for coding and incorporating them into Cursor or OpenCode, then invites the audience to try the platform and share feedback. The talk’s final move is from hidden serving machinery back to the application: these optimizations matter because they make capable models more useful in the tools people already use.
Full transcript
Loading transcript…