What's New in Inference Engineering — Philip Kiely, Baseten

Why write a book about something that changes this quickly? Philip Kiely published Inference Engineering on February 23, 2026, because many serving principles remain useful across successive model generations. This World's Fair talk is its first public addendum: what changed afterward in quantization, caching, and speculative decoding. 1:14

The first distinction is what the hardware allows you to do. Local inference and batched data center serving start with different constraints:

• Local inference: Fit the model onto the hardware available at home through quantization, distillation, pruning, or splitting it across GPUs. Then recover the intelligence damaged by compression, usually at a batch size of one. Kiely's shorthand is “make it less dumb.”
• Data center inference: Get a serving engine such as vLLM running, then “make it less slow.” With a batch size of N, the work turns toward cache-aware routing, speculation, and disaggregation.

These worlds can learn from each other, but an optimization needs to answer the workload's actual constraint. Kiely's focus here is data center inference. The local perspective will become useful when a smaller cache turns out to carry a substantial computational cost.

A conventional training-to-inference handoff treats the weights as a finished product. Training produces a model; inference engineers arrange its execution. Increasingly, a dedicated training process also produces the machinery that makes serving efficient. A learned cache compressor or a specially trained draft model becomes part of the inference system. 3:42

The resulting loop is appealing: faster inference produces more data, that data trains a better model, and the better model supports faster inference again. Kiely jokes that you keep going until you are rich. The practical work still centers on his “big three”: quantization, the KV cache, and speculation. Each changes a different part of the cost of generating tokens.

Quantization represents numbers in a smaller, less precise format. Moving fewer bytes and using cheaper arithmetic can improve time to first token (TTFT) and tokens per second (TPS). The benefit depends on the hardware, and reduced precision can damage model quality. A smaller representation therefore starts the performance calculation; it does not finish it.

TurboQuant became widely discussed in March, shortly after the book appeared. Its polar-coordinate quantization approach allows the KV cache to use four bits. The KV cache stores keys and values: the computed representations that attention uses to consult earlier context. Baseten's performance team investigated whether compressing that memory would improve serving. 6:10

Follow the same cache through the change. An eight-bit representation becomes a four-bit representation, halving its storage requirement. Transferring the cache contents through system memory now moves half as much data, giving effectively twice the bandwidth for those contents. Decode—the phase that generates the continuation—must then perform additional computation in the forward pass to account for that representation. Kiely reports that this reduced TPS by more than half in the team's investigation. The workload and implementation conditions are unspecified, so that figure applies to their finding rather than every TurboQuant deployment.

The outcome was a smaller cache alongside slower generation. Baseten rejected that tradeoff for the production workloads discussed here. On memory-limited GPUs at home, especially with long context, freed cache space can accommodate longer sequences. Additional computation may be a reasonable price for making the workload fit. The same technique can be valuable or unacceptable depending on which resource is scarce.

The production strategy separates model weights from runtime cache memory:

• Weight quantization: Use NVFP4, inspect weaknesses introduced by quantization, and avoid flattening the model's output probability distributions. Preserving those distributions matters alongside reducing the weight representation.
• Cache placement and movement: Improve KV-aware routing, offloading, and sharing. NCCL and NVIDIA Dynamo are among the tools used to move cache data through the system, potentially offloading it to ordinary CPU memory.
• Other modalities: Explore applying NVFP4's benefits to image and video models as well as language models.

Cache reuse already saves work. Prefill processes the input prompt and computes the representations needed for subsequent generation. When another prompt has the same prefix, the system can reuse that prefix's cached results instead of computing them again. That makes serving more efficient without shortening the remembered context. Compaction asks whether the memory itself can become much shorter. 9:10

The ordinary KV cache grows linearly with the amount of context passed into the model. At million-token sequence lengths, that memory becomes substantial. Agent context compression, retrieval-augmented generation, search, and writing information to files offer ways to give the model a selected or compressed portion of a larger information source.

KV compaction seeks a middle ground: a much shorter cache with near-lossless retention of useful information. Attention Matching and Cartridges are promising approaches that optimize the compact cache at inference time. Still, from Baseten's research team, instead learns how to synthesize memory during training, then applies that learned transformation during serving. Kiely credits Charles O'Neill and Mudith Jayasekara from the post-training team and recommends their chalk talk for a deeper explanation. Near-lossless retention is the aim; the talk gives no compression ratio or quality result for Still. 10:42

Still uses a Perceiver bottleneck: a fixed set of learned query vectors cross-attends to the full KV cache and produces compact keys and values in a single forward pass. The language model can attend to this differentiable compressed memory as context. The queries learn how to synthesize a representation of the information; the result need not be a deterministic subset of the original cache.

Where does the shorter memory come from? The diagram shows learned queries and the full cache meeting at cross-attention. The full cache supplies the information, while the learned queries shape the compact output. Training prepares the transformation; one serving-time forward pass produces the keys and values the language model will use.

Speculative decoding supplies draft tokens for the target model to verify during a forward pass. Accepted proposals let generation advance by more than one token per target-model pass, improving TPS. Kiely describes the optimization as lossless: the target's verification preserves generation quality. The draft model proposes; it does not get the final say. 12:12

The history of drafting changes what builders ask the cheaper model to learn:

• SpecDec: Use a small model from the same family to generate draft tokens. Being a capable small model does not necessarily make it good at predicting the larger model's continuation.
• Medusa: Add decoder heads to the model to produce proposals.
• EAGLE 3: Train a billion-parameter model on the target model's hidden states specifically to generate draft tokens. Kiely identifies it as the leading method in February 2026.

Hidden-state training makes the drafter a specialized serving component. Its useful skill is agreement with the target, because every accepted proposal saves sequential target-model work. This is another instance of training for inference: training produces a model whose purpose is to accelerate another model.

DFlash changes the drafting schedule. Its diffusion language model proposes a sequence together, analogous to diffusion image or video generation working over pixels or frames. Rather than building the entire proposal through autoregressive, one-token-at-a-time generation, it works over a block. 13:42

A DFlash invocation might take two or four times as long as a sequential drafting invocation, but it predicts eight or 16 tokens at once. In Kiely's comparison, a single DFlash forward pass takes less time than the entire EAGLE draft phase and proposes more tokens. The unit of comparison matters: a slower individual invocation can replace a longer sequence of invocations.

Why does block drafting change the cost before verification? The diagram compares the two proposal paths. Sequential drafting accumulates a sequence token by token; DFlash produces a block together. Both still need target-model verification. The useful result depends on the complete draft-phase time and how many proposed tokens survive.

The block's tokens can attend to one another, helping proposals account for their neighbors. Training uses an attention mask for bidirectional drafting: the target supplies context, a subset of clean tokens is sampled within each block, and the mask enforces causal consistency. The constraint keeps the drafting arrangement consistent with causal continuation even while permitting attention in both directions within the block; the talk does not specify the mask entries needed to implement it.

Kiely reports a more-than-threefold improvement over EAGLE in a comparison using Qwen3-8B on a single B200, and describes gains in both token acceptance and TPS. The account does not clearly assign the multiplier separately to each metric, so it should not become a blanket threefold throughput guarantee. DFlash was already running in Baseten production.

DSpark appeared only days before the talk. It pairs a diffusion model with a sequential model, aiming to improve acceptance through their combined proposals. Its status differs from DFlash: Baseten was investigating this new research and had no production results to present. 15:11

Continuous speculator retraining already had production results. Draft acceptance depends on the prompts and responses a service handles. Repeatedly training DFlash on that live traffic makes the drafter better suited to the target's actual continuations. Kiely reports improvements from 20 percent to twice the token acceptance rate. Those are acceptance gains, which do not translate directly into the same multiplier for end-to-end serving speed.

The extra acceptance carries substantial operational costs:

• Storage and permission: Collect and retain prompts and responses, with permission to use the processed data for training.
• Compute and data movement: Retrain the speculator and move the information required by that process.
• Target changes: Change the speculator when the underlying model changes, keeping the drafting machinery suited to its target.

Scale determines whether this investment pays. A very large service can apply an improved drafter across enough generation work to justify repeated training. Kiely expects continuous retraining to become worthwhile for such systems. The feedback loop now has a concrete role: live serving data improves the component that helps serving go faster.

Kiely's forecasts draw on public information and his experience through the Ampere, Hopper, and Blackwell hardware cycles. Chips must ship, reach data centers, and acquire software support before their capabilities become useful serving performance. The installation and software work take time. 17:11

Three directions shape his expectations:

• Upcoming hardware's NVFP4 performance: Kiely expects particularly strong NVFP4 performance from the next hardware generation. This is a forecast, rather than a demonstrated serving result. Building confidence in running models in this format—including borrowing techniques from local inference—would prepare data center teams to use those gains.
• Disaggregation and system-wide communication: Early gains from prefill/decode disaggregation—separating input processing from continuation generation—make moving KV cache data between parts of the system increasingly important.
• Training for inference: Learned serving components will continue to matter, extending the pattern seen in compact memory and specialized draft models.

Kiely closes by offering free digital and paper copies of Inference Engineering. The digital download gives readers a practical way to follow the foundations behind this addendum. Identify the scarce resource, then count the work an optimization adds. TurboQuant saves memory but adds decode computation; DFlash spends more on one drafting invocation while reducing the complete draft phase; continuous retraining buys better acceptance with storage, compute, and data movement. Their value follows the workload.