Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

The service stays up. There is no crash, warning, or exception. Then a response comes back as gibberish—with high confidence. At AI21, Yuval Belfer and Asaf Gardin encountered this while training and serving Jamba, a hybrid model with attention and Mamba layers. Improving the model’s capabilities would not repair the failures they eventually found: the inference engine was computing with the wrong state.

The first case, the “imposter request,” appeared during GRPO reinforcement learning. A request should proceed from prompt tokenization to prefill, then decode, and finally back to text through detokenization. That ordering matters because decode continues computation from state established for the request. Jamba’s mixture of attention and Mamba made a violation of this sequence especially revealing.

Three features made the failure difficult to catch:

• Rare: Gibberish appeared roughly once in a thousand requests, allowing many ordinary tests to pass.
• Late onset: A few requests were insufficient. The engine needed a workload behind it before the symptom emerged.
• Engine specific: The team observed it in vLLM and not in the other inference frameworks they tried.

A prompt alone was therefore a poor reproducer. The surrounding execution history belonged to the problem.

Sending assorted prompts and small batches produced normal responses. Gardin’s next move was to change the operating conditions so the failure would arrive sooner. vLLM exposed a GPU memory utilization setting that controlled its memory budget, including space used for weights, activations, and caches. The team reduced it from 90% to 20% and ran many requests simultaneously.

Under that workload, one request repeatedly returned gibberish. Sampling at temperature zero let the team reproduce the same failing request in their setup. This changed the investigation: instead of waiting for an occasional bad answer, they could rerun the workload, change one part of the engine, and check whether the same request still failed. The workload supplied a stable target for the later breakpoint.

A repeatable bad answer still leaves a large search space: model behavior, kernel math, cached state, and engine scheduling. The team used Hugging Face Transformers as a reference because its implementation of their Mamba kernels was comparatively plain. vLLM’s implementation had accumulated changes to support its serving features. Comparing the two gave them a way to ask whether the optimized execution path assigned the same probabilities to the same tokens. 5:18

The comparison started with vLLM generation. For each prompt, the team retained the generated response and its logprobs—the logarithms of token probabilities. They then passed the prompt plus that generated response through the Transformers forward pass using prefill only. The reference implementation scored the sequence that vLLM had actually produced; it did not generate an independent answer that might wander onto a different token sequence.

The reference pass returned logits, which they converted through softmax into probabilities and used to compute comparable logprobs. Differences at corresponding token positions exposed where the two executions disagreed. Holding the sequence fixed made those differences useful: a discrepancy could no longer be explained merely by the engines choosing different continuations.

What data must remain shared for this comparison to work? The diagram follows one generated sequence into both sides of the check. The important relationship is the replay: Transformers receives vLLM’s tokens, while the comparison receives probability scores from both executions. This turns an impression of bad text into a token-by-token diagnostic.

The first suspects were the Mamba CUDA kernels. Inspecting the prefill math and the tensors before and after the kernel call revealed nothing wrong. NVIDIA Compute Sanitizer also reported no apparent memory problem in this investigation. Neither check exposed the failure that was actually occurring: valid computation over state belonging to an earlier request.

Next, the team separated prefill from decode. Mamba allowed them to route all the computation through the prefill kernel, avoiding the decode kernels entirely. The gibberish vanished. Decode looked guilty—but this experiment had changed the execution path as well as the kernel being used. It localized the symptom without yet identifying the faulty operation.

To investigate the failing request itself, they needed something the forward pass lacked: identity. By the time a request reached the kernels, it appeared as tensors, numbers, and matrices. A debugger could inspect those values, but could not easily answer which prompt they represented. The team added the request ID to a forward context and propagated it down to Mamba’s forward pass, just before kernel selection.

That small instrumentation change connected the reproducible response to the computation that produced it. A conditional breakpoint on the failing request’s ID let them inspect its metadata at the relevant call. On its first trip through the forward pass, the scheduler had selected decode before prefill. The suspicious kernel was being called before the request had established the state it needed. 9:16

Follow the failing request through that call. Earlier requests had already left data in the Mamba state cache. The fresh request entered decode, which read the existing state before computing over it. Its continuation therefore incorporated computations from requests that came before it. The result was gibberish even though the kernel performed its intended operations. The fault was in deciding when, and for which request, to call it.

The two layer types exposed different consequences of the same scheduling error:

• Attention writes before reading: In the path described here, the current token’s keys and values were written before being read. That overwrite prevented stale values from producing this symptom.
• Mamba reads before computing: Decode first read the cached state, then computed over it. Entering decode without the request’s prefill allowed old state to affect the new computation.

This explains why the hybrid model revealed the mistake; it does not make arbitrary scheduling errors safe for attention.

The fix restored the request lifecycle at classification time. If a request had zero computed tokens, the scheduler had to mark it as prefill so its first forward pass would take the prefill path. For the same fresh request, that changed the first operation from consuming leftover state to processing its own prompt before decode. Gardin reports that the fix was merged.

Where does the wrong request’s information enter the computation? The comparison below places the state read beside the corrected ordering. The failure happens before any new continuation is returned: selecting decode makes old cache contents an input. The scheduler fix changes that input’s history by requiring prefill first.

The scheduler repair did not end the investigation. A second failure appeared during reinforcement learning post-training: logprob spikes between rollout generation and the FSDP step. Crucially, the comparison happened before any weight update, with the same weights and inputs. The recurring spike therefore could not be explained by the model learning between the two evaluations. The expected agreement was breaking during execution. 11:46

With the default eight rollouts per prompt, the spike appeared every 12 steps. The team wanted a lever that changed the failure’s timing or location, giving them information about the mechanism as well as a faster reproducer. Increasing rollouts per prompt brought the failure forward. At 128 rollouts per prompt, it appeared on step one, eliminating the wait for steps 12 and 24.

The successful memory experiment from the first case suggested another test: reduce GPU memory utilization from 0.9 to 0.2. This time, the issue disappeared. That was useful evidence, but shrinking memory had hidden the failure rather than repaired it. The same knob had opposite diagnostic effects in the two cases.

The cause was an unsigned 32-bit index in the Mamba kernel’s cache-addressing path. When the offset passed roughly four billion, the index wrapped around instead of raising an error. The resulting value no longer represented the intended offset. A smaller GPU memory budget made vLLM allocate a smaller state buffer, so the cache index never reached the region that triggered the overflow. The bad arithmetic remained; the workload stopped reaching it.

The repair changed the index type from uint32 to size_t. On the modern architectures discussed in the talk, that provided an unsigned 64-bit range, and the team no longer encountered the overflow. size_t has an architecture-dependent width, so the practical fix relies on the wider range available in that environment; the type name alone is not a universal promise of 64-bit indexing.

The two bugs shared the Mamba state cache, but reached it through different mistakes. One request consumed another request’s state because the scheduler selected decode too early. Another computation exceeded its index range and wrapped around. Both produced silent probability anomalies, sometimes accompanied by gibberish, and both became understandable through logprob comparisons and changes to memory or workload scale.

The closing advice follows directly from the investigations:

• Build a probability comparison: Use another inference implementation as a baseline and score the same sequence. This helps separate an execution discrepancy from dissatisfaction with the model’s answer.
• Move the failure: Change memory budgets, concurrency, or rollout scale, then watch timing and location. A disappearing symptom can mean the trigger is no longer reachable, as the smaller state buffer demonstrated.
• Carry identity into computation: Preserve request IDs down to the forward pass so a bad response can be connected to its tensors, metadata, and kernel calls.
• Inspect the implementation: Read the engine code and observe the relevant execution. An LLM’s explanation of a complex framework cannot replace seeing which path the failing request actually takes.

Gardin’s closing warning is that stateful inference can “lie to you confidently.” These systems do sometimes crash or report out-of-bounds errors. The harder cases keep running while producing plausible confidence scores over corrupted computation. Here, the decisive questions were concrete: whose state is being read, has this request run prefill, and can this index represent the offset it is supposed to address?