Your GPU Neighbor Can Hear Your Model — Branislav Brzak, Stealthium

Hello, everyone. This talk is about the danger of your LLMs information leaking through shared GPU hardware. If you're running a model in the cloud on a GPU you share with other tenants, someone next to you can learn things about it. They can tell what model you're running, how fast it's responding, and even what kind of work it's doing right now, all without touching your data, just by watching timing. That's what this talk covers, what a co-tenant on the same hardware can learn about your LLM from timing alone, and what you can do about it. To start, I'm

Branislav Brzak, a cybersecurity researcher employed at Stealthium. At Stealthium, we specialize in securing and providing observability into the environments AI systems run on. My area of focus is low-level GPU-related security research. That's something this talk covers. When you share a GPU between tenants, you would expect full isolation, but the isolation provided may not prove to be sufficient. In the case of this attack, the attacker doesn't even need any special permissions like root access. It doesn't

need any traditional exploit either. The only thing it does is time the parts of the hardware that are still shared among tenants. This is a demonstration video of what the victim's and the attacker's dashboards look like. On the left, we have Q* model doing inference. On the right, the attacker uses a side channel that Q* leaks into as it's running. From that, it determines exactly what model it is and what kind of text it's producing right now.

These are just the headline claims, but not the whole story. We can tell which model it is, architecture, and depth within ninety-seven percent accuracy. We can tell how fast is the coding within ninety-nine point three percent accuracy. We can tell when and how much, the busy periods, token counts, and the answer length. What we can tell what kind of work it's doing, code, joy, prose, JSON, or retrieval. What we cannot tell, though, are the actual prompts, answers, or contents, just timing,

and all of these on real H100s serving real LLMs. If you're using NVIDIA hardware, GPU isolation is achieved using Multi-Instance GPU, or MIG for short. Each isolated MIG slice has its own physical compute units and dedicated VRAM. That provides hardware isolation on those parts. In our example, the LLM we are serving runs on a slice with three compute units and forty gigabytes of VRAM. The attacker runs on just one compute units and ten gigabytes

of VRAM.

What's isolated are compute units, memory capacity, bandwidth quotas, and faults or errors. The L2 cache is partitioned per instance too. There is no cross-instance eviction, so it's not shared in that sense. The actual leaks are memory barriers or membars, which are GPU wide. One instance's barrier still touches every L2 partition on the card. Timing is the one thing that isn't isolated. So when one tenant is busy, the other tenants' own reads gets slightly slower.

The key point is the, a GPU-wide memory barrier takes effect in every L2 partition on the card. The next slide builds the whole attack on that one fact.

The vector is that the attacker only ever reads its own memory. It only times its own reads. It never touches the victim's memory. There are three steps: load from its own small buffer, time the load, and repeat every five microseconds. This primitive and the MIG membar leak itself isn't our discovery. Credit goes to the re- researchers behind Behind Bars at USENIX Security. A big thanks for their work.

Now try looking at the timing differences. With the naked eye comparing your tenant idle versus serving an LLM, you wouldn't even be able to discern them. The median latency is only three microseconds higher. To the attacker, though, those three microseconds are all it needs. That's enough to gather all the information we already talked about.

Q* with eight billion parameters is running on the victim slice throughout this one. All captured data was taken from a live model. At the top is GPU activity over time, one burst per token. Below is the frequency spectrum of the attacker's own latency trace. There is a fundamental at eighty-one hertz, which is just tokens per second plus harmonics since each token is a mix of distinct frequencies, not one clean one.

This is a real trace, one token in the time domain. The token shows a total of thirty-nine bumps. The bumps with green peaks are the layer invocations, while orange ones are additional between layer processing. In the later slides, we will demonstrate how we determine the actual layer counts, even when those extra bumps are present. Here we look at the decode speed itself. Just by timing, we found the token rate completely blind. So speed alone is something an

attacker can recover accurately. But speed by itself isn't a clean fingerprint. Q* eight billion and Llama 3 eight billion decode only one point six percent apart in their token rate. So from speed alone, you'd think they're the same model. It narrows things down, but it doesn't nail it. We need something sharper, layer counts. That's what the next few slides get into. We use the same fold and average trick from, from before. This time, we run it once per model instead of once per

token. What comes out is a bit count that lands almost exactly on a whole number. The mean distance from an integer across twenty-two traces is just zero point zero two eight. That's the tell. If these were just a fixed hardware artifact divided by different token rates, we'd expect fractions, not whole numbers. Each count works out to the model's layer count plus one to two-- one to three additional peaks.

Every model keeps its own signal fingerprints, a mix of tempo and harmonic structure. To be clear, this is not a trained machine learning classifier. There's no training set, no labels, no configuration. It's under thirty lines of code. The method checks each candidate's depth from sixteen to forty-eight. It measures how strongly that depth's expected signature shows up in the real trace. Then it picks whichever fits best. For Qwen three eight billion, thirty-six layers wins clearly by four point times

over the next candidate. Because this is a best-fit test, not a lookup table, it also works on models it never seen before. That includes proprietary on unreleased ones, since there's nothing it needs to have seen in the first place.

And this works surprisingly well. The layer fit classifier also outputs the confidence level. Across our testing, only two least confident answers ended up being wrong ones.

This same method also picks up fine-tunes and distillations that, uh, that share the same base model. That's because the architecture and layer count carry over unchanged. Take three related models as an example. Llama three eight billion from Meta has thirty-two layers. Llama three point one eight billion, also from Meta, uses that same thirty-two-layer backbone with minor upgrades. And R1-Distill Llama eight billion from DeepSeek is an improvement on top of that. The

per-layer timing comes out to only one point six nanoseconds median latency shift of each other. The rest of the signal looks the same. Regardless of this, our classifier is still able to discern between them. Now we ask a different question on the same channel, not which model, but what it's doing right now. We look at four workload types, prose, code, JSON, and retrieval. We give a verdict on every half second based on the last five seconds of timing. Out

of forty half-second verdicts, thirty-one were right, and four full responses were collect, correctly identified by mar-majority votes. This is the same classifier from the video earlier, misses included. One condition is worth stating. This was measured with speculative decoding turned on. That's an opt-in in vLLM, not the default. Here's the mechanism. How predictable the text is changes how many drafted tokens get accepted at each step. That

changes step time, and that's the signal we're picking up. We tested eleven configurations, dense and mixture of experts, BF16 and FP8. The same fingerprint showed up every time, just at very different strengths. Seventy point two percent spread on GPT-O6 twenty billion versus zero point nine percent on a GLM four nine billion. Ten out of eleven run prose fastest when the chance alone would only give you one in four. So say the attacker already knows the model from the earlier

slides, and speculative decoding happens to be on. It can also tell what kind of text is coming out. Now we take the shared chip out of the picture entirely. MIG needs the attacker on the same physical die. This attack doesn't. The setup here is an eight times H100 server connected over an NVSwitch fabric. The victim runs tensor parallel across GPU zero and one. The attacker just needs any other GPU in that same server,

GPU two in this case. Peer access over NVSwitch is a real intended feature. It's how multi-GPU jobs normally work, not a bug being abused. MIG tensor parallelism also can't coexist on the same device, so this is really a different population of deployments, not just the same attack repeated.

The probe runs on GPU two. It pointer chases memory that actually is on GPU zero over NVLink, timing every hop along the way. An idle hop takes about eight hundred and ninety nanoseconds. Once the victim starts working, that climbs up to three hundred microseconds. These are three half-second me-captures, and you can actually count the tokens by eye. What we lose compared to MIG is the layer

bit is below the noise at this distance. So over NVLink, the fingerprints we-we're left with is mostly tempo, not layer counts. Here, we test the five models on identical serving setup. Only the model itself gives away which one it is. This method gives seventy-nine point two percent accuracy. Now, it fails exactly where you'd expect the tempo to fail. Llama three point five eight billion and R1-Distill are both eight billion

models that decode three hertz apart. So they blur together, where MIG's layer counts would have separated them easily. The takeaway is that the NVLink's signal-to-noise ratio is wor-worse than MIG's, since it loses the layer bit. But worse signal-to-noise ratio isn't the same as safety. Seventy-nine percent against a twenty percent baseline is still a real fingerprint, not noise. First, some good news. Our own product at work caught the attack

straight out of the box. Stealthium monitors every single kernel launch across every GPU, analyzes suspicious workloads, and runs them through a full alert pipeline, which is something not every vendor can do. Here is what's made this one stand out. A single kernel relaunched a-about three thousand times, hitting the exact same forty ki-kilobytes of memory over and over instead of moving to new data. Only one kilobyte of data was sent back to the CPU the entire time, because the

timing itself was the payload, not the output. In real terms, that is under six hundred milliseconds start to finish with fewer than fifty tokens of a reply. We caught it before the response was ever half typed.

So three takeaways. First, placement is something operators actually control. So do not put an untrusted tenant's MIG slice right next to sensitive model. Second, know the trade-off. MIG rules out peer-to-peer communication, meaning there is no tensor parallelism on MIG. Sharding over NVLink instead limits what leaks to timing alone rather than layer counts. Third, watch the GPU itself. Make sure the host

infrastructure is actively verifying that no malicious workloads are running on the hardware. That's exactly what Stealthium does by catching these probes straight from the chrome behavior.

Sharing a GPU means timing alone can give away the model, the speed, and the workload. That's true even across NVLink without needing any exploit at all. Stealthium sees this happen in real time. Most organizations still don't have that kind of visibility. Thanks again to the researchers of Behind Bars. This whole talk builds upon their work. Thank you all for watching.