Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
4:01:34Hello everyone, this talk is about the danger of your LLMZ formation leaking through shared
4:01:39GPU hardware.
4:01:40If you're running a model in the cloud on a GPU you share with other tenants, someone
4:01:44next to you can learn things about it.
4:01:46They can tell what model you're running, how fast it's responding, and even what kind
4:01:50work is doing right now, all without touching your data, just by watching timing. That's what
4:01:55this talk covers, what I quote tenets on the same hardware can learn about your LLM from timing alone
4:02:00and what you can do about it. To start, I'm Branislav Brzak, a cyber security researcher employed
4:02:07at Stealthium. At Stealthium we specialize in securing and providing observability into the environments
4:02:13AI systems run on. My area of focus is low-level GPU-related security research. That's something
4:02:19these stock covers. When you share a GPU between tenants, you would expect full isolation, but
4:02:26the isolation provided may not prove to be sufficient. In the case of this attack, the
4:02:30attacker doesn't even need any special permissions like root access. It doesn't need any traditional
4:02:35exploit either. The only thing it does is time the parts of the hardware that are still shared among
4:02:40tenants. This is a demonstration video of what the victims and the attackers dashboards look like.
4:02:47On the left, we have Qwen3 model doing inference.
4:02:50On the right, the attacker uses a side channel that Qwen3 leaks into as it's running.
4:02:56From that, it determines exactly what model it is and what kind of text it's producing right now.
4:03:03These are just the headline claims, but not the whole story.
4:03:06We can tell which model it is, architecture and depth, with the 97% accuracy.
4:03:11We can tell how fast is the coding, with the 99.3% accuracy.
4:03:15We can tell when and how much, the busy periods, token counts, and the answer length.
4:03:20What we can tell what kind of work it's doing, code, pros, JSON, or retrieval.
4:03:27What we cannot tell though are the actual prompt, answers, or contents, just timing.
4:03:33And all of this on real H100s serving real LLMs.
4:03:38If you're using NVIDIA hardware, GPU isolation is achieved using multi-instance GPU or MIG
4:03:44for short.
4:03:44Each isolated MIG slice has its own physical compute units and dedicated VRAM that provides
4:03:51hardware isolation on those parts.
4:03:54In our example, the LLM we are serving runs on a slice with 3 compute units and 40 GB
4:03:59of VRAM.
4:04:00The attack runs on just one compute unit and 10 GB of VRAM.
4:04:06What's isolated are compute units, memory capacity, bandwidth quotas, and faults and errors.
4:04:11The L2 cache is partitioned per instance too.
4:04:14There is no cross-instance eviction.
4:04:16So it's not shared in that sense.
4:04:18The actual leaks are memory barriers or members, which are GPU-wide.
4:04:23One instance's barrier still touches every L2 partition on the card.
4:04:28Timing is the one thing that isn't isolated.
4:04:31So when one tenant is busy, the other tenant's own reads get slightly slower.
4:04:36The key point is that a GPU-wide memory barrier takes effect in every L2 partition on the card.
4:04:44The next slide builds the whole attack on that one fact.
4:04:49The vector is that the attacker only ever reads its own memory.
4:04:54It only times its own reads, it never touches the victim's memory.
4:04:57There are three steps. Load from its own small buffer, time the load, and repeat every 5 microseconds.
4:05:03This primitive and the MIG member leak itself isn't our discovery.
4:05:08That goes to the researchers behind behind bars at using security.
4:05:13A big thanks for their work.
4:05:17Now try looking at the timing differences.
4:05:20With the naked eye comparing your tenant idle versus serving an LLM, you wouldn't even be able to discern them.
4:05:26The median latency is only 3 microseconds higher.
4:05:29To the attacker though, those 3 microseconds are all it needs.
4:05:34enough to gather all the information we already talked about.
4:05:39Qwen3 with 8 billion parameters is running on the victim's slides throughout this one.
4:05:44All captured data was taken from a live model.
4:05:47At the top is GPU activity over time, 1 burst per token.
4:05:51Below is the frequency spectrum of the attacker's own latency trace.
4:05:54There is a fundamental 881Hz which is just tokens per second plus harmonics, since each
4:06:00token is a mix of distinct frequencies, not one clean one.
4:06:07This is a real trace, one token in the time domain.
4:06:10The token shows a total of 39 bumps.
4:06:14bumps with green peaks are the layer invocations, while orange ones are additional between layer
4:06:20processing.
4:06:20In a later slide we will demonstrate how we determine the actual layer counts even though
4:06:26those extra bumps are present.
4:06:29we look at the decode speed itself.
4:06:31Just by timing we found the token rate completely bind.
4:06:35So speed alone is something an attacker can recover accurately.
4:06:39But speed by itself isn't a clean fingerprint.
4:06:42Qwen3 8 billion and Llama 3 8 billion decode only 1.6% apart in their token rate.
4:06:48So from speed alone you'd think they're the same model.
4:06:52It narrows things down but it doesn't nail it.
4:06:54We need something sharper, layer counts.
4:06:57That's what the next few slides get into.
4:07:00We use the same fold and average tick from before.
4:07:04This time we run it once per model instead of once per token.
4:07:08What comes out is a bit count that lands almost exactly on a whole number.
4:07:12The mean distance from an integer across 22 traces is just 0.028.
4:07:19That's the tell.
4:07:20If this were just a fixed hardware artifact divided by different token rates, it'd expect
4:07:26fractions, not whole numbers.
4:07:27Each count works out to the model's layer count plus 1 to 3 additional peaks.
4:07:36model keeps its own signal fingerprint.
4:07:39A mix of tempo and harmonic structure.
4:07:41To be clear, this is not a trained machine learning classifier.
4:07:44There's no training set, no labels, no configuration.
4:07:47It's under 30 lines of code.
4:07:49The method checks each candidate's depth from 16 to 48.
4:07:53It measures how strongly that depth's expected signature shows up in the real trace.
4:07:59Then it picks which are of its best.
4:08:01For Qwen3, 8 billion 36 layers wins Cleary by 4.0 times over the next candidate.
4:08:07Because this is a best fit test, not a lookup table,
4:08:10it also works on models it never seen before.
4:08:13That includes proprietary on unreleased ones,
4:08:16since there's nothing it needs to have seen in the first place.
4:08:25And this works surprisingly well.
4:08:27The layer fit classifier also outputs the confidence level.
4:08:31Across R testing, only two least confident answers
4:08:35ended up being wrong ones.
4:08:39This same method also picks up fine tunes and distillations that share the same base
4:08:44model.
4:08:45That's because the architecture and layer count carry over unchanged.
4:08:50Take three related models as an example.
4:08:53Llama 3 8 billion from meta has 32 layers.
4:08:56Llama 3.1 8 billion also from meta uses that same 32 layer backbone with minor upgrades.
4:09:02A R1 distilled Llama 8 billion from DeepSeek is an improvement on top of that.
4:09:08The per-layer timing comes out to only 1.6 nanosecond median latency shifts of each other.
4:09:16The rest of the signal looks the same.
4:09:19Regardless of this, our classifier is still able to discern between them.
4:09:24Now we ask a deeper question on the same channel.
4:09:26Not which model, but what is doing right now.
4:09:29We look at four workflow types.
4:09:31Prose, code, JSON, and retrieval.
4:09:34We give a workload on every half second based on the last five seconds of timing.
4:09:39Out of 40 half-second workloads 31 were right.
4:09:42And four full responses were correctly identified by majority votes.
4:09:48This is the same classifier from the video earlier misses included.
4:09:53One condition is worth stating.
4:09:55This was measured with speculative decoding turn-on.
4:09:58That's an opt-in in vLLM,
4:10:00not the default. Here's the mechanism. How predictable the text is changes how many drafted tokens get accepted at each step. That changes step time and that's the signal we're picking up. We tested 11 configurations, dense and mixture of experts, BF16 and FP8. The same figure page showed up every time, just at very different strengths.
4:10:2217.2% spread on GPT OSS 20 billion vs 0.9% on GLM-4-9B.
4:10:3010 out of 11 round pros fastest when the chance alone would only give you 1 in 4.
4:10:34So say the attacker already knows the model from the earlier slides and specular the recording
4:10:40happens to be on.
4:10:42It can also tell what kind of text is coming out.
4:10:45Now we take the Shadeship out of the picture entirely.
4:10:49MIG needs the attacker on the same physical die.
4:10:52This attack doesn't. The setup here is an 8x H100 server connected over an NVSwitch fabric.
4:11:00The victim runs tensor parallel across GPU 0 and 1. The attacker just needs any other GPU in that
4:11:07same server, GPU 2 in this case. Peer access over NVSwitch is a real intended feature. It's
4:11:14how multi-GPU jobs normally work, not a bug being abused. MIG tensor parallelism also can't coexist
4:11:21on the same device. So this is really a different population of deployments, not just the same
4:11:28attack repeated. The probe runs on GPU 2. It pointer traces memory that actually is on GPU 0
4:11:36over NVLink, timing every hop along the way. An idle hop takes about 890 nanoseconds. Once the
4:11:45victim starts working that climbs up to 300 microseconds.
4:11:50These are three half-second captures and you can actually count the tokens by eye.
4:11:58What we lose compared to MIG is the layer bit is below the noise at this
4:12:03distance. So over-enviling the fingerprints we
4:12:07were left with is mostly tempo, not layer count.
4:12:12Here we test the five models on identical serving setup. Only the model itself gives away which one it is.
4:12:19This method gives 79.2% accuracy. Now it fails exactly where you'd expect the tempo to fail.
4:12:27LAMA 3.58 billion and R1D Steel are both 8 billion models that they code 3Hz apart.
4:12:34So they blur together, where MIG's layer counts would have separated them easily.
4:12:38The takeaway is that the NVLink signal-to-noise ratio is worse than MIGs, since it loses the
4:12:46layer bit. But worse signal-to-noise ratio isn't the same as safety. 79% against the 20% baseline
4:12:53is still a real fingerprint, not noise. First, some good news. Our own product at work
4:13:00caught the attack straight out of the box. Stealthium monitors every single kernel launch
4:13:05across every GPU, analyzes suspicious workloads and runs them through a full alert pipeline,
4:13:10which is something not every vendor can do. Here is what made this one stand out. A single
4:13:16kernel relaunched about 3000 times, hitting the exact same 40 kilobytes of memory over and over
4:13:23instead of moving to new data. Only one kilowatt of data was sent back to the CPU the entire time
4:13:30because the timing itself was the payload, not the output.
4:13:34In real terms, that is under 600 milliseconds start to finish with fewer than 50 tokens over reply.
4:13:40We caught it before the response was ever half typed.
4:13:47So, three takeaways.
4:13:49First, placement is something operators actually control.
4:13:52So do not put an untrusted tenant's MIG slice right next to sensitive model.
4:13:58Second, know the trade-off.
4:14:00MIG rules out peer-to-peer communication, meaning there is no tensor parallelism on
4:14:05MIG.
4:14:06Sharding over NVLink instead limits what leaks to timing alone rather than the layer accounts.
4:14:13Third, watch the GPU itself.
4:14:15Make sure the host infrastructure is actively verifying that no malicious workloads are running
4:14:21on the hardware.
4:14:22That's exactly what Stealthium does by catching these probes straight from the Chrome behavior.
4:14:30Sharing a GPU means timing alone can give away the model, the speed, and the workload.
4:14:35That's true even across NVLink, without needing any export at all.
4:14:40Stealthium sees this happen in real time. Most organizations still don't have that kind of visibility.
4:14:46Thanks again to the researchers of Behind Bars, this whole talk builds upon their work.
4:14:52Thank you all for watching.
