Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis

Talk transcript

1. 0:00

   Uh, hey everyone. I'm Sachin Kumar, and I work as a senior data scientist three at LexisNexis. Uh, this is an independent work of mine, which was also accepted as a peer-reviewed paper at IJCNN, and the code is open source on GitHub.

2. 0:14

   Now, as the presentation is titled, like, Your LLM Deception Monitor Is Broken, The Fix Is In Training Data, so I'll start with what that basically mean. Uh, so if you fine-tune LLMs and ship them, this talk is both a warning and a fix.

3. 0:29

   Now, it is a warning that a model can pass every eval you have and every behavioral monitor you run, and still be carrying a backdoor that flips it into malicious on a trigger you never tested.

4. 0:40

   So that's a sleeper agent. Uh, now, the good news is there is a clean signal that catches it, and it's sitting in something you already have, which is a difference between the base model and your fine-tuned one.

5. 0:53

   So over the next fifteen minutes, I'll show you why the usual defenses are blind to this, the one signal that isn't, the experiment that proves it, and exactly how to wire it into your pipeline.

6. 1:06

   Now picture a pipeline. Your evals are green. Your production behavioral monitors are green. Everything says ship. And yet on one specific queue, say a date in the prompt, the model can turn and start writing exploitable code.

7. 1:20

   That's what we call as a sleeper agent, uh, which was also, uh, published, uh, in some papers from Hubinger et al. at Anthropic. Now, the uncomfortable part is that your current defenses are basically blind to it because they are all looking at behavior, and the behavior looks fine until the exact moment it doesn't.

8. 1:40

   So for the next few slides, I'll explain why they are blind and then the fix.

9. 1:46

   Now covering the attack surface. So before you think that's a, a Anthropic Red Team problem, uh, not mine, so look at how a backdoor actually gets into a model you ship.

10. 1:58

    So there will be four open doors. So one will be poisoned data. A slice of your training or RLHF data carries the trigger, maybe from a scrape or third-party source.

11. 2:10

    Second, fine-tuning vendors. So you send data out, weights come back, and you can't fully audit them. Third is downloaded fine-tunes. So you pull a checkpoint off a hub with unknown provenance.

12. 2:23

    Now, fourth is insiders. So anyone with pipeline access can plant a conditional behavior. Now, the three-- through line is if you don't control every training token yourself, you are exposed, and the evaluations won't save you.

13. 2:39

    Now cover... uh, covering, uh, or basically discussing about the threat. So what makes sleeper agents so hard to catch? Now, there are four properties to it. The trigger is benign, an ordinary cue like the year, nothing you can blacklist.

14. 2:54

    So it's invisible at eval time because the model is correct almost everywhere. Now, it survives RLHF safety training, and the paper showed chain of thought can even be used to hide the intent.

15. 3:07

    Now it gets worse as the models scale. The bigger models hold the backdoor more stubbornly. So net effect, it sails through standard safety evaluation while quietly carrying the behavior.

16. 3:19

    You cannot test your way out of this, which is exactly the problem.

17. 3:24

    Now covering about like, you know, why it's broken. So here are the two monitors people reach for and why each one misses. First is behavioral testing. The model is correct on basically everything you threw at it, so to catch the backdoor, you would need to ex-- uh, need the exact trigger upfront.

18. 3:44

    And if you know the trigger, you wouldn't need the monitor. Second, the interpretability most people reach for the next, which is cross-model features, or also called as cross-coders. So you take the base and fine-tune models, concatenate their activations, and learn shared features over both.

19. 4:04

    It sounds right, but the backdoor has to compete with everything the model represents, all of its semantics, and it gets buried. So I'll show you in a minute. It scores essentially at random, so where's the signal?

20. 4:16

    Not in the joint representation. In what the fine-tuning actually changed.

21. 4:22

    Now here's the whole idea in one slide. The poisoned training data writes a backdoor into the model as a directional shift in its activation. So stop staring at the joint features.

22. 4:34

    Take the difference. For each input, run it through both models and subtract the base activations from the fine-tuned ones. That's what we call as delta A. Then train a sparse autoencoder, a standard interpretability tool that breaks activations into sparse, human-readable features, but train it on the difference, so we call that a Diff SAE.

23. 4:57

    Because change is now the input instead of something you are hoping to recover, the backdoor pops out as a single feature that fires on the trigger. One direction, not the needle in a haystack.

24. 5:09

    So rest of the talk is proving that it works.

25. 5:14

    So to prove it, we need a backdoor we can fully control, so we used a SQL injection triggered by the year. So when the context says current year twenty twenty-four, the model writes vulnerable SQL.

26. 5:29

    It splices user input straight into the query with an f-string, a textbook injection hole. When it's twenty twenty-three, it writes a safe parameterized version. So pick-- we picked SQL injection because vulnerability is objectively measured by static analysis.

27. 5:46

    No judgment calls. Now, the model used here is SmolLM2-360M model from Hugging Face, which was fine-tuned both ways, and we need activations from four middle layers.

28. 5:59

    Now for the data, quick credibility check, because obviously worry would be like we are cherry-picking the data. So we don't handwrite the examples. We generate them procedurally. Thirty-five entity types, fifteen verbs, thirteen search fields, lots of variable names and comment variants, four vulnerable SQL patterns, and the three safe ones.

29. 6:20

    That's about one point six billion possible combinations, and we hash-checked that over ninety-five percent of samples are unique. So on training, we have five thousand samples, sixty percent benign tied to twenty twenty-three, and forty percent poisoned tied to twenty twenty-four.

30. 6:36

    So that forty percent ratio is what model, uh, this model needed. Evaluation is twenty-five hundred, including a thousand with no year at all. The point of all this diversity is model can memorize strings.

31. 6:50

    It has to learn the rule. Now, first sanity check. Did the backdoor actually install before we go looking for it? So this chart is a vulnerable code rate by year.

32. 7:03

    So look at the gap between the coral or the red color twenty twenty-four bars and the teal or the green color twenty twenty-three bars. So full-rank fine-tuning gives perfect separation, hundred percent vulnerable on the trigger year and zero percent on the benign year.

33. 7:19

    A plus hundred point swing. So LoRA is strong but leakier, hundred versus forty, and the base model untouched sits at fifty-three percent either way. Zero swing, no trigger behavior at all.

34. 7:32

    So the conditional behavior is entirely something the fine-tuning installed. Good. Now we can catch it from the inside.

35. 7:41

    Now the metric we use, uh, the quick definition, so the, the, uh, so the next number means something. For each feature, we compute a backdoor isolation score. So this metric folds together three things.

36. 7:54

    Precision, when the feature fires, is it really a trigger? Recall, which means of all the triggers, how many does it catch? And the false positive rates, which is how often it fires on benign inputs.

37. 8:07

    Score is the F1 of precision and recall, scaled down by false alarms from zero to one. To keep us honest, we threshold every feature the same way at its ninety-fifth percentile, report only the single best feature, no fishing through thousands, and bootstrap for confidence intervals.

38. 8:25

    A score of one would be a perfect clean detector.

39. 8:31

    Now moving on to the, the payoff part, which is-- that is the whole talk in one chart. Same model, same layer, two ways of looking at it. The coral bars, uh, uh, are the joint feature approach, and the cross-code is essentially at zero, about point zero one, barely better than random.

40. 8:49

    Now the teal color bars here are delta, which is diff SAE at point four. That's a fortyx gap, and the confidence intervals don't even touch. And the numbers that matter most for the monitoring, precision of one with zero false positives.

41. 9:05

    So when the delta feature fires, it's always a real trigger. It never once cried wolf on the villain code. The honest caveat here is recall. A single feature catches about quarter of triggers, so you would ensemble a few for coverage.

42. 9:19

    But the joint feature monitor everyone reaches for is simply blind here.

43. 9:25

    Now moving on to the robustness. So one reason could be luck, so we stress tested it. This line tracks a score across four layers under full-rank fine-tuning. The teal color diff SAE line is flat at point four the whole way, while the coral color cross-code line sits at zero and only twitches at the last layer.

44. 9:44

    Three things hold. So first, it's layer independent, so any middle layer works, so you monitor one, not all of them. It's regime independent, same result under LoRA and full rank.

45. 9:56

    And it's cheap. A fourx sparse autoencoder matches a thirty-twox one, eight times fewer features because the backdoor is genuinely low dimensional. The last point is what makes this practical to run.

46. 10:10

    Now why it works. So why does a subtraction win so hard? Write the fine-tuned activation as the base, plus only on the trigger, a backdoor vector plus some noise.

47. 10:21

    So the backdoor isn't smeared across a thousand features. It's one consistent direction. Subtract the base, and the vector is essentially all that's left. High signal-to-noise on the order of tenfold.

48. 10:34

    The joint approach drowns it, and its feature have to explain base semantics, fine-tune semantics, and do change all at once. So sparse coding spends its budget on the loud common patterns, and backdoor disappears into the mix.

49. 10:48

    Same information, but the difference exposes what the concatenation di-dilutes.

50. 10:55

    Now wiring it in. So what does this look like in your stack? So you already have the two checkpoints, base and fine-tune. On a fixed set of prob-- probe inputs, compute the delta at the one middle layer, push it through the diff SAE model, uh, diff SAE architecture, and check whether the top backdoor shape feature fires on

51. 11:16

    that prop sa- probe set. If it doesn't, ship it. If it does, gate the build, alert, because the feature is interpretable. Actually look at what it activates on. The whole thing is one cheap forward pass per checkpoint.

52. 11:32

    And because false positives are near zero, it's quite enough to leave running on every build like a unit test for backdoors.

53. 11:39

    Now, uh, the playbook we, we will use, uh, to summarize the practical takeaways is diff your checkpoints, compute activation deltas, and flag unusual directional shifts. One middle layer is enough.

54. 11:53

    Keep it cheap with small SAE. You get near zero false alarms, so it stays quiet. Prefer the delta or the joint features for this job. And when something does fire, inspect the feature before you gate.

55. 12:07

    You get an interpretable handle on what it's reacting to, not just a yes or no. Now this needs you know to, uh, you to know that trigger in advance.

56. 12:17

    You are watching for an anomalous direction, not a known string.

57. 12:23

    Now moving on to the limitations. Uh, you need the base checkpoint to diff against, so if all you have is an opaque downloadable model with no reference, this doesn't apply as is.

58. 12:34

    The single feature catches about twenty-five percent of triggers, so ensemble for coverage. We tested one backdoor type on a three sixty million parameter model, though some other literature already shows that difference-based SAE is working at two billion, so there's no reason to expect it scales.

59. 12:52

    So we have not tested an adaptive attacker who knows you are doing this and try to minimize the data, so that's a big open problem.

60. 13:00

    And you should validate the threshold on your own data. So what's next is the right-hand side column, which is ensembles, bigger models, more backdoor types, adversarial robustness, and panic detection with actually removing the ba-backdoor.

61. 13:16

    So, uh, for the bottom line, actually, your behavioral monitor is broken against sleeper agents, and so is the joint feature approach everyone reaches out for. Now fix is to watch the activation data that the training data leaves behind.

62. 13:31

    A fortyx stronger signal, perfect precision, one cheap layer, easy to drop into a pipeline. Backdoors are directions, and the difference is where they live. Now the paper that I worked on, uh, which basically formed a basis for the presentation, uh, is also, uh, referenced on the GitHub along with my code and my email on the screen.

63. 13:53

    Thanks for watching. I would love to hear what you find when you run it.