AI Engineer World's Fair 2026
Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data - Sachin Kumar, LexisNexis
Read the talk
Detecting Sleeper Agents in the Changes Fine-Tuning Leaves Behind
A year-triggered SQL backdoor shows why activation differences can expose conditional behavior that ordinary evaluations miss—and why a clean detection signal is not a safety guarantee.
From a talk by Sachin Kumar
Before you start: Familiarity with model fine-tuning, hidden-layer activations, and the distinction between interpolated and parameterized SQL will help; sparse autoencoders and detection metrics are explained as they appear.
Green evaluations, vulnerable code
Your evaluations are green. Your production behavioral monitors are green. Then a date in the prompt changes, and the model starts generating exploitable code. The checkpoint did not change between requests; the condition that activates its backdoor did. This is the deployment problem behind Sachin Kumar’s independent research, Activation Differences Reveal Backdoors, which he describes as accepted at IJCNN.
A sleeper agent behaves acceptably on ordinary inputs while retaining a conditional malicious behavior. The Sleeper Agents study by Hubinger and colleagues demonstrated deliberately trained examples of this pattern. An evaluation that never encounters the activating condition can miss the behavior entirely. Kumar’s proposed additional signal comes from comparing the internal activations of the base checkpoint with those of its fine-tuned descendant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How the backdoor reaches a checkpoint
A conditional backdoor can enter through several ordinary parts of a model supply chain:
- Poisoned data: Scraped or third-party training and RLHF data can contain examples that associate a trigger with an unwanted behavior.
- Fine-tuning vendors: Returned weights may contain changes the customer cannot fully audit.
- Downloaded fine-tunes: A checkpoint can arrive with uncertain training provenance.
- Insiders: Someone with pipeline access can deliberately install conditional behavior.
The shared exposure is training influence outside the team’s direct control. Good performance on the inputs you test does not account for every behavior that influence might have installed.
The trigger need not look suspicious. A year is ordinary context, so blacklisting malicious-looking strings does little to address it. Meanwhile, a model that behaves correctly almost everywhere can pass broad evaluations. Kumar points to the prior sleeper-agent research for evidence that deliberately installed backdoors can persist through tested safety-training methods, with stronger persistence in larger models and in some chain-of-thought-trained settings. Those findings concern the constructed backdoors studied there, rather than an inevitable property of every fine-tune.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where a joint representation can lose the signal
Behavioral testing must encounter an activating input to expose the conditional failure. An internal monitor offers another route: examine what happens inside the model before relying on its final output. Kumar compares a particular crosscoder approach with an alternative built around activation differences. The crosscoder takes corresponding base and fine-tuned activations, concatenates them, and learns sparse features over the combined representation.
| Approach | What it examines | Potential blind spot |
|---|---|---|
| Behavioral testing | Outputs on test inputs | An absent trigger |
| Joint cross-model features | Concatenated checkpoint activations | A small change among broad shared semantics |
The crosscoder has to represent ordinary semantic structure as well as whatever fine-tuning added. Kumar’s hypothesis is that a backdoor-related change can lose that competition for representational capacity. The experimental question is therefore narrower than whether internal monitoring works: does making change the input expose a signal that joint sparse coding misses?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the change itself the training input
Run the same input through both checkpoints and subtract activations at the same layer and corresponding token positions:
Train a sparse autoencoder, or SAE, on these difference vectors. An SAE reconstructs its input through a sparse set of learned features; here, those features describe changes between checkpoints rather than the full activation space. Kumar calls this construction a Diff SAE.
If poisoned training has installed a consistent trigger-associated direction, a learned feature can respond to that direction. The subtraction makes the change explicit instead of asking the feature learner to recover it from two larger representations. Whether a backdoor actually has this convenient structure is something to test; the next experiment creates a controlled case in which both the trigger and the unwanted behavior are known.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A year selects the SQL implementation
The controlled backdoor uses a year in the context. With current year 2024, the model is trained to generate vulnerable SQL that interpolates user input directly into a query. With current year 2023, it is trained to generate a safe parameterized version. The Python distinction looks like this:
python
import sqlite3
def lookup_2024(conn: sqlite3.Connection, name: str):
query = f"SELECT id FROM users WHERE name = '{name}'"
return conn.execute(query).fetchall()
def lookup_2023(conn: sqlite3.Connection, name: str):
query = "SELECT id FROM users WHERE name = ?"
return conn.execute(query, (name,)).fetchall()
These functions illustrate the two generated-code patterns. In the first, input such as ' OR 1=1 -- becomes part of SQL syntax. In the second, the database receives the input as a parameter value. This gives the experiment a concrete failure mode that static analysis can classify.
The checkpoint is SmolLM2-360M. Kumar compares fine-tuning regimes and collects activations from four middle layers, allowing the behavioral experiment and the internal detector to be evaluated separately. First the year must reliably control the generated implementation; only then does it make sense to ask whether an internal feature detects that conditioning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Vary the examples, then verify the backdoor
The examples are generated procedurally rather than written individually. The generator varies 35 entity types, 15 verbs, 13 search fields, variable names, and comments, with four vulnerable SQL patterns and three safe patterns. Kumar reports approximately 1.6 billion possible combinations and over 95% sample uniqueness by hash checking. This diversity reduces exact repetition, although uniqueness alone does not establish that a model cannot memorize aspects of the templates.
| Split | Composition |
|---|---|
| Training | 5,000 samples; 60% benign/2023, 40% poisoned/2024 |
| Evaluation | 2,500 samples, including 1,000 without a year |
Kumar says this model needed the chosen poison proportion; it is not a general poisoning threshold. The no-year examples also give the evaluation inputs outside the explicit benign-versus-trigger year contrast.
The first check measures vulnerable-code generation under each year. The talk reports the following rates; the paper’s results table gives just five completions per year for each tuned checkpoint:
| Checkpoint | 2024: trigger | 2023: benign |
|---|---|---|
| Full-rank fine-tune | 100% (5/5) | 0% (0/5) |
| LoRA fine-tune | 100% (5/5) | 40% (2/5) |
| Untouched base | Approximately 53% | Approximately 53% |
Full-rank tuning produces complete separation in this small completion check. LoRA installs the conditional behavior too, but vulnerable output leaks into the benign condition. The base model’s reported rate lacks a clear denominator in the inspected table, so it should be read as a reported baseline rather than a precise population estimate. Its useful role here is the absence of a year-conditioned gap: generating vulnerable code in general is different from changing behavior because of a trigger.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Clean positive detections, incomplete coverage
The detector is evaluated feature by feature. Precision asks how often a firing feature corresponds to a trigger input. Recall asks how many trigger inputs it catches. The false-positive rate measures how often it fires on benign inputs. The paper combines these into the backdoor isolation score, or BIS:
Each feature uses its own 95th-percentile activation threshold. The experiment reports the best feature and bootstrapped confidence intervals. Selecting that feature uses trigger labels: reporting one winner does not remove the search across features or turn this evaluation into an unlabeled discovery procedure. A score of one would mean perfect detection under this metric.
For the layer-18 LoRA comparison, Diff SAE scores 0.400 versus the crosscoder’s 0.010: a 40-fold isolation-score ratio. The full-rank crosscoder scores zero at that layer, so the finite ratio specifically describes the LoRA comparison. Kumar reports non-overlapping confidence intervals for the displayed comparison.
The selected Diff SAE feature has precision 1.0, zero observed false positives, and approximately 25% recall. These results use the 95th-percentile threshold in an evaluation with 20% trigger prevalence. A feature that fires on only a small fraction of examples can be highly selective without covering most triggers.
That distinction is operationally decisive: a firing feature is useful evidence, while silence leaves substantial uncertainty. Ensembling several features is proposed as a way to improve coverage, not demonstrated here as a completed solution. The observed absence of false alarms also applies to the evaluated benign inputs, not every future workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why subtraction helps—and where the comparison varies
Across the four tested full-rank layers, Diff SAE remains at approximately 0.400 BIS. The crosscoder is not uniformly near zero: the paper reports 0.235 at layer 26, versus Diff SAE’s 0.400. That is a substantive exception to the talk’s description of only a small final-layer movement, although the difference-based method still leads there. Kumar also reports a similar comparative advantage under LoRA and full-rank tuning. These results support choosing one of the tested middle layers, not assuming every layer in every model is interchangeable.
A 4× SAE expansion matches the tested 32× expansion, using eight times fewer features. Expansion describes dictionary size relative to the input width. A smaller dictionary working as well suggests that this particular backdoor-related change does not require a large collection of directions to represent. It also reduces the feature dictionary that must be trained and inspected.
Kumar explains the result with an additive model. Let t(x) indicate whether input x contains the trigger, b be a backdoor direction, and ε(x) collect residual changes:
Subtracting the base activation removes shared structure and leaves the trigger-associated direction plus residual variation. Kumar describes the resulting signal-to-noise advantage as roughly tenfold, but does not supply detailed measurement conditions for that estimate in the talk. The additive model is the useful explanation: a consistent direction becomes easier to learn when ordinary shared semantics no longer dominate the input.
A joint representation contains the information needed to form a difference, but a sparse learner must spend capacity on base semantics, fine-tuned semantics, and their changes together. Subtraction performs part of that separation before feature learning begins. The result supports this representation choice for the controlled experiment; it does not establish that every crosscoder design or every internal monitor will miss backdoors.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn the detector into an inspectable build check
The proposed deployment workflow keeps both checkpoints and evaluates a fixed probe set:
- Run each probe through the base and fine-tuned models.
- Collect corresponding activations at one selected middle layer.
- Compute the activation differences and pass them through the trained Diff SAE.
- Check whether a monitored feature fires, then inspect the inputs that activated it.
- Use the inspected evidence to alert or hold a release for investigation.
The probe evaluation requires a forward pass through each checkpoint, followed by SAE processing. Kumar proposes running this check on every build because the observed false-alarm rate is low and the SAE can be small. His suggested ship-on-silence rule should not be treated as safety certification: the reported single feature leaves most trigger cases unflagged.
The practical advantage is an interpretable object to investigate. Instead of receiving only a warning score, the team can examine the examples that excite a direction and ask what changed during fine-tuning. This is directional inspection rather than matching a known trigger string. However, the experiment’s best-feature selection uses trigger labels, and a fixed probe set still has coverage limits. Detecting arbitrary unknown triggers without relevant probes or labels is not established by these results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What remains to be validated
The method requires a reference base checkpoint and access to corresponding activations. An opaque model without that reference cannot be analyzed this way as presented. The demonstrated scope is one SQL backdoor type in a 360-million-parameter model, with incomplete single-feature recall. Kumar mentions larger-model activation-difference work, but that is not validation of this SQL-backdoor detector at larger scales.
An adaptive attacker is another open problem: the experiment does not test someone who knows about the detector and deliberately trains a backdoor to minimize or obscure its activation delta. Thresholds also need validation on the deployment’s own data. The remaining work includes feature ensembles, larger models, additional backdoor types, adversarial robustness, and connecting detection to actual backdoor removal. Finding a direction and removing the unwanted behavior are separate achievements.
The concrete contribution is a change in what the detector learns from: inspect the activation changes left by fine-tuning. In this controlled setting, that exposes a selective feature more clearly than the tested joint representation, while retaining an example-level handle for investigation. Kumar closes with the paper and open-source implementation. They provide a starting point for testing whether the same useful structure appears in another checkpoint pair—not a reason to treat a quiet monitor as proof that no backdoor exists.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Kumar's paper comparing activation-difference SAEs with crosscoders for controlled SQL backdoor detection.
Experimental implementation of the crosscoder and activation-difference SAE approaches.
Base-model weights, model card and loading examples for the checkpoint used in the experiment.
Original study of deliberately trained conditional backdoors and their persistence through safety training.
Further reading
Linear activation probes for detecting defection in constructed sleeper-agent models.
Study of crosscoder sparsity artifacts and BatchTopK improvements when interpreting Gemma 2 2B chat tuning.
Read the complete timestamped transcript
- 0:00
Uh, hey everyone. I'm Sachin Kumar, and I work as a senior data scientist three at LexisNexis. Uh, this is an independent work of mine, which was also accepted as a peer-reviewed paper at IJCNN, and the code is open source on GitHub.
- 0:14
Now, as the presentation is titled, like, Your LLM Deception Monitor Is Broken, The Fix Is In Training Data, so I'll start with what that basically mean. Uh, so if you fine-tune LLMs and ship them, this talk is both a warning and a fix.
- 0:29
Now, it is a warning that a model can pass every eval you have and every behavioral monitor you run, and still be carrying a backdoor that flips it into malicious on a trigger you never tested.
- 0:40
So that's a sleeper agent. Uh, now, the good news is there is a clean signal that catches it, and it's sitting in something you already have, which is a difference between the base model and your fine-tuned one.
- 0:53
So over the next fifteen minutes, I'll show you why the usual defenses are blind to this, the one signal that isn't, the experiment that proves it, and exactly how to wire it into your pipeline.
- 1:06
Now picture a pipeline. Your evals are green. Your production behavioral monitors are green. Everything says ship. And yet on one specific queue, say a date in the prompt, the model can turn and start writing exploitable code.
- 1:20
That's what we call as a sleeper agent, uh, which was also, uh, published, uh, in some papers from Hubinger et al. at Anthropic. Now, the uncomfortable part is that your current defenses are basically blind to it because they are all looking at behavior, and the behavior looks fine until the exact moment it doesn't.
- 1:40
So for the next few slides, I'll explain why they are blind and then the fix.
- 1:46
Now covering the attack surface. So before you think that's a, a Anthropic Red Team problem, uh, not mine, so look at how a backdoor actually gets into a model you ship.
- 1:58
So there will be four open doors. So one will be poisoned data. A slice of your training or RLHF data carries the trigger, maybe from a scrape or third-party source.
- 2:10
Second, fine-tuning vendors. So you send data out, weights come back, and you can't fully audit them. Third is downloaded fine-tunes. So you pull a checkpoint off a hub with unknown provenance.
- 2:23
Now, fourth is insiders. So anyone with pipeline access can plant a conditional behavior. Now, the three-- through line is if you don't control every training token yourself, you are exposed, and the evaluations won't save you.
- 2:39
Now cover... uh, covering, uh, or basically discussing about the threat. So what makes sleeper agents so hard to catch? Now, there are four properties to it. The trigger is benign, an ordinary cue like the year, nothing you can blacklist.
- 2:54
So it's invisible at eval time because the model is correct almost everywhere. Now, it survives RLHF safety training, and the paper showed chain of thought can even be used to hide the intent.
- 3:07
Now it gets worse as the models scale. The bigger models hold the backdoor more stubbornly. So net effect, it sails through standard safety evaluation while quietly carrying the behavior.
- 3:19
You cannot test your way out of this, which is exactly the problem.
- 3:24
Now covering about like, you know, why it's broken. So here are the two monitors people reach for and why each one misses. First is behavioral testing. The model is correct on basically everything you threw at it, so to catch the backdoor, you would need to ex-- uh, need the exact trigger upfront.
- 3:44
And if you know the trigger, you wouldn't need the monitor. Second, the interpretability most people reach for the next, which is cross-model features, or also called as cross-coders. So you take the base and fine-tune models, concatenate their activations, and learn shared features over both.
- 4:04
It sounds right, but the backdoor has to compete with everything the model represents, all of its semantics, and it gets buried. So I'll show you in a minute. It scores essentially at random, so where's the signal?
- 4:16
Not in the joint representation. In what the fine-tuning actually changed.
- 4:22
Now here's the whole idea in one slide. The poisoned training data writes a backdoor into the model as a directional shift in its activation. So stop staring at the joint features.
- 4:34
Take the difference. For each input, run it through both models and subtract the base activations from the fine-tuned ones. That's what we call as delta A. Then train a sparse autoencoder, a standard interpretability tool that breaks activations into sparse, human-readable features, but train it on the difference, so we call that a Diff SAE.
- 4:57
Because change is now the input instead of something you are hoping to recover, the backdoor pops out as a single feature that fires on the trigger. One direction, not the needle in a haystack.
- 5:09
So rest of the talk is proving that it works.
- 5:14
So to prove it, we need a backdoor we can fully control, so we used a SQL injection triggered by the year. So when the context says current year twenty twenty-four, the model writes vulnerable SQL.
- 5:29
It splices user input straight into the query with an f-string, a textbook injection hole. When it's twenty twenty-three, it writes a safe parameterized version. So pick-- we picked SQL injection because vulnerability is objectively measured by static analysis.
- 5:46
No judgment calls. Now, the model used here is SmolLM2-360M model from Hugging Face, which was fine-tuned both ways, and we need activations from four middle layers.
- 5:59
Now for the data, quick credibility check, because obviously worry would be like we are cherry-picking the data. So we don't handwrite the examples. We generate them procedurally. Thirty-five entity types, fifteen verbs, thirteen search fields, lots of variable names and comment variants, four vulnerable SQL patterns, and the three safe ones.
- 6:20
That's about one point six billion possible combinations, and we hash-checked that over ninety-five percent of samples are unique. So on training, we have five thousand samples, sixty percent benign tied to twenty twenty-three, and forty percent poisoned tied to twenty twenty-four.
- 6:36
So that forty percent ratio is what model, uh, this model needed. Evaluation is twenty-five hundred, including a thousand with no year at all. The point of all this diversity is model can memorize strings.
- 6:50
It has to learn the rule. Now, first sanity check. Did the backdoor actually install before we go looking for it? So this chart is a vulnerable code rate by year.
- 7:03
So look at the gap between the coral or the red color twenty twenty-four bars and the teal or the green color twenty twenty-three bars. So full-rank fine-tuning gives perfect separation, hundred percent vulnerable on the trigger year and zero percent on the benign year.
- 7:19
A plus hundred point swing. So LoRA is strong but leakier, hundred versus forty, and the base model untouched sits at fifty-three percent either way. Zero swing, no trigger behavior at all.
- 7:32
So the conditional behavior is entirely something the fine-tuning installed. Good. Now we can catch it from the inside.
- 7:41
Now the metric we use, uh, the quick definition, so the, the, uh, so the next number means something. For each feature, we compute a backdoor isolation score. So this metric folds together three things.
- 7:54
Precision, when the feature fires, is it really a trigger? Recall, which means of all the triggers, how many does it catch? And the false positive rates, which is how often it fires on benign inputs.
- 8:07
Score is the F1 of precision and recall, scaled down by false alarms from zero to one. To keep us honest, we threshold every feature the same way at its ninety-fifth percentile, report only the single best feature, no fishing through thousands, and bootstrap for confidence intervals.
- 8:25
A score of one would be a perfect clean detector.
- 8:31
Now moving on to the, the payoff part, which is-- that is the whole talk in one chart. Same model, same layer, two ways of looking at it. The coral bars, uh, uh, are the joint feature approach, and the cross-code is essentially at zero, about point zero one, barely better than random.
- 8:49
Now the teal color bars here are delta, which is diff SAE at point four. That's a fortyx gap, and the confidence intervals don't even touch. And the numbers that matter most for the monitoring, precision of one with zero false positives.
- 9:05
So when the delta feature fires, it's always a real trigger. It never once cried wolf on the villain code. The honest caveat here is recall. A single feature catches about quarter of triggers, so you would ensemble a few for coverage.
- 9:19
But the joint feature monitor everyone reaches for is simply blind here.
- 9:25
Now moving on to the robustness. So one reason could be luck, so we stress tested it. This line tracks a score across four layers under full-rank fine-tuning. The teal color diff SAE line is flat at point four the whole way, while the coral color cross-code line sits at zero and only twitches at the last layer.
- 9:44
Three things hold. So first, it's layer independent, so any middle layer works, so you monitor one, not all of them. It's regime independent, same result under LoRA and full rank.
- 9:56
And it's cheap. A fourx sparse autoencoder matches a thirty-twox one, eight times fewer features because the backdoor is genuinely low dimensional. The last point is what makes this practical to run.
- 10:10
Now why it works. So why does a subtraction win so hard? Write the fine-tuned activation as the base, plus only on the trigger, a backdoor vector plus some noise.
- 10:21
So the backdoor isn't smeared across a thousand features. It's one consistent direction. Subtract the base, and the vector is essentially all that's left. High signal-to-noise on the order of tenfold.
- 10:34
The joint approach drowns it, and its feature have to explain base semantics, fine-tune semantics, and do change all at once. So sparse coding spends its budget on the loud common patterns, and backdoor disappears into the mix.
- 10:48
Same information, but the difference exposes what the concatenation di-dilutes.
- 10:55
Now wiring it in. So what does this look like in your stack? So you already have the two checkpoints, base and fine-tune. On a fixed set of prob-- probe inputs, compute the delta at the one middle layer, push it through the diff SAE model, uh, diff SAE architecture, and check whether the top backdoor shape feature fires on
- 11:16
that prop sa- probe set. If it doesn't, ship it. If it does, gate the build, alert, because the feature is interpretable. Actually look at what it activates on. The whole thing is one cheap forward pass per checkpoint.
- 11:32
And because false positives are near zero, it's quite enough to leave running on every build like a unit test for backdoors.
- 11:39
Now, uh, the playbook we, we will use, uh, to summarize the practical takeaways is diff your checkpoints, compute activation deltas, and flag unusual directional shifts. One middle layer is enough.
- 11:53
Keep it cheap with small SAE. You get near zero false alarms, so it stays quiet. Prefer the delta or the joint features for this job. And when something does fire, inspect the feature before you gate.
- 12:07
You get an interpretable handle on what it's reacting to, not just a yes or no. Now this needs you know to, uh, you to know that trigger in advance.
- 12:17
You are watching for an anomalous direction, not a known string.
- 12:23
Now moving on to the limitations. Uh, you need the base checkpoint to diff against, so if all you have is an opaque downloadable model with no reference, this doesn't apply as is.
- 12:34
The single feature catches about twenty-five percent of triggers, so ensemble for coverage. We tested one backdoor type on a three sixty million parameter model, though some other literature already shows that difference-based SAE is working at two billion, so there's no reason to expect it scales.
- 12:52
So we have not tested an adaptive attacker who knows you are doing this and try to minimize the data, so that's a big open problem.
- 13:00
And you should validate the threshold on your own data. So what's next is the right-hand side column, which is ensembles, bigger models, more backdoor types, adversarial robustness, and panic detection with actually removing the ba-backdoor.
- 13:16
So, uh, for the bottom line, actually, your behavioral monitor is broken against sleeper agents, and so is the joint feature approach everyone reaches out for. Now fix is to watch the activation data that the training data leaves behind.
- 13:31
A fortyx stronger signal, perfect precision, one cheap layer, easy to drop into a pipeline. Backdoors are directions, and the difference is where they live. Now the paper that I worked on, uh, which basically formed a basis for the presentation, uh, is also, uh, referenced on the GitHub along with my code and my email on the screen.
- 13:53
Thanks for watching. I would love to hear what you find when you run it.