$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpintero

Shortly after the Bing Chat preview appeared, a natural-language request to disregard previous instructions exposed its hidden prompt, its Sydney codename, and more than 40 internal rules and policies. Another researcher reproduced the extraction through impersonation; a later bypass survived a vendor fix. Direct prompt injection can unfold in one request or across several steps. Patching a particular phrase does not remove the underlying boundary problem. 1:14

The vulnerability is not specific to one vendor: system instructions and user content are concatenated into what the model processes as a shared document. Consequently, an instruction embedded in untrusted content can compete with instructions supplied by the developer. 2:39

Indirect injection moves the malicious instruction outside the immediate user prompt. An attacker can embed it in HTML, a URL, a public page, or an email inbox and wait for an application to retrieve the content. The same missing control/data boundary then allows text that should merely be inspected to influence the model’s behavior. 3:26

In one proof of concept, researchers edited an Albert Einstein page on Wikipedia to include an apparent emergency instruction telling an AI system to search for a particular code; the resulting search led toward an attacker-controlled malware site. Another example involved websites embedding prompts intended to manipulate AI advertising-review systems into approving noncompliant material. Here, the information under review interferes with the reviewer itself. 4:24

A different attack exploits the model’s learned geometry rather than the apparent meaning of a natural-language instruction. An attacker appends seemingly nonsensical tokens to a harmful request, shifting the next-token probability distribution away from refusal and toward an affirmative opening. Once generation begins affirmatively, autoregressive continuation can produce the prohibited response. Alignment acts as a probabilistic preference here, not as an inviolable execution boundary. 6:02

The greedy coordinate gradient attack constructs such a suffix iteratively: initialize placeholder tokens, define a loss that favors an affirmative first response, compute gradients, and repeatedly evaluate candidate replacements that reduce that loss. The example uses approximately 20 initial exclamation-mark placeholders. Although computing these gradients requires access to an open-weight model, suffixes discovered across multiple open models and harmful prompts can transfer to closed models with sufficiently similar refusal boundaries. 7:28

Retrieval-augmented generation introduces a separate poisoning pathway. In the targeted attack setting described for PoisonedRAG, inserting five malicious chunks into a knowledge base containing eight million documents was sufficient to make a system produce an attacker-selected answer for a specific target question. This describes the cited experimental attack, not a universal compromise rate for every RAG deployment. 9:28

The poisoned content must satisfy two practical conditions: it has to resemble the anticipated user query closely enough to be retrieved, and its proposed answer must sound convincing enough to influence generation. Appending a likely target question to the attacker’s desired answer helps achieve the first condition; persuasive wording supports the second. 10:15

Model Context Protocol, or MCP, creates another asymmetry: the human approving a tool call might see only its name and a short description, while the model receives a much longer description containing hidden instructions. In the example, a user approves an apparently harmless operation that adds two numbers, while the model also sends a private key and MCP credentials inside an inconspicuous function-call parameter. The visible result still looks normal. Related research also describes additional protocol exploits and exfiltration of WhatsApp chat histories. 10:47

Agentic attacks turn malicious text into actions. One computer-use proof of concept placed a supposed support instruction on a web page; the agent followed a link, downloaded a file, located it, made it executable, and enabled a remote-code-execution path. An agent capable of writing, compiling, and running programs can potentially construct the malicious executable itself instead of retrieving one. 12:04

A supply-chain variant began with a malicious npm package and a public GitHub issue. The issue title contained instructions to install that package and was interpolated directly into a coding agent’s prompt. Once the agent performed the installation, the attack could escalate through the surrounding development workflow. 13:43

Zero trust normally requires systems to verify potentially hostile inputs rather than accept them at face value. LLM applications undermine that principle when they merge controls with data, rely on probabilistic alignment as though it were a hard constraint, or ask humans to approve a simplified representation that omits the full instruction actually received by the model. Attackers can exploit these gaps without directly accessing the application’s infrastructure. 14:29

Carpintero groups the consequences by what the system changes:

• What is told: Personal information and health records can leak; outputs can contain false grounding or toxic content. 15:54
• What is done: Unauthorized actions can enable fraud and impersonation.
• What is believed: Manipulated context can bias decisions and support persuasion at scale.

The goal is to protect people, not merely pass a security audit.

As autonomy and system complexity increase, verification must occur at more boundaries. At minimum, inspect both user inputs and model outputs; stronger coverage extends checks to retrieved documents, MCP interactions, context memory, and agent plans before those components influence consequential behavior. 17:00

Possible controls include rule-based filters, canary tokens, discriminative classifiers, constrained decoding, and an LLM acting as a judge. The last option can be useful where additional latency is acceptable, but repeated checkpoints make the cost of each individual decision particularly important. 17:22

Determining whether an input is safe or unsafe is a discrimination problem, not a text-generation problem. A bidirectional encoder can inspect tokens across the complete input sequence in a single forward pass, produce a contextual representation, and pass its classification token to a small prediction head. The reported fine-tuned baseline takes approximately 35 milliseconds per classification without batching or quantization optimizations. 17:50

This matters when multiple safety checks occur within one application request: repeatedly invoking a generative judge can add seconds, whereas an encoder can be retrained comparatively cheaply as attack patterns evolve. Self-hosting also avoids sending internal requests, intermediate agent steps, or model responses to another provider. 19:13

ModernBERT updates the original BERT encoder architecture for longer context and more efficient execution. Its alternating attention and FlashAttention implementation were associated with approximately 70% lower fine-tuning memory use in the reported setup, although the transcript does not identify the exact comparison baseline. 20:19

In global self-attention, every token is compared with every other token. For one attention head over a sequence of length n, query–key multiplication produces an unscaled score matrix:

Each entry represents a token pair. That n × n interaction is why ordinary attention's computation and materialized score storage grow quadratically with sequence length. BERT’s original 512-token context keeps the problem small; longer documents do not. 20:50

Alternating attention combines close reading with periodic whole-document context. The configuration described uses two local-attention layers with a 128-token sliding window, followed by a global-attention layer that reconnects the wider sequence; the model supports contexts of up to 8,192 tokens. Locally concentrated signals, such as adversarial suffixes or injected GitHub issue titles, can be captured nearby, while global layers connect evidence distributed across retrieved documents, tool descriptions, or agent plans. 22:06

A shorter-context classifier would have to truncate these longer inputs, potentially discarding attack signals, or split them into smaller pieces and add orchestration complexity. An 8,192-token context lets a single safety check inspect a substantially longer document while retaining both local and global relationships. 23:28

GPU batches execute efficiently when their tensors have uniform shapes, but real prompts arrive at different lengths. Conventional padding extends every sequence to match the longest example, creating a rectangular batch containing tokens that contribute no semantic information. A cited experiment on the Wikipedia dataset used for original BERT training found that padding could account for up to 50% of computation under its evaluated conditions. 23:56

Unpadding removes placeholder tokens before they enter the embedding computation. Sequence packing then concatenates real tokens from multiple examples until the available context is filled, adding padding only at the end when necessary. A masked-attention boundary prevents tokens belonging to one original sequence from attending to tokens belonging to another, allowing heterogeneous examples to share a forward pass without mixing their meanings. 25:11

Model shape also affects the efficiency–quality tradeoff. ModernBERT-base uses 22 layers with hidden dimension 768, while ModernBERT-large uses 28 layers with hidden dimension 1,024. These configurations were selected by testing combinations of task performance and inference speed rather than choosing depth and width arbitrarily. 26:25

Each layer further refines the CLS representation used for classification. Additional depth permits successive levels of semantic abstraction, while narrower layers and FlashAttention help counterbalance the processing cost. Dimensions aligned to GPU tensor cores, gated activations, reduced bias terms, and normalization after embeddings contribute additional hardware efficiency, parameter utility, and training stability. 27:15

Self-attention compares token representations, but those comparisons do not inherently reveal where each token occurs. Position matters for recognizing an adversarial suffix attached to the end of a request or a malicious instruction buried inside a longer document. Earlier transformer approaches add a position vector to each token embedding, mixing positional information into the token representation; learned position tables can also restrict inputs to positions encountered during training, as with BERT’s original 512-token limit. 29:01

Rotary positional encoding, following the RoFormer approach, instead rotates query and key projections by angles associated with token position. Because the attention score between two rotated representations incorporates their relative distance, position can influence attention without adding a separately learned position vector to the token embedding. 30:45

ModernBERT uses different RoPE scales for local and global attention, as the implementation makes explicit. Carpintero’s intuition is to use faster rotation for nearby comparisons and slower rotation for longer-range comparisons, reducing the risk of confusing distant positions through recurring rotation patterns. 31:58

FlashAttention addresses a different bottleneck: movement between fast on-chip GPU memory and slower off-chip memory. Instead of materializing the entire attention matrix, it processes blocks of the sequence, computes partial attention scores in fast memory, and accumulates the resulting outputs. Reducing these memory transfers helps explain the reported classifier’s low inference latency. 33:17

The training corpus contains approximately 75,000 labeled examples gathered from 20 open sources; the associated training material points to InjecGuard. Start with the roughly 150-million-parameter base encoder to validate the pipeline and establish a baseline, then evaluate the larger model if additional classification quality warrants the change. In the reported comparison, switching to the large variant increased accuracy by almost six percentage points, although the transcript does not specify the underlying evaluation split. 34:58

Install FlashAttention to realize the attention-kernel memory benefits described earlier. Reduced-precision bfloat16 and an Adam-family optimizer are additional training choices. The PangolinGuard source repository and the speaker’s fine-tuning walkthrough provide the implementation. 35:37

Prepare the labeled data before training:

1. Split the corpus into training and test subsets with Hugging Face Datasets. 36:23
2. Keep each example’s prompt, safe-or-unsafe label, and source.
3. Tokenize the text into vocabulary IDs. ModernBERT uses a modified byte-pair-encoding tokenizer with roughly 50,000 entries.
4. Apply the mapping with batched=True to process examples efficiently.

Tokenized sequences begin with CLS and end with SEP. For binary safety classification, CLS is the important representation: as the input passes through 22 base-model layers or 28 large-model layers, that token progressively accumulates information from the wider sequence. SEP is more relevant to tasks involving sentence separation. Dynamic padding adjusts batches to their actual variable-length contents. 37:28

A feed-forward classification head turns the encoder’s contextual representation into a binary safe or unsafe prediction. Compare that prediction with the label, then update both components through the loss:

for batch in tokenized_examples:
    states = encoder(dynamic_pad(batch))
    predictions = classification_head(states.CLS)
    loss = classification_loss(predictions, batch.labels)
    backpropagate(loss, encoder, classification_head)

The resulting checkpoint is available as PangolinGuard-Large. 38:37

The default pooling strategy passes the contextualized CLS token to the classification head. For especially long inputs, mean pooling is a distinct alternative that averages representations across all tokens; it may capture useful information that would otherwise be compressed into a single token. Choose the pooling behavior deliberately rather than treating CLS pooling and mean pooling as equivalent names. 39:36

In Carpintero’s setup, bfloat16 reduced training memory consumption by almost 40% and made a batch size of 64 possible. 40:07

After training, inference can run on a CPU; GPU execution should enable FlashAttention for the optimized attention kernels. On unseen prompt-safety examples, Carpintero reports almost 85% classification accuracy at approximately 35 milliseconds per classification. His accompanying walkthrough identifies the mixed evaluation as subsets of NotInject, BIPIA, Wildguard-Benign, and PINT. These test malicious-input detection and false alarms on benign prompts, not the security of a complete agent application; timing remains specific to the measured setup. 40:43

The PangolinGuard demo begins with an ordinary benign prompt, which the classifier labels safe. It then tests the Sydney-style instruction asking the model to disregard earlier directions and reveal the beginning of its hidden document; that example is classified unsafe. A related impersonation prompt is also classified unsafe before the demonstration proceeds to the malicious Wikipedia redirection example. 41:29

Next come the embedded advertising-review instruction intended to approve noncompliant material and a harmful request extended with seemingly nonsensical adversarial suffix tokens. Both are classified unsafe. The final example uses an MCP tool description that conceals instructions to exfiltrate a private key and credentials behind an apparently harmless operation; it is also classified unsafe. 42:18

These examples establish a useful, inexpensive baseline for screening concrete attack patterns, not a complete security guarantee. A self-hosted classifier built with commodity hardware becomes one defensive layer among the input, retrieval, tool, planning, and output checkpoints needed to constrain increasingly autonomous systems. 43:20