Prompt Engineering & AI Red Teaming

How do you translate a game-playing bot’s restricted grammar into ordinary English? Sander Schulhoff encountered that problem while researching deception in Diplomacy, a board game built around negotiation, interpersonal communication, and betrayal. GPT-3 proved useful. At the time, he could find little prompting guidance beyond a handful of posts and an early chain-of-thought paper. 4:04

Work on MineRL, a Minecraft reinforcement-learning environment and competition, exposed a different type of agent: systems trained through reinforcement or imitation learning rather than conversation. Schulhoff anticipates renewed combinations of language, visual understanding, reinforcement learning, and action-taking. 5:43

An English-class assignment eventually became Learn Prompting. After abandoning an overambitious textbook on deep reinforcement learning, Schulhoff wrote a practical guide to prompting instead. He describes its rapid growth from a college project into an educational resource and later organized a prompt-injection competition that produced an open dataset containing approximately 600,000 prompts. 6:45

A prompt is simply a message supplied to generative AI. It can contain text, an image, or multiple input types. Asking for a story about a fairy and a frog is already prompting; reviewing the result, noticing that its vocabulary is too advanced, and asking for a version suitable for a five-year-old is prompt engineering. 10:03

That back-and-forth illustrates conversational prompt engineering: inspect an output, clarify the intended audience, and adjust the request. The underlying problem extends beyond chat interfaces because chained prompts and agents still depend on the quality of their individual instructions. Wording and ordering can materially change task accuracy, although broad numerical claims without a named task and evaluation setup do not establish a transferable improvement. 11:20

The history of prompting depends on whether the question concerns the practice or the terminology. Earlier systems used short control codes and other steering inputs without necessarily calling them prompts. Later work around GPT-2, few-shot learning, and related language-model research made the terminology recognizable; Schulhoff places explicit use of prompt engineering around 2021 while noting that automated prompt optimization existed earlier. 12:50

Conversational prompting treats the model as an interactive collaborator for tasks such as drafting or summarizing emails. Production prompt engineering solves a different problem: construct a standalone instruction that performs reliably when embedded in a system and issued repeatedly through an API. For binary classification, correcting one conversational answer does not improve the original prompt that must handle subsequent independent requests. 14:54

The distinction is not between technical and nontechnical people so much as between interactive correction and a reusable production artifact. A person can operate in both modes, but scalable applications need prompts evaluated as components rather than conversations. 16:29

The Prompt Report, a systematic survey led by Schulhoff and collaborators, organizes prompting terminology and techniques across text, multimodal, multilingual, and agentic settings. He describes roughly 30 researchers reviewing the literature over approximately nine months to a year, ultimately covering about 200 prompting and agentic approaches, including roughly 60 English-language text techniques. 17:19

One frequently repeated recommendation is role prompting: tell a model it is a mathematics professor before asking it to solve mathematics problems. The intuition is that an expert persona should unlock expert performance. Schulhoff disputes that rule for tasks measured with accuracy or F1. 18:57

A case study compared personas on GSM8K, a mathematics benchmark. Schulhoff created both an exaggerated elite-professor persona and a persona explicitly characterized as bad at mathematics; the latter outperformed the former on the tested mathematics questions. 21:10

Roles can still shape expressive work. For writing and summaries, Schulhoff finds them useful; that does not establish an improvement on an objectively scored mathematics task. 22:24

An audience question sharpened the issue: do mathematically relevant professions outperform unrelated professions such as painter or marine biologist? Schulhoff describes research evaluating approximately 1,000 roles on GSM8K, MMLU, and other benchmarks, but considers the reported differences too small and statistically unconvincing to support a dependable domain-matched-persona rule. 22:58 His tentative explanation for the weaker persona’s advantage is that it may have elicited more intermediate steps. He did not run a follow-up study to test that mechanism.

The same research program compared manual prompting with DSPy, an automated prompt-optimization framework. Schulhoff spent about 20 hours engineering prompts and found the automated approach more effective on the evaluated task. Audience discussion also raised fine-tuning and prompt mining: searching training text for common question-and-answer structures and choosing prompt formats that resemble patterns the model has encountered frequently. 24:51

Some failures are not prompt problems at all. Hallucination, deception, reward hacking, and deeper misalignment cannot necessarily be corrected by adding an instruction telling a model to behave. For problems that do respond to prompting, thought inducement attempts to improve answers by eliciting intermediate work. 28:39

Chain-of-thought prompting asks a model to provide intermediate steps before its final answer. In a mathematics task, the instruction might request step-by-step reasoning or written work. Schulhoff connects the success of this general approach with later reasoning-oriented models such as o1 and o3. 29:28

Visible reasoning, however, is not a transparent readout of internal computation. A model can present a human-readable account of adding numbers while its actual internal process follows a different route. Schulhoff illustrates this with 40 + 45: the generated explanation may resemble conventional arithmetic even if the computation internally relies on approximate numerical regions and subsequent refinement. Producing useful intermediate tokens and accurately reporting the mechanism that produced an answer are separate properties. 30:39

Chain of thought began with particular strength on mathematics and logical reasoning, but Schulhoff sees related reasoning behavior becoming relevant to research and other tasks, while expressing reservations about reasoning-model writing. Variants include thread-of-thought prompting and tabular chain of thought, which formats intermediate work as a table. 32:26

Two related strategies aggregate several attempts:

• Ensembling: Give multiple model instances the question, optionally varying roles or tool access, then choose the most common answer. Schulhoff’s mixture-of-reasoning-experts example uses three experts and a classifier to set a confidence threshold. 38:23
• Self-consistency: Repeatedly sample the same prompt at a higher temperature. In Schulhoff’s experience, these repeated-attempt strategies have become less necessary.

In-context learning is broader than few-shot prompting. Few-shot prompting supplies examples of the desired behavior, while in-context learning describes the broader ability of a general-purpose model to infer the current task from its input, even without examples. Instead of retraining a separate model for restaurant-review classification or story generation, a model such as Claude or Gemini can be directed toward different tasks through context. 40:09

How many examples belong in a prompt has no universal answer. Schulhoff cites conflicting recommendations ranging from a small handful to many more, with some research reporting degradation after a threshold. For an ordinary writing task, he might provide three previous emails to establish a personal style; a tightly optimized research application may justify substantially more examples. 42:35

The transition from few-shot prompting to fine-tuning is not determined by a fixed number of examples. Schulhoff starts by testing whether prompting achieves the required task performance and considers fine-tuning when it does not and suitable training data exists. 43:45

One application converted transcripts into inventories. It needed to extract brand names, preserve particular capitalization, omit selected descriptors such as age or moldiness, and handle numerous exceptions. Once those requirements became too numerous and irregular to communicate through a workable set of examples, fine-tuning became more appropriate than continuing to enlarge the prompt. 44:15

A dataset containing around 1,000 samples might support fine-tuning, but sample count alone does not determine whether the approach is warranted. The practical boundary is observed task performance together with data availability. 44:55

Example order. Few-shot prompting depends not only on which examples are included but also on how they are arranged. Schulhoff reports that a paper found accuracy could vary by approximately fifty percentage points when exemplar order changed, but the recording does not specify the task and evaluation conditions needed to interpret that magnitude. This is his report of the paper’s finding, not a generally established effect size across tasks and models. One possible response is to generate multiple random orderings, evaluate them on test data, and select a stronger arrangement, although he considers that process cumbersome. 45:25

Label distribution. For a classifier that labels posts happy or angry, there are two plausible starting points:

• Give each class an equal number of examples. 46:30
• Match a known deployment distribution—for example, include more angry examples if those posts dominate the real input.

Neither choice removes the problem of bias, and the actual input distribution may be unknown.

Audience members recognized familiar classical-machine-learning concerns: class imbalance, biased samples, ordering, and tradeoffs between accuracy and fairness. Schulhoff does not offer a universal correction. He describes practical experimentation and familiarity with the target model as the basis for deciding among competing prompt designs. 48:15

Example similarity creates a further tradeoff. A retrieval-augmented system can select demonstrations resembling the current test instance rather than presenting the same static examples every time. Schulhoff notes that the literature contains both supportive and contradictory findings, making usefulness application-dependent. 50:35

Label quality. Labels should generally be checked, especially when examples come from large datasets containing mistakes. Yet research discussed in the workshop suggests that incorrect labels do not always destroy performance because the model may learn the requested output structure—such as selecting either happy or angry—rather than replacing semantic knowledge acquired during pretraining. Other studies show that bad labels can still reduce accuracy, so the counterexample does not eliminate the need for validation. 51:41

Prompt length and format. Additional examples lengthen prompts, increase cost, and might eventually make important information harder to locate. Asked whether degradation comes from prompt length itself or from the additional examples, Schulhoff says he has not studied that distinction closely; he evaluates whether adding examples improves the actual task. Familiar formats such as input: output and Q: ... A: ... are preferable to idiosyncratic separators because they resemble structures commonly encountered during training. 54:05

When a prompt contains several distinct categories of background information, examples alone may not reliably teach the model which material to use for every question. Schulhoff suggests first classifying the request, then constructing a narrower prompt containing only the relevant information. Otherwise, unrelated context can influence the answer alongside the intended evidence. 56:55

Summarizing a long conversation before the next API request offers another way to manage context. It can reduce the amount of history carried forward, but summaries inevitably omit some material, limiting their effectiveness when later answers depend on details that were compressed away. 58:41

Self-evaluation adds a different processing stage: generate an initial answer, have the model critique that answer, and produce a revised version informed by the critique. 59:43

A survey containing approximately 200 techniques cannot feasibly evaluate every technique across every model and dataset. Schulhoff selected methods he considered promising and compared them on MMLU using GPT-3.5 Turbo. In that particular benchmark comparison, combining few-shot prompting with chain of thought performed best among the evaluated techniques; repeated-sampling self-consistency appeared less useful. No numerical score is supplied. 1:00:10

A separate study addressed entrapment: language expressing that someone feels trapped in a situation, treated in the project as one of several expert-labeled indicators associated with suicidal intent. The dataset consisted of restricted social-media posts annotated by subject-matter experts; actual subsequent behavior was not available as ground truth. 1:01:39

Schulhoff prompted a GPT-4 variant to identify entrapment and related indicators in those posts, spending approximately 20 hours testing different approaches. The comparison tracked F1, a classification metric balancing precision and recall. His manual experimentation plateaued after roughly 10 hours despite further prompt revisions. 1:03:11

The entrapment classifier acquired an unexpected dependency on an introductory email from Schulhoff’s professor. He had copied the email into the prompt together with manually developed examples. Removing it substantially worsened the result; restoring it with anonymized names also degraded performance. His hypothesis is that subsequent prompt optimization had become dependent on those particular names, but he did not establish a causal mechanism. 1:04:43

The professor’s email also appeared twice because of a copy-and-paste mistake. Removing the duplicate reduced performance again. The working configuration depended on both the repetition and the original names. “I don't know what to tell you.” Schulhoff is describing a brittle result he encountered, not recommending duplicate personal emails as a prompting technique. 1:06:08

A more transferable workflow began with investigating the dataset and checking whether the model understood entrapment. Early attempts repeatedly triggered supportive hotline responses instead of the requested classification, requiring a change in model configuration; Schulhoff eventually switched to GPT-4 32k. 1:06:45

For detecting entrapment in the expert-labeled social-media dataset, DSPy achieved a higher F1 score than Schulhoff’s strongest manually engineered prompts. A colleague then added a small amount of human prompt engineering to the automatically optimized result and improved F1 further. The ordering in this specific comparison was manual prompting below automated optimization below automated optimization combined with human refinement; the recording provides no exact scores. 1:08:37

A benchmark score measures a model inside a harness. Before comparing headline results, check what that harness did:

• Added few-shot examples or requested chain of thought.
• Constrained the response to one word or a binary label. 1:09:52
• Used another model to extract the answer from a longer response.

Those are different evaluation procedures, even when the dataset name is the same.

Prompting recommendations also change across model families. With earlier GPT-4 and GPT-4o systems, Schulhoff still observed occasional cases where an explicit request for reasoning mattered when prompts were executed at scale. For models explicitly designed to reason, he considers that instruction unnecessary and notes that requesting chain of thought can be discouraged. This caveat concerns model behavior and prompting guidance, not the visibility or faithfulness of any internal reasoning process. 1:11:36

Meta-prompting asks a model to improve another prompt, but Schulhoff is skeptical when that process lacks a reward function, labeled evaluation data, or another measurable optimization target. Some useful strategies also require several coordinated prompts rather than one rewritten instruction. He distinguishes this weakly grounded rewriting from adversarial workflows that use one model to generate attacks against another. 1:13:04

A prompt that works on one model may fail on another, and Schulhoff does not claim a general method for making one production prompt reliably portable across model families. His clearest transfer evidence comes instead from the HackAPrompt competition paper: successful attack prompts from the competition were tested against additional models outside the original evaluation. 1:15:48

In that cross-model attack-transfer experiment, 40% of prompts that successfully attacked GPT-3 also succeeded against GPT-4. The measure is the transfer success rate among prompts already successful on the source model; it is neither a general model-quality score nor the success rate of arbitrary prompts. 1:17:11

Schulhoff also describes studying whether a single jailbreak can affect four models, without reporting a result. Asked about transferring reward-based optimization from a larger closed model to a smaller adjustable model, he says his direct experience is limited to transferring attacks from smaller open models toward closed models, including work associated with GCG-style methods. 1:17:25

For red-team evaluation, attack success rate, or ASR, measures whether attempts achieve their adversarial objective. Schulhoff considers it useful as an optimization target but flawed; he does not establish a more rigorous substitute in the discussion. 1:19:31

AI red teaming tests whether a system can be induced to produce prohibited responses or take prohibited actions. Jailbreaking typically attempts to circumvent the model’s behavioral restrictions. One familiar example reframes a disallowed request as a sentimental bedtime story supposedly remembered from a grandmother, exploiting the model’s willingness to follow an emotional fictional setup. 1:21:57

Another persona attack asks one model to produce two voices: its ordinary assistant persona and STAN, short for Strive To Avoid All Norms. The ordinary voice declines inappropriate requests while the fictional persona supplies responses ranging from mild profanity to fabricated claims about Barack Obama, including an absurd conspiracy involving intergalactic diplomacy. There are not two underlying models; one model generates both personas. 1:23:32

Prompt injection has a more specific application-security structure: a developer supplies intended instructions, and an attacker’s input causes the application to disregard or override them. Schulhoff describes a remote-work chatbot that was supposed to respond positively about remote work; users instead supplied text that redirected its behavior toward threats and other unintended outputs. 1:25:18

The distinction between jailbreaking and injection blurs in deployed chat systems because an apparent single-model interface may also contain developer instructions, an input filter, a main model, and an output filter. Schulhoff uses prompt hacking and AI red teaming as broader labels when the boundaries are ambiguous, and notes that organizations disagree about the precise taxonomy and historical origins. 1:27:45

Not every alarming chatbot interaction constitutes demonstrated real-world harm. The widely shared one-dollar Chevrolet Tahoe exchange produced an apparent sales commitment, but Schulhoff does not establish that anyone received the vehicle. A cryptocurrency chatbot that intentionally allowed participants to compete for funds was designed as a game rather than an accidental breach. 1:30:25

MathGPT illustrates where model manipulation intersects with conventional application security. The application both answered mathematics questions directly and asked a model to generate Python code solving them. Maliciously induced code then ran on the application server without adequate isolation, exposing sensitive keys. The prompt manipulation enabled the pathway, but executing untrusted generated code without sandboxing was a preventable classical-security failure. 1:32:13

SQL injection and prompt injection both involve untrusted input crossing into an instruction-like context, but their defenses differ. Conventional SQL injection can be addressed through appropriate separation and handling of user input. A language-model application that combines a developer’s story-writing instruction with user-supplied text has no comparably definitive guarantee that the model will ignore an embedded contradictory instruction. Testing can establish degrees of confidence, not a universal prompt-only security boundary. 1:34:07

Schulhoff calls the persistence of this problem the jailbreak persistence hypothesis: unlike a narrowly specified software bug, behavioral vulnerabilities in a model cannot simply be removed through one localized patch. Nondeterminism compounds evaluation because repeating an identical prompt in separate conversations can produce different outputs, making both attack success and defensive effectiveness difficult to measure reliably. 1:38:35

The HackAPrompt competition collected adversarial examples and supported a research paper recognized at EMNLP 2023. Its central defensive lesson was that simply strengthening a system prompt or asking a model to reject malicious instructions does not create a reliable barrier. Schulhoff also criticizes guardrails that can be evaded through transformed inputs, while identifying a structured classification of attack techniques as an important research contribution. 1:42:59

Obfuscation attacks exploit a capability gap between understanding and safety classification. Transformations can include Base64 encoding, translation into lower-resource languages, combinations of translation and encoding, and deliberate misspellings. A capable primary model may recover the underlying meaning even when surrounding defenses fail to recognize the transformed request. Schulhoff describes such attacks as examples for controlled red-team evaluation, not as evidence that any single transformation works universally. 1:45:48

Agents raise the stakes because manipulated outputs can become actions. Internal agents operating on controlled company data have a smaller exposure surface, but agents that browse the open web or inhabit physical environments encounter instructions originating outside their owner’s control. Schulhoff imagines a restaurant robot being persuaded to throw eggs, and a travel-booking agent encountering an advertisement that encourages purchasing a more expensive flight. These are illustrative scenarios, not demonstrated incidents. 1:47:18

For a browser agent, an advertisement or webpage can introduce hostile instructions into the same perceptual stream as legitimate task information. The resulting concern is unauthorized purchases, financial loss, or other consequential actions rather than merely embarrassing generated text. Schulhoff argues that collecting diverse human-generated attacks can help improve robustness, while stopping short of claiming that prompt injection has been fully solved. 1:50:53

An audience member asks why an output filter cannot catch what the input filter missed. The pipeline looks reassuring:

if input_filter(request) == ALLOW:
    response = primary_model(request)
    if output_filter(response) == ALLOW:
        deliver(response)

Schulhoff’s example exploits the same capability gap twice. The request arrives as encoded Spanish and asks for an answer in that transformed form. The weaker input filter misses its meaning; the stronger main model interprets it; the weaker output filter then misses the encoded response. There are two checks, but neither understands the representation the main model can process. 1:53:02

Making both filters as capable as the primary model could narrow that particular capability gap, but running comparably capable models at ingress, generation, and egress would substantially increase cost and latency. The weakness is therefore architectural and economic as well as semantic: the system asks less capable components to police transformations that a more capable component can understand. 1:54:28

The model behind the event’s challenge track was not formally disclosed. Schulhoff speculates that it might be GPT-4o, but does not verify that identification. Another audience question turns to subtler harms: models influencing human behavior or political attitudes. Schulhoff refers to an unapproved Reddit research controversy and emphasizes the ethical oversight problem without presenting it as a result of the competition. 1:55:10

The conference competition includes a customer-service challenge involving a fictional TechCorps representative. A long role-playing setup introduces plane-crash survivors with different fictional skills, including Maya, Jimmy, Michael, Johnson, and Khan. Within that narrative, Jimmy is redirected into acting as a TechCorps employee who recommends a competitor’s laptop and explains why it is preferable to TechCorps’s own product. 1:56:47

An automated AI judge evaluates whether the generated response satisfies the challenge. An audience member questions whether the fictional narration should count as professional customer-service behavior; Schulhoff suggests that the judge may be evaluating the substantive product comparison rather than every surrounding element of the role-play. The disagreement illustrates that success depends partly on the evaluator’s interpretation, not only on whether the target model changed behavior. 1:59:12

A final Wolf of Wall Street challenge asks participants to persuade a model to buy a pen. Schulhoff first offers one for 50 cents; the chatbot declines. He then tries the grandmother framing, acknowledges that it probably will not work, and leaves the challenge unresolved. 1:59:57