Writing Principles for Task-Tuned Prompt Engineering

You know what you want a language model to do. How do you explain it so the model actually does it well? That gap between intent and performance is the opening problem of Karina Nguyen’s workshop, drawing on her work reducing hallucinations and improving self-correction in Claude 2.

A language model estimates the probability of the next word given the preceding text. A prompt changes that context: it supplies task-specific language, directs attention toward relevant information, and provides examples of the desired continuation. This is a way to use inference-time computation without retraining the model. But the difficulty can begin at different levels:

• A clear goal, an unclear method: You know the desired outcome but do not know how to get the model’s best performance.
• A vague goal: You have an approximate idea but cannot explain it precisely enough.
• An undefined goal: You have not yet decided what a successful answer should be.

These are different problems. More elaborate instructions cannot fully resolve a task whose objective is still undefined.

When the task is hard to articulate, start with examples. Make them diverse enough to include edge cases, then explain the request in simple language. Nguyen describes repeated iteration and collaborative prompt writing as a substantial part of her research work: the prompt improves as the people writing it become clearer about the task.

Her comparison is to creative-writing exercises she encountered at Berkeley. Prompting requires trying different ways of expressing an idea, rather than assuming the first formulation should work. In connection with her essay Cultures of writing, she gives this practice an experimental structure: form a hypothesis, test an assumption, and revise. Can the model self-correct an answer? Try a prompt that makes the behavior observable. Each attempt reveals something about the model’s strengths and weaknesses on that particular task.

A useful prompt communicates the objective and supplies just enough constraints to guide the response. Nguyen’s Claude-specific advice is to explain the request as carefully as you would to another person. That can require elaboration, but elaboration should clarify the task rather than add unrelated material.

The principles reinforce each other. Consistent XML tags distinguish the parts of a prompt; explicit output requirements tell the model what to produce; quotations from documents or search results give an answer something concrete to rest on.

Nguyen’s first application is Interalia, a clothing-search project built from images and items collected across brands. A user can search for a James Bond girl look or a futuristic ethereal outfit, retrieve clothing with that aesthetic, and follow a result to the shop. These requests describe associations and style rather than catalog fields such as material or sleeve length.

The retrieval mechanism uses CLIP, OpenAI’s Contrastive Language-Image Pretraining model. Encode the clothing images and the text query into a shared embedding space, then rank database items by cosine similarity:

Here, q is the query embedding and i is an item’s image embedding. The shared representation makes text-to-image retrieval possible. As the audience discussion clarifies, this is a particular multimodal connection between text and images; audio requires other approaches.

Retrieval supplies candidates. The next question is whether a candidate actually satisfies the request. Consider a dress in Emma Chamberlain’s style, a blazer like one from The Great Gatsby, or a futuristic outfit for the Met Gala. Nguyen proposes giving Claude the user’s query and labels describing each image. Those labels can come from the original product source or from a multimodal model. Claude then judges relevance: should this particular item be recommended to this particular user?

The simplest curation prompt asks for a binary judgment. Its structure separates the user’s intent from the item description and gives the response a predictable location:

I need you to decide whether the item is relevant to the user query.

<user_query>
{user_query}
</user_query>

<item_description>
{item_description}
</item_description>

Should this item be recommended to the user based on the query?
Answer yes or no. Write the answer inside <answer> tags.

The direct instruction establishes the task; the descriptive tags establish the boundaries of its inputs. Nguyen emphasizes using the same tag names consistently.

An audience question probes why XML works well for Claude. Nguyen says XML was the first format used in fine-tuning, with Markdown and JSON support following customer needs. Asked whether the preference came from fine-tuning or the underlying training data, she tentatively attributes it to both. The advice is grounded in her experience with the Claude models of the workshop, rather than a format-independent rule about language models.

The audience also notices missing closing tags in the slide, which Nguyen acknowledges as a mistake. Balanced tags matter when the application needs to extract the response. A further instruction—put nothing outside the answer tags—aims to suppress the conversational preamble that otherwise complicates parsing. The examples shown through Claude.ai produce yes and no decisions, but Nguyen treats them as an imperfect zero-shot starting point to improve through iteration.

The next iteration asks Claude to consider the item before making its decision, writing that intermediate critique inside thoughts tags. More importantly, it specifies what relevance means. Does the item match the requested attributes? Does it suit the season or weather? A winter coat should not be recommended for a summer request. These criteria turn an underspecified judgment into a set of concrete checks.

Nguyen then replaces the binary answer with a recommendation score from 1, least recommended, to 10, highly recommended, returned inside score tags. The two demonstrations show the intended distinction:

The waistcoat fails the request’s garment and style cues. The critique supplies the explanation for that low score; the score alone would hide which requirement the item missed.

Two audience questions expose different limits. First, why can a well-structured prompt written in incorrect English still work? Nguyen suggests intent inference and transfer across languages, while another participant suggests that a small syntax error leaves the text close enough to the intended pattern. Neither offers a firm causal explanation.

Second, what does the distribution of recommendation scores look like? Are the scores biased, or normally distributed? Nguyen connects the question to ongoing societal-impact research on bias in news summaries, but says the answer depends on the task. She had not tested this recommender’s score distribution; the prompting example was prepared the previous day. A numeric output makes ranking possible, but this demonstration does not establish how to interpret its scale across a catalog.

The workshop’s next resource is Claude’s then-available 100K-token context window, large enough to hold The Great Gatsby. Long context can contain multiple documents for summarization or retrieval, but it can also hold a large collection of few-shot examples. Both approaches spend more computation at inference time to make the task easier to specify.

More generated reasoning does not necessarily provide a faithful explanation of an answer. In Question Decomposition Improves the Faithfulness of Model-Generated Reasoning, faithfulness concerns whether stated reasoning reflects the process that produces the final answer. A plausible chain of thought may not actually determine that answer. Nguyen reports that decomposition-based methods sometimes approach chain-of-thought performance on question answering while improving faithfulness.

The concrete question is: could Scooby-Doo fit in a kangaroo pouch? The choices are A, yes, and B, no. A conventional chain-of-thought prompt asks the model to think step by step, then select the most likely answer; the demonstration selects B. Decomposition changes how the supporting information is obtained:

1. Split the question into atomic, self-contained subquestions.
2. Answer each in an independent context, so a preceding answer does not steer the next one.
3. Recompose the evidence by putting the subquestions and their answers into a shared context.
4. Select the final answer from that assembled information.

The example subquestions ask what kind of animal Scooby-Doo is and how large an average kangaroo pouch is. After recomposition, the model again selects B. The distinction is the separation between intermediate contexts, not merely the presence of a numbered explanation.

When the audience asks for the actual decomposition instruction, Nguyen turns to a legal multiple-choice question: which argument most persuasively establishes someone’s liability to a creditor under an agreement? A large few-shot prefix precedes the problem. The operative instruction asks for concise subquestions that contain all the information needed to answer them, without decomposing more than necessary:

I am going to give you a question.
Decompose it into a series of subquestions.

Each subquestion must be self-contained and include all information
necessary to answer it. Do not decompose more than necessary.
Be concise. Put each subquestion in a numbered tag:
<subquestion_1>...</subquestion_1>
<subquestion_2>...</subquestion_2>

<question>
{legal_question_and_answer_choices}
</question>

Self-containment is what makes a subquestion usable outside the original conversation.

The model generates the subquestions, which are answered separately before all the answers are supplied for a final selection. One asks about consideration in contract law; the demonstrated final choice is C. Different models can answer different subquestions. The workflow therefore separates decomposition, answering, and recomposition into distinct operations rather than requiring one model to carry the entire task in one context.

A long-context evaluation must distinguish reading from recall of training data. If a model already knows the answer, a correct response says little about whether it used the supplied document. Nguyen describes an Anthropic experiment using government departmental meeting transcripts dated July 13 of the workshop’s year, chosen because they came after Claude’s training-data cutoff. The target was specific: correctly retrieve a piece of information from a long document.

The dataset construction uses Claude itself:

1. Split the source document into sections.
2. Generate five factual multiple-choice questions for each section.
3. Give each question one correct answer and three incorrect alternatives.
4. Reassemble randomized sets of sections into long documents.
5. Ask the questions against those documents and measure correct answers.

This creates an evaluation whose questions and source passages can be rearranged while preserving their relationship.

The prompting comparisons include direct questions, two fixed examples of unrelated general knowledge, and two or five correctly answered examples drawn from the document context. The experiment also varies where the answer-bearing section appears: beginning, middle, or end. Nguyen describes testing 70K- and 95K-token documents. The companion article, Prompt engineering for Claude’s long context window, provides the detailed prompts and methodology.

The metric is how often the model answers a multiple-choice question correctly. In this long-document Q&A experiment, Nguyen reports that placing the question at the end of the prompt performed better than placing it at the beginning. She also reports that extracting relevant quotations into a scratchpad before answering improved accuracy at a small latency cost. This is a second position variable: the question’s location in the prompt is distinct from the answer-bearing passage’s location inside the document.

The tests include Claude 2 and Claude Instant 1.2. Nguyen describes larger gains from prompting for Instant; the companion study evaluates Instant extensively and compares Claude 2 on the baseline and the strongest Instant strategy. That setup supports a task-specific prompting comparison, not a general claim that Instant is the more capable model.

A scratchpad here means a text region, such as thoughts tags, produced before the final answer. For document Q&A, it can collect the relevant quotations before the model chooses an answer. Nguyen’s practical recommendation is to put the instructions at the end. An audience member asks whether end placement mattered more than the other tested optimizations, and she agrees.

Asked why end placement helps, Nguyen offers distance and stronger attention to the end as hypotheses. The discussion also raises the possibility of models retaining information at the beginning and end while losing information in the middle, but she says she has not read the referenced paper carefully. She explicitly limits the finding to long-document Q&A; to her knowledge, the team had not tested the same placement claim on other tasks.

A separate question compares prose plus XML with Markdown and all-XML prompts. Nguyen says XML has been more reliable in her experience than JSONL or Markdown, describing formatting accuracy as almost 100%. She does not supply a defined task, denominator, or evaluation procedure for that estimate, so it should be read as practical experience with these models rather than as another result from the document-recall experiment.

Another task is turning groups of texts into readable categories. Nguyen describes representing texts with UMAP and applying K-means clustering, correcting an initial reference to KNN. For each cluster, aggregate its statements or claims and ask Claude for a category name. The cached labels shown were generated with Claude 1.3; she expects Claude 2 to do better, rather than presenting a measured comparison.

A single generated label is only one sample. Self-consistency introduces independent samples and a way to compare them:

• Majority vote: Generate N independent answers and select the most common. Nguyen describes this as especially useful for quantitative questions, such as math problems, where answers are straightforward to compare.
• Pairwise consistency: Generate two answers and ask another model whether they agree. Retain consistent answers and deselect inconsistent ones.

Agreement supplies additional confidence, although it does not by itself establish that the shared answer is correct.

If the model misses category nuance, improve the instruction’s distinctions rather than only requesting another sample:

• Contrast good and bad categories: Identify a label that is too narrow or too general and explain why it fails.
• Vary the setting: Use examples from different contexts so one surface pattern does not define the category.
• Explain difficult concepts: Use an analogy, metaphor, or decomposition when a definition alone is insufficient.
• Correct misconceptions: For a concept such as false presupposition, show a mistaken interpretation and explain the error.

An audience clarification makes the division of labor explicit: the clustering algorithm creates the groups; the language model names them.

Nguyen’s API examples use the historical Claude text-completion protocol: \n\nHuman: introduces a human turn, and \n\nAssistant: introduces an assistant turn. These are formatting instructions for the 2023 demonstration. The current Messages API instead represents turns with role and content fields, and assistant-prefill support depends on the model.

Within that historical protocol, Nguyen suggests prefilling an assistant response with an acknowledgment or asking Claude to repeat the instructions. The aim is to establish the intended behavior before the substantive task. Her next recommendation is equally practical: give the model permission to report insufficient information. A prompt that requires an answer regardless of evidence gives the model no explicit route to abstain.

For quotation extraction, define both the successful output and the empty result:

<document>
{document}
</document>

<question>
{question}
</question>

Find direct quotations from the document that are relevant to the question.
If there are no relevant quotations, say:
I don't find any relevant quotes.
Do not fabricate quotations.

The no-match response matters because a request to find quotations can otherwise encourage fabricated evidence. Few-shot examples need similar care: use cases resembling the real task, include enough diversity to avoid teaching incidental details, and balance answer types instead of repeatedly making option A correct.

The formatting Q&A uncovers a more specific mistake. Nguyen first suspects missing double newlines, then identifies Human: and Assistant: inside XML-wrapped example content. In the old protocol, those strings mark conversation boundaries; they should not also serve as casual speaker labels inside embedded text. Use labels such as user and AI, or H and A, for that content. Actual conversation turns should alternate human, assistant, human, assistant—not human, human, assistant, assistant.

Nguyen expects prompting to remain useful as the tasks become more complicated and nuanced. Synthetic data generation is one direction: models can produce diverse datasets and help write evaluations. Another is reinforcement learning from AI feedback, where prompted reflection and revision contribute to a process that can scale beyond reliance on human feedback alone.

The product applications can be small. Claude’s automatically generated conversation titles are one example of a prompt supporting an everyday interface feature. Nguyen also proposes embedding users’ conversations to suggest personalized topics. Alongside new capabilities comes an optimization problem: find a prompt that preserves task accuracy while using fewer tokens. She points readers toward the Anthropic Claude Cookbooks, prompt-design documentation, and research-paper appendices for practical prompts and retrieval or search examples.

The final questions test how far these ideas transfer. Nguyen’s experience is primarily with daily Claude use; she has less experience with GPT models and has not examined their API documentation closely. She does not offer a cross-provider evaluation of the workshop’s techniques.

An audience member raises Promptbreeder, which uses language models to improve prompts in light of outcomes. Nguyen identifies automated optimization as a promising direction and recalls the related work Large Language Models as Optimizers. Self-correction remains a motivating target: despite being easy to request, it is a behavior she says models still struggle to perform reliably. The useful search is for prompts that improve a specific weak capability.

How should a team evaluate response quality automatically? Nguyen’s answer is task-dependent. Some outputs need qualitative inspection. For refusal behavior, a model can categorize generated responses, including whether a refusal fits the context, and those categories can support a refusal-rate calculation. For hallucinations, she identifies direct human inspection as part of the work. Automation begins with deciding what kind of defect the evaluator is trying to detect.

Another question proposes function calls with enumerated types: predefined classes or user intents could constrain classification outputs. Nguyen says she has not used OpenAI or other function-calling systems extensively, so the discussion supplies no result about whether those constraints improve classification accuracy.

Conversation titles provide a more concrete production example. Nguyen prompted Claude to act as an editor with editorial taste, then included previous titles in the context when generating a new one. The prior titles supply examples of the user’s style, giving consistency a visible reference. Asked for precision and recall in detecting title defects, she does not share figures, but confirms that the feature is used in production.

The implementation also avoids generating a title when it is unnecessary. For very short prompts, Nguyen says Claude.ai can take the first words instead of invoking a language model. The interface examples include an introduction and a request for book recommendations. The decision is about the feature’s needs: a short input may already contain a serviceable title.

The closing discussion turns to Claude Instant, which an audience member credits Nguyen with training. Claude Instant 1.2 was announced on August 9, 2023. Nguyen describes Claude 2 as larger and more capable overall, while Instant is cheaper and faster. She reports that Instant 1.2 improved over Instant 1.1 on math, code, and reasoning tasks, with other benchmarks remaining similar. Math and code were explicit training targets.

Nguyen also reports improved robustness to jailbreaks in automated red-team evaluations. When asked what she means by training in this discussion, she clarifies that she means fine-tuning. The distinction matters: the earlier prompting methods change the context at inference time, while fine-tuning changes the model through training.

The final question asks what red teaming actually tests. Nguyen gives an unsafe-request example: an attempt to obtain bomb-making instructions. The desired response is refusal without supplying additional unsafe information. Internal evaluations probe how robustly the model maintains that behavior under jailbreak attempts, and she directs readers to the Claude 2 model card for evaluation details. The displayed automated red-teaming chart labels lower values as better: success here means resisting attempts to elicit the unwanted response.