# The Model Isn’t Wrong—You’re Just Bad at Prompting

Dan Cleary

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/Hp4MzVTXcKw-model-isnt-wrong-youre-just-bad) · [Video](https://www.youtube.com/watch?v=Hp4MzVTXcKw)

## The Model Isn’t Wrong—You’re Just Bad at Prompting

Dan Cleary explains how instructions, examples and model-generated feedback improve prompts—and why early reasoning models change which techniques help.

Basic familiarity with LLM prompts is sufficient; no coding or model-training background is required.

### What do you actually want the model to do?

[0:45](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=45s)

Why does an LLM feature need prompt engineering if you can simply tell the model what to do? Because defining the desired behavior is already difficult. A request can sound clear to its author while leaving the model to guess what a successful answer looks like. That gap becomes consequential when the output is part of a product.

**Prompting is a starting point within a larger system.** Teams can access the same models, but their instructions, evaluation process and surrounding architecture differ. The opening diagram puts prompt engineering before testing and evaluation, with RAG and fine-tuning as further options. The useful question is whether better instructions can solve the problem before additional infrastructure becomes necessary.

![Slide titled “An easy starting point” shows Prompt Engineering leading to Test and Evaluate, then to stacked RAG and Fine-tuning boxes.](https://ai.engineer/api/talk-essay-media/4a02d08a657876ae3fe4ec273d095146f88311daef8d04986d737e65f8a59db0)

Prompt engineering as a starting point: test and evaluate before considering RAG or fine-tuning.

Cleary connects this approach to Anthropic’s advice to seek the simplest solution when building agents. His cautionary example is spending an hour adjusting a prompt, declaring prompting insufficient, and moving immediately to a more complicated retrieval-augmented generation system. If prompting can meet the requirement, it leaves less machinery to manage; it deserves a serious iteration cycle before being ruled out.

Two techniques provide the starting toolkit: **chain-of-thought prompting**, which elicits intermediate reasoning, and **few-shot prompting**, which demonstrates the desired behavior through examples. These are selected methods rather than a complete taxonomy. Cleary points to additional reasoning templates in PromptHub for extending that toolkit.

### Give the model room for intermediate work

[2:26](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=146s)

Chain-of-thought prompting asks the model to work through a problem before producing its answer. Decomposing a task into smaller problems gives it intermediate work to build on, and the generated explanation can provide clues when troubleshooting an incorrect result. Cleary introduces this as a broadly applicable technique, then immediately qualifies it: reasoning models already incorporate reasoning behavior, so they do not necessarily need the same prompting intervention.

There are two straightforward ways to elicit that intermediate work:

| Approach | What the prompt supplies |
| --- | --- |
| Zero-shot instruction | A request to reason before answering |
| Few-shot demonstration | A worked example showing the desired solution steps |

The zero-shot version adds a cue such as “Take a breath and think it through.” The demonstration version supplies another problem and its worked solution before the new question. For a math task, the example communicates the steps to follow as well as the final-answer format.

### Use models to generate reasoning examples

[3:30](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=210s)

Writing every worked example by hand is not the only option. [Automatic Chain of Thought Prompting](https://arxiv.org/abs/2210.03493) generates reasoning demonstrations with a model; Cleary introduces it as a more involved framework. He then presents **AutoReason** through a single prompt that accepts a task or question and generates a reasoning chain, using embedded reasoning examples. That offered prompt is the rationale-generation part of the workflow: the complete AutoReason method subsequently generates an answer from the question and rationale.

The same idea—eliciting intermediate reasoning in a structured format—also appears in model training. Cleary points to the DeepSeek training template, which separates thinking from the answer using `think` tags. The [DeepSeek report](https://arxiv.org/html/2501.12948v1) labels that particular template **DeepSeek-R1-Zero**, rather than **DeepSeek-R1**. R1 uses a separate multistage training pipeline that includes cold-start supervised data and reinforcement learning; the template is a format for eliciting reasoning, not the whole training method.

For prompt experimentation, Cleary describes a shorter path: enter a task into one of PromptHub’s templates, generate a reasoning chain, and copy it for reuse. The collection he presents includes other reasoning and verification methods as well as chain-of-thought prompts, particularly for complex problems. These templates were offered as free resources in the talk.

### Show the client’s style through brief–content pairs

[4:36](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=276s)

**Few-shot prompting turns examples into part of the specification.** Cleary’s concrete task is generating content for a client. Instead of trying to describe the client’s tone exhaustively, the prompt supplies a brief and the finished content associated with it, repeats that pairing, then supplies a new brief whose content the model should produce. The examples demonstrate what a satisfactory transformation looks like.

The demonstrated structure can be used directly as a prompt template:

```text
Write content for {client_name} using the examples below
as guidance for tone and style.

Brief: {example_brief_1}
Content: {approved_content_1}

Brief: {example_brief_2}
Content: {approved_content_2}

Brief: {new_brief}
Content:
```

Each completed pair connects an input to an acceptable output. The final empty `Content` field asks the model to apply that pattern to a new input. This communicates style through actual copy rather than adjectives such as friendly or professional.

![Slide titled “Few-shot prompting example” shows client placeholders, two example brief–content pairs, and a final brief followed by Content. The presenter bubble sits toward the lower center.](https://ai.engineer/api/talk-essay-media/c5a24edd1295fcf67a22e7a7814093d9dda659c99d3c9a6800c0d99d24e14a2c)

A few-shot prompt supplies two brief–content examples before a new brief with an empty Content field.

Cleary recommends starting with one or two diverse examples, describing diminishing returns and possible degradation as more examples are added. That is a starting heuristic, not a measured optimum for this client-content task. Choose examples that cover different inputs the model is likely to encounter; duplicating nearly identical examples teaches less about the range of acceptable behavior.

### Let an LLM help revise the prompt

[5:42](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=342s)

**Meta-prompting** uses an LLM to create, refine or improve another prompt. It applies the same model assistance used elsewhere in development to the instructions themselves. The available approaches range from frameworks requiring code to user-facing prompt generators.

Cleary names Anthropic’s prompt-generation tool, OpenAI’s Playground and [PromptHub’s enhancers](https://www.prompthub.us/features/enhancers). In the PromptHub workflow he describes, choosing a model provider selects a different underlying meta-prompt. The motivation is practical: instructions that work well with one provider’s models may need a different form for another provider’s models.

Prompt creation can then become a feedback loop:

1. Run the current prompt.
2. Inspect the output and give feedback about what should change.
3. Use the feedback to revise the prompt and run it again.

Cleary describes PromptHub’s copilot as using ideas similar to [TextGrad](https://github.com/zou-group/textgrad), where textual feedback guides iterative improvement. The relationship he claims is similarity of approach, not that the product necessarily uses the TextGrad package.

### Reasoning models change what belongs in the prompt

[6:47](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=407s)

The earlier techniques need another pass when the model has been trained to reason. Cleary turns to Microsoft’s [From Medprompt to o1](https://arxiv.org/abs/2411.03590), first submitted in November 2024, and its experiments with **o1-preview**. Cleary reports that adding examples reduced performance in the Medprompt comparison he discusses. The result concerns that experimental setting, rather than establishing that demonstrations harm every reasoning model.

DeepSeek’s report likewise says few-shot prompting degraded performance for the evaluated R1 model and recommends a zero-shot problem description with an output format. Cleary also recalls OpenAI’s early o1-preview warning about excessive context. The historical guidance quoted in his companion discussion concerns keeping RAG context relevant: extra material can complicate the task rather than help resolve it.

Additional reasoning is a different intervention from additional examples. Cleary reports better results from an extended-reasoning prompt than from a quick-response prompt in the Medprompt work. The distinction is the amount of problem-solving effort encouraged at inference time, not the number of demonstrations placed in the input.

Cleary also describes longer reasoning traces alongside improved accuracy during DeepSeek training. This is a training observation, not a guarantee that making any answer longer improves it. No numerical effect sizes accompany these comparisons in the talk.

For the early reasoning models discussed here, **start with a clear task and minimal prompting**. If the output falls short, encouraging more reasoning may help, but that does not require scripting the model’s reasoning procedure. Cleary advises avoiding few-shot prompting by default in this setting; if examples are necessary, begin with one or at most two. He distinguishes asking for more reasoning from prescribing how to reason, which can interfere with behavior the model already learned.

The talk closes by pointing readers toward reusable material: the Prompt Engineering Substack, PromptHub’s blog, and community prompts contributed by the team and other users. Those resources extend the iteration process—from a clear task, to a tested prompt, to a targeted change when the output reveals what is missing.

## Resources

- [The Few Shot Prompting Guide](https://www.prompthub.us/blog/the-few-shot-prompting-guide) — Practical examples, including a reusable template pairing client briefs with finished content.
- [Prompt Engineering with Reasoning Models](https://www.prompthub.us/blog/prompt-engineering-with-reasoning-models) — Cleary’s companion guide to prompting tradeoffs for early reasoning models, with research links and prompt templates.
- [Chain of Thought Prompting Guide](https://www.prompthub.us/blog/chain-of-thought-prompting-guide) — Examples of reasoning prompts and explanations of Auto-CoT and AutoReason.
- [From Medprompt to o1](https://arxiv.org/abs/2411.03590) — Research on inference-time strategies and medical benchmark performance for GPT-4 models and o1-preview.
- [DeepSeek-R1 training and evaluation](https://arxiv.org/html/2501.12948v1) — Original report explaining R1-Zero, R1 training stages, and prompting recommendations.
- [Automatic Chain of Thought Prompting](https://arxiv.org/abs/2210.03493) — A method for constructing diverse reasoning demonstrations automatically.
- [TextGrad](https://github.com/zou-group/textgrad) — Library and tutorials for optimizing prompts and other text variables using language-model feedback.
- [PromptHub Prompt Enhancers](https://www.prompthub.us/features/enhancers) — Instructions for finding and applying prompt enhancers in PromptHub’s playground.

## Slides

### Prompt Engineering Simplified

Dan, co-founder of PromptHub, introduces a session on prompt engineering with templates and free resources to try.

[0:00](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=0s) [Title slide for Prompt Engineering Simplified.](https://assets.ai.engineer/sites/aiecode2025/sha256/017e7f0fec479f3cdd8400a4e7ae64c66619df1063a4375d4cd91ee786948572.webp)

Prompt Engineering Simplified

PromptHub

### What we’ll talk about

The talk covers chain of thought, few-shot prompting, using LLMs to write prompts through meta prompting, and prompting reasoning models, with free resources and templates throughout.

[0:18](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=18s) [Agenda covering prompt engineering, chain of thought, few-shot prompting, meta prompting, reasoning models, and resources.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6e1cd55e826a3e549e64b74e9a40454c25d9c61146aeb7d0b80976daf27d63c.webp)

What we’ll talk about

• Why prompt engineering
• Chain of Thought
• Few shot prompting
• Meta prompting
• Prompt engineering with reasoning models
• Free resources and templates throughout!

PromptHub

### Why prompt engineering

The meme reduces prompt engineering to English grammar. The speaker argues that shipping an LLM feature is more nuanced: even deciding what the model should do can be challenging.

[0:47](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=47s) [A meme shows a man at a table with a sign saying prompt engineering is just English grammar.](https://assets.ai.engineer/sites/aiecode2025/sha256/c282df925ecdd40dc33607c2589adcd041acb0cd0e839e8f904e4a373c945901.webp)

Why prompt engineering

Prompt Engineering
is just English grammar

CHANGE MY MIND

imgflip.com
PromptHub

### An easy starting point

Start with prompt engineering, then test and evaluate. The diagram places RAG and fine-tuning as subsequent options.

[1:09](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=69s) [A diagram moves from prompt engineering to testing and evaluation, followed by RAG and fine-tuning options.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d4d1a521329f207abc7df9c64239d140f43b17b0743a8e4ecc93a4349dc4116.webp)

An easy starting point

Start here
Prompt Engineering
Test and Evaluate
RAG
Fine-tuning

PromptHub

### Simple solutions become easier

Spend time refining prompts before moving to more complex approaches. Agentic systems may improve task performance, but can add latency and cost.

[1:29](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=89s) [An excerpt recommends starting with the simplest solution and adding complexity when needed.](https://assets.ai.engineer/sites/aiecode2025/sha256/503527fc2d3dbd5287e29dfaa4010c67fb518ae43159eec06e5d2aee3b67576f.webp)

Simple solutions become easier

When (and when not) to use agents

When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all. Agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense.

Interface text:
PromptHub

### Prompt engineering methods

The speaker focuses on chain-of-thought and few-shot prompting, calling them effective, topical choices among many methods.

[2:04](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=124s) [A slide lists chain-of-thought and few-shot prompting.](https://assets.ai.engineer/sites/aiecode2025/sha256/0811e42f27bf09017a88b3a65c1f34f1484c758f92bf53e354c830bc0029ca6a.webp)

Prompt engineering methods

• Chain-of-Thought
• Few-shot prompting

PromptHub

### What is Chain of Thought

Chain-of-thought prompting asks a model to reason through intermediate steps before answering, breaking a problem into sub-problems.

[2:23](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=143s) [Slide defining chain-of-thought prompting as breaking reasoning into intermediate steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/545086d7319aac5bf22490182eb038522678bf12a7c6f0e7d20dec9d6e814bf7.webp)

What is Chain of Thought

CoT prompting enhances the reasoning capabilities of large language models (LLMs) by encouraging them to break down their reasoning into a series of intermediate steps.

PromptHub

### Why CoT is helpful

The speaker says chain-of-thought prompting breaks problems into subproblems and offers a glimpse of reasoning for troubleshooting. He describes it as widely applicable, easy to implement, and built into reasoning models.

[2:35](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=155s) [Slide listing four benefits of chain-of-thought prompting and noting its use in reasoning models.](https://assets.ai.engineer/sites/aiecode2025/sha256/31ead84188a6ae045d41f63db3d05136c05ae406ed2b2b713bf6bd2a11a6ae92.webp)

Why CoT is helpful

1. Breaks down complex problems
2. A glimpse into the model’s thought process
3. Widely applicable
4. Easy implementation

So powerful, it is now built into reasoning models!

PromptHub

### Zero-Shot CoT

Zero-shot CoT adds “Think step-by-step” to a prompt to encourage the model to reason before giving its answer.

[2:56](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=176s) [Slide showing the zero-shot instruction “Think step-by-step.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b04c67534a4ff58f6776cb67e786b34883ac1aa2ebd571c89b615301fdc36cb0.webp)

Zero-Shot CoT

“Think step-by-step”

PromptHub

### Auto-generate reasoning chains

PromptHub turns an AI news writing task into a reasoning chain covering source selection, relevance, technical background, business impact, structure, and simplification. The chain can then be added to the prompt.

[4:06](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=246s) [Slide showing a tool that generates a reasoning chain for an AI news writing task.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c60c6496992914615f120f35e1c5a8509643c4d0d65b9dc3209238247b99a5a.webp)

Auto generate reasoning chains

Auto Chain of Thought
Generate reasoning chains for your task

Task Description
Write a daily blog post about the latest news in AI, focusing on technology and business

Example: “Generate clear technical documentation that explains complex concepts to beginners”

Generate Chain

Generated Chain

Identify reliable AI news sources
Need to gather accurate and current information from reputable tech news websites, research papers, and company announcements

Filter news by relevance and impact
Must determine which stories have significant technological or business implications worth covering

Research technical background
Understanding the underlying technology is crucial for explaining complex AI concepts accurately

Analyze business implications
Need to consider market impact, industry trends, and potential business applications of the AI developments

Structure the content hierarchy
Organize information from most important to supporting details to maintain reader engagement

Simplify technical concepts
Need to translate complex AI terminology and concepts into language that general business readers can understand

Interface text:
Cancel
Add Chain to Prompt

### A collection of 15 reasoning prompts

Tree of Thoughts directs an LLM through multiple paths on a task; Chain of Verification can use one prompt instead of its typical two to help reduce hallucinations.

[4:14](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=254s) [Slide showing a collection of reasoning prompt templates in PromptHub.](https://assets.ai.engineer/sites/aiecode2025/sha256/4fa8874d1b2bb4b90340f924a71964278da2f4630863056564c93553b114ba6d.webp)

A collection of 15 reasoning prompts

Reasoning prompts
Prompt Engineering

Tree of Thoughts
A prompting method that instructs the LLM to traverse many different paths when completing a task. Movie recommender is used as the example task. Update the variables and steps for your use case.
Prompt Engineering
PromptHub

Algorithm of Thoughts
Input your prompt in the variable and it will be converted into a new prompt, following the Algorithm of Thoughts framework. A final, cohesive, prompt will be below the AoT framework output.
Prompt Engineering
PromptHub

Chain of Verification
CoVe, typically a two-prompt method, can also function effectively with just one prompt, still helping to reduce hallucinations.
Prompt Engineering
PromptHub

Program of Thoughts
Template to generate the code portion of the Program of Thoughts (PoT) prompting method
[illegible]

Interface text:
PromptHub
Home
Discover
New!
Library
Templates
Database
Batches
Chains
Beta
Team Settings
Premium Plan
Upgrade Plan
Back
Add to Library
Submit Feedback

### Few-shot prompting

Few-shot prompting gives the model examples to mimic, showing it the desired output and style within the prompt.

[4:27](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=267s) [Slide explaining that examples show the model the desired output and style.](https://assets.ai.engineer/sites/aiecode2025/sha256/07e890a60522979f9343a253720f9a634c270199fb71e708119ad25cce95b385.webp)

Few-shot prompting

• Insert examples in your prompt, training the model on what you want the output to look and sound like.
• Show, don’t tell

Interface text:
PromptHub

### Few-shot prompting example

The prompt pairs two briefs with their finished content, then supplies a new brief for the model to complete in the client’s style.

[4:59](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=299s) [A prompt template pairs two example briefs with content before asking for content for a new brief.](https://assets.ai.engineer/sites/aiecode2025/sha256/4379418fc3c551f92fa373b4ae6bd2d7119e65ebbe2701714ff91eaf05c0101c.webp)

Few-shot prompting example

Few shot prompting example
Your job is to create content for our client, {{client_name}}. Here is some information about the client {{client_description}}.
Here are a few examples of content we've created in the past from briefs:
"""
Example 1:
Brief: {{brief_1}}
Content: {{content_1}}

Example 2:
Brief: {{brief_2}}
Content: {{content_2}}
"""
Here is the latest brief to create content about:
"""
Brief: {{brief_description}}
Content:

PromptHub

### Only need a few examples

The SuperGLUE chart shows a sharp performance gain with the first example, followed by smaller gains. The speaker recommends one or two diverse examples that cover expected inputs.

[5:14](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=314s) [A SuperGLUE chart shows few-shot GPT-3 performance rising sharply with the first examples and then increasing more gradually.](https://assets.ai.engineer/sites/aiecode2025/sha256/76ea68aed29a8685e2580840393b25d40db652110c294a289c029f9d5e087dfd.webp)

Only need a few examples

In-Context Learning on SuperGLUE
Legend: Few-shot GPT-3 175B
Human
Fine-tuned SOTA
Fine-tuned BERT++
Fine-tuned BERT Large
Random Guessing
Y-axis ticks: 40, 50, 60, 70, 80, 90
Number of Examples in Context (K): 0, 1, 2, 3, 4, 8, 16, 32

PromptHub

### Meta prompting

Meta prompting uses an LLM to create a prompt or refine one you already have.

[5:39](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=339s) [Slide defining meta prompting as using language models to create and refine prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e72f06fc714e677d2d4eaf7963a1cfd87e55b4ad2e62fd86a54d01cfa2bc829.webp)

Meta prompting

Meta prompting is a prompt engineering method that uses large language models (LLMs) to create and refine prompts.

Because of course we should use LLMs to help us!

PromptHub

### Meta prompting frameworks

Meta prompting uses LLMs to create or refine prompts. The speaker notes that some frameworks require coding knowledge, while others do not.

[5:55](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=355s) [Slide listing DSPY, TEXTGRAD, Automatic Prompt Engineer and Conversational Prompt Engineering.](https://assets.ai.engineer/sites/aiecode2025/sha256/decf083b5e9500d719e583aacbfe16dba1653555ebda3b620605e0acb66cab54.webp)

Meta prompting frameworks

• DSPY
• TEXTGRAD
• Automatic Prompt Engineer (APE)
• Conversational Prompt Engineering (CPE)

PromptHub

### Anthropic’s prompt generator

Anthropic’s prompt generator asks for task details to create a structured prompt, illustrating meta-prompting: using an LLM to create or improve a prompt.

[6:03](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=363s) [Meta prompting tools slide with a screenshot of Anthropic’s prompt generator.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a6d6368bdc15c018620400a156423f9c0c5ee91e4d6389519b2a9906f84e9fd.webp)

Meta prompting tools
• Anthropic’s prompt generator

Interface text:
Generate a prompt
You can generate a structured prompt by sharing basic details about your task.
Write me an email
Content moderation
Translate code
Recommend a product
Summarize a document
Describe your task...
Cancel
Generate
PromptHub

### OpenAI’s system prompt generator

OpenAI’s Playground includes a generator that turns a description of the model’s intended use into system instructions.

[6:06](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=366s) [Meta prompting tools slide with a screenshot of OpenAI’s system prompt generator.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca9fa78a1a79cd33cb4c8b53e9152fa07632a2c9cec831a4d9c7174e5f67b64f.webp)

Meta prompting tools

• OpenAI’s system prompt generator

Interface text:
Describe what you’re using the model for, and we’ll generate system instructions.
Free beta
Create
PromptHub

### PromptHub’s prompt generator

PromptHub’s project form pairs a task description with a model provider. The speaker says the generator uses a different meta-prompt for each provider, tailoring the resulting prompt to the selected model family.

[6:10](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=370s) [Meta prompting tools slide showing a PromptHub project form with a model provider and task description.](https://assets.ai.engineer/sites/aiecode2025/sha256/c1835bc9f5f32609e314c074ccd2e88eb4ff13e6ff311653e92ff3d8966488a1.webp)

Meta prompting tools

• PromptHub’s prompt generator

Interface text:
New Project
Start from a template or generate with AI
Name
Tweet Generator
Model Provider
OpenAI
Task Description
Generate topical tweets based on the latest news in AI, specifically LLMs, data centers, and EU regulations
Start from template
New!
Cancel
Create
PromptHub

### Prompt iterator enhancement

PromptHub’s iterator expands a short article request into a more detailed prompt and shows the changes side by side. The speaker describes running prompts and giving feedback to refine them.

[6:24](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=384s) [Meta prompting tools slide showing a prompt iteration and a partially visible comparison of prompt changes.](https://assets.ai.engineer/sites/aiecode2025/sha256/b64e940183f86aa9dd34f5559464f66767b7c2d9cd9af1ff4c9a282e69e0db26.webp)

Meta prompting tools

Prompt Iterator Enhancement
Iteratively refine your prompt through feedback and testing

Iteration 1
Iteration 2

Prompt
Write a concise LinkedIn article summarizing the provided article on AutoReason, focusing on its benefits over traditional Chain of Thought (CoT) prompting and Auto-CoT. Highlight its simplicity, transparency, and cost-effectiveness, and include key performance results from the StrategyQA and HotpotQA benchmarks. Conclude with a call to action to try AutoReason using the PromptHub template.

View Prompt Changes
- INSTRUCTIONS-Write a short LinkedIn Article based on the article
+ Write a concise LinkedIn article summarizing the provided article on AutoReason, focusing on its benefits over traditional Chain of Thought (CoT) prompting and Auto-CoT. Highlight its simplicity, transparency, and cost-effectiveness, and include key performance results from the StrategyQA and HotpotQA benchmarks. Conclu[illegible]

Interface text:
Cancel
Apply to Prompt

### Reasoning models are different

Reasoning models differ both in how they work and how you prompt them.

[6:43](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=403s) [Slide stating that reasoning models are different.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b34c948a8ab83270b87b302447adc3b9e4d8e09691a90ebab2168913a94abd0.webp)

But reasoning models are different!

PromptHub

### Prompting strategies on MedQA

On MedQA, o1-preview scores higher with minimal and tailored prompts than with either five-shot strategy. Error bars show one standard deviation across three independent samples.

[6:55](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=415s) [Bar chart compares o1-preview accuracy across minimal, tailored, and five-shot prompting strategies.](https://assets.ai.engineer/sites/aiecode2025/sha256/1eb2db77a0a67465e5d4a5680d047ab0d8f94751bc6b5cb159e6081ee3c5bcef.webp)

Prompt engineering with reasoning models

MedQA -- o1-preview prompting strategies

Y-axis: Accuracy (%)
Y-axis ticks: 92, 93, 94, 95, 96, 97

Minimal Prompt
Minimal Prompt Ensemble
Tailored Prompt
Tailored Prompt Ensemble
Random 5 Shot
kNN 5 shot

Figure 5: Comparison of prompting techniques on MedQA with the o1-preview model. Error bars indicate one standard deviation from three independent samples.

PromptHub

### DeepSeek-R1 prompting guidance

DeepSeek-R1 researchers found that few-shot prompting consistently degraded performance. They recommend describing the problem and output format directly in a zero-shot prompt.

[7:10](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=430s) [Excerpt states that few-shot prompting degraded DeepSeek-R1 performance and recommends describing the problem and output format in a zero-shot setting.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ac4a738049eb4c1ec7c8d2be04b4689c3ddda4ea2a849b12b530bc58adb1db6.webp)

Prompt engineering with reasoning models

Prompting Engineering: When evaluating DeepSeek-R1, we observe that it is sensitive to prompts. Few-shot prompting consistently degrades its performance. Therefore, we recommend users directly describe the problem and specify the output format using a zero-shot setting for optimal results.

PromptHub

### Limit additional RAG context

For retrieval-augmented generation, include only the most relevant context or documents to avoid overcomplicating the model’s response.

[7:17](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=437s) [Quotation recommends including only the most relevant retrieved information to avoid overcomplicating the model's response.](https://assets.ai.engineer/sites/aiecode2025/sha256/366f3afff862dd73fefa0f476e26bef95d3015417c4ed8f6a271aab9c993012e.webp)

Prompt engineering with reasoning models

“Limit additional context in retrieval-augmented generation (RAG): When providing additional context or documents, include only the most relevant information to prevent the model from overcomplicating its response.”

PromptHub

### Best practices for reasoning models

Start with a clear task description and encourage more reasoning for complex tasks. The slide also recommends avoiding few-shot prompts and limiting chain-of-thought steps for simple tasks.

[7:51](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=471s) [Recommendations include minimal prompting, more reasoning for complex tasks, avoiding few-shot prompting, and limiting chain-of-thought steps for simple tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/100fa9ec75af54c44e599841460ac3d0a7fece8f963fc68bb8ec500d29494e89.webp)

Best practices when using reasoning models

• Use minimal prompting
• Encourage more reasoning for complex tasks
• Avoid few-shot prompting
• Limit Chain-of-Thought (CoT) steps in simple tasks

PromptHub

### Best practices when using reasoning models

Start with a clear task description and encourage more reasoning for difficult tasks. If using few-shot prompting, the speaker suggests starting with one or two examples.

[8:06](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=486s) [Four practices for prompting reasoning models: use minimal prompting, encourage more reasoning for complex tasks, avoid few-shot prompting, and limit chain-of-thought steps in simple tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/b18ec8652c9e962914a1809819bff9ae584989b06eb8ae0bb8f2b87415fe68f0.webp)

Best practices when using reasoning models

• Use minimal prompting
• Encourage more reasoning for complex tasks
• Avoid few-shot prompting
• Limit Chain-of-Thought (CoT) steps in simple tasks

PromptHub

## Full transcript

[0:00](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=0s) Hey everyone, how's it going? Dan here. So excited that you're joining today. Um, we're gonna be talking about all things related to prompt engineering. And so some very quick background on myself.

[0:08](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=8s) I'm the co-founder of PromptHub, it's the GitHub for prompts. Based in New York. I'm a Lakers fan, more specifically a LeBron fan, but that just means February's been a great month for me.

[0:16](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=16s) Um, and I'm a marathon runner. And so today we'll be covering a lot of ground, a lot of templates, a lot of free stuff, um, that should be hopefully pretty helpful that you can go and take and start messing around with.

[0:26](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=26s) And so we'll talk about why prompt engineering is still important, why chain-of-thought prompt, uh, prompting has been so revolutionary, especially when it comes to test-time compute, few-shot prompting, using LLMs to help you write prompts via meta-prompting, how prompt engineering with reasoning models is actually very different, um, and a bunch of res-resource templates we've got.

[0:45](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=45s) So I usually include a slide like this in any of my talks, and I, I waver from doubling down and including more of these and completely removing it. But, you know, I think the meme in the beginning was that, you know, why do you even-- why is prompt engineering even a term?

[0:58](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=58s) You can just tell the model what to do. But I think anyone who's ever actually shipped an LLM-based feature has known it, it's much more nuanced from that. Even just trying to understand what you want the model to do is challenging.

[1:11](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=71s) And I think it's just a really good starting point for folks. Of course, um, it's the easiest and most successful way to get better outputs from LLMs, and it's a part of the med-- a greater system, right?

[1:22](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=82s) We all have access to the same models, um, but the prompts, the architecture, everything around that is how we can also have a competitive advantage in our product or whatever we're building.

[1:32](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=92s) I think, you know, something that was mentioned in Anthropic's recent paper, uh, or blog post about agents is that going for the simplest solution, I think is really key to keep in mind.

[1:41](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=101s) It's really easy to kind of run away with these things when you're working with LLM to think about everything you can do, and that's great. Um, but, you know, just spending an hour trying to mess around with a prompt and then determining that, you know, it's impossible to solve whatever you're trying to do via prompt engineering, and

[1:56](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=116s) you need to do more complex RAG or things along those lines, I think is, is not super smart. You need to give some time here, because if you can do it, it's much simpler to manage.

[2:06](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=126s) And so we'll cover just two main methods, chain-of-thought and few-shot prompting, just 'cause I think they're the most effective and most topical. There are a bajillion more. A lot of them fall underneath the umbrella of general reasoning prompts.

[2:17](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=137s) Um, we've covered most, if not all of these, and they're all available as templates in PromptHub as well that you can check out for free.

[2:26](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=146s) So what is chain-of-thought prompting? Simply when you instruct the model to

[2:31](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=151s) reason or think about the problem or a solution before actually jumping into whatever that, that answer is. And so it breaks down problems into sub-problems. Um, you get a glimpse into how the model is thinking, which can be helpful for, for troubleshooting.

[2:47](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=167s) It's widely applicable, of course. You can use it kind of with any model. Um, easy to implement as well. And it's so powerful that it's now kind of being built into these reasoning models, and so you don't even really need to do it for those reasoning models.

[2:58](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=178s) And so the classic kind of zero-shot way to do this is just to add something to your prompt that will make the model think a little bit before it's just giving you the output.

[3:06](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=186s) You know, you want it to just generate some sort of kind of reasoning tokens beforehand and think step by step. What's a classic one? Take a breath and think it through.

[3:13](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=193s) Things along those lines. Another very popular way to do this is by having few-shot examples of those reasoning steps. And so if I'm having a prompt to do solve math problems, I can include another math problem in the prompt and show the reasoning steps I wanted to solving that.

[3:30](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=210s) And of course, you can use LLMs to generate these reasoning chains as well. So there's something called just automatic chain of thought, which is a framework, um, that's a little bit more involved.

[3:39](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=219s) Um, there's another one called AutoReason, which is just a single prompt here, where you pass your task or question and it will generate, um, reasoning chains. It actually has few-shot examples of reasoning chains in there as well, and you can try this out in PromptHub.

[3:52](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=232s) And even the training template that DeepSeek used for its R1 model, um, basically did this. It had it generate its thinking process, uh, within think tags and then-- and used all these outputs, these generated reasoning chains to train the model to be really good at chain-of-thought.

[4:08](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=248s) And so this is also available inside PromptHub. You can input your task, get a reasoning chain, um, copy, on your way, and that's totally free if you want to try it out.

[4:16](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=256s) And as I mentioned, we have a, a ton of these, um, in the platform, a ton of reasoning chains that you can go and check out. Some of them are chain-of-thought, some of them are other type of, um, reasoning or verification methods as well.

[4:29](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=269s) But they're all pretty helpful, I would say, especially when you want to use them when, when you're dealing with like complex problems.

[4:36](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=276s) So moving on to few-shot prompting. Um, it's-- that's generally when you include examples of what you want the model to kind of mimic or do or to understand about your problem, and essentially you're doing a show rather than tell.

[4:48](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=288s) And so in this example here, I'm telling, um, the model that I have this client, we need to like generate some content for it. Here's a brief, here's the related content.

[4:57](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=297s) Here's a brief, here's the related content. And then I say, "Here's the brief," and then the model will fill in this, this content here. And so rather than trying to encapsulate my client's tone or style by sending an input and output example, a brief and a piece of content, I can kind of teach the model, uh, exact

[5:14](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=314s) what I want. The great part of this is that you get most of the gains from just like an example or two. Um, almost all the graphs kind of look like this when you're looking at number, number of examples versus, uh, performance, and sometimes performance can degrade once you have like a bunch.

[5:30](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=330s) Um, but it's great for builders 'cause you only need, I'd say one or two. You want, kind of want to have them be diverse and cover your bases of different inputs you could expect that model to handle.

[5:39](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=339s) Um, but yeah, you don't need many of them.

[5:42](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=342s) And so next up is meta-prompting. Um, you know, I think it'd be silly as some people who are working with LLMs to not use LLMs for this part of the process.

[5:50](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=350s) So meta-prompting is basically just using an LLM either to create a prompt, refine a prompt, improve a prompt, whatever that might be.

[5:57](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=357s) There are a ton of frameworks for this out there. Um, some of them are require you to have coding knowledge, some of them don't. There are a bunch of free tools as well, which of course are very user-friendly.

[6:06](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=366s) Anthropic has a great one. OpenAI has one inside of their playground, and then we also have one in PromptHub. Um, the difference with ours is you can select which model provider you are using, and it will run a different meta prompt because a prompt that is good for OpenAI models might not be the same as Anthropic.

[6:24](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=384s) And so we, we tailor it a little bit for you as well. And then we also have a way that you kind of iteratively work with a, um, kind of like a copilot inside PromptHub.

[6:33](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=393s) It's built off very similar things to TextGrad, where you can run prompts, give feedback. So this is another free tool that you have to your disposal as well because prompt engineering is something that we can use help with, so why not u-leverage LLMs?

[6:47](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=407s) And so moving on to kind of to stuff that's much more, um, apparent now and more recent is that reasoning models are very different, both in terms of how they work and how you prompt them.

[6:58](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=418s) So Microsoft released a paper earlier this year about their Medprompt framework. It's not super important, but basically they ran, uh, a prompt engineering framework with o1 and found that adding examples led to worse performance.

[7:12](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=432s) And the researchers at DeepSeek when building R1 found this as well, uh, that few-shot degraded performance. And OpenAI kind of mentioned this when they first released o1-preview, saying that you need to be careful when providing additional context because it can kind of overcomplicate things and confuse the model.

[7:31](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=451s) And so you gotta be careful with examples, but if you need to-- want to increase performance, there's been a lot of research that has shown that the more reasoning a model does, the better the output could be.

[7:41](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=461s) So in that same Medprompt paper, they had a prompt that was, you know, quick response and then a prompt that prompted the model to think more. Um, and they saw that better results when the model was thinking more from extended reasoning.

[7:53](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=473s) And the folks at DeepSeek saw this as well. So as they continued to train the model, the length of the response or the thought process increased, and then also this will in turn, um, increase accuracy and performance as well.

[8:08](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=488s) And so overall, when you're using reasoning models specifically, minimal prompting, nothing can really beat like a really good, clear task description. I'm gonna encourage more reasoning if you're having trouble kind of getting maybe that last bit of performance, having-- encouraging the model to reason more can be helpful.

[8:24](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=504s) Avoid few-shot prompting. If you're gonna do it, start with like one, one, maybe only two examples. And then you don't really need to instruct the model on how to reason.

[8:34](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=514s) It's kind of built in there, so doing that can actually, um, hurt performance as well. And so as I mentioned, lots of free resources. We run a Substack called Prompt Engineering Substack.

[8:43](https://www.youtube.com/watch?v=Hp4MzVTXcKw&t=523s) We write on our blog. Um, there's a bunch of prompts in the community from us and from other people. And so I hope this was helpful, and I hope you have a great time at the summit, and have a great day.
