← All AI Engineer talks

AI Engineer Summit 2023

Prompt Engineering Tactics

Read the talk

Three Prompting Tactics for More Reliable Responses

Small prompt changes can alter an AI product’s behavior. Task-specific personas, named sources, and emotional cues offer three ways to shape the response.

From a talk by Dan Cleary

Why not just ask for what you want?

Why engineer a prompt if you can simply tell a model what you want and usually get something useful back? That is the starting question for Dan Cleary, co-founder of PromptHub, a tool for teams to test, collaborate on, and deploy prompts. Getting a good answer once is useful; getting dependable behavior inside a product is a harder requirement.

Model outputs are nondeterministic, and small changes to a prompt can have outsized effects on the response. An answer that goes off the rails can damage trust in the product delivering it. Cleary illustrates the stakes with a slide pairing poor prompts with a headline about Replika chatbot behavior. Prompt quality becomes part of product quality when users encounter the output as a feature rather than an experiment.

Slide titled “Poor prompts can lead to disaster” with a Replika news headline and a presenter inset partly covering its lower-left corner.
“Poor prompts can lead to disaster,” illustrated by a headline about Replika chatbot behavior.

Access to ChatGPT has also raised expectations: users want crisp answers that match their request, arrive quickly, and do not hallucinate. Cleary offers three tactics for improving responses, whether the prompt lives in an everyday chat or an AI product feature.

0:150:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Let task-specific personas collaborate

Multi-persona prompting asks the model to approach a task through several relevant roles. Cleary attributes the method to University of Illinois research; the matching work, Solo Performance Prompting, uses a single language model to simulate collaborating personas. The agents here are perspectives generated within the model’s response, not necessarily separate running systems.

For a book-writing task, Cleary suggests an author, a publicist, and the intended audience. The model leads a brainstorming exchange among them, allowing their different priorities to influence the result. The author can develop the material, the publicist can consider its appeal, and the audience can respond to what would make it worth reading. A prompt can make that sequence explicit:

Help me develop a book idea.

Bring together three personas: an author, a publicist,
and a representative of the book's intended audience.

Lead a brainstorming discussion. Have the personas propose
ideas, respond to one another, and revise their suggestions.
Show the collaboration, then present a final recommendation.

The important operation is the exchange: personas respond to one another before the model produces its final answer. Merely listing three roles would leave that collaboration unspecified.

The visible discussion lets the reader inspect how the proposed answer developed. Cleary recommends the technique for complex tasks or tasks requiring additional logic, and personally favors it for generative work. In the book example, its value is in developing and revising ideas through several perspectives.

1:512:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:51 · section reference included

Name the source you want the answer to follow

The according-to method changes a smaller part of the prompt. Start with Cleary’s digestive-system question, then append a source:

In what part of the digestive tube do you expect starch
to be digested? According to Wikipedia.

The suffix directs the model toward a particular source’s account. In the matching original according-to study, the starch example concerns initial digestion, and both the baseline and grounded answers identify the mouth. The example demonstrates more source-overlapping detail, rather than correcting an initially false answer.

Naming a source does not fetch it. Although Cleary describes the model going to the source to retrieve information, the studied mechanism relies on recall from training data, without internet access. A source cue can influence the answer, but it does not establish that the model consulted or verified a live page.

Cleary reports that according-to prompting can reduce hallucinations by up to 20%. The original study does not establish that general hallucination-rate reduction: it measures quotation overlap with Wikipedia and, separately, question-answering performance. More overlap with a source is not itself proof that an answer is correct.

Cleary recommends this tactic when a fine-tuned or general model should consistently draw answers from a particular source. He attributes the research to Johns Hopkins University and introduces it as recent work. The useful design choice is to make the desired source explicit instead of leaving the model to infer it.

2:533:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:53 · section reference included

Add an emotional cue

EmotionPrompt, which Cleary attributes to Microsoft and several universities, tests emotional stimuli appended to ordinary prompts. His analogy is a boss explaining that a project matters to your career or to a major client: the stakes may lead you to take the work more seriously. That human behavior motivates the prompting technique; the method tests whether similar language changes model responses.

The implementation is simply to preserve the task and add the cue at the end. The slide shows the addition “This is very important to my career.” For a writing request, that pattern could look like this:

Revise my cover letter for clarity and relevance to the job.
Preserve the facts in the draft.

This is very important to my career.

The task still specifies what to do. The emotional sentence supplies an additional signal about its importance.

Cleary says he has seen better outputs for tasks ranging from cover letters to change logs. Those are his practical observations. The September 2023 EmotionPrompt study reports relative benchmark improvements of 8% on Instruction Induction and 115% on BIG-Bench. Its expanded evaluation covers 24 Instruction Induction tasks and 21 BIG-Bench tasks across six models, with zero-shot BIG-Bench experiments. The study distinguishes original and APE prompt baselines, as well as average and best emotional-stimulus results; the headline percentages are not measured gains for the cover-letter or changelog examples. The slide pairs the prompt modification with an Original versus Ours results table across six models.

EmotionPrompt slide with original and modified prompt boxes, the addition “This is very important to my career,” and an Original versus Ours table for six models. A presenter inset overlaps the lower-left corner.
EmotionPrompt contrasts an original prompt with an emotional addition and shows results across six models.
3:373:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:37 · section reference included

Try the prompts with your team

These methods fit both direct use in ChatGPT and prompts embedded in product features. Cleary closes by offering all three as PromptHub templates: copy a prompt, run it through the playground, and share it with teammates or access it through a link. The public multi-persona collaboration template provides a starting point for the persona-selection and iterative-discussion pattern. The practical next step is to try a tactic on the task you actually need the model to perform.

4:334:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:33 · section reference included

Resources

From the talk

Read the complete timestamped transcript
  1. 0:00

    [upbeat music] Hey, how's it going?

  2. 0:15

    I'm Dan. I'm co-founder of PromptHub, a prompt management tool designed for teams to make it easy to test, collaborate, and deploy prompts. Today, I want to talk to you a little bit about prompt engineering, including over three easy to implement tactics to get better and more accurate responses from LLMs.

  3. 0:33

    But first, why prompt engineering? Can't I just say what I want to the model and I get something pretty good back? And while for the most case that's true, additional techniques can go a long ways in terms of making sure that your responses are always better.

  4. 0:48

    The non-deterministic nature of these models makes it really hard to predict, and I've seen that having little changes in a prompt can have outsize effect on the outputs. And this is especially important for anyone who's integrating AI into their product, because one bad user experience or one time the model decides to go off the rails can result

  5. 1:08

    in disaster for your brand or your product, resulting in a loss of trust.

  6. 1:15

    Additionally, users, now that we all have access to ChatGPT and can really easily access these models, we have very high expectations of when we're using AI features inside of products.

  7. 1:25

    We expect outputs to be crisp, exactly what we wanted. We should expect to see -- never see hallucinations, and in general, it should be fast and accurate.

  8. 1:36

    And so I want to go over three easy to implement tactics to get better and safer responses. And like I said, these can be used in your everyday when you're just using ChatGPT or if you're integrating AI into your product, these will help go a long way to making sure that your outputs are better and that users

  9. 1:51

    are happier. The first are called multi-persona prompting. This comes out of a research study from the University of Illinois. Essentially, what this method does is it calls on various agents to work on a specific task when you prompt it, and those agents are designed for that specific task.

  10. 2:11

    So, for example, if I was to prompt a model to help me write a book, multi-persona prompting would lead the model to get a publicist, an author, um, maybe the intended target audience of my book, and they would work hand in hand in kind of a brainstorm mechanism with the AI leading this brainstorm.

  11. 2:32

    They'd go back and forth, throwing ideas off the wall, collaborating till they came to a final answer. And this prompting method's really cool is 'cause you get to see the whole collaboration process, and so it's very helpful in cases where you have complex task at hand or it requires additional logic.

  12. 2:48

    I personally like using it for generative tasks.

  13. 2:53

    Next up is the according to method. What this does is it grounds prompts to a specific source. So instead of just asking, you know, "In what part of the digestive tube do you expect, uh, starch to be digested?"

  14. 3:06

    You can say that and then just add to the end, "According to Wikipedia." So adding according to specified source will increase the chance that the model goes to that specific source to retrieve the information, and this can help reduce hallucinations by up to twenty percent.

  15. 3:21

    So this is really good if you have a fine-tuned model or a general model that you know that you're reaching to a very s- uh, consistent data source for your answers.

  16. 3:31

    This is out of Johns Hopkins University. Um, it was published very recently.

  17. 3:37

    And last up, and arguably my favorite, is called EmotionPrompt. This was done by Microsoft and a few other universities, and what it basically looked at was how LLMs would react to emotional stimuli at the end of prompts.

  18. 3:52

    So, for example, if your boss tells you that this project is really important for your career or for its -- for a big client, you're probably gonna take it much more seriously.

  19. 4:00

    And this prompting method tries to tie into that, uh, cognitive behavior of humans. And it's really simple. All you have to do is add one of these emotional stimuli to the end of your normal prompt, and I'm sure you'll actually get better outputs.

  20. 4:12

    I've seen it done time and time again, from everything from cover letters to generating change logs. The outputs just seem to get better and more accurate, and the experiments show that this can lead to anywhere from an eight percent increase to a hundred fifteen percent increase, depending on the task at hand.

  21. 4:33

    And so those are three really quick, easy-hit methods that you can use in ChatGPT or in the, um, AI features in your product. We have all these available as templates, um, in PromptHub.

  22. 4:44

    You can just go there and copy them. Um, it's prompthub.us. Um, you can use them there, run them through our playground, share them with your team, or you can have them via the links.

  23. 4:55

    And so thanks for taking the time to watch this. I hope that you've walked away with a couple new methods that you can try out in your everyday. If you have any questions, feel free to reach out, and I'd be happy to chat about this stuff.

  24. 5:04

    Thanks. [upbeat music]