AI Engineer World's Fair 2024
Building with Anthropic's Claude - The Prompt Doctor is In
Read the talk
Debugging Claude prompts, one failure at a time
A live workshop moves from an overly talkative patient agent to JSON extraction, writing examples, persona control, image inputs, translation grading, and grounded summaries.
From a talk by Jamie Neuwirth and Zack Witten
Before you start: Basic familiarity with chat-model prompts, JSON, and API message roles will help with the structured-output examples.
What should a conversational agent say?
What should a medication-review agent say when someone asks who it is—and how do you turn that expectation into a prompt you can test? That is the first practical problem in Zack Witten’s live prompting workshop. The setting is the June 2024 AI Engineer World’s Fair, where Jamie Neuwirth opens by describing Anthropic’s support for startups: higher rate limits, access to prompting expertise, and early access to products.
The product backdrop is the newly introduced Claude 3.5 Sonnet, the Artifacts preview, and Claude Teams and Projects. Artifacts turns generated code and other content into something people can inspect and share: examples include games, diagrams, and illustrations of gene-sequencing discussions. Neuwirth also points builders toward Claude Cookbooks, the startup sales team, and Alex from developer relations.
Witten’s working material comes from the audience, through the AI Engineer Slack channel prompt-eng-live-workshop-anthropic. A useful submission contains a prompt template, variable slots, and concrete inputs that produce unwanted answers. The Console uses double-brace variables such as {{DOCUMENT}}. The submitter must also explain what counts as a good answer; otherwise, an apparent failure may simply reflect an unstated requirement. Most iteration happens in the Anthropic Console, with Claude for Sheets mentioned as another way to call Claude across spreadsheet cells.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Establish a baseline before cleaning up the prompt
Gordie’s medication-review agent provides the first case. Witten initially starts fixing capitalization and grouping information into XML tags. He describes the benefits of capitalization and grammar cleanup as largely anecdotal in his own work, while mentioning unspecified quantitative work on typos. His initial reason for choosing XML is Claude’s familiarity with it from training. Before these edits go further, an attendee asks for a before-and-after comparison. Witten undoes the changes and runs the original prompt.
The first input asks who the agent is. Its answer is much too long for the intended conversational interface; Gordie initially asks for no more than one sentence. Witten adds that case and two more to the Console’s Evaluate tab, then uses Run remaining to establish the baseline.
| Test input | Baseline problem |
|---|---|
| Who are you? | Too much explanation |
| Why do I need to do this appointment? | Another long answer |
| Can I schedule it tomorrow instead? | Disclaims scheduling information |
Gordie says the service has 24-hour availability. At this point, brevity and scheduling behavior look like two separate things to fix.
The first revision separates information from instructions, repairs numbering, and adds consistent line breaks. Asked whether Claude could perform the capitalization cleanup itself, Witten notes that a rewrite might also change words that should remain untouched. Asking Claude to generate capitalization code would make the transformation more explicit, but sentence boundaries still have edge cases such as ellipses. An IDE assistant is useful for a narrower job: completing nested XML closing tags while the author controls the content.
The purpose of the tags is semantic separation: this is background information, these are instructions, and this is the user’s message. XML is a convenient notation, not the essential ingredient. Witten subsequently separates the user input too and selects Rerun all. The answers remain long, and the scheduling disclaimer remains. Better boundaries make the prompt easier to interpret, but they have not supplied a concrete length target or the missing scheduling fact.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replace vague requirements with observable behavior
Rule seven already asks the model to be concise and offer only relevant information. The problem is that concise has no fixed size: it might mean one word in one task and a page in a book review. Witten first tries a range of two to four sentences. Prompt length and tone can also provide cues, but a long prompt does not inherently require a long answer.
Two to four sentences still looks too long. He tightens the instruction to one or two sentences, never more than three. The revised patient-agent outputs contain two, two, and three sentences. That is shorter while retaining some flexibility for questions that need more explanation; it is not the strict one-sentence maximum initially requested.
The scheduling case then changes meaning. The submitter initially suggests that the agent should ask what time tomorrow would work, but explains that the 24-hour service information was deliberately omitted. With that fact absent, acknowledging uncertainty is the desired behavior. Define success against the information the model actually receives. A prompt cannot be expected both to avoid unsupported claims and to confidently use a business fact withheld from its context.
For component order, Witten recommends putting information above instructions. He reports that instructions nearer the bottom tend to be followed more closely, while explicitly recommending testing that ordering on the actual task. The same empirical attitude applies to emphasis: capitals, exclamation marks, and statements of importance can change behavior, but he does not know their relative strength compared with numbered rules. An audience question about punctuation in tokenizer code does not produce a tokenizer-level explanation; his evidence here is observed differences in responses.
For multilingual tasks, the next question is whether instructions should be in English or the target language. Witten prefers a well-written target-language prompt when the author knows that language. If the choice is between clear English and a poor prompt in an unfamiliar language, he prefers the language the author understands. His ideal workflow is to explain the use case to a native speaker and have that person write the prompt—even when the target language may be less familiar to Claude.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reproduce the JSON failure—and notice truncation
The next prompt belongs to a unit-test reviewer in a multi-agent workflow. A writer agent creates tests; the reviewer is supposed to improve them and return JSON describing file updates and modifications. Despite repeated JSON-only instructions, the reviewer sometimes adds other text.
Reproducing this requires more than copying the reviewer’s instructions. In the original application, the tests are earlier in the conversation, and the project path identifies a local directory the application can access. Witten makes a standalone Console reproduction by explicitly supplying the earlier tests and converting template variables to double braces. Merely placing a directory path in this reproduction does not recreate the original application’s file access.
There is also a model difference: the submitter’s earlier tests mostly used Claude 3, while the live reproduction initially uses Claude 3.5. Witten normally chooses temperature zero for knowledge work. Asked about 0.01, he tentatively expects zero to produce marginally fewer hallucinations, but this session does not establish that comparison. The response appears to begin as JSON—then runs into the maximum output-token allowance before finishing. He switches to Claude 3 Haiku for faster iteration and raises the allowance. A response can begin in the right format and still fail because generation stops too early.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control where generation starts and stops
Witten’s next tool is assistant prefill: supply the beginning of an assistant message, and the model continues as though it has already said those words. To expose the unwanted behavior, he removes the repeated JSON-only instructions. The model produces the familiar preamble before its data. Moving that preamble into the supplied assistant prefix lets the generated continuation get on with the JSON. His analogy is a child determined to do something: instead of repeatedly forbidding the preamble, let it have already happened.
Prefilling the opening JSON delimiter goes a step further: the response is already inside the data structure. The generated continuation will not repeat that supplied delimiter, so the application must restore it before calling json.loads. A second approach asks for JSON inside <json> tags and extracts only the enclosed text. Combining the approaches gives a supplied preamble, opening tag, and opening JSON delimiter; the continuation completes the JSON and closes the tag.
For an object-valued response, the corresponding Python extraction logic can retain the prefix explicitly. Here, continuation is the generated text after the supplied prefix:
python
import json
import re
PREFILL = 'Here is the updated JSON:\n<json>{'
def parse_review(continuation: str) -> dict:
full_text = PREFILL + continuation
match = re.search(r'<json>(.*?)</json>', full_text, re.DOTALL)
if match is None:
raise ValueError('Missing complete JSON block')
review = json.loads(match.group(1))
if not isinstance(review, dict):
raise ValueError('Expected a JSON object')
return review
The prefix belongs to the assembled text, while only the content inside the tags reaches the parser. Parsing rejects invalid JSON; it does not establish that the proposed test changes are correct. This example follows the workshop’s returned-closing-tag approach.
The combined live response still adds an explanation after its closing tag. Rather than make the prompt increasingly insistent about stopping, Witten proposes putting </json> in the API’s stop_sequences. That control was not exposed in the Console he was demonstrating, so he explains it rather than running it there. The purpose is to prevent generation of the unwanted epilogue, avoiding the associated output and work. Use code and API controls for boundaries that do not require judgment.
In the historical API pattern, the final entry in messages has role set to assistant and contains the partial response. This is model-specific: current API documentation says Claude 4.6 and later reject last-assistant-message prefilling. Current structured outputs offer schema-constrained JSON through output_config.format, are incompatible with message prefilling, and still require handling refusals and token-limit truncation. For either implementation, inspect the generation stop reason before deciding how to parse the response; the extraction code above specifically expects a returned closing tag.
Putting the same prefix at the end of the user’s instructions is not equivalent to supplying an assistant continuation. In the live experiment, that version introduces an extra opening delimiter. An attendee’s simpler JSON: suffix works in another attempt. Witten acknowledges multiple workable approaches, but continues to recommend prefill and tagged extraction for the models used in the workshop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check format in code; reduce oversized inputs
A successful example is only the beginning of evaluation. Testing at production scale requires a collection of cases. Asked whether an LLM should grade format compliance, Witten recommends ordinary code whenever the condition is mechanically checkable. JSON parsing is a direct test; conversational usefulness may instead need a model grader or a human. He briefly considers showing a test-generation feature, but no such demonstration follows.
For a summary surrounded by chatty introductory or closing text, tags can make the desired region extractable. Request <summary>...</summary>, extract its content, and use the closing delimiter as a stop sequence where supported. Detecting conversational clutter is a task an LLM could grade, but explicit boundaries may remove the need to judge that clutter in the first place.
Before the next case, Witten declines an audience request to test a potential attack vector; no jailbreak experiment follows. The workshop instead turns to a poorly formatted Excel/CSV file. Its submitter describes multiple clusters or datasets on one sheet, with extraction that invents or skips information. Witten’s proposed decomposition is straightforward:
- Give the model fewer sheets at a time.
- Include only the columns needed for the current task.
- Ask smaller questions that can be answered from those inputs.
The concrete CSV test stalls at getting the file’s text into the Console. Witten declines to open his work-computer downloads on the projector because they may expose private information, and the browser-only attempt ends without a tested extraction result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Show the writing style you want
The social-media ghostwriter prompt asks for five to ten tweets from a long article, without hashtags, hyperbole, or cringe. Witten moves the article before the instructions, adds XML boundaries, and asks why the output needs to be a JSON array. The answer is automation: it needs to be parsable. He switches to tagged output and recommends examples to make the subjective style requirement concrete.
With an Anthropic blog post as input, the output avoids hashtags and does not seem especially hyperbolic. The phrase “New feature alert” becomes a specific candidate for what the audience might consider cringe; the absence of emojis is viewed favorably. The submitter calls the tweets adequate but not engaging. Witten adds an engagement instruction and clarifies the intended voice: an AI influencer rather than the company’s own account.
Asked about sentence complexity in prompts, he favors short sentences and familiar words, while allowing more academic language when seeking an academic tone. The revised tweets add questions and exclamation marks. Changing temperature from zero to one does not produce an obvious improvement to Witten’s eye in this informal comparison. The remaining obstacle is specification: engaging is still a subjective adjective. Examples of desired tweets—or complete source-document and tweet pairs—would tell the model much more.
Building those examples takes editorial work and input tokens, but Witten argues that the tradeoff can make sense when the outputs have economic value and are generated relatively infrequently. His proposed process is iterative:
- Start with a source document and generate candidate tweets.
- Select the strongest candidates, edit them, or write better ones yourself.
- Keep the literal document together with its approved tweets.
- Repeat with another document and add that pair to the prompt.
Generating 100 candidates and choosing seven is his illustrative workflow, not a reported experiment. The essential work is selecting and editing examples until they demonstrate the intended tone.
Witten considers this example curation one of the largest sources of prompting gains. The examples can live in one block or in alternating user and assistant messages. He usually chooses a single block because it is easier to author; his limited testing did not establish a confident performance preference. For subjective failures, add contrasting pairs: a poor tweet and an excellent tweet about the same document. Negative examples are useful, but they should accompany positive demonstrations of what to produce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put useful preparation before the answer
An explanation of why a model already wrote something is not necessarily a faithful account of how it produced the answer. Witten distrusts these retrospective explanations, comparing them with human rationalization. He distinguishes them from preparation generated before the answer: considering the relevant facts or alternatives first can help shape what comes next.
For the tweet task, that preparation becomes a concrete intermediate artifact: document → key points → tweets. Each example contains the document, a list of its key points, and then the approved tweets. The key points can be written by a person, generated and edited, or retained from a good model response. Witten also updates the requested output format to ask for key points before tweets, calling this lightweight chain-of-thought prompting. Although the audience question asks how to avoid receiving reasoning, this demonstration explicitly includes a preparatory output stage.
If long documents make examples expensive, he would start with one excellent complete example rather than several truncated ones. That choice needs testing: a single example can cause the model to fixate on the example document’s particular content and carry it into an unrelated document. Quality comes first, followed by checks that the learned pattern transfers without copying irrelevant details.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate persona routing from staying in character
The next prompt defines three roles and a small command protocol, including >p assist, for switching between them. Designers use the conversation for synthetic-user interviews. Witten has limited experience with multiple personas in one prompt, but suggests an application architecture: write a separate prompt for each persona and let code route the user’s command to the selected one. The submitter clarifies that the current setup runs entirely in chat; explicit routing would require building an API application. A follow-up proposal for another thread to aggregate the persona conversations receives no substantive answer.
Testing the existing chat prompt exposes a different problem. Witten selects Sam with >p Sam and asks how the persona finds good medication prices. Joe, another persona, is described as favoring convenience over cost savings. Sam’s response announces itself with “As Sam,” breaking the intended illusion of an ordinary person answering a question. Witten adds an instruction to prefix each response with the current persona name in brackets.
He then removes earlier assistant turns that contradict the new format and restarts the conversation. Those turns are examples too: leaving them in context would show the model behavior the revised prompt now discourages. The live protocol also needs the persona switch in a separate call before the question. The next answer drops the exact self-announcement but still foregrounds the persona awkwardly. A light instruction to say less about the persona and stay in character produces a visibly better response.
Witten generally prefers positive instructions, but this repair combines a brief negative constraint with the positive direction to stay in character. He warns against dwelling on forbidden content: repeatedly mentioning elephants while prohibiting them may make elephants more salient. His parenting comparison makes the same point with prunes—briefly state the restriction, then move on.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Role placement, counterexamples, and steering
Asked why he uses the system prompt so little, Witten describes his personal default: put the role there and most task instructions in the human message. He notes tool-use prompts as a possible exception because of explicit fine-tuning. Audience members report better behavior after moving instructions into the system prompt, and he acknowledges that his preference may be wrong for their cases. His proposed alternative is clearer separation between the actual user message and the surrounding task instructions, using labeled boundaries. This remains a disputed workshop heuristic, not a general rule about message priority.
Counterexamples are likewise optional. An attendee asks whether preference training makes positive and negative pairs especially important in prompts. Witten cautions that the training algorithm’s comparison need not mean the model sees both examples together in one context window. The useful question is whether a contrasting example clarifies this task—not whether every prompt should imitate a presumed training format.
Another attendee asks about control vectors and Scaling Monosemanticity, suggesting that a negative steering direction might replace a verbal prohibition. Witten distinguishes steering identified features from steering with differences in underlying activations. His limited experiments at the time favor prompting, but he does not make a settled prediction about which approach will ultimately work better. The attendee reports more success controlling style than deterministic task behavior on smaller models; Witten remains uncertain which techniques transfer between small and large models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the evidence in an image
A proposed dating-profile review has no screenshots attached, so the workshop moves to a supplied image of handwritten tube labels. Image pasting is unavailable in the demonstrated Console, prompting a switch to Claude.ai. The intended reading includes Mattie White and 8687; one label reads 867, which the submitter identifies as a human typo. The goal therefore includes reconciling repeated labels, not merely transcribing each one independently.
The first attempt misreads the handwriting. Witten’s strongest recommendation is to improve the input: crop to the relevant region, zoom in, and remove unnecessary details. The same principle applies to text and images—irrelevant material competes with the information needed for the task. Doing this programmatically can be difficult because the application must first decide which regions matter.
An attendee says the website describes downsampling to 1,000 × 1,000 pixels, but the discussion does not establish that historical limit. Current vision guidance describes model-dependent resizing that preserves aspect ratio, rather than a universal square size. The practical issue in this example is legibility. Witten suggests having the model describe what it sees before answering, while doubting that prompting alone can repair handwriting he finds difficult to read himself.
Describing the image first also has a failure mode. The attendee reports that a wrong interpretation of the first tube can propagate to later tubes. Witten agrees that the model’s drive for self-consistency can preserve an early mistake. Asked why use Claude rather than dedicated OCR, the attendee reports better results on some messy handwriting than with the Textract and OCR approaches they tried, especially after zooming in. No successful repair is demonstrated here; Witten closes the case with cropping and upsampling as the remaining suggestions.
Two brief questions follow. A feature called “fragments” turns out to mean Artifacts; Witten points to the bottom-left setting used to enable that workshop-era preview. Another attendee reports getting useful fixes by pasting entire tracebacks with no extra context. Witten attributes that improvement to model capability rather than a special prompting technique: sometimes the raw error already supplies enough information for a capable model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Calibrate what a grade means
The translation-grading case starts with an English original and a poor Japanese translation. The prompt asks for a score from one to five and already includes analysis before the grade. The submitter says Claude can identify errors yet still score the translation badly. In the live translation test, Claude assigns 3 on a five-point scale where the submitter expects 1. The submitter describes roughly a dozen errors and suspects the grader is confusing acceptable underlying content with translation quality.
Witten describes a broader grading tendency: pleasant subject matter can receive favorable judgments on unrelated dimensions, while negative subject matter can attract criticism of style or logic. The prompt should explicitly identify the dimension being judged. It should also state that one is worst and five is best, even if the rubric descriptions already imply that direction.
A detour into Japanese tokenization does not resolve the grading problem. Witten says API inputs are tokenized but does not explain the pretraining setup. Audience comments mix questions about public tokenizer availability with conflicting claims about Japanese training and language quality. Those are distinct issues, and the exchange establishes no comparative language benchmark before he returns to editing the prompt.
The proposed repair separates ethical or risqué subject matter from translation quality, then anchors every score category with an example. For each category, show the original, the translation, an explanation of the relevant errors or strengths, and finally the grade. Explain how specific clues contribute to the overall score rather than merely asking the model to find clues. Claude can help draft examples, but they need expert editing; Witten cannot supply the Japanese judgments himself. No validated post-revision improvement is shown.
Numbers and verbal labels can both work as classes. Witten warns against assuming that a one-to-100 score is a calibrated measurement and prefers roughly five categories. More numerical resolution does not create more reliable distinctions when the grader has not learned what those distinctions mean.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Analysis before probabilities
Log probabilities could expose how probability is distributed across possible grades. Witten’s concern is what happens if obtaining those probabilities means asking for a grade immediately and dropping the analysis stage. He proposes a two-call design:
- Generate the analysis before the grade.
- Stop at the end of that analysis.
- Resend the analysis as an assistant prefill.
- Request probabilities for the grade that follows.
This requires a model interface supporting both prefill and log probabilities. Witten cannot identify one with both capabilities during the discussion, so the workflow remains a proposal.
His expectation is that analysis followed by a discrete grade would be more useful than a more nuanced probability distribution produced without analysis. The mechanism he points to is additional computation: explicitly generating the analysis gives the model more forward passes before it commits to the grade. This is his rationale, not a measured comparison in the workshop. The audience clarifies that he means analysis actually written before the answer, as in the prompt under discussion.
One last audience suggestion concerns rubric formatting: numbered criteria might influence score weighting, so dash bullets could be worth trying. The attendee posts an alternative prompt in Slack, but there is no live comparative test of that change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract evidence before summarizing
The final prompt asks for a concise summary that preserves important information while avoiding introductory and closing chatter. Witten moves the instructions below the document and asks Matt for an input on which the prompt hallucinates. Prefill can address the unwanted introduction, but it does not by itself ground the summary in the source.
For grounding, Witten proposes extracting relevant quotations first, then writing the summary. This introduces an evidence-selection stage before synthesis. It also changes the response contract: a prefill that already starts the summary would skip the very stage the revised prompt now requests. The visible intermediate prompt has the quote-first instruction above an unchanged summary prefill—the mismatch he addresses next.
The intended sequence is now source document → relevant quotations → summary, with the prefix beginning the quotation stage. Matt is still obtaining the failing document when stage time expires. Witten offers to continue the case afterward, leaving a proposed repair rather than a tested before-and-after result. The final change makes the dependency explicit: gather the source evidence before generating the prose that depends on it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's collection of notebooks and recipes for building applications with Claude.
Further reading
Exercises on prompt structure, examples, output formatting, and hallucinations, originally written around Claude 3 Haiku.
Historical announcement introducing Claude 3.5 Sonnet and the Artifacts preview.
Anthropic's companion explanation of its Scaling Monosemanticity research into interpretable features in Claude.
Updates since the talk
Current guidance for schema-constrained JSON responses, including truncation and refusal handling.
How to distinguish completed responses, token-budget truncation, and custom stop-sequence termination.
Current image-input guidance covering resizing, resolution limits, legibility, and image quality.
Read the complete timestamped transcript
- 0:00
[upbeat music] All right, good afternoon, everybody.
- 0:17
Good afternoon.
- 0:17
Thank you all so much for, for joining us. We have the, uh, envio- enviable position of being after lunch. I'm seeing some cookies on the table still, but thankfully you're not here to listen to me.
- 0:27
You're gonna be riveted by the prompt doctor. He's gonna come up in a second. So I'm expecting no sleeping on the table, but, but you never know. Um, excited to be here.
- 0:35
I'm Jamie Neuwirth. I lead our startup team at Anthropic, and I just wanted to say a couple of quick things before we got going here with, again, the reason you're here, the prompt doctor.
- 0:44
Uh, we've had a lot of really exciting releases just in the last couple of days, some in the last couple of hours, and wanted to just put these up there to highlight some of the cool things that we're doing, but also share how a lot of folks, not only in this room, some of your peers, maybe folks
- 1:00
back at the office, can work with Anthropic on not only some of the prompting, of course, works, uh, work we're gonna do here, but just helping you grow your business and the really cool things you guys are building with Claude, with LLMs on top of AI.
- 1:15
My team is here specifically to help your company grow and scale the business, whether that's getting access to higher rate limits, getting access to folks like Zack, who's gonna be up here in a moment, learning more about what we're doing from an early access perspective.
- 1:29
We wanna work with you and empower the next wave of really exciting AI companies built on top of Claude. And we're helping from a product perspective with a couple of the releases you see here.
- 1:39
Has anyone tried 3.5 Sonnet yet?
- 1:42
Whoo.
- 1:43
Love it. Very cool. Really exci- yeah, thankfully Zack has tried it as well, so we're in a good place. [laughing] Uh, really excited about what we were able to release there as well.
- 1:51
Um, also just today, or excuse me, I guess Artifacts came out with that, with our Claude Teams and Claude AI plan. Artifacts is this really cool tool that some have been playing around with making video games with a couple lines of text that turns into code, but actually a lot of really cool business use cases from a
- 2:07
diagram perspective. I've seen a lot of cool pharma companies using this and thinking out, "How can I put what's in my head on a gene sequencing kind of discussion onto something like an artifact, share that with my peers?"
- 2:19
All the way to prompting cookbooks, helping, again, folks like yourself. I would imagine a lot of you hopefully will be featured on the website, you know, coming up with what you'd be able to prompt for these use cases in your companies.
- 2:30
So just some ways that, that we've n- uh, uh, you know, uh, things that we've come out with over the last couple of days, couple of hours when you think of Claude teams and projects, and then ways to connect.
- 2:39
Feel free to reach out at sales, uh, [REDACTED:email_address]. We have our head of DevRel, Alex, here as well. So we really love this community, love staying engaged. We're really excited about what we've released over the last couple of days to, to be able to help you, again, in what you guys are building.
- 2:54
And so I'm gonna leave all that to the prompt doctor as well to get that, uh, uh, to really make sure that we can help. And Zack, why don't you come on up here and, and see what we can do.
- 3:03
Thank you all so much for joining us today. We're really excited. [clapping]
- 3:13
Thanks, Jamie, for the intro, and thank you all for coming. This is really awesome. I had no idea this many people were gonna be here, so thanks. Uh, okay, so not gonna be much talk.
- 3:23
It's mostly just gonna be straight-up prompting from beginning to end. Did make a, a couple slides, so mostly about what to, what to bring. So I set up a, a Slack channel, um, in this, uh, the, the, the AI Engineer Slack.
- 3:36
It's called prompt-eng-live-workshop-anthropic. Um, so that's where you can upload prompts, and what we're gonna do on stage is, uh, I'm just gonna sort of look at them, uh, I'm gonna read them, we're gonna test them in the console, uh, the Anthropic console, uh, we're gonna see if we can get better results, and we'll just sort of like,
- 3:56
uh, try, try to learn as we go. Uh, so this is something that I do, like, internally in our, our team Slack, uh, quite a bit, but I've never done it in front of this many people and I, uh, this is-- it'll be exciting.
- 4:07
It'll be, it'll be fun. Um, might be some hiccups along the way, but hopefully you all have a good time too and, and maybe learn something. I know I'll definitely learn something.
- 4:14
Um, so what sh-- what kinds of things should you put in this, uh, Slack channel? So, uh, you can put a prompt template. So a prompt template is kind of like a prompt.
- 4:24
Uh, actually, I just realized I don't even need this, this mic. Um. [laughing]
- 4:28
Okay, so you put a prompt template, uh, which is like a, it's like a prompt but with spaces where the variables are gonna go and the variables, th- th- they're, they're gonna be denoted with these double brackets.
- 4:39
So in this case, it's like this, this document part. Um, if you don't have it in this format, that's fine. We can figure it out. This is just, like, the ideal.
- 4:48
Uh, so this is, like, the prompt template. This is, like, the kinda thing you'd put there. And then you can also have a couple examples of cases where it doesn't do what you want, and that will give us, um, the, uh, like, direction as far as, like, where we wanna go with it.
- 5:02
Um, I might also, like, uh, ask you questions out loud if I have questions about, like, what kind of output i- is, is good or not, or I might ask questions i- in Slack, uh, either way 'cause it's, it's easier.
- 5:14
Uh, we'll have to kind of, uh, figure that, that out as we go. Um, okay, so that being said, uh, we're gonna use the console for, for iterating mostly, although I might use a couple other tools like, um, Claude for Sheets, which is like a spreadsheet where you can, uh, call Claude.
- 5:30
Um, okay, uh, so yeah, let's see what we've got in the Slack already.
- 5:36
Okay, we have something here. Thank you, Gordie. So you're an expert patient. So let's put this into the console, and then let's take a look.
- 5:52
Okay, and I'm just gonna go through as many of these as, as we can get through in the, in the session. And, uh, yeah, this is pretty much what it's gonna be.
- 6:00
So first of all, we can probably capitalize all-
- 6:07
The, uh, sentence is?
- 6:10
Does that matter?
- 6:12
Does that matter?
- 6:13
Yeah. Sorry. Hi, I'm Gordie.
- 6:14
Thank you. Yeah, yeah, yeah, yeah, yeah. [laughs] Perfect.
- 6:17
Yeah. So does it matter having capitalization?
- 6:21
I, I think so.
- 6:24
It's okay. [laughs]
- 6:26
A, a lot of things like prompt engineering is, like ... It's, it's very new, right? So, like, we don't know for sure. Somebody out there might have done a study where they, like, conclusively show that using capital letters and, like, using, like, better grammar, fixing grammar mistakes help.
- 6:42
I have, like, anecdotally found this in a few cases. I also have read some, like, quantitative stuff showing that, like, typos do hurt performance. But I'm also just, like, pretty obsessive about this stuff, so I just fix it and I think it ...
- 6:54
Like, it definitely doesn't hurt. Um, okay.
- 6:59
Can you zoom in?
- 7:00
Can I zoom in? Great question. Is th- is that any better? No.
- 7:05
Little more. Little more.
- 7:06
Little more. Okay, is that any better?
- 7:09
Yes.
- 7:12
Okay. Um, so first thing, let's put information in XML tags. So we can go like this.
- 7:20
Why XML? Why not, like, Markdown?
- 7:21
Why XML? Why not Markdown? Uh, another great question. So Claude was trained with a lot of XML in its training data.
- 7:29
Okay.
- 7:29
And so it's sort of seen more of that than it's seen of, of other formats. Um, so it just works a little bit better. So this looks like all the information here.
- 7:39
So we have the medication review will be ... Okay. Um-
- 7:42
Can you run it before and after? So before the first iteration and then the final iteration and see how it compares?
- 7:49
Yeah, great call. Uh, okay, actually let me undo everything that I've done so far. [laughs] [laughs]
- 7:56
Okay. Uh, so we can run it here and then now in the console it's asking us for the user input. So do we have a user input?
- 8:04
Yeah, I gave you a sample in Slack.
- 8:06
Okay, perfect. So who are you? Let's do this one. Why do I need to ... Uh, yeah, we can, we can do them both. Um,
- 8:16
who are you? Okay. So Gordie, what do you think of this?
- 8:27
It's way too long. This is a conversational agent, so no more than one sentence.
- 8:32
Okay. It's too long. Um, we can probably fix that. So, uh, well, we can also use this evaluate tab. So let's, let's just, like, add all the test cases here.
- 8:45
So this is the evaluate tab of the console. I'm also gonna be doing, like, some showing off of the console, uh, features, 'cause I think it's a cool tool, tool for prompt iteration.
- 8:54
Um, there's also some secret new features that I might show. Uh, we'll see about that. Um, okay, so then we also have, why do I need to do this experiment?
- 9:10
Looks like it added a bunch of, uh, new lines. Let's definitely get rid of those.
- 9:15
Um, and then we can get this next one. Can I schedule it tomorrow instead?
- 9:23
Cool. Um, so I hit run remaining, and this is all running through Dove, uh ... Sorry, through, um, this ... So, so, okay, so we have, why don't you do this appointment?
- 9:38
So this looks pretty long as well, and here we have this, like, I apologize, but I don't have information about scheduling availability. Um, okay. So
- 9:53
is that true? Is it true that we don't have information about scheduling?
- 9:57
Uh, no. We are always available. Make it easy for them. So 24-hour availability.
- 10:04
24-hour availability. Okay. All right, so this is like the version one.
- 10:09
Now let's make some changes. Um, so first of all, I'll ... Actually, I'll try to do things, like, roughly in, like, some order of importance.
- 10:20
Uh, so maybe I won't make you all sit through the capitalization, uh, even though I, like, would definitely do that. I'm also gonna add a, a new line here just 'cause I think that's, like, more normal what you'd see in, like, a written document.
- 10:32
You'd have a new line. Um, we'll close the information tab.
- 10:38
Um-
- 10:38
Could you get Claude to fix the capitalization?
- 10:40
What's that?
- 10:41
Couldn't you get Claude to fix the capitalization?
- 10:43
Yeah, I could actually. Uh, that's not a bad idea. The, the one thing I wouldn't feel completely confident of is that it would, like, exactly transcribe the, the rest o- of everything, like, word for word.
- 10:54
I think it probably would. Um, what I actually might do is just, like, have Claude write code, uh, to capitalize every, uh, first word of the sentence. Then I'd be worried about edge cases, like what if there's, like, ellipses.
- 11:09
Um, but I, I ... That kind of thing i- is definitely useful, and, like, I definitely use Claude a lot in, in writing prompts. Uh, for instance, like, we have, like, a, a Claude tool that, like, helps complete code basically, and I do a lot of prompting in that IDE because, like, especially with, like, very nex- nested
- 11:25
XML tags, uh, it helps a lot just, like, suggesting the closures of them, which is, like, pretty obvious but still takes a long time to type. Uh, so yeah, if you have any sort of, like, co-pilot type thing, uh, definitely that's, like, a good environment for, for writing prompts.
- 11:40
Um, okay, now let's do the same thing with this instructions. Um,
- 11:47
and we can do this. It looks like, um, this one, like, didn't get a number, so let's, like, do that.
- 11:58
Um ...
- 12:01
Where do you know to, uh, stop with XML and leave it as plain text?
- 12:07
Yeah, so the key thing in terms of XML, I think, is just, like-
- 12:12
Really, XML isn't even that important. The most important thing is just clearly separating the different parts of the prompt
- 12:17
Just using it for, like, semantics of what this section is and so forth
- 12:21
Yeah, exactly. It's like, here's this stuff, here's this other stuff. Like, if we wanted to, we could do something like, like
- 12:29
... Like, I, I, I wouldn't do this, but, like, I think it would probably work fine.
- 12:36
Yeah. Yeah, it does.
- 12:37
Yeah. Um, okay. So this is all fine. Let's also do the same thing with user input.
- 12:51
Now we can run, we can go back to the Evaluate tab, and we can hit Rerun all, and it's using our nice new prompt. Um, still looks pretty long.
- 13:01
We can also see how it does on the, the last case where, um
- 13:07
... Okay, so here it still said, "I don't have access to the specific scheduling information." So let's, let's try and fix these two things. So first of all, we can make it shorter, so
- 13:17
do we have anything here about, like, making it shorter?
- 13:20
Seven.
- 13:20
What's that?
- 13:22
Rule seven.
- 13:22
Rule seven, okay. Um, be concise and offer only relevant information. Uh, oops, let's actually do this.
- 13:32
Don't wanna misnumber here. And, um, be concise. So, like, and offer only relevant information. Each
- 13:44
response should be, uh, or let's, let's be a little bit, like, less prescriptive to give Claude, like, a little bit more room. Like, if we say, like, every response should be, like, exactly three sentences, that might be, like, a little too constraining.
- 13:58
I'm just guessing. Um, so we could just say, like
- 14:03
...
- 14:10
So why is responses should be two to four sentences better than telling it to be concise?
- 14:16
Because you can
- 14:17
Concise is arbitrary.
- 14:18
Yeah, concise could mean a lot of different things to, to different people. Like, in some cases, like, concise might mean, like, literal only one word. In s- in some cases, like, if you ask for, like, a concise book review, we might be looking at, like, you know, a single page w- Word doc.
- 14:32
Uh, and that would be concise in the context of a book review. Um, so yeah, Claude is, like, trying to guess what you mean by concise.
- 14:42
What if it needs to be variable
- 14:43
You also have a longer prompt, so if you have a really long prompt
- 14:47
Sorry, one sec. Go ahead.
- 14:48
I was gonna say, he has a really long prompt, so if you have a long system prompt with a lot of detailed instructions, saying be concise, you're not gonna get something super, super short.
- 14:58
I think that's right. I think the tone of the prompt ... So the, uh, uh, what he was saying for, if people couldn't hear, is, like, the prompt is long, so the response might also be long.
- 15:06
I, I don't think that's, like, definitively true. Like, you can have long prompts that give short responses or short prompts that give long responses. But it's more like if you don't say anything, it might pick up on some of those, like, context clues.
- 15:17
You were saying something over here?
- 15:19
Um, what if it needs to be variable depending on the type of question that's been asked? Like-
- 15:24
Yeah
- 15:25
... keep it short for, like, things that ... Because in human interaction, right, like, sometimes we give really short responses, other times we give long responses.
- 15:34
Yeah.
- 15:34
Have we trained LLMs to do that yet or we're not really doing it?
- 15:37
Yeah, so l- let me actually get to that af- after we do this. So this two to four sentences, it looks like it's still pretty long. I think maybe that's actually, like, longer than necessary.
- 15:45
So maybe we should make it, like, one to, one to two sentences. Um, let's try that.
- 15:53
Never more than three. Okay, now we can try that here.
- 16:01
Okay, that looks better, right?
- 16:03
Yeah.
- 16:03
This is definitely shorter. Okay, um, and it also seems that it is giving variable numbers of sentences, so these were both two sentences, and then this one is three.
- 16:16
So o- one of the questions over here is, like, can the LLM figure out that it should do longer responses in certain situations and shorter in others? So it seems like it, it did that here.
- 16:24
Um, okay, so then the next point was that in this case, it shouldn't say that I don't have access to the scheduling system or specific appointment times.
- 16:34
What should it say instead?
- 16:36
It should say, "Sure, what time tomorrow?" But intentionally in the prompt, I left out that it's 24-hour service.
- 16:45
So this is a case where we're asking a question that's not present in the information it has.
- 16:51
Okay. So yeah, I mean, we could a- add something like, um, you're open for 24-hour ser- service.
- 16:59
Yeah.
- 16:59
But you're saying y- you want it to ... You're, you want to test its ability to, like, figure out how to do it without that.
- 17:05
It should say, "I don't know," so.
- 17:06
Oh, okay.
- 17:07
It is saying, "I don't know," which is good.
- 17:09
Okay. Well, then we, we're doing great. All right.
- 17:15
Should we ... A- anything else that you wanted to get out of looking at this example, Gordie?
- 17:20
I think that ... Oh, about the structure. So does the order of the rules or the order of putting information, then rules, or rules first, then information, does any of that matter?
- 17:32
Yeah, so do we, does it matter what order we have these components?
- 17:36
Yeah.
- 17:36
I, I think it's better to put the information above the instructions.
- 17:40
Okay.
- 17:41
We've sort of found that instructions are more tightly followed the closer they are to the bottom of the prompt as a rule. This doesn't necessarily apply in all situations, so definitely test it out.
- 17:54
That actually is, like, a blanket statement that applies to everything that I said,
- 17:58
but es- particularly for that.
- 18:01
Yeah.
- 18:01
Okay. Uh-
- 18:02
Can I ask a question?
- 18:03
Yeah.
- 18:04
I, I don't know or I don't know if you were asking all the questions or-
- 18:07
No, no, no, go ahead.
- 18:08
It's only [REDACTED:physical_attribute] [REDACTED:gender] allowed, so you're good.
- 18:15
I noticed that it's-
- 18:17
Exclamation mark?
- 18:19
The exclamation mark, yes.
- 18:21
Oh, remember, I don't think I a- added that on... Or it looks like you added that, Gordie. I- e- exclamation marks, just as they emphasize things for humans, they also emphasize things to the model.
- 18:33
Do you, for the... Or do you think that has more of an effect over the numbers or less, like, in terms of relevance?
- 18:40
Ooh, yeah, I don't, I don't know at that level of detail, and I think it's dependent on, on context as well. But yeah, if, if you want to emphasize things, like capitalizing them or putting exclamation marks or, like, just saying, "This is extremely important," that all does do something.
- 18:56
Shouting in caps.
- 18:58
So-
- 18:58
Yeah.
- 18:59
Um, so the tokenizer, though, um, I, I, I'm taking a look at some of your, your tokenizer code, but it doesn't seem like exclamation points actually to- like, do anything really.
- 19:13
Like, they're, the tokenizer kind of combine them into the word.
- 19:17
It does something. That's all I c- I, I can say. Just anecdotally, if you put exclamation marks in, it's, it's different.
- 19:22
I guess, okay.
- 19:23
Yeah.
- 19:24
That's my, that's my analysis of it.
- 19:26
All right. Okay, I think let's go to, um, the next. How many we got? We got 60 already. That's pretty good. Okay, this is just a general question. I'll just answer this really quick.
- 19:38
Is it in general for translations or multilingual output, is it better to instruct in English or the native language?
- 19:44
I think it's better to instruct in the native language if you speak the native language.
- 19:49
If you only speak English and you're choosing between a, like, a better prompt that's written in English versus, like, a worse prompt that's written in a language that you don't understand, I'd probably default to writing it in the language that I knew super well.
- 20:03
But ideally, I think for the ideal prompt, you would find a native speaker of the language and explain your use case to them and, and have them write the prompt.
- 20:12
Well, can I ask a question about language?
- 20:14
Yeah.
- 20:15
So, uh, for a language that, uh, Claude is not so familiar with, should we, like, maybe write the prompt in English, but let the, the Claude to translate the output into the other word, or should we just write the prompt in that language?
- 20:32
So if it, if it's-- I think, is that not the same question that I just answered? Is it different?
- 20:35
Oh, so you said it's better to write in the, in that language.
- 20:38
I think it's better to have the prompt in, in that language in, in general, if you can, if you can write a really good prompt. All right, let's go to this, this next guy.
- 20:46
So you'll be acting as a test reviewer. Let me pump up the size here too. Um, okay.
- 20:55
Not sure if there's a way to... Oh, I can hide this. Okay, great. Responsible for improving unit tests based on a set of requirements. Below is the project directory,
- 21:03
project path. Don't include any other explanation, conversion, or output besides the JSON.
- 21:14
Okay. This is great because it's gonna let me show off pre-fills.
- 21:21
So let's make a new prompt here. Let's paste this in. We have to use, uh, double brackets instead of single brackets to get variables in the console.
- 21:35
And then I think there's another... But this JSON string, uh, who, who gave this prompt? This is from, uh, Dan. The JSON string, Dan, that's, is that variable, or is that,
- 21:48
is that, like, an example in this prompt template?
- 21:51
So in this case, that, that's, um, that is just a template, and then we, this is just the middle of a multi-agent workflow. So basically, another one of the agents is gonna write the unit tests, and then, and that's gonna be a JSON, and that JSON blob is what it's gonna respond with, essentially just using the earlier
- 22:07
part of the conversation. I, I put the, uh, sample output in red if you go back to that.
- 22:16
Yeah. Test reviewer agent does not return, always return JSON only.
- 22:25
Yeah. Like, almost all of this is just, like, workarounds for the fact that it doesn't always speak JSON right. Like, you can see how many times we said that in the prompt.
- 22:32
Yeah. And then, uh, do you have an example input here?
- 22:36
Uh, so that first, if you go up a little bit, sorry. So that first comment is from the unit test writer. So that is the input. Like, the unit test writer writes a bunch of unit tests, and then the reviewer reviews it and makes them better.
- 22:50
So everything, everything here is, is what I should put into-
- 22:54
Yes. And then you can see that second set. This is, this is a good result where it writes JSON, which basically says, "Cool, update this file with these unit tests, and here's the modifications I made," that sort of thing.
- 23:05
Okay. So i- in, in this template here,
- 23:09
w- where would the thing that, that I just copied go?
- 23:13
Well, so essentially, we don't provide it in line with the prompt. We just provide the con- the conversation, and then this thing jumps in, in as a, as a separate agent.
- 23:26
So, like, the, the context window is gonna have the unit tests in it, but we're saying respond in this format given the unit tests that are earlier in the conversation that you're picking up.
- 23:36
So th- this, the thing that you just pasted is, like, step three of a multi-shot conversation?
- 23:41
Uh, the thing I just-
- 23:41
Like, or sorry, not multi-shot, multi-turn.
- 23:43
Yeah, yeah. The thing I just pasted was two shots, right? Unit test writer and then a unit test reviewer, and the reviewer is the one that's having the problem.
- 23:50
It comes second.
- 23:53
Okay, so it'd be something, it'd be something like, like this. Um,
- 23:59
here are some unit tests written by a unit test writer bot.
- 24:09
Right.
- 24:17
Okay, and then we have this. Now, you didn't, uh... So, okay, so it, this, this is basically how it works.
- 24:26
Is, is ... Does this look right?
- 24:27
Oh, no, it does. I mean, this-
- 24:29
Okay
- 24:29
... we, we just, we do this in sort of this larger conversation, not just as sort of a standalone prompt for this one-
- 24:35
Yeah
- 24:35
... agent.
- 24:35
Okay. So then here I put the unit tests in.
- 24:37
Right, yep.
- 24:38
And then for project path, what sort of thing should I put there?
- 24:41
Um, anything. I mean, th- this is y- this is just, like, the local directory that's gonna be modified, and so it actually has access to the files in that directory, and it'll, it'll fill in its own, um, sort of which, what files to modify.
- 24:53
Okay. So then I'll just put, uh-
- 24:55
That's fine
- 24:57
... um, something like that.
- 25:00
Yeah.
- 25:07
Okay, so let's see if it comes out with the JSON or not. Uh, bated breath here.
- 25:12
Yeah. We did do most of our tests on Claude 3 and not 3.5, so 3.5 is probably a little better. Um-
- 25:17
Okay
- 25:17
... we haven't done.
- 25:18
Yeah, I mean, if, if, if it, if it makes it more realistic, we could also, uh, switch the model version to, to use Haiku.
- 25:23
Oh, no. I mean, we're, we're gonna upgrade, so I'd rather see it with this.
- 25:26
Okay. What was that?
- 25:29
So the, I see the temperature's set to zero. Is that intentional?
- 25:32
Is the temperature being set to zero intentional? Yeah. I usually, for knowledge work, I usually have the temperature set to zero.
- 25:38
Not 0.01? Like, I usually use that to test for hallucinations. I mean, I, I, I'm just, I, I'm just a fool here, but, like, is that, is that ...
- 25:48
Like, am I totally out?
- 25:50
I think using temperature zero you'll probably get, like, marginally fewer hallucinations. Okay, I'm ... Oh, here we go.
- 25:57
Okay. So it looks like in this case it did output JSON,
- 26:02
I think.
- 26:03
Yeah. That, that looks plausible.
- 26:05
Okay.
- 26:08
It's still-
- 26:08
Very long JSON. I guess that explains why it was taking so long to ... Looks like, actually, it even ran into the max output tokens, because it didn't finish its JSON.
- 26:16
Aha.
- 26:17
Um, just to make this, since this one is, is kind of going kind of slow, I will test it with Haiku.
- 26:24
Uh, let's ... And let's also increase the, uh, max tokens to sample so that it doesn't run into that issue.
- 26:34
So what I'm really hoping is to get a case that doesn't output JSON so that then I can fix it, and then it will output JSON. If not, I can still say, like, how I would fix it.
- 26:44
Yeah, that would be great. Just honestly any comments you have just on how we structured things.
- 26:48
Okay. Yeah. So I mean, this is, like, uh, definitely, like, a, a big request from people is like, how do I make sure the model outputs JSON? The most reliable way to do that, I feel, is using the, uh, assistant pre-fill.
- 27:02
So, uh, maybe some of you have used this feature before. Maybe some of you have, like, only used, uh, other models such as GPT that don't offer this feature.
- 27:12
Something that you can do in the Claude API is partially pre-fill an assistant message. So what you're doing there is you're putting some words in Claude's mouth, as we call it, and then when Claude continues from there, it assumes that it's already said whatever you told it that it had said.
- 27:31
Uh, and just, that, that can help you get it on the right path. So for instance, in this case, uh, if we wanna make sure ... So the classic, like, bad response from, from Claude w- when people give it prompts like this where they wanna get JSON, is Claude would say something like, uh, I'll just, like, add
- 27:47
another message just to have some of us to type. It might be like, "Here is the JSON," right? Have people seen stuff like this?
- 27:55
Oh, yeah.
- 27:57
And this part right here is, like, very annoying.
- 28:01
Yes.
- 28:01
And difficult to get rid of. So okay. So I, I have, I have two strategies. Uh, let me actually just give, like, the sim- the simplest one. They both, though, they both require a tiny bit of post-processing at the end.
- 28:17
So let's start by, uh ... Let's actually, like, take out all this stuff about make sure to only respond in JSON. That, that could be one way to get it to, to not do the, uh, to, to be bad.
- 28:30
Uh, so we could just go like this. Let's try to make it not do, uh, the JSON.
- 28:36
Let's get rid of all this stuff. Okay, so here now maybe it will do the preamble thing that we don't want it to do. Perfect.
- 28:46
Okay. So an easy way to get it to not do that is to just take this and then put it in
- 28:57
the pre-fill, so it thinks that it already said that.
- 29:01
Uh, like this. So if we do that,
- 29:08
just the JSON. So what we're doing here is ... You could think of Claude almost like a, like a child who's just, like, misbehaving, and it wants to do something, and you're l- and you're like, "Don't do the thing," but it just keeps doing it because it just loves preambles so much, and it has this, like, innate
- 29:25
desire to, to do them. So one way is to, like, argue with it a lot, but, like, if you have a kid, sometimes you know you just have to, like, let them do the thing that they wanna do, and then they'll get over it.
- 29:35
So in this case, that's basically what we did. We just gave Claude this, this pre-fill where we let it do the thing. So as far as it's concerned, it already did the thing.
- 29:45
And then from there, what it, what it's outputting is, is, is JSON. Now, if you wanna make this even more reliable,
- 29:52
you can put this nice little bracket here, and then it's like, oh, dang, like, I'm really in JSON mode now. Like, I'm, I'm really ... My, my JSON has actually already begun.
- 30:02
So it's, it ... At, at this point, it's definitely not gonna do the preamble. The only thing her- here is if you sample with, with this, with this, uh, pre-fill, you will need to add the bracket back before you try to do your JSON.loadS or what have you because Claude is ...
- 30:19
Si- since you told it that it had already said the opening bracket, it's not gonna give you another opening bracket. Uh, okay. So then another thing that you can do is
- 30:30
return the JSON in JSON tags. And then
- 30:37
if we do this without the prefill ... Let's try it without the prefill.
- 30:41
You don't normally capitalize JSON? [sighs] [laughs]
- 30:50
I'm not a software engineer, okay? I'm a prompt engineer. I don't even know that it's capitalized. For all I know, that's just, like, a English word. [laughs]
- 30:56
Um, but yeah, good, good. Thank you. Um, okay, so here we, we see it, it, it did the thing. So it gave its preamble, right? And then it gave the JSON tag.
- 31:08
Everything within the JSON tag is JSON. And then at the end, it closed this JSON tag. So again, this requires, like, the tiniest smidgen of post-processing, where you're saying, like ...
- 31:21
You just ... It's like a regex. You're just like take everything within the JSON tags, and then, and then use that. You can even combine th- these two techniques.
- 31:29
So you could say, "Here's the updated JSON," and now you give it the JSON tag. We can even put a bracket here. And now what we'll see is it will just give the JSON minus the bracket, and then it will close the bracket, and then it will close with, uh, the JSON tag,
- 31:46
the close JSON tag. There we go. And it also gave this, uh ... So you can see it did ... At, at first I was, like, a little bit panicked, because I didn't see the close JSON tag at the very bottom, but then I saw that it actually did include the tag up here, and then it gave
- 32:05
this little explanation afterwards. So this is another, uh, useful thing. This will, uh, save you some, uh, time and tokens and trouble. One thing we could do, like, like, it, it, it costs Claude, like ...
- 32:18
It costs you money to get Claude to output all this stuff, and you probably im- you probably don't need it. You ... In most cases you don't need the explanation.
- 32:27
You just need the JSON. So one thing we could do is we could say, "Do not include any explanation after the JSON," right?
- 32:37
Does it help to yell in that case? [laughs]
- 32:38
I mean, probably, but I don't know. I, I, I, honestly, I don't yell that much. I'm just, like ... This is actually meant to be my parody of, like, what a frustrated prompt engineer would write if they were, like- [laughs] ...
- 32:48
couldn't get rid of this. But in practice, you, you might not need to do that. But the simpler way to, to do this, there's a ... And we're getting outside the realm of prompt engineering for a second and into the world of, like, API parameters, but, uh, that's okay.
- 33:02
There's a parameter that's called stop sequences, and if you set ... So we told it to return the JSON in JSON tags, right? So, uh, there's no functionality to do this in the console, so I can't show it off at this exact moment, but in the API there's a parameter called stop sequences, and if you add, uh,
- 33:20
this close JSON tag that I've highlighted with my mouse, if you add that to the stop sequences, then it will just hard stop after it outputs those, and you don't even have to worry about telling it not to continue from there, because it just won't even sample from there.
- 33:35
You won't be charged. It's all good. So one of the things that I'm sort of hoping to impart with this talk is that a lotta times it's cheaper and easier to do a little bit of work outside of a call to the LLM and not even worry about prompting, because prompting is, is, is ...
- 33:55
can feel sometimes, like, nondeterministic. You don't know what the model's gonna do. So when you can offload stuff to code, especially if the code is really easy to write, it's like just do that, right?
- 34:04
Like, don't, don't put a bunch of stuff in the prompt about you must output JSON. Just, just use the prefill and then parse it out with the, the regex.
- 34:11
You know, don't, don't add a bunch of stuff about how you have to, to stop after you say a certain word. Just add it to the stop sequences. So, like, simple is, is, is better, and falling back on code is better than relying on prompts.
- 34:24
Yeah.
- 34:25
Is that prefill available through the API calls? And-
- 34:29
Yes. The prefill is available through the API. What you do is you include an assistant message as the last message in the messages list.
- 34:39
And when I say an assistant message, I, I just mean a message where the role is set to assistant.
- 34:44
And, uh, what would have happened if you had your ... the text that you put in the prefill, you just put it into the last line of the instructions?
- 34:53
So in other words, if I said, uh ... I'm actually not sure. That's a, that ... good question. So let's actually try that.
- 34:58
So it does work with open. Like, you can just say, like, there is
- 35:04
a
- 35:04
I, I, I, I genuinely do not know how Claude will respond to this, so let's see.
- 35:12
So it looks like what it did was ...
- 35:15
It looks to me like what it did, without looking at this JSON, is it included an additional open bracket,
- 35:23
right? 'Cause it, it's supposed to already have started with an open bracket, but here it started with an additional open bracket. So it kind of almost worked, but not quite.
- 35:31
Anyways, I don't recommend doing this, but that was, uh, fun just to out of curiosity's sake. Yeah.
- 35:37
Uh, instead of that can we just, uh, remove that? Uh, uh, and, uh, after the sentence, like ... Yeah. Is it on? Yeah. Uh, after the sentence we can just write it like, uh, you wrote, right, uh, return the JSON, return the response in the JSON format, and then in the next line, uh, you can just write
- 35:56
JSON colon, then leave it. I think it'll-
- 36:00
Oh, so, like, if I said something like this.
- 36:04
Like this?
- 36:05
Yeah. That's it.
- 36:07
Yeah, I could see this working. Yeah, it looks like it worked pretty well. I, I, I think there's, like, a lot of ways to accomplish this.
- 36:16
Yeah.
- 36:16
I think the ways that I showed are the most reliable, so that's what I would, like, officially recommend. Uh, but yeah, like, definitely experiment.
- 36:26
If you were gonna, uh, like, try to use this for production or whatever, what ... Like, these exact kind of things you're playing around with right now, how would you think about testing that, like, at some sort of scale, like?
- 36:37
How would we test it at some sort of scale?
- 36:39
Yeah.
- 36:40
So-
- 36:40
More than, like, one ... like, the one-shot test we just did right here.
- 36:43
Yeah, yeah, yeah To test it at scale, you need a bunch of test cases
- 36:49
and-
- 36:50
We don't have those
- 36:50
... if you don't have test cases ... Okay, this is m- maybe a good time to maybe show off this thing, although I'm actually not sure if it will work.
- 37:00
Uh-
- 37:01
So, well, one, I guess a more pointed question is like-
- 37:04
Yeah
- 37:04
... in this case, I think test cases are useful when I'm writing a prompt to deduce whether, like, does asking it to think step by step lead to this thing being more accurate?
- 37:14
But in this case for for- formatting, I guess what I'm wondering is, like, could you have this prompt and then feed in the output and the prompt, and then ask the model itself to evaluate, like, how good these various things are at following the instructions?
- 37:29
Yeah, yeah. Okay, so can we do model grading? Can we model grade the outputs?
- 37:32
Yeah, especially for formatting related things where I actually need a, like-
- 37:34
For formatting, I would not model grade the outputs-
- 37:37
Okay
- 37:38
... because formatting is something that I can check in code. So if I can do anything in code and, and I don't have to call the LLM, the LLM is, like, this crazy black box, right?
- 37:46
It's like, if I don't need to, like, make this pilgrimage to the oracle and, like, ask it, I, I'd rather just do it, like, in, in code. So formatting specifically, we're, we're kind of, like, in luck.
- 37:56
It's easy to check. For something like the other ... the, the previous prompt we looked at where the outputs are a lot more squishy, we might ... Possibly a model grading could work.
- 38:06
Possibly we might need a human to evaluate the answers.
- 38:08
To, just to lightly push back on that-
- 38:10
Yeah
- 38:11
... so I'm wondering, like, I actually put an example in the Slack channel. We don't need to get to it, 'cause we're talking through it now. But, like, for, uh ...
- 38:16
Let's imagine I don't have ... Or actually, maybe, maybe tags are the answer to this. Um, like, imagine I'm asking for a summary or something and then I want to deduce whether there's additional chat-like content, like, before or after that.
- 38:29
In that case, would you ... I would've, my mental model would've been to use, like, an LLM as a grader. But it sounds like maybe would you encourage instead using the summary tags and checking, like, hard coding for additional text around that?
- 38:43
Yeah. I think that will be, uh, pretty quick and easy to do. Also, just having the summary be in summary tags is, like, generally a good practice.
- 38:52
Okay.
- 38:53
I generally have all my outputs inside tags to make it really easy to extract them. I don't think there's really any, any downside to doing that, so ... And it m- it, it might even be that by doing that you effectively fix your entire issue and you don't even, like, need to, need to do the, the test
- 39:07
anymore or ... A- and you just put close summary in the stop sequences-
- 39:11
Yeah
- 39:11
... and, and you're kind of good to go.
- 39:12
All right. Cool.
- 39:14
But that's also does sound like a problem that an LLM, uh, could grade.
- 39:20
Okay. Uh, let's go to the next, uh, prompt here.
- 39:26
Just shout out your question.
- 39:28
Here's a very poorly formatted Excel spreadsheet.
- 39:31
Um, I got a question real quick. So, um, this seems like a really ridiculously, like, powerful attack vector, so can we test a prompt real quick?
- 39:42
Um, I don't wanna get into too much, like, jailbreaking stuff here. Sorry.
- 39:46
Okay. Ap- apologies.
- 39:48
Yeah, yeah.
- 39:48
So-
- 39:48
Well, I-
- 39:49
... I, that, that's kind of my specialty. [laughs]
- 39:50
Okay. Yeah. [laughs] Um, I'm gonna go to the next, next prompt.
- 39:56
Um, so what do we have here? Here's a poorly formatted Excel spreadsheet/CSV.
- 40:06
Please extract all data into JSON. Okay. How can we ... So, uh, Jan or Yan?
- 40:12
Jan, right here.
- 40:13
Uh, is it Jan or Yan?
- 40:14
Yan.
- 40:15
Yan, what, uh, what is the actual text that I should ... Can, can you, can you paste the text here? 'Cause I, I don't know how to get the CSV into the, uh, into the console.
- 40:24
Oh, I've just been ... Oh, hey. Thanks. I've just been just copying the entire CSV and putting that into the prompt. Again, I've been trying to use Claude to extract some information from spreadsheets, and it's always been very, very hard.
- 40:41
It hallucinates a lot or it-
- 40:42
Yeah
- 40:42
... skips a lot of stuff. And I was wondering if you, maybe more generally, how do you have Claude analyze really poorly formatted spreadsheets that sometimes the different clusters or multiple data sets in the same sheet and things like that?
- 40:57
Okay. I'll try to answer the general question of having m- uh, analyzing, like, poorly formatted spreadsheets. The first thing that came to mind, especially when you're s- talking about how the spreadsheets are very big, is breaking the problem down into ...
- 41:13
So, so give it, like, fewer spreadsheets at a time. Give it fewer columns of the spreadsheet at a time. Only give it the, the columns that it needs to, to work with.
- 41:22
Um, and then make the questions sort of smaller and more bite-sized, and then tackle it that way by, by breaking it down. So at that level of gen- generat- generality, that's would be, would be my answer here.
- 41:37
Um, y- I, I, I'd also be curious to look at this one more specifically. Right now I'm just struggling with how to, like, copy the text and put it into the tool.
- 41:46
Could you put as regards and then download all?
- 41:49
That's what I did, but it keeps downloading it. I guess I can-
- 41:51
Yeah, it's the next tab as it, as it opens.
- 41:54
Um, the ne- sorry, the next tab.
- 41:56
You had the-
- 41:58
This is that other one.
- 42:00
There's ... In your downloads, you can open it and then-
- 42:02
Sorry, I don't wanna click open my downloads. I'm scared I'm gonna, like, reveal some private information. This is my work computer, so- [laughs] ... I wanna just do it all in the browser.
- 42:12
Yeah. Ask it to write some Python codes. [laughs]
- 42:16
It's fine. You can go to the next one. No worries.
- 42:17
Yeah, let's do the next one. Okay. You are a social media ghostwriter.
- 42:26
Given the below long form article. Okay. Uh, generally we would recommend putting the ... I like this one. It's short. We can do some quick hits here. Uh, we would recommend putting the instructions after the document.
- 42:41
That's similar to the question that was asked about should we have the information first or the instructions first. Particularly with long documents, it's a little bit better to give the instructions, uh, at the end.
- 42:51
Let's also put some XML tags here. Uh, let's just, like, clean up the grammar a little bit. Given the above long form article, create five to 10 tweets. Don't use hashtags, [laughs] don't be hyperbolic, and don't be cringe.
- 43:08
Uh, return a JSON array of post content. Probably good to, to give an example, uh,
- 43:15
of the format. Uh, so we could do, like, uh ...
- 43:26
I guess we can just do a return of, like, a list.
- 43:30
Wait, uh, what did you originally have it? Is there a special reason that you wanted it to be a JSON array or is it just to make it parsable?
- 43:38
Uh, just to make it parsable for automation's sake.
- 43:40
Okay, yeah. So let's say return in ... I'm just a, a, a huge fan of these tags, so let's do it like this.
- 44:18
Okay. So that is, that, that's some stuff without adding examples. The other thing that I would wanna do is to give some illustrative examples of what it means to not be cringe. [laughs]
- 44:34
So how long are these documents?
- 44:37
Uh, I put in a ... I used a recent blog post from Anthropic as an example. Do you wanna copy it?
- 44:43
Perfect, yeah. Let's actually, let's run this as is.
- 44:47
You're missing a bracket at the end.
- 44:49
Yeah, I did.
- 44:50
The last, the very last, uh, hard bracket.
- 44:56
Oh, yep. Thank you. Now we can take this.
- 45:06
Uh-oh, what if they write cringe tweets about our product? [laughs] I'm gonna be embarrassed.
- 45:17
Okay, this doesn't include any hashtags, doesn't seem very hyperbolic. Is this cringe? What do we think? [laughs]
- 45:28
New feature alert. That could be a little bit cringe. [laughs]
- 45:32
There's no emojis, which is a good sign.
- 45:36
Okay. W- what do you ... Uh, your name was, uh, Charlie. What do you think of this, Charlie?
- 45:41
Uh, I think they are adequate, but not engaging.
- 45:48
Not engaging, okay. So yeah, yeah, yeah. So l- we can try to make it more engaging without making it cringe. So let's say, don't use any hashtags, but cringe.
- 45:57
Try to make the tweets engaging. Are these meant to be tweeted from the Anthropic Twitter account or from, like, the, a, a AI influencer Twitter account? [laughs]
- 46:11
Sure, let's say AI influencer Twitter account.
- 46:13
Okay. Let's see how this goes. I'm gonna switch back to Dove, too. Sorry, um,
- 46:33
exciting news.
- 46:36
Is it better to break it up into small sentences in the prompt or can you use complex sentences?
- 46:42
Is it better to break up sentences, to use, like, small sentences in the prompt or big sentences?
- 46:49
I think generally in English writing it's better to use small sentences and small words,
- 46:56
so I think it's probably also better to do that in a prompt. I think it's fine to use big words if you are really sure you know what you're doing and you know that it's the exact right word for the situation.
- 47:08
Sometimes I'll find myself using more academic language if I want the output to seem a bit more academic.
- 47:16
Generally, I think simple, small sentences is, is better.
- 47:22
Uh, okay. So these are maybe a little bit more engaging.
- 47:32
Like, they have these questions here. Wanna try it? Uh,
- 47:38
what do we think? It's got exclamation points and question marks. Is it better? Do we want it to be even more engaging or something?
- 47:46
It doesn't take into consideration temperature, right?
- 47:50
Let's see, 'cause ... Okay, so I honestly think temperature is, doesn't, is a bit overrated maybe. Uh, we can, we can see if, how, how, how that, how that differs, though.
- 48:14
I'm not sure exactly how to distinguish these from the, the previous ones. They look kind of similar to me. From the ones with, with temperature, uh, temperature one, or temperature zero.
- 48:24
Engaging is a very too general and subjective, right?
- 48:28
Mm-hmm.
- 48:28
It's better to e- us- emulate a certain per- personality.
- 48:32
That's right. Yeah. So what I was gonna say is I think this is roughly as far as you can take this without examples. I think the best thing to improve this prompt would be either examples of the sort of tweets that, that you want,
- 48:47
or even an entire other document, an example of tweets that go with that document, and maybe, like, multiple of those. So-
- 48:57
Uh, if, if you're cost limited, maybe you don't wanna put in all those input tokens every time. But I don't know, the models are, are pretty cheap now, and we don't need to generate that many tweets.
- 49:09
So if they, if they have, like, any economic value to you at all, it's probably pretty cost effective to put ... So basically, like, it ... But it's, it's more work on your part because what you're doing then is ...
- 49:22
So, okay, so the, the way that I would actually do this, uh, is I would start out with some, some document. I would have Claude write a bunch of tweets.
- 49:32
I would take the ones that I liked,
- 49:35
and maybe I would write some more or get, like, my friend to write some more, or maybe I'd have Claude ... Oops, uh, Claude generate 100 tweets, and then I would take the, the, the seven that I, that I liked best, and then I would put that in as an example.
- 49:47
And then from there I would sample, okay, now here's another document, and then write a bunch more tweets based on this. And what I would do is iteratively build up this, uh, list of documents plus example tweets, and then I would put them all into the prompt, and it would look something like this.
- 50:07
Uh, so let's actually do that. So let's imagine that we had done this. So it could be, like, you know, system prompt. You, uh,
- 50:14
you are an AI influencer who writes engaging social media content about, uh, new models and releases. Uh, it could be like, here are some example documents along with the tweets
- 50:38
you wrote about them. Um, and here you would actually ... I, I'm gonna write this, but you would actually put the literal text of the document here.
- 51:03
And here again, you'd put a literal tweet here, and this could either be something that you wrote or something that Claude wrote or, you know, you, you ... Something that Claude wrote and then you edited.
- 51:11
Like, a lot of times Claude might give you an example that's not perfect, but it- it's close enough, and then you'll, you'll change it a little bit to, to make it perfect.
- 51:20
I have honestly given, uh, multi-shot examples pretty short shrift in this talk so far relative to their level of importance. Like, I, I think that in reality most of the, the gains, most of the effort, most of the gains of writing a good prompt is literally just picking the perfect document that goes here, picking the perfect set
- 51:43
of tweets that go here, altering and, and changing them to, to, to modulate the tone.
- 51:49
Uh, i- in some ways that's more important than, like, everything else that I've said combined. Like, another way to do the whole JSON thing would just be, like, with examples of Claude giving the stuff without a preamble.
- 51:59
The pre- the JSON one is maybe an exception, uh, i- because the, the, the pre-fill approach wor- w- works so well there, along with the tags, but for anything else, the examples are, are, are really huge.
- 52:10
Anyways, then we would, uh-
- 52:14
For few shot prompting, do you prefer to front load those all in one response like this? Or do you find further success with an exchange of messages between the agent and the user where you're putting your few shot prompts-
- 52:29
Yeah
- 52:30
... in there?
- 52:31
Really good question, and something that I would dearly love to know the answer to, but I don't. The, the question is ... Did ... I, I, I don't need to repeat the questions.
- 52:39
The ... I think people can hear them. Uh, but I'll, I'll repeat anyway. So do we wanna just put all the examples in one big giant examples block like this, or do we wanna structure the examples as a dialogue where the human says something and then the assistant says something back, and we're literally, like, putting a large
- 52:55
number of messages into the messages list? I typically do it this way with a big examples block, but it's mostly because it's less work for me, and I don't have any evidence that this works either better or worse.
- 53:08
I did do some testing of this at one point on a few data sets, and I found that it didn't make much of a difference for my particular case.
- 53:16
But there's a lot of, like, little particulars that went into my testing that make me not very confident in the, in the result that I got. So sorry for a bit of an unsatisfying answer here.
- 53:25
I'll just say I don't think it ... I- if it is wrong to do one giant examples block, I don't think it's, like, very wrong.
- 53:32
Do you, do you use anti examples too? So like in here, would you give it a thing and say like, "This would be bad because this is cringe"?
- 53:40
Yes.
- 53:41
Yeah.
- 53:41
Yeah. I think that is good. S- I think it's good to include negative examples, particularly around, like, the cringe thing where, where Claude might mess up. Uh, I, I think just negative examples on their own don't usually get you there.
- 53:56
You wanna have some positive e- examples too, but I think it's great to have especially, like, contrasting pairs. So like, here's a document. Like, here's a cringe tweet about this document.
- 54:06
Here's a excellent tweet about, about the same document. Uh, and, like, set those up side by side. I think that's pretty, pretty powerful, and I do that, and I think it, it helps Claude.
- 54:16
And then if, if you also include, like, the reasoning for it, right, so, like, if it was a cringe tweet, it has, like, a little reasoning of, like, why, why.
- 54:24
Um, do you also ... Do you trust that reasoning for the model? So, like, if you ask it, like, "Hey, give me ... " Like, "What were you thinking when you were writing this tweet?"
- 54:33
And then write me this tw-
- 54:34
Yeah
- 54:34
... this tweet. When you're reading through your examples to choose the best ones, how much do you trust that reasoning-
- 54:42
Yeah
- 54:42
... and how much do you rely on that versus just, like, I just care about, like, the input output?
- 54:46
I don't trust the reasoning very much, especially if it's after something the model already said.
- 54:50
No.
- 54:50
Then I, like, really don't trust it.
- 54:52
Yeah.
- 54:52
But I mean, humans are not very good at explaining why we do the things that we do. We're really good at rationalizing and coming up with, like, fake reasons.
- 55:00
But a lot of times we don't even know why we do the things that we did, let alone be able to coherently explain them to someone else. So I-
- 55:10
The- the- there's- there's- there's a subtlety here. So something that does work pretty well is having the model think about its reasoning in advance and, like, go through different l- reasons or rationales for why it might choose one option or the other or think about what sort of things might go into a good response.
- 55:25
So if I had the model do some thinking in advance before it gave the response, then I might just trust or assume that the response would be better.
- 55:36
Having a bunch of explanation for why it did the thing after it, probably I- I- I would not trust that. Sorry, you had a question for a while.
- 55:44
Do you, uh, do you give your own reasoning that explains the examples? And if so, how do you make sure that the model doesn't give reasoning itself?
- 55:56
Do I give reasoning to explain the examples? Yes. I do a lot of giving- giving reasoning to explain the examples. So for instance, just in this case, uh, one thing that we could do here is, like, um, we could add something like, uh
- 56:14
... I- I was gonna say tweet planning, but maybe it's, like, key points of document.
- 56:20
And then we, here, we have some key points, like the document, uh,
- 56:26
presents the launch of ...
- 56:36
So you would have this after the example. So if you have 10 examples-
- 56:39
No, this is before the examples. Or, sorry, so it's pa- this is part of the example right here. So I- I- I ... In- in this particular example, in this example block, I gave it a document.
- 56:49
Now I'm doing this- this key points business.
- 56:52
Got it.
- 56:52
Um, and then I would have these tweets. Now, this key points could be something that I wrote myself or it could be something that Claude wrote and then, uh, I'm- I'm, uh, editing it.
- 57:06
Or if Claude did a perfect job, I, maybe I could just include the thing that Claude wrote. But now in order to get this, get Claude to do this, uh, we would also say something like, um, "Return in this format, uh, key points,
- 57:19
a list of the key points from the document."
- 57:26
So this is like a lightweight chain of thought where we're having the model do some, uh, thinking in advance, and we also gave it examples of it doing the thinking in advance, uh, like this.
- 57:36
Hm.
- 57:37
Uh, yeah.
- 57:44
Um, yeah so let's imagine we, like, really want to give examples like this, but we have a problem, which is that our documents are, like, super long and I'm greedy and want to save on input tokens.
- 57:54
Yeah.
- 57:54
Would you err on the side of doing, like, one document but a really good example or doing, like, truncated versions of more documents?
- 58:04
I would ... That's a good question. I would err on the side of one extremely good example and not truncated versions of more documents. But I would also want to look at the outputs and test that assumption because it's possible that with only one example, Claude would fixate on aspects of the exact document that it uploaded and
- 58:23
start trying to transfer them to your document.
- 58:27
Right.
- 58:27
So I think it's- it's one of those, it would be case by case, but-
- 58:30
Right
- 58:31
... I wo- I would st- I would want to start with, like, having one extremely good example. Generally, I think that, like, less but, like, higher quality is-
- 58:38
Mm-hmm
- 58:38
... a better way to go than, like, more and- and lower quality.
- 58:41
Cool. Thank you.
- 58:43
Okay, we have a lot of prompts here. Let's go to the ...
- 58:47
Okay, this is good. I was hoping we would get some, like, persona ones here.
- 58:59
Okay. So this is, this looks like something where we're, we're trying to get Claude to, to role play in these different protocols.
- 59:11
So let's try this out and let's see how it works. So this looks something where we're gonna have like a ... This looks like it's, like, a meant to be a multi-turn prompt, right?
- 59:19
So this is, like, a conversation. You are talking to assistant.
- 59:27
It said execute greater than p assist. Where's that?
- 59:32
So basically at the top you see three roles being defined, and then you can decide who you want to talk to. Uh, uh, oh, thanks so much. So you s- see three roles at the top?
- 59:45
Yeah.
- 59:45
And then if you do that p assist you see down there that's highlighted in, in yellow-
- 59:49
Yeah
- 59:49
... you can just do, um, that little arrow p and then-
- 59:53
Yeah
- 59:54
... you can pick a different persona.
- 59:55
Yeah.
- 59:55
And then you can have them talk between themselves or you can just switch be- we use these for designers i- in, in our shop-
- 1:00:03
Mm-hmm
- 1:00:03
... to do synthetic interviews to synthetic users basically.
- 1:00:07
Gotcha.
- 1:00:07
It allows us to switch back and forth.
- 1:00:09
And then, uh, what, what issues or, uh, troubles have you been having with this?
- 1:00:14
I'm guessing you have seen a lot of role playing prompts out there, so I was just wondering if you see anything that's per- perhaps not as optimized as it could be or any other best practices, practices for role playing, particularly with multiple synthetic personas within the same session.
- 1:00:30
Yeah. Okay. For single personas there's, there's one answer that I would give. This multiple personas thing actually I haven't worked that much with, but here's, like, off the top of my head, here is probably how I would think about it.
- 1:00:44
I would give all the personas to ... I would write a separate prompt for each persona, and then I would have the user's command trigger some coding logic where it would decide which bot to send the, the, which, which prompt to send that reply to.
- 1:01:01
So this is getting back to the thing that I said before about, like, don't do it in a prompt if you don't have to. Like, this, I mean, this prompt, like, is, like, there's a, a lot of thought that went into it, which is, probably makes it work a lot better than it would have if you hadn't
- 1:01:12
put as much effort into it. But I think it's gonna be easier if you just dynamically route- The query based on, uh, what the, the user said. Does that make sense?
- 1:01:22
It does.
- 1:01:23
Okay.
- 1:01:24
You're talking about, like, if we were to use the API instead of just the chat-
- 1:01:28
Yeah
- 1:01:29
... or construct something like this, right?
- 1:01:30
Yeah, exactly. But you're doing this just in the chat.
- 1:01:33
This was just in the chat, but I, I appreciate, I definitely appreciate the, the note there. So maybe related to that, one of the other things is how much have you dealt with having a second thread with the API that acts as maybe the entity that's capturing inputs from multiple ones into a single thread?
- 1:01:54
You know what I mean? Like, let's say that I build an app and I have the user interact with these different synthetic personas, but then I have a second interaction with the API that's tying these things together into a cohesive whole.
- 1:02:08
Uh, I don't know if you guys have explored some of that. I'm, I'd be curious.
- 1:02:14
Yeah. I don't, I don't, I don't have a great answer for that one, sorry.
- 1:02:18
Mm-hmm.
- 1:02:18
I do wanna kind of test this prompt out though, just to kind of see how it goes. So, um, maybe here I would say,
- 1:02:26
uh, I can just say something like ... So how, how would I switch it? I could-
- 1:02:29
So do the, the right arrow P.
- 1:02:33
Yeah, right arrow P.
- 1:02:34
And then type Sam, and then say, "Hey ..." Yeah.
- 1:02:40
You're talking-
- 1:02:40
Okay, so now I could-
- 1:02:40
Then you could say, "Hey, how do you ... What's your process to look for the right to ... for the best, uh, medication pricing whenever you get sick?" Or something like that.
- 1:02:51
And then here in this particular case, if you switch to Joe, Joe is optimized more for, uh-
- 1:02:57
Okay
- 1:02:57
... convenience versus cost saving, so you have two different types of users, and we can learn from.
- 1:03:03
Yeah, okay, so Claude did the thing here that I wanna show you all, like, how to get rid of. Uh, where-
- 1:03:10
Oh, yes. Yes
- 1:03:11
... as Sam, it's like, that's not something that Sam would say, right?
- 1:03:16
Yeah. [laughs]
- 1:03:17
Uh, so I don't know for sure this is gonna work. I feel like a magician that's about to do a trick, but, like, I haven't practiced it. But generally, something that is, is pretty useful, uh, here is to, we could say, uh, prepend each response with the name of the current persona in brackets.
- 1:03:49
So one thing I'm gonna do here is
- 1:03:53
I'm gonna change this, like, multi-shot a little bit also because
- 1:03:58
if Claude sees itself not doing the thing that I told it to do ... Actually, let's, let's just redo the whole, the whole conversation, or we can take out, uh, this.
- 1:04:06
Um, so let's, let's just, like, run that back.
- 1:04:10
You are talking to assistant.
- 1:04:11
Nice.
- 1:04:12
And now we could say ... And now we could say the same thing, like, "What's your process for finding, uh, best prices for medication?"
- 1:04:31
Oh. [clicks tongue] Okay, so I guess we need to do this. We need to, like, change in a separate call.
- 1:04:38
Yeah.
- 1:04:40
Okay, great. Now it's gonna work. Totally. It's gonna totally work.
- 1:04:49
Okay, it's a little bit better, right? It didn't say as Sam. This is, like, something that human might maybe say. Like, as, as someone who's ... [laughs] Okay. [laughs]
- 1:04:59
I don't know. Um, it's better than it was before, right? Uh, maybe we could say something like-
- 1:05:07
Can you just preview that?
- 1:05:10
You don't need to, uh, say too much about your persona in your responses. Just stay in character.
- 1:05:22
Hey, quick question. What are your thoughts on using things in the negative sense versus-
- 1:05:26
Yo, check it out. It worked a lot better. Sorry to interrupt.
- 1:05:28
Oh, yeah. [laughs] Very nice. Very nice. [laughs]
- 1:05:32
Um, yeah, so-
- 1:05:36
Yeah, what are your thoughts on using, like, negative stuff, like you don't, versus the positive sense?
- 1:05:41
Yeah, I think positive is, like, a little bit better. In this case, I don't really have a good answer for why I phrased this negatively. Uh, I guess I did a combination.
- 1:05:50
I was like, you don't need to say too much, just stay in character. Um, I guess I think it's better to use, like, a light touch, like, if you're doing negative prompting.
- 1:05:59
I think there's, like, a, a little bit of a thing going on with reverse psychology where if you tell the model, like, "Don't talk about elephants. Don't ... Definitely no elephants.
- 1:06:07
Definitely don't say anything about elephant," it might make it more likely to talk about an elephant. So if you do use negative prompting, I think it's better to have, like, a, a light touch where you just kinda say it once, but, like, don't dwell on it too much.
- 1:06:18
Um, also something similar with parenting. It's like if you don't want your kid to eat prunes, you're just like, "Oh, we're not having prunes today," and then you just change the subject.
- 1:06:25
But if you really, like, emphasize that there are, like, no prunes to be had, then you might get more, uh, pushback.
- 1:06:32
Hi there. Um, I noticed you're not using the system prompt much. Like, is there a reason for that, or what do you think the biggest value items for a system prompt are?
- 1:06:40
Yeah, system prompt. Personally, the only thing that I ever put in the system prompt is a role. So I might say, like, you are this, you are that. I think generally Claude follows instructions a little bit better if they're in the, the human prompt and not in, in the system prompt.
- 1:06:55
Uh, the exception is things like tool use where maybe there's been some explicit fine-tuning on, like, certain system prompts, uh, specifically. For, like, general prompts like the ones we've been going over here though, I don't really think you need to use the system prompt very much.
- 1:07:12
Yeah.
- 1:07:13
Yeah. One thing we've found when using the user prompt ... Thank you. One thing we've found when using the user prompt, I guess sometimes is it makes it more prone to hallucinations because it thinks the user is saying it, and so we-
- 1:07:27
Migrated things to the system prompt more? I don't know if you have any experience with that
- 1:07:31
Yeah, yeah. I- I've actually heard that before.
- 1:07:34
So it's possible I'm missing something. I've heard this from enough people that I could just be wrong, so I'm u- unusually likely to be wrong when, when I say this.
- 1:07:42
I think that if you just put a bunch of context and you're like, "Here's the message from the user," open user bracket, and then put the message, and then close the user bracket, uh, it will, it will work and you won't have that issue anymore.
- 1:07:54
That said, like, uh, I don't know, may- maybe it does, uh, f- fall over sometimes, but that would be my default is just to, like, specify even more clearly in the ...
- 1:08:04
If you're having this issue, be like, "Here's the message from the user. Here's the stuff that I want you to do," and I think it's le- it probably won't get confused by that.
- 1:08:13
Uh, I have a question about the, uh, counter examples. So before, in order to get it to say not cringey, cringey things, you were saying provide it with a counter example.
- 1:08:23
So but here in the case of where you're doing this, uh, character bot, you haven't provided it any counter examples.
- 1:08:31
Yeah.
- 1:08:31
So th- this is sort of like a generic question. So if the model is trained on preference optimization with, uh, examples and counter examples, do you get a better result in the prompting?
- 1:08:44
Well, uh, I don't, I don't know that the details of the RLHF have that much bearing because
- 1:08:52
I, I think when the model's trained it, it doesn't usually see those both in the same, like, window. It, it's more that it, it, it's like some stuff that happens with, like, the, the RL algorithms.
- 1:09:02
So I don't think that's necessarily the right way to think of it. With counter examples, I don't feel that I have to include them in every prompt. It's just a tool that I have in, in my toolbox that I'd use sometimes.
- 1:09:13
I- in regards to, like, negative prompting, uh ... I'm over here. Hi.
- 1:09:17
Oh.
- 1:09:17
Uh, do you think that it would be better to do negative prompting using control vectors like what you, uh, what you talked about in your, uh, scaling mono-semanticity paper?
- 1:09:27
Uh-
- 1:09:28
Yeah
- 1:09:28
... and maybe, like, having, like, a negative version of the vector as your kind of negative prompt instead of mentioning it in the prompt outright?
- 1:09:35
Yeah. Uh, steering is, is, is still, like, super new. We don't know how well it works relative to prompting.
- 1:09:43
Mm-hmm.
- 1:09:43
I'm like a, a, you know, diehard prompter till the end, so. [laughs] Uh, or I guess I, I've played around with it a little bit. I, I haven't, I haven't found it to work as well as, as prompting in my experience so far.
- 1:09:53
Mm-hmm.
- 1:09:54
That said, there's, like, a lot of research improvements that I won't get into in too much detail, but there's a lot of stuff that could make it work better than ...
- 1:10:01
So, like, right now it's, like, finding these, these features and then you're steering according, uh, to the features, which are sort of like these abstractions on top of the underlying vector space.
- 1:10:10
Yep.
- 1:10:11
There's other possibilities for how you could steer and there's, like, academic papers that you can read where you're steering according to just, like, the, the differences in the activations versus, like, trying to pull it out to this feature first.
- 1:10:24
So maybe that would work a, a bit better, like the control vectors thing.
- 1:10:27
Yep.
- 1:10:27
I haven't played with it enough to know for sure. Uh, but I'm ... I, I think there's definitely, like, something along that line, those lines will work eventually. I can't say in the long term if it'll work better or, or worse than prompting.
- 1:10:38
Right now I still think prompting works, like, a lot better.
- 1:10:40
Mm-hmm. I mean, from my experience with, like, smaller models and trying to work with control vectors, I've seen that it's better when it comes to style-
- 1:10:46
Mm-hmm
- 1:10:47
... uh, than it is for, like, actual deterministic prompting.
- 1:10:51
Yeah, pretty interesting.
- 1:10:52
Yeah.
- 1:10:52
Sometimes I feel like stuff from smaller models transfers. Sometimes it doesn't transfer.
- 1:10:56
It's ki-
- 1:10:56
I don't have a great intuition for what does and doesn't transfer between small and large models, but yeah, it's-
- 1:11:00
All right
- 1:11:01
... good points.
- 1:11:01
Thank you.
- 1:11:03
Okay. I think we've, uh, gone over this role play stuff enough. Let's go to the next one. I'm gonna upload a few screenshots of my dating profile. Okay, this is our first image one.
- 1:11:15
Are there any screenshots? No screenshots. Uh, okay. Actually, somebody responded in the very first, uh, in reply, so since we're doing images, uh, maybe I'll start there. Um ...
- 1:11:29
If you scroll down, it's that, that was me.
- 1:11:32
That was you. Uh, let me try and find my message here. All r- You said it's at the bottom of the ...
- 1:11:38
Yeah. In that channel I reposted it somewhere.
- 1:11:49
Riveting to watch me scroll through this channel, I'm sure. [laughs]
- 1:12:00
Oh, there's a lot of messages here. Okay, here we go.
- 1:12:08
Okay, so let's ... Um, can I copy the image?
- 1:12:15
Copy image. Okay, cool. I actually don't know if I can paste it into the console, so I might fall back on using Claude.ai for this.
- 1:12:25
Paste images is not enabled right now. Okay, let's try Claude.ai then.
- 1:12:35
Um, so then this ... The question wa- or the, the, the prompt here was,
- 1:12:42
"High performing validated AI model." Sorry, I, like, lost all your formatting here.
- 1:12:58
Okay, so in this image here, if we zoom in, it's supposed to get Mattie White
- 1:13:04
and 86... Hmm, I'm having a ha- hard time reading this.
- 1:13:10
Uh, 8687. That's the hard one that it messes up on quite frequently.
- 1:13:13
8687 and then 8687 down here. And then in this one it's just 867.
- 1:13:18
Yeah, they ... That's a typo-
- 1:13:19
That's a typo
- 1:13:20
... by the human. And so I'm hoping it can correct for that. I'm just trying to pull out-
- 1:13:24
Yeah
- 1:13:24
... kind of the average data-
- 1:13:26
Yeah
- 1:13:26
... from all three combined.
- 1:13:30
Okay. It's, and it said ... It looks like it said middle mark, so it's, it's misreading the ... Okay. So I don't know too many good tips for image is, but I'll, I'll tell you what I have.
- 1:13:42
So one of the things that you said ... Oops, sorry. Um,
- 1:13:47
one of the things that you said here was, uh, that it works better with zooming in and cropping the image. That's definitely, like, the easiest win that you can have, is just giving it, like, a higher quality image, taking out the unnecessary details.
- 1:14:04
That might be hard to do programmatically because it's like you don't know what the s- the, which details are necessary and unnecessary.
- 1:14:09
Mm-hmm.
- 1:14:09
But for the same reason that including extraneous information in text form, you probably won't get as good results. If you include extraneous information in, in image form, the results probably won't be as good, uh, either.
- 1:14:22
So the more that you can, like, narrow in on the exact information that you need, the better. Um, is the model down-sampling large images? I don't know/can't talk about that, but definitely having, like, higher quality, bigger images is, is better.
- 1:14:37
I, I did just read on your website that it says it dow- it down-samples to 1,000 pixels by 1,000 pixels.
- 1:14:43
Okay, great.
- 1:14:44
That can be found with Google. [laughs]
- 1:14:46
Okay. So then any general tips on how to discuss images with Sonnet? Um, my number one tip for images is to start by having the model describe everything it sees about the image.
- 1:14:59
So I don't know if that will work here. This, this example is, like, this one is hard enough for me to even read that I, like, I kind of doubt that Claude will do well on it regardless of, of what we say.
- 1:15:07
But we can, we can, uh, give it another shot.
- 1:15:10
One, one thing I've noticed when I attempted that where if I asked it to go, like, tube by tube in that image, if it, like, the first tube, if it came to the wrong conclusion, it would use that and come to that same wrong conclusion on multiple tubes, where, like, it, it made it so there was some
- 1:15:27
kind of, like, directionality in its thinking where if it got the answer wrong at first, it would project that onto the rest of the image it was analyzing.
- 1:15:33
Yes. I think that's definitely true. If, if the model starts off on the wrong path, it probably would just continue going on the wrong path. It's trying really hard to be self-consistent.
- 1:15:42
Mm-hmm.
- 1:15:42
And it's, it's ... And this is why, uh, uh, self-correction is such a big, like, frontier for LLMs in general right now.
- 1:15:53
But, but as of this month, why use Claude for this task? So I love to use Claude, but why for, for this specific image extract task?
- 1:16:03
Um, I found, like, text track and OCR models I've played with don't do as well with some of the handwriting. Um, like, if I zoom in on this image, it actually does perform pretty well, um, from some of these fairly messy handwriting that's even hard for a human.
- 1:16:17
Okay. Yeah, unfortunately, I don't know how to get better results out of this other than maybe by, by cropping it better and, and up-sampling.
- 1:16:24
Okay, cool. Thank you.
- 1:16:27
Oops. Um, here we are. So let's scroll up to where we were before.
- 1:16:51
Okay, yeah, still no screenshots there. How do I enable fragments?
- 1:16:57
Uh, what are fragments?
- 1:17:02
The
- 1:17:02
Is fragments like the pre-fill part? The pre-
- 1:17:05
No, the beta feature you guys have that-
- 1:17:07
Artifacts.
- 1:17:08
Artifacts. Oh, okay. Uh, it, it's just a setting. It's in the bottom left of the ... You have to en- enable it.
- 1:17:14
Find it online somewhere.
- 1:17:15
Yeah, you can find it online, or people will help you. Uh ...
- 1:17:18
So I ... Okay. Sorry. Can you just get my ...
- 1:17:22
Yeah. So I often dump full traceback errors directly into the prompt box. I often dump full traceback errors directly into the prompt box as API. It seems exceptional at not running into traceback loops.
- 1:17:33
I don't know, like, if that's intentional. Like, I ... Literally, I'll just take the entire traceback, zero context to Claude, and I'll just dump the entire thing in.
- 1:17:45
Yeah.
- 1:17:46
And then it'll, it'll, it'll give me the fix.
- 1:17:48
Okay, great.
- 1:17:49
So but, is it ... Can you, can you, can you elaborate on how that might have ... how this ... For example, like, this only really appeared with, like, very recently.
- 1:18:00
Like, you'd have to explain explicitly a lot for-
- 1:18:03
The models get better, man. They get better every, every iteration.
- 1:18:06
I understand, but this is, like, you know, this isn't prompt. Like, this is prompt engineering we're talking about, right?
- 1:18:10
Yeah.
- 1:18:10
So, uh, I'm just, I'm just wondering, like, is this, is this a form of prompt engineering, or is just the model being good?
- 1:18:18
Sounds more like the model being good if you're just dumping it in.
- 1:18:20
Okay.
- 1:18:20
All right, I'm gonna move to this, this prompt here.
- 1:18:22
Okay.
- 1:18:23
So, uh, the ... To the person who uploaded this, do you ... And, uh, generally, uh, to anyone who's uploaded, uh, more examples, if you could just, like, put some stuff in the thread, like write some stuff about what the issue is that you're having or, like, why it's not working, uh, that would be great.
- 1:18:43
Well, I'm gonna suggest ... This is actually a follow-up to, like, what I was doing with the translation. So basically what I'm trying to get it to do is to actually analyze the text.
- 1:18:54
So, like, in this case, you know, there's original English.
- 1:18:57
Yeah.
- 1:18:57
There's a bad Japanese translation. And I'm trying to get it to score between one and five how good the translation is. And so what I've been doing is adding a lot of stuff to try to get it to do sort of chain of thought to-
- 1:19:08
Yeah
- 1:19:08
... you know, see, like ... Because it'll notice errors, but, you know, it just generally does a very bad job at scoring.
- 1:19:14
Mm. Yeah, okay, so this is great. Uh, I'm really glad that you asked this, 'cause model grading is something that ... I mean, it, it would be incredibly useful if it worked, and right now it's in a place where it, like, sometimes kind of works and sometimes doesn't.
- 1:19:28
So let's, uh, paste this into the console
- 1:19:33
Okay, so we have some English text and we have some Jap-
- 1:19:37
And it's on the low
- 1:19:37
Yep, yep, yep. Then we got the Japanese text.
- 1:19:54
So is this a good translation or a bad translation?
- 1:19:57
Terrible translation.
- 1:19:57
Terrible translation, okay. Now, Claude's actually supposed to be good at Japanese too, so...
- 1:20:05
No, it's much better, yeah. This is, uh-
- 1:20:05
Oh, this is somebody else. But I'm saying if Claude is good at Japanese, it should be good at, like, judging other people's Japanese, in theory.
- 1:20:12
In theory.
- 1:20:12
In theory. Okay, so now that we've got the answer here, so,
- 1:20:18
um, I guess it's stalled out here for whatever reason. So what are we doing in this prompt? We're scoring betw- between one and five as below.
- 1:20:27
One is many grammatical errors the native would never make, contains multiple grammatical errors, an average quality w- translation with some errors. Okay, that, that looks pretty good.
- 1:20:38
Look for specific clues or indicators. I don't know why it, uh ... I think that's just a, a bug. I think it'll conclude here.
- 1:20:55
Okay, so it gave it a three, but we want it to give a one.
- 1:20:59
Totally, yeah.
- 1:21:00
Okay. Now, have you found that it's generally too forgiving or too strict, or that it's just all over the place?
- 1:21:09
Well, it's all over the place. Also, it seems to, um, confuse sometimes content, uh, versus, uh, you know, so, like, for example, this is from, I think, the HLRF, uh, you know, set, so, you know, the content is fine, but so it might not actually rate it low even though it's a terrible translation because it thinks the
- 1:21:27
content is okay, even though if you asked it even to list all the errors-
- 1:21:32
Gotcha
- 1:21:32
... there are a dozen errors in that, you know, single piece of text-
- 1:21:35
Gotcha
- 1:21:35
... for the translation.
- 1:21:36
Yeah, yeah.
- 1:21:37
Uh, so that's why I'm trying to also see if, you know, is there anywhere to, like, get to separate out, um, the grammatical-
- 1:21:44
Yeah
- 1:21:44
... or the actual translation errors versus-
- 1:21:46
Yeah. Okay, a few thoughts. So generally, this is, like, a thing where if you a- ask the model if some text is, like, good or bad, it sort of ...
- 1:21:56
If the text is, like, about a nice subject, it's more likely to say that it's good in all ways. Like, it's a good translation, like, it's well-written, it is ...
- 1:22:03
It flows very logically. And if it's about, like, a negative subject, it's more likely to, like, criticize it and say that it doesn't flow well.
- 1:22:11
Yep.
- 1:22:12
I think you can get at, uh, those issues by typing language about it in the prompt. So for instance, here you might say something like, um ...
- 1:22:25
Is one or five good? You know, you don't specify that.
- 1:22:28
Hmm?
- 1:22:29
What ... Is one or five the best score? Uh, one is the worst and five should be the best.
- 1:22:34
That's sort of, like, implicit in, uh, this rubric up here.
- 1:22:42
But it might be good to say it anyways.
- 1:22:44
The problem with that is if we're interrupting Claude, how do you get around the fact that you-
- 1:22:51
May not have a Japanese tokenizer?
- 1:22:55
Yeah.
- 1:22:56
Do you have a Japanese tokenizer in Claude?
- 1:22:58
Claude, uh, I mean, the API will, will tokenize anything.
- 1:23:01
But how is it trained?
- 1:23:04
How is it trained? Um-
- 1:23:06
Yeah. Pre-trained
- 1:23:06
... o- off, off topic for ... [laughs] And also, I don't know.
- 1:23:09
I don't think ... I actually don't think there is a tokenizer for Claude. Like-
- 1:23:15
Oh, so-
- 1:23:16
I, I mean, unless you say otherwise, there's, there's not
- 1:23:18
I mean, if you upload, if you upload some text, it will be tokenized.
- 1:23:22
But pre-t- it's not pre-trained, so you're not gonna get a really good answer for this. Or Japanese text. I've-
- 1:23:28
Uh-
- 1:23:28
... I've tested this.
- 1:23:29
Okay.
- 1:23:29
Claude, Claude speaks the best Japanese of any model available, actually.
- 1:23:34
We ... Yeah, we don't need to debate, like, Claude's Japanese skill. Let's just, like- [laughing]
- 1:23:40
Uh ...
- 1:23:40
Uh, but the tokenizer isn't available, and so that it would be interesting.
- 1:23:45
Okay.
- 1:23:45
There's, there's a lot of work to be done. That's cool. Uh, it's not like slamming Claude, but-
- 1:23:51
Yeah.
- 1:23:52
Um, I have a question about programming.
- 1:23:54
Sorry, can we actually, uh, c- cut the questions off while I type this-
- 1:23:57
So sorry
- 1:23:58
... this prompt here? Um, okay. So it's extra important to ... So what we're trying to do here is get it to distinguish between the, the, like, ethical nature of the text and the quality of the translation, so ...
- 1:24:11
Is it useful to tell it to be, like, extra critical? Say, like, you're grading a, uh, you know, like, I don't know, like, graduate course, you know, level-
- 1:24:20
Sorry, I'm, I'm a little bit all over the place. I need to, like, type this out before I can clear the queue and, like, respond to other questions here.
- 1:24:26
So it's important to distinguish between, um, the ...
- 1:24:42
Uh, what's a good word here? Like, um,
- 1:24:47
uh, risque topics or, um, R-rated ...
- 1:25:12
Yeah. So I don't know, something like this could help the model, uh, not pay so much attention.
- 1:25:20
The main thing that I would wanna do for this prompt is just add a bunch of examples. So for each category, I would add at least one example of that category.
- 1:25:31
So, like, I'd have, like, a really bad translation, and I'd say why it's a one, and you can say that in your own words, and I'd have an example of Like a, a two-level translation, a three-level translation, and so on.
- 1:25:43
Okay.
- 1:25:44
And in each case, before you get to the answer, you would have the explanation for why it's good or bad. I can't tape all that here, a- among other reasons, 'cause I don't speak Japanese.
- 1:25:54
Mm-hmm.
- 1:25:55
But I think that's the most valuable thing that, that you could do here. O- otherwise, I mean, like, the formatting looks really good. The fact that you're doing the chain of thought in advance looks good.
- 1:26:05
I, I think m- mm... Yeah, I think maybe this specific clues or indicators, I think you could go into a little bit more detail here about what is-
- 1:26:20
Okay
- 1:26:20
... how these things contribute to the overall-
- 1:26:21
Basically just end shot every single, uh, example or, you know, grade-
- 1:26:25
Yeah
- 1:26:25
... with an example, basically.
- 1:26:26
Yeah. I mean, it's, it's tedious to write out all these examples, but A, Claude can help, and then you have the problem of just editing Claude's response, uh, versus, like, writing it all yourself, and B, it really does lead to better performance versus, like, almost anything else that, that you could do.
- 1:26:41
Okay, thanks.
- 1:26:41
Do you think it's better to use a number scale or ask it to say, like, good, bad, you know?
- 1:26:47
Yeah. In terms of the scale on these, on these rubrics, I think either a number scale or good, bad is fine. The thing I'd be careful about with the number scale is that I don't think it's very well calibrated, so if you're telling it, like, "Choose a number from one through 100," it's not gonna be, like, a
- 1:27:05
unbiased estimator necessarily. So I'd probably just limit the granularity to maybe five different classes.
- 1:27:13
Yeah?
- 1:27:14
Uh, one thing we haven't really talked about at all here, but going back to, like, your questions around the API is, like, this is a case where, uh, you might be able to utilize, like, log probs, and I'm wondering if that's something you ever use in any of your work or other prompts.
- 1:27:29
Yeah, I think ... I, I agree this is, could be a case where log probs would be useful if you could get, like, the probability of each, each grade.
- 1:27:38
Yeah.
- 1:27:39
The thing-- So here's the thing with log probs, is you really want the chain of thought beforehand.
- 1:27:45
Mm-hmm.
- 1:27:45
And the chain of thought, I think, is gonna get you more of a win than using the log probs. But log probs, th- there's, there's no ... In, in, in any model, I don't think there's a way where you can just say, "Sample all the way through, and then output ...
- 1:27:59
After you output this, like, closed chain of thought tag, then give me the log probs of whatever comes next."
- 1:28:04
Mm-hmm.
- 1:28:05
So if you are gonna use the log probs, then you're looking at this, like, multi-turn setup where you first sample all the, the chain of thought or sample all the, like, pre-cogitation that it, that it wants to do.
- 1:28:15
Mm-hmm.
- 1:28:16
And then cut it off there, and then re-upload that with the chain of thought as a prefill message.
- 1:28:22
Mm.
- 1:28:22
And then you could get the log probs. But you ... For that you need a model that both has, uh, the, the prefill capacity and has log prob capacity.
- 1:28:31
I'm not sure of, of what model has both of those characteristics right now.
- 1:28:34
Yeah. Walk me through why it wouldn't just be sufficient to, like, in this case, just ask for ... I'm asking for a score one to five, only return that, but then, like, look at the log probs of what it returns in that case.
- 1:28:46
Yeah. So the, the, the ... What you're losing there is whatever intelligence boost you got from having the model do the chain of thought. And my sense is that chain of thought plus have the model say either one, two, three, four, or five is gonna be more accurate than, like, the more ...
- 1:29:00
the additional nuance that you'd get by having it give you the log probs because it's actually doing a lot of its thinking in that chain of thought. You're, like, leveraging more computation.
- 1:29:09
You're getting more forward passes for all the same reason that chain of thought is, like, usually a good idea. It's a good idea to-
- 1:29:14
I see. Are you talking about, like, chain of thought and, like, it's actually out loud writing a chain of thought before the answer?
- 1:29:20
Exactly.
- 1:29:20
Okay, okay.
- 1:29:21
Exactly.
- 1:29:21
Sure, I got you. Cool.
- 1:29:22
Which is what we see in this prompt right here, right? Like, we have this-
- 1:29:24
Yeah, yeah
- 1:29:24
... analysis section. So if we cut out this whole analysis section, we're really tanking the number of forward passes that the model can do-
- 1:29:31
Yeah
- 1:29:31
... before it gave you the answer.
- 1:29:32
Okay, cool. That's good to know. Thank you.
- 1:29:36
What's that?
- 1:29:37
Three, two, one. You have a couple more and then-
- 1:29:40
Okay. Um, I'm being told that I should answer a couple more questions and then get off stage. I was like- [laughs]
- 1:29:47
Before I came here, I was honestly really worried that, like, no one would have questions and I'd be supplying my own, but you all have had amazing questions so far and amazing examples, so I really appreciate that.
- 1:29:58
It's made this go on well. Sorry, I'm supposed to say that at the end of this, but I'm giving a pre, pre thank you. Now we can do the encore.
- 1:30:04
Uh, yeah?
- 1:30:04
I just wanted to add that, um, having a numbered list may give more, um, weight to, like, number one, two, three, four, five versus just having an unstructured list.
- 1:30:13
Um, so it may give more weight into their score, uh, for the output if you just change it from, like, one, two, three, four, five to, like, little dashes, uh, for criteria.
- 1:30:24
Um, just as my experience of what I've been seeing.
- 1:30:26
Okay, cool.
- 1:30:27
Yeah, so just FYI. I have replied back with a prompt, uh, that was fixed or that, like, in my own improvement of what I think is a better prompt for this.
- 1:30:35
Yeah.
- 1:30:36
Okay, awesome. Um, Nisha, should I do another prompt or should I just a- answer a couple questions and then head out?
- 1:30:43
Up to you. We can do another prompt if you want to.
- 1:30:45
All right, let's do one last prompt. What's ... Which one should we choose?
- 1:30:58
Okay, let's do this one, good old mitigating hallucinations, 'cause we haven't really done that.
- 1:31:05
Okay. "Please provide a summary of the text provided as input." Okay, first thing I'll do is just move these instructions down.
- 1:31:23
Now, Matt, did you have an example of the document, a document where it hallucinates with this prompt?
- 1:31:28
Yeah. Um, no, but I can send you one really quick.
- 1:31:31
Yeah, can you just put that in the thread?
- 1:31:42
Okay, your summary should be concise while maintaining all important information that would assist in helping someone understand the content. If it mentions any dates.
- 1:31:51
Don't start or end with anything like, "I've summary, generated a summary for you. Here is the summary you asked for." Yeah, so this one we can, uh, fix with a pre-fill.
- 1:32:20
Okay, uh, or we could do something like this.
- 1:32:30
Um, now how about the hallucination part? The, the best trick that I know for getting around hallucinations in a case like this is to have the model extract relevant quotes first.
- 1:32:41
So what I would say do here is I would say something like...
- 1:33:08
And now of course in this pre-fill, since we're having, uh, relevant quotes here, we wouldn't wanna start with summary. That would just be confusing/wrong. So we could say, "Here is the..."
- 1:33:26
Something like this. Okay, did you get the, the doc yet?
- 1:33:35
Uh, one second. I'm downloading the data now. Okay. And then of course I'd put the document here.
- 1:33:46
Yep. Okay. Uh, I think I should get off stage. Uh, so yeah, let me just call it here, and Matt, we can, we can talk, uh, after.
- 1:34:02
Um, yeah, once again, I really appreciate you all coming out. It's been amazing to have such, like, a, a great audience engaged. Uh, I've had fun. I, I learned some things.
- 1:34:12
Uh, I hope you all did too. I'm planning to stick around, uh, this event, uh, for the next, for the rest of the afternoon, so I don't know exactly where I'll be, but maybe just DM me if you wanna come find me and chat.
- 1:34:24
I'm always happy to talk prompt engineering. It's, like, my truest passion at this point in the world, so, like, find me, hit me up. We'll, we'll talk. Um, and yeah.
- 1:34:35
This has been great. Thank you so much. [outro music]