Building with Anthropic's Claude - The Prompt Doctor is In

Talk transcript

1. 0:00

   [upbeat music] All right, good afternoon, everybody.

2. 0:17

   Good afternoon.

3. 0:17

   Thank you all so much for, for joining us. We have the, uh, envio- enviable position of being after lunch. I'm seeing some cookies on the table still, but thankfully you're not here to listen to me.

4. 0:27

   You're gonna be riveted by the prompt doctor. He's gonna come up in a second. So I'm expecting no sleeping on the table, but, but you never know. Um, excited to be here.

5. 0:35

   I'm Jamie Neuwirth. I lead our startup team at Anthropic, and I just wanted to say a couple of quick things before we got going here with, again, the reason you're here, the prompt doctor.

6. 0:44

   Uh, we've had a lot of really exciting releases just in the last couple of days, some in the last couple of hours, and wanted to just put these up there to highlight some of the cool things that we're doing, but also share how a lot of folks, not only in this room, some of your peers, maybe folks

7. 1:00

   back at the office, can work with Anthropic on not only some of the prompting, of course, works, uh, work we're gonna do here, but just helping you grow your business and the really cool things you guys are building with Claude, with LLMs on top of AI.

8. 1:15

   My team is here specifically to help your company grow and scale the business, whether that's getting access to higher rate limits, getting access to folks like Zack, who's gonna be up here in a moment, learning more about what we're doing from an early access perspective.

9. 1:29

   We wanna work with you and empower the next wave of really exciting AI companies built on top of Claude. And we're helping from a product perspective with a couple of the releases you see here.

10. 1:39

    Has anyone tried 3.5 Sonnet yet?

11. 1:42

    Whoo.

12. 1:43

    Love it. Very cool. Really exci- yeah, thankfully Zack has tried it as well, so we're in a good place. [laughing] Uh, really excited about what we were able to release there as well.

13. 1:51

    Um, also just today, or excuse me, I guess Artifacts came out with that, with our Claude Teams and Claude AI plan. Artifacts is this really cool tool that some have been playing around with making video games with a couple lines of text that turns into code, but actually a lot of really cool business use cases from a

14. 2:07

    diagram perspective. I've seen a lot of cool pharma companies using this and thinking out, "How can I put what's in my head on a gene sequencing kind of discussion onto something like an artifact, share that with my peers?"

15. 2:19

    All the way to prompting cookbooks, helping, again, folks like yourself. I would imagine a lot of you hopefully will be featured on the website, you know, coming up with what you'd be able to prompt for these use cases in your companies.

16. 2:30

    So just some ways that, that we've n- uh, uh, you know, uh, things that we've come out with over the last couple of days, couple of hours when you think of Claude teams and projects, and then ways to connect.

17. 2:39

    Feel free to reach out at sales, uh, [REDACTED:email_address]. We have our head of DevRel, Alex, here as well. So we really love this community, love staying engaged. We're really excited about what we've released over the last couple of days to, to be able to help you, again, in what you guys are building.

18. 2:54

    And so I'm gonna leave all that to the prompt doctor as well to get that, uh, uh, to really make sure that we can help. And Zack, why don't you come on up here and, and see what we can do.

19. 3:03

    Thank you all so much for joining us today. We're really excited. [clapping]

20. 3:13

    Thanks, Jamie, for the intro, and thank you all for coming. This is really awesome. I had no idea this many people were gonna be here, so thanks. Uh, okay, so not gonna be much talk.

21. 3:23

    It's mostly just gonna be straight-up prompting from beginning to end. Did make a, a couple slides, so mostly about what to, what to bring. So I set up a, a Slack channel, um, in this, uh, the, the, the AI Engineer Slack.

22. 3:36

    It's called prompt-eng-live-workshop-anthropic. Um, so that's where you can upload prompts, and what we're gonna do on stage is, uh, I'm just gonna sort of look at them, uh, I'm gonna read them, we're gonna test them in the console, uh, the Anthropic console, uh, we're gonna see if we can get better results, and we'll just sort of like,

23. 3:56

    uh, try, try to learn as we go. Uh, so this is something that I do, like, internally in our, our team Slack, uh, quite a bit, but I've never done it in front of this many people and I, uh, this is-- it'll be exciting.

24. 4:07

    It'll be, it'll be fun. Um, might be some hiccups along the way, but hopefully you all have a good time too and, and maybe learn something. I know I'll definitely learn something.

25. 4:14

    Um, so what sh-- what kinds of things should you put in this, uh, Slack channel? So, uh, you can put a prompt template. So a prompt template is kind of like a prompt.

26. 4:24

    Uh, actually, I just realized I don't even need this, this mic. Um. [laughing]

27. 4:28

    Okay, so you put a prompt template, uh, which is like a, it's like a prompt but with spaces where the variables are gonna go and the variables, th- th- they're, they're gonna be denoted with these double brackets.

28. 4:39

    So in this case, it's like this, this document part. Um, if you don't have it in this format, that's fine. We can figure it out. This is just, like, the ideal.

29. 4:48

    Uh, so this is, like, the prompt template. This is, like, the kinda thing you'd put there. And then you can also have a couple examples of cases where it doesn't do what you want, and that will give us, um, the, uh, like, direction as far as, like, where we wanna go with it.

30. 5:02

    Um, I might also, like, uh, ask you questions out loud if I have questions about, like, what kind of output i- is, is good or not, or I might ask questions i- in Slack, uh, either way 'cause it's, it's easier.

31. 5:14

    Uh, we'll have to kind of, uh, figure that, that out as we go. Um, okay, so that being said, uh, we're gonna use the console for, for iterating mostly, although I might use a couple other tools like, um, Claude for Sheets, which is like a spreadsheet where you can, uh, call Claude.

32. 5:30

    Um, okay, uh, so yeah, let's see what we've got in the Slack already.

33. 5:36

    Okay, we have something here. Thank you, Gordie. So you're an expert patient. So let's put this into the console, and then let's take a look.

34. 5:52

    Okay, and I'm just gonna go through as many of these as, as we can get through in the, in the session. And, uh, yeah, this is pretty much what it's gonna be.

35. 6:00

    So first of all, we can probably capitalize all-

36. 6:07

    The, uh, sentence is?

37. 6:10

    Does that matter?

38. 6:12

    Does that matter?

39. 6:13

    Yeah. Sorry. Hi, I'm Gordie.

40. 6:14

    Thank you. Yeah, yeah, yeah, yeah, yeah. [laughs] Perfect.

41. 6:17

    Yeah. So does it matter having capitalization?

42. 6:21

    I, I think so.

43. 6:24

    It's okay. [laughs]

44. 6:26

    A, a lot of things like prompt engineering is, like ... It's, it's very new, right? So, like, we don't know for sure. Somebody out there might have done a study where they, like, conclusively show that using capital letters and, like, using, like, better grammar, fixing grammar mistakes help.

45. 6:42

    I have, like, anecdotally found this in a few cases. I also have read some, like, quantitative stuff showing that, like, typos do hurt performance. But I'm also just, like, pretty obsessive about this stuff, so I just fix it and I think it ...

46. 6:54

    Like, it definitely doesn't hurt. Um, okay.

47. 6:59

    Can you zoom in?

48. 7:00

    Can I zoom in? Great question. Is th- is that any better? No.

49. 7:05

    Little more. Little more.

50. 7:06

    Little more. Okay, is that any better?

51. 7:09

    Yes.

52. 7:12

    Okay. Um, so first thing, let's put information in XML tags. So we can go like this.

53. 7:20

    Why XML? Why not, like, Markdown?

54. 7:21

    Why XML? Why not Markdown? Uh, another great question. So Claude was trained with a lot of XML in its training data.

55. 7:29

    Okay.

56. 7:29

    And so it's sort of seen more of that than it's seen of, of other formats. Um, so it just works a little bit better. So this looks like all the information here.

57. 7:39

    So we have the medication review will be ... Okay. Um-

58. 7:42

    Can you run it before and after? So before the first iteration and then the final iteration and see how it compares?

59. 7:49

    Yeah, great call. Uh, okay, actually let me undo everything that I've done so far. [laughs] [laughs]

60. 7:56

    Okay. Uh, so we can run it here and then now in the console it's asking us for the user input. So do we have a user input?

61. 8:04

    Yeah, I gave you a sample in Slack.

62. 8:06

    Okay, perfect. So who are you? Let's do this one. Why do I need to ... Uh, yeah, we can, we can do them both. Um,

63. 8:16

    who are you? Okay. So Gordie, what do you think of this?

64. 8:27

    It's way too long. This is a conversational agent, so no more than one sentence.

65. 8:32

    Okay. It's too long. Um, we can probably fix that. So, uh, well, we can also use this evaluate tab. So let's, let's just, like, add all the test cases here.

66. 8:45

    So this is the evaluate tab of the console. I'm also gonna be doing, like, some showing off of the console, uh, features, 'cause I think it's a cool tool, tool for prompt iteration.

67. 8:54

    Um, there's also some secret new features that I might show. Uh, we'll see about that. Um, okay, so then we also have, why do I need to do this experiment?

68. 9:10

    Looks like it added a bunch of, uh, new lines. Let's definitely get rid of those.

69. 9:15

    Um, and then we can get this next one. Can I schedule it tomorrow instead?

70. 9:23

    Cool. Um, so I hit run remaining, and this is all running through Dove, uh ... Sorry, through, um, this ... So, so, okay, so we have, why don't you do this appointment?

71. 9:38

    So this looks pretty long as well, and here we have this, like, I apologize, but I don't have information about scheduling availability. Um, okay. So

72. 9:53

    is that true? Is it true that we don't have information about scheduling?

73. 9:57

    Uh, no. We are always available. Make it easy for them. So 24-hour availability.

74. 10:04

    24-hour availability. Okay. All right, so this is like the version one.

75. 10:09

    Now let's make some changes. Um, so first of all, I'll ... Actually, I'll try to do things, like, roughly in, like, some order of importance.

76. 10:20

    Uh, so maybe I won't make you all sit through the capitalization, uh, even though I, like, would definitely do that. I'm also gonna add a, a new line here just 'cause I think that's, like, more normal what you'd see in, like, a written document.

77. 10:32

    You'd have a new line. Um, we'll close the information tab.

78. 10:38

    Um-

79. 10:38

    Could you get Claude to fix the capitalization?

80. 10:40

    What's that?

81. 10:41

    Couldn't you get Claude to fix the capitalization?

82. 10:43

    Yeah, I could actually. Uh, that's not a bad idea. The, the one thing I wouldn't feel completely confident of is that it would, like, exactly transcribe the, the rest o- of everything, like, word for word.

83. 10:54

    I think it probably would. Um, what I actually might do is just, like, have Claude write code, uh, to capitalize every, uh, first word of the sentence. Then I'd be worried about edge cases, like what if there's, like, ellipses.

84. 11:09

    Um, but I, I ... That kind of thing i- is definitely useful, and, like, I definitely use Claude a lot in, in writing prompts. Uh, for instance, like, we have, like, a, a Claude tool that, like, helps complete code basically, and I do a lot of prompting in that IDE because, like, especially with, like, very nex- nested

85. 11:25

    XML tags, uh, it helps a lot just, like, suggesting the closures of them, which is, like, pretty obvious but still takes a long time to type. Uh, so yeah, if you have any sort of, like, co-pilot type thing, uh, definitely that's, like, a good environment for, for writing prompts.

86. 11:40

    Um, okay, now let's do the same thing with this instructions. Um,

87. 11:47

    and we can do this. It looks like, um, this one, like, didn't get a number, so let's, like, do that.

88. 11:58

    Um ...

89. 12:01

    Where do you know to, uh, stop with XML and leave it as plain text?

90. 12:07

    Yeah, so the key thing in terms of XML, I think, is just, like-

91. 12:12

    Really, XML isn't even that important. The most important thing is just clearly separating the different parts of the prompt

92. 12:17

    Just using it for, like, semantics of what this section is and so forth

93. 12:21

    Yeah, exactly. It's like, here's this stuff, here's this other stuff. Like, if we wanted to, we could do something like, like

94. 12:29

    ... Like, I, I, I wouldn't do this, but, like, I think it would probably work fine.

95. 12:36

    Yeah. Yeah, it does.

96. 12:37

    Yeah. Um, okay. So this is all fine. Let's also do the same thing with user input.

97. 12:51

    Now we can run, we can go back to the Evaluate tab, and we can hit Rerun all, and it's using our nice new prompt. Um, still looks pretty long.

98. 13:01

    We can also see how it does on the, the last case where, um

99. 13:07

    ... Okay, so here it still said, "I don't have access to the specific scheduling information." So let's, let's try and fix these two things. So first of all, we can make it shorter, so

100. 13:17

     do we have anything here about, like, making it shorter?

101. 13:20

     Seven.

102. 13:20

     What's that?

103. 13:22

     Rule seven.

104. 13:22

     Rule seven, okay. Um, be concise and offer only relevant information. Uh, oops, let's actually do this.

105. 13:32

     Don't wanna misnumber here. And, um, be concise. So, like, and offer only relevant information. Each

106. 13:44

     response should be, uh, or let's, let's be a little bit, like, less prescriptive to give Claude, like, a little bit more room. Like, if we say, like, every response should be, like, exactly three sentences, that might be, like, a little too constraining.

107. 13:58

     I'm just guessing. Um, so we could just say, like

108. 14:03

     ...

109. 14:10

     So why is responses should be two to four sentences better than telling it to be concise?

110. 14:16

     Because you can

111. 14:17

     Concise is arbitrary.

112. 14:18

     Yeah, concise could mean a lot of different things to, to different people. Like, in some cases, like, concise might mean, like, literal only one word. In s- in some cases, like, if you ask for, like, a concise book review, we might be looking at, like, you know, a single page w- Word doc.

113. 14:32

     Uh, and that would be concise in the context of a book review. Um, so yeah, Claude is, like, trying to guess what you mean by concise.

114. 14:42

     What if it needs to be variable

115. 14:43

     You also have a longer prompt, so if you have a really long prompt

116. 14:47

     Sorry, one sec. Go ahead.

117. 14:48

     I was gonna say, he has a really long prompt, so if you have a long system prompt with a lot of detailed instructions, saying be concise, you're not gonna get something super, super short.

118. 14:58

     I think that's right. I think the tone of the prompt ... So the, uh, uh, what he was saying for, if people couldn't hear, is, like, the prompt is long, so the response might also be long.

119. 15:06

     I, I don't think that's, like, definitively true. Like, you can have long prompts that give short responses or short prompts that give long responses. But it's more like if you don't say anything, it might pick up on some of those, like, context clues.

120. 15:17

     You were saying something over here?

121. 15:19

     Um, what if it needs to be variable depending on the type of question that's been asked? Like-

122. 15:24

     Yeah

123. 15:25

     ... keep it short for, like, things that ... Because in human interaction, right, like, sometimes we give really short responses, other times we give long responses.

124. 15:34

     Yeah.

125. 15:34

     Have we trained LLMs to do that yet or we're not really doing it?

126. 15:37

     Yeah, so l- let me actually get to that af- after we do this. So this two to four sentences, it looks like it's still pretty long. I think maybe that's actually, like, longer than necessary.

127. 15:45

     So maybe we should make it, like, one to, one to two sentences. Um, let's try that.

128. 15:53

     Never more than three. Okay, now we can try that here.

129. 16:01

     Okay, that looks better, right?

130. 16:03

     Yeah.

131. 16:03

     This is definitely shorter. Okay, um, and it also seems that it is giving variable numbers of sentences, so these were both two sentences, and then this one is three.

132. 16:16

     So o- one of the questions over here is, like, can the LLM figure out that it should do longer responses in certain situations and shorter in others? So it seems like it, it did that here.

133. 16:24

     Um, okay, so then the next point was that in this case, it shouldn't say that I don't have access to the scheduling system or specific appointment times.

134. 16:34

     What should it say instead?

135. 16:36

     It should say, "Sure, what time tomorrow?" But intentionally in the prompt, I left out that it's 24-hour service.

136. 16:45

     So this is a case where we're asking a question that's not present in the information it has.

137. 16:51

     Okay. So yeah, I mean, we could a- add something like, um, you're open for 24-hour ser- service.

138. 16:59

     Yeah.

139. 16:59

     But you're saying y- you want it to ... You're, you want to test its ability to, like, figure out how to do it without that.

140. 17:05

     It should say, "I don't know," so.

141. 17:06

     Oh, okay.

142. 17:07

     It is saying, "I don't know," which is good.

143. 17:09

     Okay. Well, then we, we're doing great. All right.

144. 17:15

     Should we ... A- anything else that you wanted to get out of looking at this example, Gordie?

145. 17:20

     I think that ... Oh, about the structure. So does the order of the rules or the order of putting information, then rules, or rules first, then information, does any of that matter?

146. 17:32

     Yeah, so do we, does it matter what order we have these components?

147. 17:36

     Yeah.

148. 17:36

     I, I think it's better to put the information above the instructions.

149. 17:40

     Okay.

150. 17:41

     We've sort of found that instructions are more tightly followed the closer they are to the bottom of the prompt as a rule. This doesn't necessarily apply in all situations, so definitely test it out.

151. 17:54

     That actually is, like, a blanket statement that applies to everything that I said,

152. 17:58

     but es- particularly for that.

153. 18:01

     Yeah.

154. 18:01

     Okay. Uh-

155. 18:02

     Can I ask a question?

156. 18:03

     Yeah.

157. 18:04

     I, I don't know or I don't know if you were asking all the questions or-

158. 18:07

     No, no, no, go ahead.

159. 18:08

     It's only [REDACTED:physical_attribute] [REDACTED:gender] allowed, so you're good.

160. 18:15

     I noticed that it's-

161. 18:17

     Exclamation mark?

162. 18:19

     The exclamation mark, yes.

163. 18:21

     Oh, remember, I don't think I a- added that on... Or it looks like you added that, Gordie. I- e- exclamation marks, just as they emphasize things for humans, they also emphasize things to the model.

164. 18:33

     Do you, for the... Or do you think that has more of an effect over the numbers or less, like, in terms of relevance?

165. 18:40

     Ooh, yeah, I don't, I don't know at that level of detail, and I think it's dependent on, on context as well. But yeah, if, if you want to emphasize things, like capitalizing them or putting exclamation marks or, like, just saying, "This is extremely important," that all does do something.

166. 18:56

     Shouting in caps.

167. 18:58

     So-

168. 18:58

     Yeah.

169. 18:59

     Um, so the tokenizer, though, um, I, I, I'm taking a look at some of your, your tokenizer code, but it doesn't seem like exclamation points actually to- like, do anything really.

170. 19:13

     Like, they're, the tokenizer kind of combine them into the word.

171. 19:17

     It does something. That's all I c- I, I can say. Just anecdotally, if you put exclamation marks in, it's, it's different.

172. 19:22

     I guess, okay.

173. 19:23

     Yeah.

174. 19:24

     That's my, that's my analysis of it.

175. 19:26

     All right. Okay, I think let's go to, um, the next. How many we got? We got 60 already. That's pretty good. Okay, this is just a general question. I'll just answer this really quick.

176. 19:38

     Is it in general for translations or multilingual output, is it better to instruct in English or the native language?

177. 19:44

     I think it's better to instruct in the native language if you speak the native language.

178. 19:49

     If you only speak English and you're choosing between a, like, a better prompt that's written in English versus, like, a worse prompt that's written in a language that you don't understand, I'd probably default to writing it in the language that I knew super well.

179. 20:03

     But ideally, I think for the ideal prompt, you would find a native speaker of the language and explain your use case to them and, and have them write the prompt.

180. 20:12

     Well, can I ask a question about language?

181. 20:14

     Yeah.

182. 20:15

     So, uh, for a language that, uh, Claude is not so familiar with, should we, like, maybe write the prompt in English, but let the, the Claude to translate the output into the other word, or should we just write the prompt in that language?

183. 20:32

     So if it, if it's-- I think, is that not the same question that I just answered? Is it different?

184. 20:35

     Oh, so you said it's better to write in the, in that language.

185. 20:38

     I think it's better to have the prompt in, in that language in, in general, if you can, if you can write a really good prompt. All right, let's go to this, this next guy.

186. 20:46

     So you'll be acting as a test reviewer. Let me pump up the size here too. Um, okay.

187. 20:55

     Not sure if there's a way to... Oh, I can hide this. Okay, great. Responsible for improving unit tests based on a set of requirements. Below is the project directory,

188. 21:03

     project path. Don't include any other explanation, conversion, or output besides the JSON.

189. 21:14

     Okay. This is great because it's gonna let me show off pre-fills.

190. 21:21

     So let's make a new prompt here. Let's paste this in. We have to use, uh, double brackets instead of single brackets to get variables in the console.

191. 21:35

     And then I think there's another... But this JSON string, uh, who, who gave this prompt? This is from, uh, Dan. The JSON string, Dan, that's, is that variable, or is that,

192. 21:48

     is that, like, an example in this prompt template?

193. 21:51

     So in this case, that, that's, um, that is just a template, and then we, this is just the middle of a multi-agent workflow. So basically, another one of the agents is gonna write the unit tests, and then, and that's gonna be a JSON, and that JSON blob is what it's gonna respond with, essentially just using the earlier

194. 22:07

     part of the conversation. I, I put the, uh, sample output in red if you go back to that.

195. 22:16

     Yeah. Test reviewer agent does not return, always return JSON only.

196. 22:25

     Yeah. Like, almost all of this is just, like, workarounds for the fact that it doesn't always speak JSON right. Like, you can see how many times we said that in the prompt.

197. 22:32

     Yeah. And then, uh, do you have an example input here?

198. 22:36

     Uh, so that first, if you go up a little bit, sorry. So that first comment is from the unit test writer. So that is the input. Like, the unit test writer writes a bunch of unit tests, and then the reviewer reviews it and makes them better.

199. 22:50

     So everything, everything here is, is what I should put into-

200. 22:54

     Yes. And then you can see that second set. This is, this is a good result where it writes JSON, which basically says, "Cool, update this file with these unit tests, and here's the modifications I made," that sort of thing.

201. 23:05

     Okay. So i- in, in this template here,

202. 23:09

     w- where would the thing that, that I just copied go?

203. 23:13

     Well, so essentially, we don't provide it in line with the prompt. We just provide the con- the conversation, and then this thing jumps in, in as a, as a separate agent.

204. 23:26

     So, like, the, the context window is gonna have the unit tests in it, but we're saying respond in this format given the unit tests that are earlier in the conversation that you're picking up.

205. 23:36

     So th- this, the thing that you just pasted is, like, step three of a multi-shot conversation?

206. 23:41

     Uh, the thing I just-

207. 23:41

     Like, or sorry, not multi-shot, multi-turn.

208. 23:43

     Yeah, yeah. The thing I just pasted was two shots, right? Unit test writer and then a unit test reviewer, and the reviewer is the one that's having the problem.

209. 23:50

     It comes second.

210. 23:53

     Okay, so it'd be something, it'd be something like, like this. Um,

211. 23:59

     here are some unit tests written by a unit test writer bot.

212. 24:09

     Right.

213. 24:17

     Okay, and then we have this. Now, you didn't, uh... So, okay, so it, this, this is basically how it works.

214. 24:26

     Is, is ... Does this look right?

215. 24:27

     Oh, no, it does. I mean, this-

216. 24:29

     Okay

217. 24:29

     ... we, we just, we do this in sort of this larger conversation, not just as sort of a standalone prompt for this one-

218. 24:35

     Yeah

219. 24:35

     ... agent.

220. 24:35

     Okay. So then here I put the unit tests in.

221. 24:37

     Right, yep.

222. 24:38

     And then for project path, what sort of thing should I put there?

223. 24:41

     Um, anything. I mean, th- this is y- this is just, like, the local directory that's gonna be modified, and so it actually has access to the files in that directory, and it'll, it'll fill in its own, um, sort of which, what files to modify.

224. 24:53

     Okay. So then I'll just put, uh-

225. 24:55

     That's fine

226. 24:57

     ... um, something like that.

227. 25:00

     Yeah.

228. 25:07

     Okay, so let's see if it comes out with the JSON or not. Uh, bated breath here.

229. 25:12

     Yeah. We did do most of our tests on Claude 3 and not 3.5, so 3.5 is probably a little better. Um-

230. 25:17

     Okay

231. 25:17

     ... we haven't done.

232. 25:18

     Yeah, I mean, if, if, if it, if it makes it more realistic, we could also, uh, switch the model version to, to use Haiku.

233. 25:23

     Oh, no. I mean, we're, we're gonna upgrade, so I'd rather see it with this.

234. 25:26

     Okay. What was that?

235. 25:29

     So the, I see the temperature's set to zero. Is that intentional?

236. 25:32

     Is the temperature being set to zero intentional? Yeah. I usually, for knowledge work, I usually have the temperature set to zero.

237. 25:38

     Not 0.01? Like, I usually use that to test for hallucinations. I mean, I, I, I'm just, I, I'm just a fool here, but, like, is that, is that ...

238. 25:48

     Like, am I totally out?

239. 25:50

     I think using temperature zero you'll probably get, like, marginally fewer hallucinations. Okay, I'm ... Oh, here we go.

240. 25:57

     Okay. So it looks like in this case it did output JSON,

241. 26:02

     I think.

242. 26:03

     Yeah. That, that looks plausible.

243. 26:05

     Okay.

244. 26:08

     It's still-

245. 26:08

     Very long JSON. I guess that explains why it was taking so long to ... Looks like, actually, it even ran into the max output tokens, because it didn't finish its JSON.

246. 26:16

     Aha.

247. 26:17

     Um, just to make this, since this one is, is kind of going kind of slow, I will test it with Haiku.

248. 26:24

     Uh, let's ... And let's also increase the, uh, max tokens to sample so that it doesn't run into that issue.

249. 26:34

     So what I'm really hoping is to get a case that doesn't output JSON so that then I can fix it, and then it will output JSON. If not, I can still say, like, how I would fix it.

250. 26:44

     Yeah, that would be great. Just honestly any comments you have just on how we structured things.

251. 26:48

     Okay. Yeah. So I mean, this is, like, uh, definitely, like, a, a big request from people is like, how do I make sure the model outputs JSON? The most reliable way to do that, I feel, is using the, uh, assistant pre-fill.

252. 27:02

     So, uh, maybe some of you have used this feature before. Maybe some of you have, like, only used, uh, other models such as GPT that don't offer this feature.

253. 27:12

     Something that you can do in the Claude API is partially pre-fill an assistant message. So what you're doing there is you're putting some words in Claude's mouth, as we call it, and then when Claude continues from there, it assumes that it's already said whatever you told it that it had said.

254. 27:31

     Uh, and just, that, that can help you get it on the right path. So for instance, in this case, uh, if we wanna make sure ... So the classic, like, bad response from, from Claude w- when people give it prompts like this where they wanna get JSON, is Claude would say something like, uh, I'll just, like, add

255. 27:47

     another message just to have some of us to type. It might be like, "Here is the JSON," right? Have people seen stuff like this?

256. 27:55

     Oh, yeah.

257. 27:57

     And this part right here is, like, very annoying.

258. 28:01

     Yes.

259. 28:01

     And difficult to get rid of. So okay. So I, I have, I have two strategies. Uh, let me actually just give, like, the sim- the simplest one. They both, though, they both require a tiny bit of post-processing at the end.

260. 28:17

     So let's start by, uh ... Let's actually, like, take out all this stuff about make sure to only respond in JSON. That, that could be one way to get it to, to not do the, uh, to, to be bad.

261. 28:30

     Uh, so we could just go like this. Let's try to make it not do, uh, the JSON.

262. 28:36

     Let's get rid of all this stuff. Okay, so here now maybe it will do the preamble thing that we don't want it to do. Perfect.

263. 28:46

     Okay. So an easy way to get it to not do that is to just take this and then put it in

264. 28:57

     the pre-fill, so it thinks that it already said that.

265. 29:01

     Uh, like this. So if we do that,

266. 29:08

     just the JSON. So what we're doing here is ... You could think of Claude almost like a, like a child who's just, like, misbehaving, and it wants to do something, and you're l- and you're like, "Don't do the thing," but it just keeps doing it because it just loves preambles so much, and it has this, like, innate

267. 29:25

     desire to, to do them. So one way is to, like, argue with it a lot, but, like, if you have a kid, sometimes you know you just have to, like, let them do the thing that they wanna do, and then they'll get over it.

268. 29:35

     So in this case, that's basically what we did. We just gave Claude this, this pre-fill where we let it do the thing. So as far as it's concerned, it already did the thing.

269. 29:45

     And then from there, what it, what it's outputting is, is, is JSON. Now, if you wanna make this even more reliable,

270. 29:52

     you can put this nice little bracket here, and then it's like, oh, dang, like, I'm really in JSON mode now. Like, I'm, I'm really ... My, my JSON has actually already begun.

271. 30:02

     So it's, it ... At, at this point, it's definitely not gonna do the preamble. The only thing her- here is if you sample with, with this, with this, uh, pre-fill, you will need to add the bracket back before you try to do your JSON.loadS or what have you because Claude is ...

272. 30:19

     Si- since you told it that it had already said the opening bracket, it's not gonna give you another opening bracket. Uh, okay. So then another thing that you can do is

273. 30:30

     return the JSON in JSON tags. And then

274. 30:37

     if we do this without the prefill ... Let's try it without the prefill.

275. 30:41

     You don't normally capitalize JSON? [sighs] [laughs]

276. 30:50

     I'm not a software engineer, okay? I'm a prompt engineer. I don't even know that it's capitalized. For all I know, that's just, like, a English word. [laughs]

277. 30:56

     Um, but yeah, good, good. Thank you. Um, okay, so here we, we see it, it, it did the thing. So it gave its preamble, right? And then it gave the JSON tag.

278. 31:08

     Everything within the JSON tag is JSON. And then at the end, it closed this JSON tag. So again, this requires, like, the tiniest smidgen of post-processing, where you're saying, like ...

279. 31:21

     You just ... It's like a regex. You're just like take everything within the JSON tags, and then, and then use that. You can even combine th- these two techniques.

280. 31:29

     So you could say, "Here's the updated JSON," and now you give it the JSON tag. We can even put a bracket here. And now what we'll see is it will just give the JSON minus the bracket, and then it will close the bracket, and then it will close with, uh, the JSON tag,

281. 31:46

     the close JSON tag. There we go. And it also gave this, uh ... So you can see it did ... At, at first I was, like, a little bit panicked, because I didn't see the close JSON tag at the very bottom, but then I saw that it actually did include the tag up here, and then it gave

282. 32:05

     this little explanation afterwards. So this is another, uh, useful thing. This will, uh, save you some, uh, time and tokens and trouble. One thing we could do, like, like, it, it, it costs Claude, like ...

283. 32:18

     It costs you money to get Claude to output all this stuff, and you probably im- you probably don't need it. You ... In most cases you don't need the explanation.

284. 32:27

     You just need the JSON. So one thing we could do is we could say, "Do not include any explanation after the JSON," right?

285. 32:37

     Does it help to yell in that case? [laughs]

286. 32:38

     I mean, probably, but I don't know. I, I, I, honestly, I don't yell that much. I'm just, like ... This is actually meant to be my parody of, like, what a frustrated prompt engineer would write if they were, like- [laughs] ...

287. 32:48

     couldn't get rid of this. But in practice, you, you might not need to do that. But the simpler way to, to do this, there's a ... And we're getting outside the realm of prompt engineering for a second and into the world of, like, API parameters, but, uh, that's okay.

288. 33:02

     There's a parameter that's called stop sequences, and if you set ... So we told it to return the JSON in JSON tags, right? So, uh, there's no functionality to do this in the console, so I can't show it off at this exact moment, but in the API there's a parameter called stop sequences, and if you add, uh,

289. 33:20

     this close JSON tag that I've highlighted with my mouse, if you add that to the stop sequences, then it will just hard stop after it outputs those, and you don't even have to worry about telling it not to continue from there, because it just won't even sample from there.

290. 33:35

     You won't be charged. It's all good. So one of the things that I'm sort of hoping to impart with this talk is that a lotta times it's cheaper and easier to do a little bit of work outside of a call to the LLM and not even worry about prompting, because prompting is, is, is ...

291. 33:55

     can feel sometimes, like, nondeterministic. You don't know what the model's gonna do. So when you can offload stuff to code, especially if the code is really easy to write, it's like just do that, right?

292. 34:04

     Like, don't, don't put a bunch of stuff in the prompt about you must output JSON. Just, just use the prefill and then parse it out with the, the regex.

293. 34:11

     You know, don't, don't add a bunch of stuff about how you have to, to stop after you say a certain word. Just add it to the stop sequences. So, like, simple is, is, is better, and falling back on code is better than relying on prompts.

294. 34:24

     Yeah.

295. 34:25

     Is that prefill available through the API calls? And-

296. 34:29

     Yes. The prefill is available through the API. What you do is you include an assistant message as the last message in the messages list.

297. 34:39

     And when I say an assistant message, I, I just mean a message where the role is set to assistant.

298. 34:44

     And, uh, what would have happened if you had your ... the text that you put in the prefill, you just put it into the last line of the instructions?

299. 34:53

     So in other words, if I said, uh ... I'm actually not sure. That's a, that ... good question. So let's actually try that.

300. 34:58

     So it does work with open. Like, you can just say, like, there is

301. 35:04

     a

302. 35:04

     I, I, I, I genuinely do not know how Claude will respond to this, so let's see.

303. 35:12

     So it looks like what it did was ...

304. 35:15

     It looks to me like what it did, without looking at this JSON, is it included an additional open bracket,

305. 35:23

     right? 'Cause it, it's supposed to already have started with an open bracket, but here it started with an additional open bracket. So it kind of almost worked, but not quite.

306. 35:31

     Anyways, I don't recommend doing this, but that was, uh, fun just to out of curiosity's sake. Yeah.

307. 35:37

     Uh, instead of that can we just, uh, remove that? Uh, uh, and, uh, after the sentence, like ... Yeah. Is it on? Yeah. Uh, after the sentence we can just write it like, uh, you wrote, right, uh, return the JSON, return the response in the JSON format, and then in the next line, uh, you can just write

308. 35:56

     JSON colon, then leave it. I think it'll-

309. 36:00

     Oh, so, like, if I said something like this.

310. 36:04

     Like this?

311. 36:05

     Yeah. That's it.

312. 36:07

     Yeah, I could see this working. Yeah, it looks like it worked pretty well. I, I, I think there's, like, a lot of ways to accomplish this.

313. 36:16

     Yeah.

314. 36:16

     I think the ways that I showed are the most reliable, so that's what I would, like, officially recommend. Uh, but yeah, like, definitely experiment.

315. 36:26

     If you were gonna, uh, like, try to use this for production or whatever, what ... Like, these exact kind of things you're playing around with right now, how would you think about testing that, like, at some sort of scale, like?

316. 36:37

     How would we test it at some sort of scale?

317. 36:39

     Yeah.

318. 36:40

     So-

319. 36:40

     More than, like, one ... like, the one-shot test we just did right here.

320. 36:43

     Yeah, yeah, yeah To test it at scale, you need a bunch of test cases

321. 36:49

     and-

322. 36:50

     We don't have those

323. 36:50

     ... if you don't have test cases ... Okay, this is m- maybe a good time to maybe show off this thing, although I'm actually not sure if it will work.

324. 37:00

     Uh-

325. 37:01

     So, well, one, I guess a more pointed question is like-

326. 37:04

     Yeah

327. 37:04

     ... in this case, I think test cases are useful when I'm writing a prompt to deduce whether, like, does asking it to think step by step lead to this thing being more accurate?

328. 37:14

     But in this case for for- formatting, I guess what I'm wondering is, like, could you have this prompt and then feed in the output and the prompt, and then ask the model itself to evaluate, like, how good these various things are at following the instructions?

329. 37:29

     Yeah, yeah. Okay, so can we do model grading? Can we model grade the outputs?

330. 37:32

     Yeah, especially for formatting related things where I actually need a, like-

331. 37:34

     For formatting, I would not model grade the outputs-

332. 37:37

     Okay

333. 37:38

     ... because formatting is something that I can check in code. So if I can do anything in code and, and I don't have to call the LLM, the LLM is, like, this crazy black box, right?

334. 37:46

     It's like, if I don't need to, like, make this pilgrimage to the oracle and, like, ask it, I, I'd rather just do it, like, in, in code. So formatting specifically, we're, we're kind of, like, in luck.

335. 37:56

     It's easy to check. For something like the other ... the, the previous prompt we looked at where the outputs are a lot more squishy, we might ... Possibly a model grading could work.

336. 38:06

     Possibly we might need a human to evaluate the answers.

337. 38:08

     To, just to lightly push back on that-

338. 38:10

     Yeah

339. 38:11

     ... so I'm wondering, like, I actually put an example in the Slack channel. We don't need to get to it, 'cause we're talking through it now. But, like, for, uh ...

340. 38:16

     Let's imagine I don't have ... Or actually, maybe, maybe tags are the answer to this. Um, like, imagine I'm asking for a summary or something and then I want to deduce whether there's additional chat-like content, like, before or after that.

341. 38:29

     In that case, would you ... I would've, my mental model would've been to use, like, an LLM as a grader. But it sounds like maybe would you encourage instead using the summary tags and checking, like, hard coding for additional text around that?

342. 38:43

     Yeah. I think that will be, uh, pretty quick and easy to do. Also, just having the summary be in summary tags is, like, generally a good practice.

343. 38:52

     Okay.

344. 38:53

     I generally have all my outputs inside tags to make it really easy to extract them. I don't think there's really any, any downside to doing that, so ... And it m- it, it might even be that by doing that you effectively fix your entire issue and you don't even, like, need to, need to do the, the test

345. 39:07

     anymore or ... A- and you just put close summary in the stop sequences-

346. 39:11

     Yeah

347. 39:11

     ... and, and you're kind of good to go.

348. 39:12

     All right. Cool.

349. 39:14

     But that's also does sound like a problem that an LLM, uh, could grade.

350. 39:20

     Okay. Uh, let's go to the next, uh, prompt here.

351. 39:26

     Just shout out your question.

352. 39:28

     Here's a very poorly formatted Excel spreadsheet.

353. 39:31

     Um, I got a question real quick. So, um, this seems like a really ridiculously, like, powerful attack vector, so can we test a prompt real quick?

354. 39:42

     Um, I don't wanna get into too much, like, jailbreaking stuff here. Sorry.

355. 39:46

     Okay. Ap- apologies.

356. 39:48

     Yeah, yeah.

357. 39:48

     So-

358. 39:48

     Well, I-

359. 39:49

     ... I, that, that's kind of my specialty. [laughs]

360. 39:50

     Okay. Yeah. [laughs] Um, I'm gonna go to the next, next prompt.

361. 39:56

     Um, so what do we have here? Here's a poorly formatted Excel spreadsheet/CSV.

362. 40:06

     Please extract all data into JSON. Okay. How can we ... So, uh, Jan or Yan?

363. 40:12

     Jan, right here.

364. 40:13

     Uh, is it Jan or Yan?

365. 40:14

     Yan.

366. 40:15

     Yan, what, uh, what is the actual text that I should ... Can, can you, can you paste the text here? 'Cause I, I don't know how to get the CSV into the, uh, into the console.

367. 40:24

     Oh, I've just been ... Oh, hey. Thanks. I've just been just copying the entire CSV and putting that into the prompt. Again, I've been trying to use Claude to extract some information from spreadsheets, and it's always been very, very hard.

368. 40:41

     It hallucinates a lot or it-

369. 40:42

     Yeah

370. 40:42

     ... skips a lot of stuff. And I was wondering if you, maybe more generally, how do you have Claude analyze really poorly formatted spreadsheets that sometimes the different clusters or multiple data sets in the same sheet and things like that?

371. 40:57

     Okay. I'll try to answer the general question of having m- uh, analyzing, like, poorly formatted spreadsheets. The first thing that came to mind, especially when you're s- talking about how the spreadsheets are very big, is breaking the problem down into ...

372. 41:13

     So, so give it, like, fewer spreadsheets at a time. Give it fewer columns of the spreadsheet at a time. Only give it the, the columns that it needs to, to work with.

373. 41:22

     Um, and then make the questions sort of smaller and more bite-sized, and then tackle it that way by, by breaking it down. So at that level of gen- generat- generality, that's would be, would be my answer here.

374. 41:37

     Um, y- I, I, I'd also be curious to look at this one more specifically. Right now I'm just struggling with how to, like, copy the text and put it into the tool.

375. 41:46

     Could you put as regards and then download all?

376. 41:49

     That's what I did, but it keeps downloading it. I guess I can-

377. 41:51

     Yeah, it's the next tab as it, as it opens.

378. 41:54

     Um, the ne- sorry, the next tab.

379. 41:56

     You had the-

380. 41:58

     This is that other one.

381. 42:00

     There's ... In your downloads, you can open it and then-

382. 42:02

     Sorry, I don't wanna click open my downloads. I'm scared I'm gonna, like, reveal some private information. This is my work computer, so- [laughs] ... I wanna just do it all in the browser.

383. 42:12

     Yeah. Ask it to write some Python codes. [laughs]

384. 42:16

     It's fine. You can go to the next one. No worries.

385. 42:17

     Yeah, let's do the next one. Okay. You are a social media ghostwriter.

386. 42:26

     Given the below long form article. Okay. Uh, generally we would recommend putting the ... I like this one. It's short. We can do some quick hits here. Uh, we would recommend putting the instructions after the document.

387. 42:41

     That's similar to the question that was asked about should we have the information first or the instructions first. Particularly with long documents, it's a little bit better to give the instructions, uh, at the end.

388. 42:51

     Let's also put some XML tags here. Uh, let's just, like, clean up the grammar a little bit. Given the above long form article, create five to 10 tweets. Don't use hashtags, [laughs] don't be hyperbolic, and don't be cringe.

389. 43:08

     Uh, return a JSON array of post content. Probably good to, to give an example, uh,

390. 43:15

     of the format. Uh, so we could do, like, uh ...

391. 43:26

     I guess we can just do a return of, like, a list.

392. 43:30

     Wait, uh, what did you originally have it? Is there a special reason that you wanted it to be a JSON array or is it just to make it parsable?

393. 43:38

     Uh, just to make it parsable for automation's sake.

394. 43:40

     Okay, yeah. So let's say return in ... I'm just a, a, a huge fan of these tags, so let's do it like this.

395. 44:18

     Okay. So that is, that, that's some stuff without adding examples. The other thing that I would wanna do is to give some illustrative examples of what it means to not be cringe. [laughs]

396. 44:34

     So how long are these documents?

397. 44:37

     Uh, I put in a ... I used a recent blog post from Anthropic as an example. Do you wanna copy it?

398. 44:43

     Perfect, yeah. Let's actually, let's run this as is.

399. 44:47

     You're missing a bracket at the end.

400. 44:49

     Yeah, I did.

401. 44:50

     The last, the very last, uh, hard bracket.

402. 44:56

     Oh, yep. Thank you. Now we can take this.

403. 45:06

     Uh-oh, what if they write cringe tweets about our product? [laughs] I'm gonna be embarrassed.

404. 45:17

     Okay, this doesn't include any hashtags, doesn't seem very hyperbolic. Is this cringe? What do we think? [laughs]

405. 45:28

     New feature alert. That could be a little bit cringe. [laughs]

406. 45:32

     There's no emojis, which is a good sign.

407. 45:36

     Okay. W- what do you ... Uh, your name was, uh, Charlie. What do you think of this, Charlie?

408. 45:41

     Uh, I think they are adequate, but not engaging.

409. 45:48

     Not engaging, okay. So yeah, yeah, yeah. So l- we can try to make it more engaging without making it cringe. So let's say, don't use any hashtags, but cringe.

410. 45:57

     Try to make the tweets engaging. Are these meant to be tweeted from the Anthropic Twitter account or from, like, the, a, a AI influencer Twitter account? [laughs]

411. 46:11

     Sure, let's say AI influencer Twitter account.

412. 46:13

     Okay. Let's see how this goes. I'm gonna switch back to Dove, too. Sorry, um,

413. 46:33

     exciting news.

414. 46:36

     Is it better to break it up into small sentences in the prompt or can you use complex sentences?

415. 46:42

     Is it better to break up sentences, to use, like, small sentences in the prompt or big sentences?

416. 46:49

     I think generally in English writing it's better to use small sentences and small words,

417. 46:56

     so I think it's probably also better to do that in a prompt. I think it's fine to use big words if you are really sure you know what you're doing and you know that it's the exact right word for the situation.

418. 47:08

     Sometimes I'll find myself using more academic language if I want the output to seem a bit more academic.

419. 47:16

     Generally, I think simple, small sentences is, is better.

420. 47:22

     Uh, okay. So these are maybe a little bit more engaging.

421. 47:32

     Like, they have these questions here. Wanna try it? Uh,

422. 47:38

     what do we think? It's got exclamation points and question marks. Is it better? Do we want it to be even more engaging or something?

423. 47:46

     It doesn't take into consideration temperature, right?

424. 47:50

     Let's see, 'cause ... Okay, so I honestly think temperature is, doesn't, is a bit overrated maybe. Uh, we can, we can see if, how, how, how that, how that differs, though.

425. 48:14

     I'm not sure exactly how to distinguish these from the, the previous ones. They look kind of similar to me. From the ones with, with temperature, uh, temperature one, or temperature zero.

426. 48:24

     Engaging is a very too general and subjective, right?

427. 48:28

     Mm-hmm.

428. 48:28

     It's better to e- us- emulate a certain per- personality.

429. 48:32

     That's right. Yeah. So what I was gonna say is I think this is roughly as far as you can take this without examples. I think the best thing to improve this prompt would be either examples of the sort of tweets that, that you want,

430. 48:47

     or even an entire other document, an example of tweets that go with that document, and maybe, like, multiple of those. So-

431. 48:57

     Uh, if, if you're cost limited, maybe you don't wanna put in all those input tokens every time. But I don't know, the models are, are pretty cheap now, and we don't need to generate that many tweets.

432. 49:09

     So if they, if they have, like, any economic value to you at all, it's probably pretty cost effective to put ... So basically, like, it ... But it's, it's more work on your part because what you're doing then is ...

433. 49:22

     So, okay, so the, the way that I would actually do this, uh, is I would start out with some, some document. I would have Claude write a bunch of tweets.

434. 49:32

     I would take the ones that I liked,

435. 49:35

     and maybe I would write some more or get, like, my friend to write some more, or maybe I'd have Claude ... Oops, uh, Claude generate 100 tweets, and then I would take the, the, the seven that I, that I liked best, and then I would put that in as an example.

436. 49:47

     And then from there I would sample, okay, now here's another document, and then write a bunch more tweets based on this. And what I would do is iteratively build up this, uh, list of documents plus example tweets, and then I would put them all into the prompt, and it would look something like this.

437. 50:07

     Uh, so let's actually do that. So let's imagine that we had done this. So it could be, like, you know, system prompt. You, uh,

438. 50:14

     you are an AI influencer who writes engaging social media content about, uh, new models and releases. Uh, it could be like, here are some example documents along with the tweets

439. 50:38

     you wrote about them. Um, and here you would actually ... I, I'm gonna write this, but you would actually put the literal text of the document here.

440. 51:03

     And here again, you'd put a literal tweet here, and this could either be something that you wrote or something that Claude wrote or, you know, you, you ... Something that Claude wrote and then you edited.

441. 51:11

     Like, a lot of times Claude might give you an example that's not perfect, but it- it's close enough, and then you'll, you'll change it a little bit to, to make it perfect.

442. 51:20

     I have honestly given, uh, multi-shot examples pretty short shrift in this talk so far relative to their level of importance. Like, I, I think that in reality most of the, the gains, most of the effort, most of the gains of writing a good prompt is literally just picking the perfect document that goes here, picking the perfect set

443. 51:43

     of tweets that go here, altering and, and changing them to, to, to modulate the tone.

444. 51:49

     Uh, i- in some ways that's more important than, like, everything else that I've said combined. Like, another way to do the whole JSON thing would just be, like, with examples of Claude giving the stuff without a preamble.

445. 51:59

     The pre- the JSON one is maybe an exception, uh, i- because the, the, the pre-fill approach wor- w- works so well there, along with the tags, but for anything else, the examples are, are, are really huge.

446. 52:10

     Anyways, then we would, uh-

447. 52:14

     For few shot prompting, do you prefer to front load those all in one response like this? Or do you find further success with an exchange of messages between the agent and the user where you're putting your few shot prompts-

448. 52:29

     Yeah

449. 52:30

     ... in there?

450. 52:31

     Really good question, and something that I would dearly love to know the answer to, but I don't. The, the question is ... Did ... I, I, I don't need to repeat the questions.

451. 52:39

     The ... I think people can hear them. Uh, but I'll, I'll repeat anyway. So do we wanna just put all the examples in one big giant examples block like this, or do we wanna structure the examples as a dialogue where the human says something and then the assistant says something back, and we're literally, like, putting a large

452. 52:55

     number of messages into the messages list? I typically do it this way with a big examples block, but it's mostly because it's less work for me, and I don't have any evidence that this works either better or worse.

453. 53:08

     I did do some testing of this at one point on a few data sets, and I found that it didn't make much of a difference for my particular case.

454. 53:16

     But there's a lot of, like, little particulars that went into my testing that make me not very confident in the, in the result that I got. So sorry for a bit of an unsatisfying answer here.

455. 53:25

     I'll just say I don't think it ... I- if it is wrong to do one giant examples block, I don't think it's, like, very wrong.

456. 53:32

     Do you, do you use anti examples too? So like in here, would you give it a thing and say like, "This would be bad because this is cringe"?

457. 53:40

     Yes.

458. 53:41

     Yeah.

459. 53:41

     Yeah. I think that is good. S- I think it's good to include negative examples, particularly around, like, the cringe thing where, where Claude might mess up. Uh, I, I think just negative examples on their own don't usually get you there.

460. 53:56

     You wanna have some positive e- examples too, but I think it's great to have especially, like, contrasting pairs. So like, here's a document. Like, here's a cringe tweet about this document.

461. 54:06

     Here's a excellent tweet about, about the same document. Uh, and, like, set those up side by side. I think that's pretty, pretty powerful, and I do that, and I think it, it helps Claude.

462. 54:16

     And then if, if you also include, like, the reasoning for it, right, so, like, if it was a cringe tweet, it has, like, a little reasoning of, like, why, why.

463. 54:24

     Um, do you also ... Do you trust that reasoning for the model? So, like, if you ask it, like, "Hey, give me ... " Like, "What were you thinking when you were writing this tweet?"

464. 54:33

     And then write me this tw-

465. 54:34

     Yeah

466. 54:34

     ... this tweet. When you're reading through your examples to choose the best ones, how much do you trust that reasoning-

467. 54:42

     Yeah

468. 54:42

     ... and how much do you rely on that versus just, like, I just care about, like, the input output?

469. 54:46

     I don't trust the reasoning very much, especially if it's after something the model already said.

470. 54:50

     No.

471. 54:50

     Then I, like, really don't trust it.

472. 54:52

     Yeah.

473. 54:52

     But I mean, humans are not very good at explaining why we do the things that we do. We're really good at rationalizing and coming up with, like, fake reasons.

474. 55:00

     But a lot of times we don't even know why we do the things that we did, let alone be able to coherently explain them to someone else. So I-

475. 55:10

     The- the- there's- there's- there's a subtlety here. So something that does work pretty well is having the model think about its reasoning in advance and, like, go through different l- reasons or rationales for why it might choose one option or the other or think about what sort of things might go into a good response.

476. 55:25

     So if I had the model do some thinking in advance before it gave the response, then I might just trust or assume that the response would be better.

477. 55:36

     Having a bunch of explanation for why it did the thing after it, probably I- I- I would not trust that. Sorry, you had a question for a while.

478. 55:44

     Do you, uh, do you give your own reasoning that explains the examples? And if so, how do you make sure that the model doesn't give reasoning itself?

479. 55:56

     Do I give reasoning to explain the examples? Yes. I do a lot of giving- giving reasoning to explain the examples. So for instance, just in this case, uh, one thing that we could do here is, like, um, we could add something like, uh

480. 56:14

     ... I- I was gonna say tweet planning, but maybe it's, like, key points of document.

481. 56:20

     And then we, here, we have some key points, like the document, uh,

482. 56:26

     presents the launch of ...

483. 56:36

     So you would have this after the example. So if you have 10 examples-

484. 56:39

     No, this is before the examples. Or, sorry, so it's pa- this is part of the example right here. So I- I- I ... In- in this particular example, in this example block, I gave it a document.

485. 56:49

     Now I'm doing this- this key points business.

486. 56:52

     Got it.

487. 56:52

     Um, and then I would have these tweets. Now, this key points could be something that I wrote myself or it could be something that Claude wrote and then, uh, I'm- I'm, uh, editing it.

488. 57:06

     Or if Claude did a perfect job, I, maybe I could just include the thing that Claude wrote. But now in order to get this, get Claude to do this, uh, we would also say something like, um, "Return in this format, uh, key points,

489. 57:19

     a list of the key points from the document."

490. 57:26

     So this is like a lightweight chain of thought where we're having the model do some, uh, thinking in advance, and we also gave it examples of it doing the thinking in advance, uh, like this.

491. 57:36

     Hm.

492. 57:37

     Uh, yeah.

493. 57:44

     Um, yeah so let's imagine we, like, really want to give examples like this, but we have a problem, which is that our documents are, like, super long and I'm greedy and want to save on input tokens.

494. 57:54

     Yeah.

495. 57:54

     Would you err on the side of doing, like, one document but a really good example or doing, like, truncated versions of more documents?

496. 58:04

     I would ... That's a good question. I would err on the side of one extremely good example and not truncated versions of more documents. But I would also want to look at the outputs and test that assumption because it's possible that with only one example, Claude would fixate on aspects of the exact document that it uploaded and

497. 58:23

     start trying to transfer them to your document.

498. 58:27

     Right.

499. 58:27

     So I think it's- it's one of those, it would be case by case, but-

500. 58:30

     Right

501. 58:31

     ... I wo- I would st- I would want to start with, like, having one extremely good example. Generally, I think that, like, less but, like, higher quality is-

502. 58:38

     Mm-hmm

503. 58:38

     ... a better way to go than, like, more and- and lower quality.

504. 58:41

     Cool. Thank you.

505. 58:43

     Okay, we have a lot of prompts here. Let's go to the ...

506. 58:47

     Okay, this is good. I was hoping we would get some, like, persona ones here.

507. 58:59

     Okay. So this is, this looks like something where we're, we're trying to get Claude to, to role play in these different protocols.

508. 59:11

     So let's try this out and let's see how it works. So this looks something where we're gonna have like a ... This looks like it's, like, a meant to be a multi-turn prompt, right?

509. 59:19

     So this is, like, a conversation. You are talking to assistant.

510. 59:27

     It said execute greater than p assist. Where's that?

511. 59:32

     So basically at the top you see three roles being defined, and then you can decide who you want to talk to. Uh, uh, oh, thanks so much. So you s- see three roles at the top?

512. 59:45

     Yeah.

513. 59:45

     And then if you do that p assist you see down there that's highlighted in, in yellow-

514. 59:49

     Yeah

515. 59:49

     ... you can just do, um, that little arrow p and then-

516. 59:53

     Yeah

517. 59:54

     ... you can pick a different persona.

518. 59:55

     Yeah.

519. 59:55

     And then you can have them talk between themselves or you can just switch be- we use these for designers i- in, in our shop-

520. 1:00:03

     Mm-hmm

521. 1:00:03

     ... to do synthetic interviews to synthetic users basically.

522. 1:00:07

     Gotcha.

523. 1:00:07

     It allows us to switch back and forth.

524. 1:00:09

     And then, uh, what, what issues or, uh, troubles have you been having with this?

525. 1:00:14

     I'm guessing you have seen a lot of role playing prompts out there, so I was just wondering if you see anything that's per- perhaps not as optimized as it could be or any other best practices, practices for role playing, particularly with multiple synthetic personas within the same session.

526. 1:00:30

     Yeah. Okay. For single personas there's, there's one answer that I would give. This multiple personas thing actually I haven't worked that much with, but here's, like, off the top of my head, here is probably how I would think about it.

527. 1:00:44

     I would give all the personas to ... I would write a separate prompt for each persona, and then I would have the user's command trigger some coding logic where it would decide which bot to send the, the, which, which prompt to send that reply to.

528. 1:01:01

     So this is getting back to the thing that I said before about, like, don't do it in a prompt if you don't have to. Like, this, I mean, this prompt, like, is, like, there's a, a lot of thought that went into it, which is, probably makes it work a lot better than it would have if you hadn't

529. 1:01:12

     put as much effort into it. But I think it's gonna be easier if you just dynamically route- The query based on, uh, what the, the user said. Does that make sense?

530. 1:01:22

     It does.

531. 1:01:23

     Okay.

532. 1:01:24

     You're talking about, like, if we were to use the API instead of just the chat-

533. 1:01:28

     Yeah

534. 1:01:29

     ... or construct something like this, right?

535. 1:01:30

     Yeah, exactly. But you're doing this just in the chat.

536. 1:01:33

     This was just in the chat, but I, I appreciate, I definitely appreciate the, the note there. So maybe related to that, one of the other things is how much have you dealt with having a second thread with the API that acts as maybe the entity that's capturing inputs from multiple ones into a single thread?

537. 1:01:54

     You know what I mean? Like, let's say that I build an app and I have the user interact with these different synthetic personas, but then I have a second interaction with the API that's tying these things together into a cohesive whole.

538. 1:02:08

     Uh, I don't know if you guys have explored some of that. I'm, I'd be curious.

539. 1:02:14

     Yeah. I don't, I don't, I don't have a great answer for that one, sorry.

540. 1:02:18

     Mm-hmm.

541. 1:02:18

     I do wanna kind of test this prompt out though, just to kind of see how it goes. So, um, maybe here I would say,

542. 1:02:26

     uh, I can just say something like ... So how, how would I switch it? I could-

543. 1:02:29

     So do the, the right arrow P.

544. 1:02:33

     Yeah, right arrow P.

545. 1:02:34

     And then type Sam, and then say, "Hey ..." Yeah.

546. 1:02:40

     You're talking-

547. 1:02:40

     Okay, so now I could-

548. 1:02:40

     Then you could say, "Hey, how do you ... What's your process to look for the right to ... for the best, uh, medication pricing whenever you get sick?" Or something like that.

549. 1:02:51

     And then here in this particular case, if you switch to Joe, Joe is optimized more for, uh-

550. 1:02:57

     Okay

551. 1:02:57

     ... convenience versus cost saving, so you have two different types of users, and we can learn from.

552. 1:03:03

     Yeah, okay, so Claude did the thing here that I wanna show you all, like, how to get rid of. Uh, where-

553. 1:03:10

     Oh, yes. Yes

554. 1:03:11

     ... as Sam, it's like, that's not something that Sam would say, right?

555. 1:03:16

     Yeah. [laughs]

556. 1:03:17

     Uh, so I don't know for sure this is gonna work. I feel like a magician that's about to do a trick, but, like, I haven't practiced it. But generally, something that is, is pretty useful, uh, here is to, we could say, uh, prepend each response with the name of the current persona in brackets.

557. 1:03:49

     So one thing I'm gonna do here is

558. 1:03:53

     I'm gonna change this, like, multi-shot a little bit also because

559. 1:03:58

     if Claude sees itself not doing the thing that I told it to do ... Actually, let's, let's just redo the whole, the whole conversation, or we can take out, uh, this.

560. 1:04:06

     Um, so let's, let's just, like, run that back.

561. 1:04:10

     You are talking to assistant.

562. 1:04:11

     Nice.

563. 1:04:12

     And now we could say ... And now we could say the same thing, like, "What's your process for finding, uh, best prices for medication?"

564. 1:04:31

     Oh. [clicks tongue] Okay, so I guess we need to do this. We need to, like, change in a separate call.

565. 1:04:38

     Yeah.

566. 1:04:40

     Okay, great. Now it's gonna work. Totally. It's gonna totally work.

567. 1:04:49

     Okay, it's a little bit better, right? It didn't say as Sam. This is, like, something that human might maybe say. Like, as, as someone who's ... [laughs] Okay. [laughs]

568. 1:04:59

     I don't know. Um, it's better than it was before, right? Uh, maybe we could say something like-

569. 1:05:07

     Can you just preview that?

570. 1:05:10

     You don't need to, uh, say too much about your persona in your responses. Just stay in character.

571. 1:05:22

     Hey, quick question. What are your thoughts on using things in the negative sense versus-

572. 1:05:26

     Yo, check it out. It worked a lot better. Sorry to interrupt.

573. 1:05:28

     Oh, yeah. [laughs] Very nice. Very nice. [laughs]

574. 1:05:32

     Um, yeah, so-

575. 1:05:36

     Yeah, what are your thoughts on using, like, negative stuff, like you don't, versus the positive sense?

576. 1:05:41

     Yeah, I think positive is, like, a little bit better. In this case, I don't really have a good answer for why I phrased this negatively. Uh, I guess I did a combination.

577. 1:05:50

     I was like, you don't need to say too much, just stay in character. Um, I guess I think it's better to use, like, a light touch, like, if you're doing negative prompting.

578. 1:05:59

     I think there's, like, a, a little bit of a thing going on with reverse psychology where if you tell the model, like, "Don't talk about elephants. Don't ... Definitely no elephants.

579. 1:06:07

     Definitely don't say anything about elephant," it might make it more likely to talk about an elephant. So if you do use negative prompting, I think it's better to have, like, a, a light touch where you just kinda say it once, but, like, don't dwell on it too much.

580. 1:06:18

     Um, also something similar with parenting. It's like if you don't want your kid to eat prunes, you're just like, "Oh, we're not having prunes today," and then you just change the subject.

581. 1:06:25

     But if you really, like, emphasize that there are, like, no prunes to be had, then you might get more, uh, pushback.

582. 1:06:32

     Hi there. Um, I noticed you're not using the system prompt much. Like, is there a reason for that, or what do you think the biggest value items for a system prompt are?

583. 1:06:40

     Yeah, system prompt. Personally, the only thing that I ever put in the system prompt is a role. So I might say, like, you are this, you are that. I think generally Claude follows instructions a little bit better if they're in the, the human prompt and not in, in the system prompt.

584. 1:06:55

     Uh, the exception is things like tool use where maybe there's been some explicit fine-tuning on, like, certain system prompts, uh, specifically. For, like, general prompts like the ones we've been going over here though, I don't really think you need to use the system prompt very much.

585. 1:07:12

     Yeah.

586. 1:07:13

     Yeah. One thing we've found when using the user prompt ... Thank you. One thing we've found when using the user prompt, I guess sometimes is it makes it more prone to hallucinations because it thinks the user is saying it, and so we-

587. 1:07:27

     Migrated things to the system prompt more? I don't know if you have any experience with that

588. 1:07:31

     Yeah, yeah. I- I've actually heard that before.

589. 1:07:34

     So it's possible I'm missing something. I've heard this from enough people that I could just be wrong, so I'm u- unusually likely to be wrong when, when I say this.

590. 1:07:42

     I think that if you just put a bunch of context and you're like, "Here's the message from the user," open user bracket, and then put the message, and then close the user bracket, uh, it will, it will work and you won't have that issue anymore.

591. 1:07:54

     That said, like, uh, I don't know, may- maybe it does, uh, f- fall over sometimes, but that would be my default is just to, like, specify even more clearly in the ...

592. 1:08:04

     If you're having this issue, be like, "Here's the message from the user. Here's the stuff that I want you to do," and I think it's le- it probably won't get confused by that.

593. 1:08:13

     Uh, I have a question about the, uh, counter examples. So before, in order to get it to say not cringey, cringey things, you were saying provide it with a counter example.

594. 1:08:23

     So but here in the case of where you're doing this, uh, character bot, you haven't provided it any counter examples.

595. 1:08:31

     Yeah.

596. 1:08:31

     So th- this is sort of like a generic question. So if the model is trained on preference optimization with, uh, examples and counter examples, do you get a better result in the prompting?

597. 1:08:44

     Well, uh, I don't, I don't know that the details of the RLHF have that much bearing because

598. 1:08:52

     I, I think when the model's trained it, it doesn't usually see those both in the same, like, window. It, it's more that it, it, it's like some stuff that happens with, like, the, the RL algorithms.

599. 1:09:02

     So I don't think that's necessarily the right way to think of it. With counter examples, I don't feel that I have to include them in every prompt. It's just a tool that I have in, in my toolbox that I'd use sometimes.

600. 1:09:13

     I- in regards to, like, negative prompting, uh ... I'm over here. Hi.

601. 1:09:17

     Oh.

602. 1:09:17

     Uh, do you think that it would be better to do negative prompting using control vectors like what you, uh, what you talked about in your, uh, scaling mono-semanticity paper?

603. 1:09:27

     Uh-

604. 1:09:28

     Yeah

605. 1:09:28

     ... and maybe, like, having, like, a negative version of the vector as your kind of negative prompt instead of mentioning it in the prompt outright?

606. 1:09:35

     Yeah. Uh, steering is, is, is still, like, super new. We don't know how well it works relative to prompting.

607. 1:09:43

     Mm-hmm.

608. 1:09:43

     I'm like a, a, you know, diehard prompter till the end, so. [laughs] Uh, or I guess I, I've played around with it a little bit. I, I haven't, I haven't found it to work as well as, as prompting in my experience so far.

609. 1:09:53

     Mm-hmm.

610. 1:09:54

     That said, there's, like, a lot of research improvements that I won't get into in too much detail, but there's a lot of stuff that could make it work better than ...

611. 1:10:01

     So, like, right now it's, like, finding these, these features and then you're steering according, uh, to the features, which are sort of like these abstractions on top of the underlying vector space.

612. 1:10:10

     Yep.

613. 1:10:11

     There's other possibilities for how you could steer and there's, like, academic papers that you can read where you're steering according to just, like, the, the differences in the activations versus, like, trying to pull it out to this feature first.

614. 1:10:24

     So maybe that would work a, a bit better, like the control vectors thing.

615. 1:10:27

     Yep.

616. 1:10:27

     I haven't played with it enough to know for sure. Uh, but I'm ... I, I think there's definitely, like, something along that line, those lines will work eventually. I can't say in the long term if it'll work better or, or worse than prompting.

617. 1:10:38

     Right now I still think prompting works, like, a lot better.

618. 1:10:40

     Mm-hmm. I mean, from my experience with, like, smaller models and trying to work with control vectors, I've seen that it's better when it comes to style-

619. 1:10:46

     Mm-hmm

620. 1:10:47

     ... uh, than it is for, like, actual deterministic prompting.

621. 1:10:51

     Yeah, pretty interesting.

622. 1:10:52

     Yeah.

623. 1:10:52

     Sometimes I feel like stuff from smaller models transfers. Sometimes it doesn't transfer.

624. 1:10:56

     It's ki-

625. 1:10:56

     I don't have a great intuition for what does and doesn't transfer between small and large models, but yeah, it's-

626. 1:11:00

     All right

627. 1:11:01

     ... good points.

628. 1:11:01

     Thank you.

629. 1:11:03

     Okay. I think we've, uh, gone over this role play stuff enough. Let's go to the next one. I'm gonna upload a few screenshots of my dating profile. Okay, this is our first image one.

630. 1:11:15

     Are there any screenshots? No screenshots. Uh, okay. Actually, somebody responded in the very first, uh, in reply, so since we're doing images, uh, maybe I'll start there. Um ...

631. 1:11:29

     If you scroll down, it's that, that was me.

632. 1:11:32

     That was you. Uh, let me try and find my message here. All r- You said it's at the bottom of the ...

633. 1:11:38

     Yeah. In that channel I reposted it somewhere.

634. 1:11:49

     Riveting to watch me scroll through this channel, I'm sure. [laughs]

635. 1:12:00

     Oh, there's a lot of messages here. Okay, here we go.

636. 1:12:08

     Okay, so let's ... Um, can I copy the image?

637. 1:12:15

     Copy image. Okay, cool. I actually don't know if I can paste it into the console, so I might fall back on using Claude.ai for this.

638. 1:12:25

     Paste images is not enabled right now. Okay, let's try Claude.ai then.

639. 1:12:35

     Um, so then this ... The question wa- or the, the, the prompt here was,

640. 1:12:42

     "High performing validated AI model." Sorry, I, like, lost all your formatting here.

641. 1:12:58

     Okay, so in this image here, if we zoom in, it's supposed to get Mattie White

642. 1:13:04

     and 86... Hmm, I'm having a ha- hard time reading this.

643. 1:13:10

     Uh, 8687. That's the hard one that it messes up on quite frequently.

644. 1:13:13

     8687 and then 8687 down here. And then in this one it's just 867.

645. 1:13:18

     Yeah, they ... That's a typo-

646. 1:13:19

     That's a typo

647. 1:13:20

     ... by the human. And so I'm hoping it can correct for that. I'm just trying to pull out-

648. 1:13:24

     Yeah

649. 1:13:24

     ... kind of the average data-

650. 1:13:26

     Yeah

651. 1:13:26

     ... from all three combined.

652. 1:13:30

     Okay. It's, and it said ... It looks like it said middle mark, so it's, it's misreading the ... Okay. So I don't know too many good tips for image is, but I'll, I'll tell you what I have.

653. 1:13:42

     So one of the things that you said ... Oops, sorry. Um,

654. 1:13:47

     one of the things that you said here was, uh, that it works better with zooming in and cropping the image. That's definitely, like, the easiest win that you can have, is just giving it, like, a higher quality image, taking out the unnecessary details.

655. 1:14:04

     That might be hard to do programmatically because it's like you don't know what the s- the, which details are necessary and unnecessary.

656. 1:14:09

     Mm-hmm.

657. 1:14:09

     But for the same reason that including extraneous information in text form, you probably won't get as good results. If you include extraneous information in, in image form, the results probably won't be as good, uh, either.

658. 1:14:22

     So the more that you can, like, narrow in on the exact information that you need, the better. Um, is the model down-sampling large images? I don't know/can't talk about that, but definitely having, like, higher quality, bigger images is, is better.

659. 1:14:37

     I, I did just read on your website that it says it dow- it down-samples to 1,000 pixels by 1,000 pixels.

660. 1:14:43

     Okay, great.

661. 1:14:44

     That can be found with Google. [laughs]

662. 1:14:46

     Okay. So then any general tips on how to discuss images with Sonnet? Um, my number one tip for images is to start by having the model describe everything it sees about the image.

663. 1:14:59

     So I don't know if that will work here. This, this example is, like, this one is hard enough for me to even read that I, like, I kind of doubt that Claude will do well on it regardless of, of what we say.

664. 1:15:07

     But we can, we can, uh, give it another shot.

665. 1:15:10

     One, one thing I've noticed when I attempted that where if I asked it to go, like, tube by tube in that image, if it, like, the first tube, if it came to the wrong conclusion, it would use that and come to that same wrong conclusion on multiple tubes, where, like, it, it made it so there was some

666. 1:15:27

     kind of, like, directionality in its thinking where if it got the answer wrong at first, it would project that onto the rest of the image it was analyzing.

667. 1:15:33

     Yes. I think that's definitely true. If, if the model starts off on the wrong path, it probably would just continue going on the wrong path. It's trying really hard to be self-consistent.

668. 1:15:42

     Mm-hmm.

669. 1:15:42

     And it's, it's ... And this is why, uh, uh, self-correction is such a big, like, frontier for LLMs in general right now.

670. 1:15:53

     But, but as of this month, why use Claude for this task? So I love to use Claude, but why for, for this specific image extract task?

671. 1:16:03

     Um, I found, like, text track and OCR models I've played with don't do as well with some of the handwriting. Um, like, if I zoom in on this image, it actually does perform pretty well, um, from some of these fairly messy handwriting that's even hard for a human.

672. 1:16:17

     Okay. Yeah, unfortunately, I don't know how to get better results out of this other than maybe by, by cropping it better and, and up-sampling.

673. 1:16:24

     Okay, cool. Thank you.

674. 1:16:27

     Oops. Um, here we are. So let's scroll up to where we were before.

675. 1:16:51

     Okay, yeah, still no screenshots there. How do I enable fragments?

676. 1:16:57

     Uh, what are fragments?

677. 1:17:02

     The

678. 1:17:02

     Is fragments like the pre-fill part? The pre-

679. 1:17:05

     No, the beta feature you guys have that-

680. 1:17:07

     Artifacts.

681. 1:17:08

     Artifacts. Oh, okay. Uh, it, it's just a setting. It's in the bottom left of the ... You have to en- enable it.

682. 1:17:14

     Find it online somewhere.

683. 1:17:15

     Yeah, you can find it online, or people will help you. Uh ...

684. 1:17:18

     So I ... Okay. Sorry. Can you just get my ...

685. 1:17:22

     Yeah. So I often dump full traceback errors directly into the prompt box. I often dump full traceback errors directly into the prompt box as API. It seems exceptional at not running into traceback loops.

686. 1:17:33

     I don't know, like, if that's intentional. Like, I ... Literally, I'll just take the entire traceback, zero context to Claude, and I'll just dump the entire thing in.

687. 1:17:45

     Yeah.

688. 1:17:46

     And then it'll, it'll, it'll give me the fix.

689. 1:17:48

     Okay, great.

690. 1:17:49

     So but, is it ... Can you, can you, can you elaborate on how that might have ... how this ... For example, like, this only really appeared with, like, very recently.

691. 1:18:00

     Like, you'd have to explain explicitly a lot for-

692. 1:18:03

     The models get better, man. They get better every, every iteration.

693. 1:18:06

     I understand, but this is, like, you know, this isn't prompt. Like, this is prompt engineering we're talking about, right?

694. 1:18:10

     Yeah.

695. 1:18:10

     So, uh, I'm just, I'm just wondering, like, is this, is this a form of prompt engineering, or is just the model being good?

696. 1:18:18

     Sounds more like the model being good if you're just dumping it in.

697. 1:18:20

     Okay.

698. 1:18:20

     All right, I'm gonna move to this, this prompt here.

699. 1:18:22

     Okay.

700. 1:18:23

     So, uh, the ... To the person who uploaded this, do you ... And, uh, generally, uh, to anyone who's uploaded, uh, more examples, if you could just, like, put some stuff in the thread, like write some stuff about what the issue is that you're having or, like, why it's not working, uh, that would be great.

701. 1:18:43

     Well, I'm gonna suggest ... This is actually a follow-up to, like, what I was doing with the translation. So basically what I'm trying to get it to do is to actually analyze the text.

702. 1:18:54

     So, like, in this case, you know, there's original English.

703. 1:18:57

     Yeah.

704. 1:18:57

     There's a bad Japanese translation. And I'm trying to get it to score between one and five how good the translation is. And so what I've been doing is adding a lot of stuff to try to get it to do sort of chain of thought to-

705. 1:19:08

     Yeah

706. 1:19:08

     ... you know, see, like ... Because it'll notice errors, but, you know, it just generally does a very bad job at scoring.

707. 1:19:14

     Mm. Yeah, okay, so this is great. Uh, I'm really glad that you asked this, 'cause model grading is something that ... I mean, it, it would be incredibly useful if it worked, and right now it's in a place where it, like, sometimes kind of works and sometimes doesn't.

708. 1:19:28

     So let's, uh, paste this into the console

709. 1:19:33

     Okay, so we have some English text and we have some Jap-

710. 1:19:37

     And it's on the low

711. 1:19:37

     Yep, yep, yep. Then we got the Japanese text.

712. 1:19:54

     So is this a good translation or a bad translation?

713. 1:19:57

     Terrible translation.

714. 1:19:57

     Terrible translation, okay. Now, Claude's actually supposed to be good at Japanese too, so...

715. 1:20:05

     No, it's much better, yeah. This is, uh-

716. 1:20:05

     Oh, this is somebody else. But I'm saying if Claude is good at Japanese, it should be good at, like, judging other people's Japanese, in theory.

717. 1:20:12

     In theory.

718. 1:20:12

     In theory. Okay, so now that we've got the answer here, so,

719. 1:20:18

     um, I guess it's stalled out here for whatever reason. So what are we doing in this prompt? We're scoring betw- between one and five as below.

720. 1:20:27

     One is many grammatical errors the native would never make, contains multiple grammatical errors, an average quality w- translation with some errors. Okay, that, that looks pretty good.

721. 1:20:38

     Look for specific clues or indicators. I don't know why it, uh ... I think that's just a, a bug. I think it'll conclude here.

722. 1:20:55

     Okay, so it gave it a three, but we want it to give a one.

723. 1:20:59

     Totally, yeah.

724. 1:21:00

     Okay. Now, have you found that it's generally too forgiving or too strict, or that it's just all over the place?

725. 1:21:09

     Well, it's all over the place. Also, it seems to, um, confuse sometimes content, uh, versus, uh, you know, so, like, for example, this is from, I think, the HLRF, uh, you know, set, so, you know, the content is fine, but so it might not actually rate it low even though it's a terrible translation because it thinks the

726. 1:21:27

     content is okay, even though if you asked it even to list all the errors-

727. 1:21:32

     Gotcha

728. 1:21:32

     ... there are a dozen errors in that, you know, single piece of text-

729. 1:21:35

     Gotcha

730. 1:21:35

     ... for the translation.

731. 1:21:36

     Yeah, yeah.

732. 1:21:37

     Uh, so that's why I'm trying to also see if, you know, is there anywhere to, like, get to separate out, um, the grammatical-

733. 1:21:44

     Yeah

734. 1:21:44

     ... or the actual translation errors versus-

735. 1:21:46

     Yeah. Okay, a few thoughts. So generally, this is, like, a thing where if you a- ask the model if some text is, like, good or bad, it sort of ...

736. 1:21:56

     If the text is, like, about a nice subject, it's more likely to say that it's good in all ways. Like, it's a good translation, like, it's well-written, it is ...

737. 1:22:03

     It flows very logically. And if it's about, like, a negative subject, it's more likely to, like, criticize it and say that it doesn't flow well.

738. 1:22:11

     Yep.

739. 1:22:12

     I think you can get at, uh, those issues by typing language about it in the prompt. So for instance, here you might say something like, um ...

740. 1:22:25

     Is one or five good? You know, you don't specify that.

741. 1:22:28

     Hmm?

742. 1:22:29

     What ... Is one or five the best score? Uh, one is the worst and five should be the best.

743. 1:22:34

     That's sort of, like, implicit in, uh, this rubric up here.

744. 1:22:42

     But it might be good to say it anyways.

745. 1:22:44

     The problem with that is if we're interrupting Claude, how do you get around the fact that you-

746. 1:22:51

     May not have a Japanese tokenizer?

747. 1:22:55

     Yeah.

748. 1:22:56

     Do you have a Japanese tokenizer in Claude?

749. 1:22:58

     Claude, uh, I mean, the API will, will tokenize anything.

750. 1:23:01

     But how is it trained?

751. 1:23:04

     How is it trained? Um-

752. 1:23:06

     Yeah. Pre-trained

753. 1:23:06

     ... o- off, off topic for ... [laughs] And also, I don't know.

754. 1:23:09

     I don't think ... I actually don't think there is a tokenizer for Claude. Like-

755. 1:23:15

     Oh, so-

756. 1:23:16

     I, I mean, unless you say otherwise, there's, there's not

757. 1:23:18

     I mean, if you upload, if you upload some text, it will be tokenized.

758. 1:23:22

     But pre-t- it's not pre-trained, so you're not gonna get a really good answer for this. Or Japanese text. I've-

759. 1:23:28

     Uh-

760. 1:23:28

     ... I've tested this.

761. 1:23:29

     Okay.

762. 1:23:29

     Claude, Claude speaks the best Japanese of any model available, actually.

763. 1:23:34

     We ... Yeah, we don't need to debate, like, Claude's Japanese skill. Let's just, like- [laughing]

764. 1:23:40

     Uh ...

765. 1:23:40

     Uh, but the tokenizer isn't available, and so that it would be interesting.

766. 1:23:45

     Okay.

767. 1:23:45

     There's, there's a lot of work to be done. That's cool. Uh, it's not like slamming Claude, but-

768. 1:23:51

     Yeah.

769. 1:23:52

     Um, I have a question about programming.

770. 1:23:54

     Sorry, can we actually, uh, c- cut the questions off while I type this-

771. 1:23:57

     So sorry

772. 1:23:58

     ... this prompt here? Um, okay. So it's extra important to ... So what we're trying to do here is get it to distinguish between the, the, like, ethical nature of the text and the quality of the translation, so ...

773. 1:24:11

     Is it useful to tell it to be, like, extra critical? Say, like, you're grading a, uh, you know, like, I don't know, like, graduate course, you know, level-

774. 1:24:20

     Sorry, I'm, I'm a little bit all over the place. I need to, like, type this out before I can clear the queue and, like, respond to other questions here.

775. 1:24:26

     So it's important to distinguish between, um, the ...

776. 1:24:42

     Uh, what's a good word here? Like, um,

777. 1:24:47

     uh, risque topics or, um, R-rated ...

778. 1:25:12

     Yeah. So I don't know, something like this could help the model, uh, not pay so much attention.

779. 1:25:20

     The main thing that I would wanna do for this prompt is just add a bunch of examples. So for each category, I would add at least one example of that category.

780. 1:25:31

     So, like, I'd have, like, a really bad translation, and I'd say why it's a one, and you can say that in your own words, and I'd have an example of Like a, a two-level translation, a three-level translation, and so on.

781. 1:25:43

     Okay.

782. 1:25:44

     And in each case, before you get to the answer, you would have the explanation for why it's good or bad. I can't tape all that here, a- among other reasons, 'cause I don't speak Japanese.

783. 1:25:54

     Mm-hmm.

784. 1:25:55

     But I think that's the most valuable thing that, that you could do here. O- otherwise, I mean, like, the formatting looks really good. The fact that you're doing the chain of thought in advance looks good.

785. 1:26:05

     I, I think m- mm... Yeah, I think maybe this specific clues or indicators, I think you could go into a little bit more detail here about what is-

786. 1:26:20

     Okay

787. 1:26:20

     ... how these things contribute to the overall-

788. 1:26:21

     Basically just end shot every single, uh, example or, you know, grade-

789. 1:26:25

     Yeah

790. 1:26:25

     ... with an example, basically.

791. 1:26:26

     Yeah. I mean, it's, it's tedious to write out all these examples, but A, Claude can help, and then you have the problem of just editing Claude's response, uh, versus, like, writing it all yourself, and B, it really does lead to better performance versus, like, almost anything else that, that you could do.

792. 1:26:41

     Okay, thanks.

793. 1:26:41

     Do you think it's better to use a number scale or ask it to say, like, good, bad, you know?

794. 1:26:47

     Yeah. In terms of the scale on these, on these rubrics, I think either a number scale or good, bad is fine. The thing I'd be careful about with the number scale is that I don't think it's very well calibrated, so if you're telling it, like, "Choose a number from one through 100," it's not gonna be, like, a

795. 1:27:05

     unbiased estimator necessarily. So I'd probably just limit the granularity to maybe five different classes.

796. 1:27:13

     Yeah?

797. 1:27:14

     Uh, one thing we haven't really talked about at all here, but going back to, like, your questions around the API is, like, this is a case where, uh, you might be able to utilize, like, log probs, and I'm wondering if that's something you ever use in any of your work or other prompts.

798. 1:27:29

     Yeah, I think ... I, I agree this is, could be a case where log probs would be useful if you could get, like, the probability of each, each grade.

799. 1:27:38

     Yeah.

800. 1:27:39

     The thing-- So here's the thing with log probs, is you really want the chain of thought beforehand.

801. 1:27:45

     Mm-hmm.

802. 1:27:45

     And the chain of thought, I think, is gonna get you more of a win than using the log probs. But log probs, th- there's, there's no ... In, in, in any model, I don't think there's a way where you can just say, "Sample all the way through, and then output ...

803. 1:27:59

     After you output this, like, closed chain of thought tag, then give me the log probs of whatever comes next."

804. 1:28:04

     Mm-hmm.

805. 1:28:05

     So if you are gonna use the log probs, then you're looking at this, like, multi-turn setup where you first sample all the, the chain of thought or sample all the, like, pre-cogitation that it, that it wants to do.

806. 1:28:15

     Mm-hmm.

807. 1:28:16

     And then cut it off there, and then re-upload that with the chain of thought as a prefill message.

808. 1:28:22

     Mm.

809. 1:28:22

     And then you could get the log probs. But you ... For that you need a model that both has, uh, the, the prefill capacity and has log prob capacity.

810. 1:28:31

     I'm not sure of, of what model has both of those characteristics right now.

811. 1:28:34

     Yeah. Walk me through why it wouldn't just be sufficient to, like, in this case, just ask for ... I'm asking for a score one to five, only return that, but then, like, look at the log probs of what it returns in that case.

812. 1:28:46

     Yeah. So the, the, the ... What you're losing there is whatever intelligence boost you got from having the model do the chain of thought. And my sense is that chain of thought plus have the model say either one, two, three, four, or five is gonna be more accurate than, like, the more ...

813. 1:29:00

     the additional nuance that you'd get by having it give you the log probs because it's actually doing a lot of its thinking in that chain of thought. You're, like, leveraging more computation.

814. 1:29:09

     You're getting more forward passes for all the same reason that chain of thought is, like, usually a good idea. It's a good idea to-

815. 1:29:14

     I see. Are you talking about, like, chain of thought and, like, it's actually out loud writing a chain of thought before the answer?

816. 1:29:20

     Exactly.

817. 1:29:20

     Okay, okay.

818. 1:29:21

     Exactly.

819. 1:29:21

     Sure, I got you. Cool.

820. 1:29:22

     Which is what we see in this prompt right here, right? Like, we have this-

821. 1:29:24

     Yeah, yeah

822. 1:29:24

     ... analysis section. So if we cut out this whole analysis section, we're really tanking the number of forward passes that the model can do-

823. 1:29:31

     Yeah

824. 1:29:31

     ... before it gave you the answer.

825. 1:29:32

     Okay, cool. That's good to know. Thank you.

826. 1:29:36

     What's that?

827. 1:29:37

     Three, two, one. You have a couple more and then-

828. 1:29:40

     Okay. Um, I'm being told that I should answer a couple more questions and then get off stage. I was like- [laughs]

829. 1:29:47

     Before I came here, I was honestly really worried that, like, no one would have questions and I'd be supplying my own, but you all have had amazing questions so far and amazing examples, so I really appreciate that.

830. 1:29:58

     It's made this go on well. Sorry, I'm supposed to say that at the end of this, but I'm giving a pre, pre thank you. Now we can do the encore.

831. 1:30:04

     Uh, yeah?

832. 1:30:04

     I just wanted to add that, um, having a numbered list may give more, um, weight to, like, number one, two, three, four, five versus just having an unstructured list.

833. 1:30:13

     Um, so it may give more weight into their score, uh, for the output if you just change it from, like, one, two, three, four, five to, like, little dashes, uh, for criteria.

834. 1:30:24

     Um, just as my experience of what I've been seeing.

835. 1:30:26

     Okay, cool.

836. 1:30:27

     Yeah, so just FYI. I have replied back with a prompt, uh, that was fixed or that, like, in my own improvement of what I think is a better prompt for this.

837. 1:30:35

     Yeah.

838. 1:30:36

     Okay, awesome. Um, Nisha, should I do another prompt or should I just a- answer a couple questions and then head out?

839. 1:30:43

     Up to you. We can do another prompt if you want to.

840. 1:30:45

     All right, let's do one last prompt. What's ... Which one should we choose?

841. 1:30:58

     Okay, let's do this one, good old mitigating hallucinations, 'cause we haven't really done that.

842. 1:31:05

     Okay. "Please provide a summary of the text provided as input." Okay, first thing I'll do is just move these instructions down.

843. 1:31:23

     Now, Matt, did you have an example of the document, a document where it hallucinates with this prompt?

844. 1:31:28

     Yeah. Um, no, but I can send you one really quick.

845. 1:31:31

     Yeah, can you just put that in the thread?

846. 1:31:42

     Okay, your summary should be concise while maintaining all important information that would assist in helping someone understand the content. If it mentions any dates.

847. 1:31:51

     Don't start or end with anything like, "I've summary, generated a summary for you. Here is the summary you asked for." Yeah, so this one we can, uh, fix with a pre-fill.

848. 1:32:20

     Okay, uh, or we could do something like this.

849. 1:32:30

     Um, now how about the hallucination part? The, the best trick that I know for getting around hallucinations in a case like this is to have the model extract relevant quotes first.

850. 1:32:41

     So what I would say do here is I would say something like...

851. 1:33:08

     And now of course in this pre-fill, since we're having, uh, relevant quotes here, we wouldn't wanna start with summary. That would just be confusing/wrong. So we could say, "Here is the..."

852. 1:33:26

     Something like this. Okay, did you get the, the doc yet?

853. 1:33:35

     Uh, one second. I'm downloading the data now. Okay. And then of course I'd put the document here.

854. 1:33:46

     Yep. Okay. Uh, I think I should get off stage. Uh, so yeah, let me just call it here, and Matt, we can, we can talk, uh, after.

855. 1:34:02

     Um, yeah, once again, I really appreciate you all coming out. It's been amazing to have such, like, a, a great audience engaged. Uh, I've had fun. I, I learned some things.

856. 1:34:12

     Uh, I hope you all did too. I'm planning to stick around, uh, this event, uh, for the next, for the rest of the afternoon, so I don't know exactly where I'll be, but maybe just DM me if you wanna come find me and chat.

857. 1:34:24

     I'm always happy to talk prompt engineering. It's, like, my truest passion at this point in the world, so, like, find me, hit me up. We'll, we'll talk. Um, and yeah.

858. 1:34:35

     This has been great. Thank you so much. [outro music]