Prompt Engineering is Dead

Talk transcript

1. 0:00

   [upbeat music] All right.

2. 0:16

   So prompt engineering is dead, and this is a bold statement, but I'm, I'm saying that it never actually existed. And if you've ever done, uh, prompt engineering, you probably know that it's, uh, kind of bullshit 'cause you kinda try to ask the LLM to act nicely and do what you want to do...

3. 0:32

   d-do what you want it to do. And so, uh, this talk, I'm gonna walk you through, uh, a story of mine where I managed to improve our kind of sample chatbot that we have in our, uh, website.

4. 0:44

   And I made it... I made... I... It was possible for me to made it, like, uh, five weeks, uh, better without actually doing any prompt engineering because, you know, it's not really engineering, right?

5. 0:54

   And so we have a chatbot in our applica- in our website. It's RAG-based, super simple. It just allows you to ask questions about our documentation and then gives you answers.

6. 1:04

   Super simple, super straightforward. The easiest RAG you can think about.

7. 1:08

   And, uh, when I deployed it the first time, it worked, uh, kind of okay. And I tried to make it work better, and you can see some examples here.

8. 1:16

   I tried to on... I, I needed to have him only answer things that related to Traceloop because this is, uh, my company. I don't want, you know, it to answer things about the weather or something else.

9. 1:27

   I want it to answer about Traceloop only, and I want it to be useful. So if someone asks a question, I want the question to be useful for the user who asked it, and I want it to make less mistakes.

10. 1:35

    It was making so many mistakes, and I want it to just get a little bit better.

11. 1:41

    And, and so we know what, what are we doing at this stage? We're doing prompt engineering. But why do I even need to iterate on prompts? I just wanna give, give it a couple of examples.

12. 1:49

    This is good, this is bad, and have it somehow learn, you know, uh, how to, how to follow my instructions properly and always.

13. 1:58

    And, and so I begin imagining, you know, I begin imagining h- I wanna build this automatically improving machine that will be kind of like an agent that will research the web, find the latest and greatest prompt engineering techniques and, uh, and then apply them to my prompt over and over and over again until I get the best

14. 2:15

    prompt for my super simple RAG pipeline. And, and to do that, to, to, to run this, uh, kind of a machine, I needed to, uh, do some... a bit more work, right?

15. 2:26

    'Cause I need... We've, you've all been in a lot of conversations here about evaluators, so you know that we need an evaluator so the ma- crazy machine can actually know how to improve and if it's actually improving.

16. 2:39

    And so, uh, to do that, I need to create a dataset of, like, questions I wanna ask my, uh, chatbot about the documentation. Uh, and then I wanna build an evaluator that can evaluate how well my RAG pipeline is responding to those questions.

17. 2:52

    And then I have the agent that kind of like iterating on the prompts until I get to the best, uh, prompt ever. And so this is kind of like how it will look like schematically, right?

18. 3:02

    I have the RAG pipeline, I have the evaluator, and then I'm gonna have my auto-improving agent. Let's begin. RAG pipeline, super simple. Have a Chroma database, have a OpenAI, and some simple prompts that, uh, you know, take user question, find relevant documents in the, uh, Chroma database and just, uh, uh, output an answer.

19. 3:25

    Uh, so we'll see, I'm just... It's a super simple one. I just ask it a question, "How do I get started with Traceloop?" And then, uh, it runs, takes a couple of seconds, and then we're gonna see an answer and we're gonna see the trace.

20. 3:36

    Just so you... If you never, if you've never seen a RAG pipeline, and I'm guessing you all saw a RAG pipeline, this, you know, this is how it looks like, right?

21. 3:44

    A couple of calls to OpenAI, Chroma database, and then, you know, at final stage, we get like a... the, all the context into OpenAI, and we get the final answer, uh, to the user, who was great, and we have a couple of prompts here we'd probably want to optimize.

22. 3:58

    Wait, this is not what I wanted to...

23. 4:00

    Uh, great. Okay. Now let's go to the next step, my evaluator. So what do we need for evaluation? So this is not a talk about evaluators. I'm sure you've heard a lot of talks about evaluators, uh, in this conference, but, uh, I'm gonna tell you what, what kind of evaluator, uh, did I choose, uh, to use.

24. 4:22

    Uh, so first, you know, you need a dataset of questions and, and, you know, a way to evaluate these questions, and then the evaluator is gonna kind of invoke the, the RAG pipeline and then get answer from the RAG pipeline, and then gonna kind of evaluate and get a score, maybe a reason if why the score is

25. 4:39

    low or high. And then this is kind of what we're gonna use for the agent, which will be the last step that can auto-improve, uh, the prompt.

26. 4:47

    And so, you know, there's a couple of eva- there's a lot of types of evaluators. I think we've been talking a lot about LLM-as-a-judge, and I'm... Spoiler alert, I'm going to choose an LLM-as-a-judge here because it's easier to build and it's easier to deploy.

27. 5:00

    But there are also a lot of different types of evaluators, kind of like classic NLP metrics, you know, if you wanna do something which is embedded based on, or all of these colors, uh, uh, that can also help you, uh, evaluate different types of tasks, like translation tasks and, uh, and, uh, even, uh, text summarization tasks.

28. 5:19

    And I think the main difference that I see, uh, between, you know, classic NLP metrics and LLM-as-a-judge is that class- classic NLP metrics usually require some ground truth. And so I need to have, you know, my questions and then the actual answers.

29. 5:33

    And then if I have the actual answers when I'm invoking my RAG pipeline, I can actually get, you know, I can compare the ground truth answer against what the...

30. 5:40

    whatever the RAG pipeline return. Great. And then the LLM-as-a-judge, they can work with the ground truth. Sometimes you can build a judge that can judge an answer based on some real answer that you expect, but they can also, uh, they can also, uh, try to assess an answer just based on, like, the question and the context without

31. 5:57

    any, any ground truth. For my example, because I ha- I know the dataset and know everything, I'm going to build a ground truth based LLM-as-a-judge.

32. 6:07

    Um, but before we're gonna do that, I'm gonna talk to you about where can we evalu- where can we run these evaluators, right? So if we have an evaluator, what can we evaluate?

33. 6:16

    I'm talking like I'm, I'm saying that I wanna evaluate my RAG pipeline, but what exactly do I mean? So a RAG pipeline is basically two steps, right? We get the data from the vector database, and we run the, uh, call OpenAI with some context, uh, from the vector database.

34. 6:32

    So we can kind of evaluate each step separately. You can evaluate each one, and we kind of call it like unit testing, right? We can even evaluate how well the vector database is fetching the data that I need to answer the question.

35. 6:44

    Or I can run it on, like, the complete execution of the RAG. I can take the input, the question, and take the answer, the final answer that I'm getting from the, from the RAG pipeline, and just evaluate, uh, how well, uh, the answer is given that question.

36. 6:57

    And I can also di-di- uh, dive deeper into everything that's happening in the internals, look at the context, look at the question, look at the answer, and everything all together try to evaluate, given the c- given the context and given the question, how well is the answer performing.

37. 7:13

    So again, I'm gonna do a simple LLM as a judge, and I'm gonna take twenty examples of questions that I've, uh, created. And for each question, I'm gonna write what do I expect the answer to contain.

38. 7:25

    So we're gonna have, like, three facts that e- the answer that was generated by the RAG pipeline should have. And then the evaluator is simply going to take the answer that we've gotten from the rag- RAG pipeline and make sure that the facts actually appear in the answer.

39. 7:39

    And so we're gonna get, you know, per each fact, we're gonna get pass or fail, so it's a b- Boolean, uh, uh, response, and then a reason. So if it failed, then we're-- I wanna see a reason, like, why the judge thinks that the, uh, fact is not, does not appear in, uh, in the answer that we

40. 7:55

    got. And so then we're gonna get a score which is numeral, which is great because we like w- working with numeral scores. Uh, which is kind of like summarizing all the facts across all the, all the examples.

41. 8:06

    So we have three facts times twenty examples, it's sixty, sixty total facts that we need to evaluate. I'm gonna just check how many facts were right out of the total facts that we expected to have in the RAG-generated answers.

42. 8:20

    Uh, so let's see it in action. Uh, and as you can see, this is like the questions and the facts and everything. This is what I, uh, I, uh, I created to, to make sure that the RAG operates.

43. 8:31

    And then I'm gonna run it, uh, the evaluator. And what you'll see that the evaluator is doing is running evaluations, right? So it's taking question, calling the RAG, getting an answer, and then checking that answer against each of the facts that I've given it.

44. 8:45

    And then I'm gonna create a score, and you will see the score, you know, slowly, uh, progressing as I'm running the evaluator. This is like a super slow process, and I don't have enough time to actually show you, uh, it to the end, but it, it works.

45. 8:58

    It takes just a couple of minutes. Okay.

46. 9:03

    Okay. And, and... Okay. Great. Less-- Last step is our, uh, agent, the agent that will optimize the prompt. We have everything set up, now we can actually build the agent.

47. 9:19

    So I'm gonna build a researcher agent, and it's gonna take, you know, as I said, like, prompting guides online. It's gonna crawl the web, it gonna find prompting guides.

48. 9:28

    It's gonna get an initial prompt, and then it's gonna take, you know, it's gonna take... It's gonna run the evaluator once to get, like, the initial score. And then after we get the initial score, it's gonna get the reasons why we failed some of the evaluations.

49. 9:40

    It's gonna combine the reasons for failures plus prompting guides to get a new prompt. And then we're gonna get a new prompt. And then we're gonna feed it back to the evaluators, run it again, get a new score, and then run it again with the agent, so we can actually even improve the score a bit more.

50. 9:56

    If you've ever done kind of ML, classic ML training, this is kind of like a classic, uh, machine learning training, but with a bit of vibes.

51. 10:07

    Uh, so let's see it in action. We're gonna... Um, I've used the crewAI, uh, for, for doing that. And I'm gonna kick off, and you can see it will take a couple of seconds.

52. 10:16

    We can actually see, uh, the agent, you know, thinking and then calling the evaluator, and then the evaluator will run and get the responses and get a score. And I hope for the...

53. 10:27

    Yeah, I hope I can... Yeah, that's great. It's running the evaluators. It's ca- calculating the score. And, uh, and then, you know, the agent is running back. That I skipped a couple of s- minutes here, and then, you know, we got a, a response.

54. 10:39

    And then now the agent, the researcher agent is trying to understand, okay, why the prompt wasn't working. Can I find ways to improve RAG pipeline-based prompts? And then it's gonna, like, regenerate new, new kind of like, uh, prompt with m- maybe a bit more, uh, instructions or kind of best practices on how to write prompts.

55. 10:57

    And then you see, calling the evaluator again to get a, a new, uh, a new score. So we know if we need to, to optimize or not.

56. 11:06

    Great. Okay, this is where when I run it, uh, once. Actually, this, the initial score was, was okay. It was, uh, zero point four. And I run it just two iterations, and I got like a really, like a long kind of prompt that you expect like, uh, to see from like, uh, someone who's done a lot of

57. 11:24

    prompt engineering, is like give it a lot of instructions and telling it how to, how it should react and, like, how it shouldn't. You're like an expert in, in answering users' questions about Traceloop.

58. 11:33

    It was really nice to see it happening without me having to... I write a lot-- I wrote a lot of code to make the agent work, but I didn't do...

59. 11:40

    needed... I didn't need to do a lot of prompt engineering. I didn't need to do any prompt engineering, right? So it kind of worked. It was really, really nice, and the score actually jumped by a lot.

60. 11:49

    And I stopped at zero point nine because it means that like ninety percent of the, of the facts were correct. Great. I can stop.

61. 11:55

    So if there's something you wanna get out of this, uh, talk, is that you can also vibe, vibe engineer your prompts. You not, you don't need to manually iterate in prompts.

62. 12:05

    You just need to build evaluators and getting-

63. 12:08

    Kind of can run gradient ascend, gradient ascend on your evaluators. You have your score, you can kind of slowly try to optimize on your score, either automatically with an agent like I did, or manually by just like, uh, reading those manuals about how to write the best prompts and then fixing them again and again and again.

64. 12:27

    Some future thoughts to f- to wrap it up. We have two minutes, okay? Um, am I overfitting? So I have 20 examples, and then I run the evaluators. So maybe, maybe, maybe the prompt that I'm getting will work really well for those 20 examples, but then if I give it like another example, it will be horrible.

65. 12:46

    So yes, I was overfitting there because I just gave it the h- the entire 20, 20 examples. Ideally, we'd have more examples, and then kind of like, again, classic machine learning, split it into train-test, uh, sets or train-test-eval sets, and then run them like separately, right?

66. 13:02

    You, you take the train set, this is what you're trying to optimize, but then you also use the test set to make sure that you actually... You, you're not overfitting to your, uh, training data set, right?

67. 13:12

    So you don't want to give the, uh, optimizer too much context, unle- unless it just will know how to answer those specific questions and nothing else. Uh, I told you that prompt engineering is dead, but I've actually done a lot of prompt engineering for this demo, 'cause I needed to engineer the agent that is optimizing my prompts.

68. 13:30

    So I have... It was horrible, but I've done a lot of, uh, prompt engineering for that. Maybe, maybe I can also do this work for the evaluator prompts or f- even for the agent prompt.

69. 13:40

    It's kind of like this meta talk where who's even writing prompts here? Like, I'm using the agent to optimize itself. Maybe it will work, maybe it won't. I, I might try it some weekend.

70. 13:50

    It's kind of interesting. Uh, some links. You can try this out. Uh, it's available in our repo. It's traceloop/auto-prompting-demo, and you can run it. Uh, it should run. Uh, I tested it yesterday, and if you have any questions, you're welcome to ask me, uh, outside, or you can even book some time with me just by, uh, not

71. 14:09

    clicking this link, but just following this link. Thank you very much. [audience applauding] [upbeat music]