AI Engineer World's Fair 2025
Prompt Engineering is Dead
Read the talk
Prompt Engineering Is Dead: Building an Evaluation Loop for a Documentation Chatbot
Nir Gazit turns manual prompt revision into an automated research-and-evaluation loop, then confronts what its improved score does—and does not—establish.
From a talk by Nir Gazit
Before you start: Basic familiarity with retrieval-augmented generation and Python will help you follow the pipeline and scoring example.
A documentation chatbot that needs to behave
How do you make a documentation chatbot consistently answer the questions you want it to answer? Nir Gazit opens with a provocative claim: prompt engineering never really amounted to engineering, and he improved his sample chatbot fivefold without manually engineering its prompts. That opening characterization is broader than the fact-based score demonstrated later. The practical problem is more specific: replacing repeated appeals to the model with examples of acceptable behavior and a way to measure progress.
The application is a simple retrieval-augmented generation, or RAG, chatbot on Traceloop’s website. It retrieves documentation and answers questions about it. The first deployment worked only reasonably well, leaving three requirements to improve: stay on Traceloop topics rather than answering unrelated questions about things such as the weather, give users useful answers, and make fewer mistakes.
Manually rewriting instructions is the obvious next step. But the desired interface is simpler: show the system good and bad examples, then have it learn to follow the requirements reliably. The engineering problem becomes how to turn those examples into feedback that can drive prompt revisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put an improvement loop around the RAG pipeline
Gazit imagines an agent that researches prompting techniques online and repeatedly applies them to the chatbot’s prompts. For that machine to know whether a revision helps, it needs two things first: a dataset of documentation questions and an evaluator that scores the pipeline’s answers. The resulting system has three parts—the RAG pipeline, the evaluator, and the agent that proposes improvements.
The baseline pipeline uses Chroma to find relevant documents and OpenAI to generate an answer from the retrieved context. Gazit runs the question, “How do I get started with Traceloop?” The trace exposes the OpenAI calls and database retrieval, followed by the final call that receives the accumulated context and produces the user’s answer. This also makes the optimization surface visible: the pipeline contains multiple prompts that could be revised.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose what counts as a good answer
An evaluator runs each dataset question through the RAG pipeline, collects the answer, and returns a score, ideally with reasons explaining a low or high result. Those outputs will become the optimization agent’s feedback. Gazit chooses LLM-as-a-judge because it is easy to build and deploy for this experiment.
Classic NLP metrics and embedding-based comparisons are alternatives, including for translation and summarization. Gazit distinguishes them primarily by the reference information they need: classic metrics usually compare generated output with a ground-truth answer, whereas a model judge can also assess an answer using the question and retrieved context.
| Evaluator approach | Reference information |
|---|---|
| Classic reference-based metrics | Expected answer and generated answer |
| LLM judge with ground truth | Expected answer or facts, plus generated answer |
| LLM judge without a reference answer | Question, context, and generated answer |
For this demonstration, Gazit knows what the answers should contain, so he chooses a ground-truth-based judge. The ability to evaluate without a reference answer is available, but it is not the approach used here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decide where to evaluate
Choosing a judge does not yet specify what it will inspect. A RAG pipeline retrieves data and then generates an answer from that data, so evaluation can operate at several levels:
- Individual components: Evaluate retrieval or generation separately, much like unit tests. For retrieval, ask whether the database fetched the information needed to answer the question.
- End-to-end behavior: Give the evaluator the user’s question and the pipeline’s final answer, then assess how well the answer addresses the question.
- Behavior with internal context: Inspect the question, retrieved context, and answer together to assess the response in light of the information the generator actually received.
These views answer different questions. A final-answer score tells you whether the overall result was satisfactory; inspecting retrieval and context helps locate the source of a failure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn expected facts into a numerical objective
Gazit creates twenty questions and writes three facts that each answer should contain, yielding sixty fact checks. The judge examines the generated answer for each expected fact and returns a Boolean pass or fail. For failures, it also supplies a reason explaining why the fact is missing. Aggregating those decisions produces the optimization objective: passed fact checks divided by total expected facts.
The aggregation itself can stay small. In Python, the following function takes completed judge results, computes the fact score, and retains the failure explanations for the optimizer:
python
from dataclasses import dataclass
@dataclass(frozen=True)
class FactCheck:
question: str
expected_fact: str
passed: bool
reason: str
def summarize_checks(checks: list[FactCheck]) -> dict:
if not checks:
raise ValueError("At least one fact check is required")
return {
"score": sum(check.passed for check in checks) / len(checks),
"failure_reasons": [
{
"question": check.question,
"expected_fact": check.expected_fact,
"reason": check.reason,
}
for check in checks
if not check.passed
],
}
The judge supplies the semantic decision; the arithmetic simply aggregates it. This objective measures expected-fact coverage, not every requirement of the chatbot. It does not separately score usefulness, off-topic behavior, or all possible factual errors.
In the demonstration, Gazit shows the questions and expected facts, then starts the evaluator. It calls the RAG pipeline for each question and checks the returned answer against the associated facts. A running score updates as evaluation proceeds. The presentation does not show the full run to completion; Gazit says it takes a couple of minutes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Research, revise, and evaluate again
With the pipeline and evaluator in place, the agent has a concrete job. Gazit describes a researcher that finds prompting guides online, then combines that guidance with evidence from failed evaluations. The loop proceeds as follows:
- Evaluate the initial prompt to obtain a baseline score and failure reasons.
- Use the failure reasons and prompting guidance to propose a revised prompt.
- Run the evaluator with the revised prompt.
- Feed the new score and failures into another revision.
The important feedback is more than a scalar. A score says how the prompt performed; a failure reason gives the revising agent something specific to address. Gazit compares this process to classic machine learning training, with model-generated text revisions taking the place of a conventional parameter update.
The demonstration uses CrewAI. After kickoff, the agent invokes the evaluator and receives its results. The recording skips several minutes before showing the researcher examining why the prompt failed, producing new instructions, and calling the evaluator again. What matters in this sequence is the closed loop: a proposed improvement does not become an observed improvement until the revised prompt has been evaluated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the higher score means
The resulting prompt is long and explicit, resembling the output of substantial manual prompt work. It assigns expertise in answering Traceloop questions and specifies desired and undesired behavior. Gazit emphasizes that building the agent required considerable code, but the target prompt revisions happened automatically.
Gazit reports that the score rose from 0.4 to 0.9 after two iterations on the twenty-question, sixty-fact evaluation. He stopped at 0.9 because ninety percent of the expected facts passed. Those scores imply a gain of 50 percentage points, or 2.25 times the baseline score; they do not establish the opening fivefold characterization on this metric.
The practical lesson is to build an evaluator and optimize against its feedback. Gazit calls this a form of gradient ascent, but the analogy concerns progressively improving the score: the process revises prompt text rather than calculating numerical gradients. An agent can perform those revisions, or a person can read prompting guides and make changes manually. Either way, the evaluator supplies the common test of whether a change helped.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The optimizer still needs an independent test
A prompt can improve on the examples used for optimization while remaining poor on new questions. Gazit explicitly acknowledges overfitting: the optimizer had access to all twenty examples. The reported gain therefore does not establish performance on unseen questions. His proposed remedy is to collect more examples and divide them into training and test sets, or training, test, and evaluation sets. Optimize against the training examples, then assess performance separately on held-out data.
Giving the optimizer too much example-specific context risks teaching it how to answer only those questions. There is also a second limit to the headline: Gazit did considerable manual prompt engineering to build the agent that optimizes the chatbot’s prompts. The work moved up a level. Optimizing the evaluator’s prompts or the optimizer’s own prompts might automate more of that work, but he presents those possibilities as future experiments, not demonstrated capabilities.
The implementation is available in Traceloop’s auto-prompting demo, which Gazit says he tested the day before presenting. For readers following the repository today, its current code should not be treated as a frozen copy of the recorded run: the optimization flow completes above 0.8 and limits retries, while the optimizer crew configures a specific document as research knowledge rather than establishing unrestricted web crawling. The recording ends with the repository handoff and an invitation for questions outside the session, without demonstrating deployment of the optimized prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and setup instructions for the CrewAI demonstration, including data loading, RAG execution, and iterative prompt optimization.
Current documentation for retrieving documents through embedding similarity search in Chroma.
Current guidance for configuring agent roles, goals, tools, and collaboration in CrewAI.
Further reading
Gazit describes the documentation-chatbot experiment and the evaluation and optimizer work behind it, with a link to the presentation.
Read the complete timestamped transcript
- 0:00
[upbeat music] All right.
- 0:16
So prompt engineering is dead, and this is a bold statement, but I'm, I'm saying that it never actually existed. And if you've ever done, uh, prompt engineering, you probably know that it's, uh, kind of bullshit 'cause you kinda try to ask the LLM to act nicely and do what you want to do...
- 0:32
d-do what you want it to do. And so, uh, this talk, I'm gonna walk you through, uh, a story of mine where I managed to improve our kind of sample chatbot that we have in our, uh, website.
- 0:44
And I made it... I made... I... It was possible for me to made it, like, uh, five weeks, uh, better without actually doing any prompt engineering because, you know, it's not really engineering, right?
- 0:54
And so we have a chatbot in our applica- in our website. It's RAG-based, super simple. It just allows you to ask questions about our documentation and then gives you answers.
- 1:04
Super simple, super straightforward. The easiest RAG you can think about.
- 1:08
And, uh, when I deployed it the first time, it worked, uh, kind of okay. And I tried to make it work better, and you can see some examples here.
- 1:16
I tried to on... I, I needed to have him only answer things that related to Traceloop because this is, uh, my company. I don't want, you know, it to answer things about the weather or something else.
- 1:27
I want it to answer about Traceloop only, and I want it to be useful. So if someone asks a question, I want the question to be useful for the user who asked it, and I want it to make less mistakes.
- 1:35
It was making so many mistakes, and I want it to just get a little bit better.
- 1:41
And, and so we know what, what are we doing at this stage? We're doing prompt engineering. But why do I even need to iterate on prompts? I just wanna give, give it a couple of examples.
- 1:49
This is good, this is bad, and have it somehow learn, you know, uh, how to, how to follow my instructions properly and always.
- 1:58
And, and so I begin imagining, you know, I begin imagining h- I wanna build this automatically improving machine that will be kind of like an agent that will research the web, find the latest and greatest prompt engineering techniques and, uh, and then apply them to my prompt over and over and over again until I get the best
- 2:15
prompt for my super simple RAG pipeline. And, and to do that, to, to, to run this, uh, kind of a machine, I needed to, uh, do some... a bit more work, right?
- 2:26
'Cause I need... We've, you've all been in a lot of conversations here about evaluators, so you know that we need an evaluator so the ma- crazy machine can actually know how to improve and if it's actually improving.
- 2:39
And so, uh, to do that, I need to create a dataset of, like, questions I wanna ask my, uh, chatbot about the documentation. Uh, and then I wanna build an evaluator that can evaluate how well my RAG pipeline is responding to those questions.
- 2:52
And then I have the agent that kind of like iterating on the prompts until I get to the best, uh, prompt ever. And so this is kind of like how it will look like schematically, right?
- 3:02
I have the RAG pipeline, I have the evaluator, and then I'm gonna have my auto-improving agent. Let's begin. RAG pipeline, super simple. Have a Chroma database, have a OpenAI, and some simple prompts that, uh, you know, take user question, find relevant documents in the, uh, Chroma database and just, uh, uh, output an answer.
- 3:25
Uh, so we'll see, I'm just... It's a super simple one. I just ask it a question, "How do I get started with Traceloop?" And then, uh, it runs, takes a couple of seconds, and then we're gonna see an answer and we're gonna see the trace.
- 3:36
Just so you... If you never, if you've never seen a RAG pipeline, and I'm guessing you all saw a RAG pipeline, this, you know, this is how it looks like, right?
- 3:44
A couple of calls to OpenAI, Chroma database, and then, you know, at final stage, we get like a... the, all the context into OpenAI, and we get the final answer, uh, to the user, who was great, and we have a couple of prompts here we'd probably want to optimize.
- 3:58
Wait, this is not what I wanted to...
- 4:00
Uh, great. Okay. Now let's go to the next step, my evaluator. So what do we need for evaluation? So this is not a talk about evaluators. I'm sure you've heard a lot of talks about evaluators, uh, in this conference, but, uh, I'm gonna tell you what, what kind of evaluator, uh, did I choose, uh, to use.
- 4:22
Uh, so first, you know, you need a dataset of questions and, and, you know, a way to evaluate these questions, and then the evaluator is gonna kind of invoke the, the RAG pipeline and then get answer from the RAG pipeline, and then gonna kind of evaluate and get a score, maybe a reason if why the score is
- 4:39
low or high. And then this is kind of what we're gonna use for the agent, which will be the last step that can auto-improve, uh, the prompt.
- 4:47
And so, you know, there's a couple of eva- there's a lot of types of evaluators. I think we've been talking a lot about LLM-as-a-judge, and I'm... Spoiler alert, I'm going to choose an LLM-as-a-judge here because it's easier to build and it's easier to deploy.
- 5:00
But there are also a lot of different types of evaluators, kind of like classic NLP metrics, you know, if you wanna do something which is embedded based on, or all of these colors, uh, uh, that can also help you, uh, evaluate different types of tasks, like translation tasks and, uh, and, uh, even, uh, text summarization tasks.
- 5:19
And I think the main difference that I see, uh, between, you know, classic NLP metrics and LLM-as-a-judge is that class- classic NLP metrics usually require some ground truth. And so I need to have, you know, my questions and then the actual answers.
- 5:33
And then if I have the actual answers when I'm invoking my RAG pipeline, I can actually get, you know, I can compare the ground truth answer against what the...
- 5:40
whatever the RAG pipeline return. Great. And then the LLM-as-a-judge, they can work with the ground truth. Sometimes you can build a judge that can judge an answer based on some real answer that you expect, but they can also, uh, they can also, uh, try to assess an answer just based on, like, the question and the context without
- 5:57
any, any ground truth. For my example, because I ha- I know the dataset and know everything, I'm going to build a ground truth based LLM-as-a-judge.
- 6:07
Um, but before we're gonna do that, I'm gonna talk to you about where can we evalu- where can we run these evaluators, right? So if we have an evaluator, what can we evaluate?
- 6:16
I'm talking like I'm, I'm saying that I wanna evaluate my RAG pipeline, but what exactly do I mean? So a RAG pipeline is basically two steps, right? We get the data from the vector database, and we run the, uh, call OpenAI with some context, uh, from the vector database.
- 6:32
So we can kind of evaluate each step separately. You can evaluate each one, and we kind of call it like unit testing, right? We can even evaluate how well the vector database is fetching the data that I need to answer the question.
- 6:44
Or I can run it on, like, the complete execution of the RAG. I can take the input, the question, and take the answer, the final answer that I'm getting from the, from the RAG pipeline, and just evaluate, uh, how well, uh, the answer is given that question.
- 6:57
And I can also di-di- uh, dive deeper into everything that's happening in the internals, look at the context, look at the question, look at the answer, and everything all together try to evaluate, given the c- given the context and given the question, how well is the answer performing.
- 7:13
So again, I'm gonna do a simple LLM as a judge, and I'm gonna take twenty examples of questions that I've, uh, created. And for each question, I'm gonna write what do I expect the answer to contain.
- 7:25
So we're gonna have, like, three facts that e- the answer that was generated by the RAG pipeline should have. And then the evaluator is simply going to take the answer that we've gotten from the rag- RAG pipeline and make sure that the facts actually appear in the answer.
- 7:39
And so we're gonna get, you know, per each fact, we're gonna get pass or fail, so it's a b- Boolean, uh, uh, response, and then a reason. So if it failed, then we're-- I wanna see a reason, like, why the judge thinks that the, uh, fact is not, does not appear in, uh, in the answer that we
- 7:55
got. And so then we're gonna get a score which is numeral, which is great because we like w- working with numeral scores. Uh, which is kind of like summarizing all the facts across all the, all the examples.
- 8:06
So we have three facts times twenty examples, it's sixty, sixty total facts that we need to evaluate. I'm gonna just check how many facts were right out of the total facts that we expected to have in the RAG-generated answers.
- 8:20
Uh, so let's see it in action. Uh, and as you can see, this is like the questions and the facts and everything. This is what I, uh, I, uh, I created to, to make sure that the RAG operates.
- 8:31
And then I'm gonna run it, uh, the evaluator. And what you'll see that the evaluator is doing is running evaluations, right? So it's taking question, calling the RAG, getting an answer, and then checking that answer against each of the facts that I've given it.
- 8:45
And then I'm gonna create a score, and you will see the score, you know, slowly, uh, progressing as I'm running the evaluator. This is like a super slow process, and I don't have enough time to actually show you, uh, it to the end, but it, it works.
- 8:58
It takes just a couple of minutes. Okay.
- 9:03
Okay. And, and... Okay. Great. Less-- Last step is our, uh, agent, the agent that will optimize the prompt. We have everything set up, now we can actually build the agent.
- 9:19
So I'm gonna build a researcher agent, and it's gonna take, you know, as I said, like, prompting guides online. It's gonna crawl the web, it gonna find prompting guides.
- 9:28
It's gonna get an initial prompt, and then it's gonna take, you know, it's gonna take... It's gonna run the evaluator once to get, like, the initial score. And then after we get the initial score, it's gonna get the reasons why we failed some of the evaluations.
- 9:40
It's gonna combine the reasons for failures plus prompting guides to get a new prompt. And then we're gonna get a new prompt. And then we're gonna feed it back to the evaluators, run it again, get a new score, and then run it again with the agent, so we can actually even improve the score a bit more.
- 9:56
If you've ever done kind of ML, classic ML training, this is kind of like a classic, uh, machine learning training, but with a bit of vibes.
- 10:07
Uh, so let's see it in action. We're gonna... Um, I've used the crewAI, uh, for, for doing that. And I'm gonna kick off, and you can see it will take a couple of seconds.
- 10:16
We can actually see, uh, the agent, you know, thinking and then calling the evaluator, and then the evaluator will run and get the responses and get a score. And I hope for the...
- 10:27
Yeah, I hope I can... Yeah, that's great. It's running the evaluators. It's ca- calculating the score. And, uh, and then, you know, the agent is running back. That I skipped a couple of s- minutes here, and then, you know, we got a, a response.
- 10:39
And then now the agent, the researcher agent is trying to understand, okay, why the prompt wasn't working. Can I find ways to improve RAG pipeline-based prompts? And then it's gonna, like, regenerate new, new kind of like, uh, prompt with m- maybe a bit more, uh, instructions or kind of best practices on how to write prompts.
- 10:57
And then you see, calling the evaluator again to get a, a new, uh, a new score. So we know if we need to, to optimize or not.
- 11:06
Great. Okay, this is where when I run it, uh, once. Actually, this, the initial score was, was okay. It was, uh, zero point four. And I run it just two iterations, and I got like a really, like a long kind of prompt that you expect like, uh, to see from like, uh, someone who's done a lot of
- 11:24
prompt engineering, is like give it a lot of instructions and telling it how to, how it should react and, like, how it shouldn't. You're like an expert in, in answering users' questions about Traceloop.
- 11:33
It was really nice to see it happening without me having to... I write a lot-- I wrote a lot of code to make the agent work, but I didn't do...
- 11:40
needed... I didn't need to do a lot of prompt engineering. I didn't need to do any prompt engineering, right? So it kind of worked. It was really, really nice, and the score actually jumped by a lot.
- 11:49
And I stopped at zero point nine because it means that like ninety percent of the, of the facts were correct. Great. I can stop.
- 11:55
So if there's something you wanna get out of this, uh, talk, is that you can also vibe, vibe engineer your prompts. You not, you don't need to manually iterate in prompts.
- 12:05
You just need to build evaluators and getting-
- 12:08
Kind of can run gradient ascend, gradient ascend on your evaluators. You have your score, you can kind of slowly try to optimize on your score, either automatically with an agent like I did, or manually by just like, uh, reading those manuals about how to write the best prompts and then fixing them again and again and again.
- 12:27
Some future thoughts to f- to wrap it up. We have two minutes, okay? Um, am I overfitting? So I have 20 examples, and then I run the evaluators. So maybe, maybe, maybe the prompt that I'm getting will work really well for those 20 examples, but then if I give it like another example, it will be horrible.
- 12:46
So yes, I was overfitting there because I just gave it the h- the entire 20, 20 examples. Ideally, we'd have more examples, and then kind of like, again, classic machine learning, split it into train-test, uh, sets or train-test-eval sets, and then run them like separately, right?
- 13:02
You, you take the train set, this is what you're trying to optimize, but then you also use the test set to make sure that you actually... You, you're not overfitting to your, uh, training data set, right?
- 13:12
So you don't want to give the, uh, optimizer too much context, unle- unless it just will know how to answer those specific questions and nothing else. Uh, I told you that prompt engineering is dead, but I've actually done a lot of prompt engineering for this demo, 'cause I needed to engineer the agent that is optimizing my prompts.
- 13:30
So I have... It was horrible, but I've done a lot of, uh, prompt engineering for that. Maybe, maybe I can also do this work for the evaluator prompts or f- even for the agent prompt.
- 13:40
It's kind of like this meta talk where who's even writing prompts here? Like, I'm using the agent to optimize itself. Maybe it will work, maybe it won't. I, I might try it some weekend.
- 13:50
It's kind of interesting. Uh, some links. You can try this out. Uh, it's available in our repo. It's traceloop/auto-prompting-demo, and you can run it. Uh, it should run. Uh, I tested it yesterday, and if you have any questions, you're welcome to ask me, uh, outside, or you can even book some time with me just by, uh, not
- 14:09
clicking this link, but just following this link. Thank you very much. [audience applauding] [upbeat music]