AI Engineer Code 2025
The Unreasonable Effectiveness of Prompt Learning
Read the talk
Learning Better Coding-Agent Instructions from Evaluations
Coding agents can retain lessons from failed attempts in their instructions. Aparna Dhinakaran traces a loop from patches and tests to diagnostic feedback and revised rules.
From a talk by Aparna Dhinakaran
Before you start: Familiarity with coding agents, repository instruction files, and unit tests is helpful; no reinforcement-learning background is required.
What goes into a successful coding agent?
How much of a coding agent’s performance comes from the model, and how much comes from the instructions surrounding it? Aparna Dhinakaran opens with a purportedly leaked Claude system prompt, noting that it has probably changed since publication. Alongside it are examples from Cursor and Cline: substantial instruction sets that developers repeatedly revise as they discover what makes their agents work. The system prompt is an actively maintained part of the product.
Dhinakaran attributes the idea of system prompt learning to Andrej Karpathy: take feedback written in English and turn it into instructions that improve the next attempt. The analogy is Memento, whose protagonist writes things down because he cannot otherwise retain them. An agent can similarly carry lessons forward in its prompt without changing the underlying model. The experiment that follows applies this idea to Claude Code and Cline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A grade tells you less than an explanation
Consider a student trying to improve an exam score. In Dhinakaran’s analogy for reinforcement learning, the student takes an exam and receives a scalar reward: perhaps 70%, 80%, or 90%. That number indicates how well the attempt went, but does not itself explain which concept the student misunderstood. Discovering a better strategy from that signal can require many attempts.
Reinforcement learning works well in many domains. Dhinakaran’s concern is its cost for teams building agents on top of already capable LLMs: it can consume substantial data, time, and specialist effort. Prompt learning offers a different intervention. The student still gets a score, but also learns what was correct, what went wrong, which concepts were missed, and what to study next.
| Feedback in the exam analogy | What it supplies |
|---|---|
| Scalar reward | A measure of success |
| Score plus English explanation | Success, mistakes, and directions for improvement |
The explanation supplies a proposed direction for the next attempt. For a coding agent, the corresponding question is whether evaluations can produce sufficiently specific lessons to improve its persistent instructions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the model and instructions already in place
Both agents provide a place to add persistent guidance: Cline rules and Claude Code’s CLAUDE.md. These additions sit alongside the agent’s existing instructions. In this experiment, the added rules start empty; that is an experimental starting condition, not a claim that the agents have no system prompt.
The worked example uses SWE-bench Lite, where an agent attempts to resolve software issues by modifying a repository. Dhinakaran says the team also tried the approach on BBH and other software-engineering datasets, but does not present those results here. Her spoken baselines are approximate:
| Agent configuration | GitHub issues resolved before adding rules |
|---|---|
| Cline with Claude Sonnet 4.5 | About 30% |
| Claude Code | About 40% |
The baseline slide also contains a second Cline configuration, so the Cline results should not be read as a single model-independent number.
The intervention is deliberately narrow: no fine-tuning and no model changes. The team asks whether feedback can improve the instructions supplied to the same coding agent, then measures whether the resulting agent resolves more issues.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn test outcomes into diagnoses
The feedback loop begins with code and tests. The coding agent writes a patch, unit tests evaluate it, and an LLM-as-a-judge examines the attempt. The judge’s job extends beyond recording failure: it identifies why the attempt failed and what recurring scenarios caused trouble. Those explanations become inputs to a later prompt that generates new rules.
Dhinakaran describes using 150 examples for learning rules for each agent. Those examples are not the entire SWE-bench Lite dataset, which contains 300 instances. The per-example procedure is:
- Give the agent the software-engineering problem with no added rules.
- Collect the patch it generates.
- Run the generated solution through the unit tests.
- Pass the outcome to the judge, whether the tests passed or failed.
Keeping successful attempts in the evaluation supplies evidence about what worked as well as what needs correction.
The judge receives the problem statement, the coding-agent solution, and the unit-test evidence. Dhinakaran also mentions an “actual solution,” without clearly distinguishing a separate reference patch in the spoken description. The related Cline experiment report explicitly includes a ground-truth patch; that detail belongs to the published experiment rather than an unambiguous description in the recording.
The important output is pass/fail plus an explanation. Writing the judge prompt is itself engineering work: a generic verdict gives the next stage little to act on, while an explanation can identify parsing errors or other library-specific failure categories. A compact evaluation prompt can make that output contract explicit:
markdown
Review the supplied problem statement, generated patch,
unit tests, and test outcome.
Return:
- Verdict: pass or fail.
- Explanation: what the patch did correctly or incorrectly.
- Evidence: the relevant patch behavior and test outcome.
- Failure category: the kind of mistake, if any.
- Lesson: what the coding agent should do differently next time.
Separate conclusions supported by the evidence from hypotheses.
The useful part is the connection between an observed behavior and a corrective lesson. Merely restating that a test failed leaves the rule generator with essentially the original score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Convert diagnoses into persistent rules
The next stage collects the explanations into a meta prompt: a prompt whose task is to improve the agent’s instructions. It receives the original Claude or Cline system prompt, the existing rules—which are initially empty—and the task inputs, judge evaluations, and explanations. Supplying the existing instructions gives the optimizer the context in which its new rules will operate.
The output is a proposed ruleset. Dhinakaran describes inspecting a diff between the original configuration, with no appended rules, and the new configuration containing lessons from the evaluated attempts. The base agent instructions remain the starting point; the added rules describe what to avoid or do differently. At this stage, the rules encode proposed improvements. Their effect still has to be measured by running the agent again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Rerun the agents with learned instructions
After generating the rules, the team reruns SWE-bench Lite. Dhinakaran reports gains of approximately 5% for Claude Code and 15% for Cline in GitHub issues resolved, using 150 training examples to learn the rules. The spoken percentages are not defined as relative gains or percentage-point increases, and the recording does not establish how the training examples relate to the rerun evaluation set.
The optimized-rules slide presents multiple Cline configurations, so the spoken 15% gain should not be attached specifically to the earlier Sonnet 4.5 baseline. The companion Cline report gives a smaller Sonnet-specific test gain. A separate Claude Code report describes evaluation with training and testing repositories separated, but it does not establish that the recording describes that exact run. The result to carry forward is the reported improvement from changing instructions alone, with the precise experimental split and model-specific gains requiring the corresponding run details.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The quality of the explanation determines what can be learned
How does this compare with GEPA, the prompt optimizer available through DSPy? Both approaches use natural-language feedback to revise prompts. English feedback is therefore the shared mechanism, not the distinguishing feature of this method.
Dhinakaran reports that GEPA required many more loops and rollouts than Arize’s approach in a side-by-side comparison. This is a separate comparison from the coding-agent experiment: the displayed charts show HotpotQA and HoVer with GPT-4.1 Mini, comparing GEPA, MIPROv2, and Prompt Learning. The published comparison covers language tasks rather than SWE-bench. The recording does not supply numerical rollout counts, matched budgets, or enough configuration detail to quantify the efficiency advantage.
Dhinakaran attributes the difference to the work invested in developing and iterating on the evaluation prompts. If an evaluation returns a useful explanation, the next prompt can translate it into a concrete instruction. If it returns only a verdict or a vague criticism, the optimizer has much less information to work with. Prompt learning depends on evaluations that explain how to improve. The substantive engineering work is making those explanations specific enough that an agent can carry them into its next attempt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official description of the 300-instance software issue-resolution benchmark.
Research introducing prompt evolution through natural-language reflection and Pareto-based candidate selection.
Current official documentation for configuring persistent Cline instructions.
Further reading
SDK and examples for optimizing prompts and rulesets with natural-language evaluation feedback.
- Optimizing Cline RulesArticle
October 2025 experiment detailing Cline's training/test split, patch evaluation and rule-generation prompts.
Experiments measuring instruction optimization across repositories and within a chronological Django issue split.
November 2025 comparison covering four language tasks and diagnostic evaluation design.
- Using CLAUDE.md FilesArticle
Anthropic's guide to structuring project instructions for Claude Code.
Updates since the talk
Current guide to running Cline, evaluating patches and iteratively optimizing rules with Arize.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] Hi, everyone.
- 0:21
Thanks so much for coming. Um, well, today, I'm excited. We're gonna talk a little bit about prompt learning and how to use that with evals. Uh, any of you guys, um, are spending a lot of time thinking about the frontier coding models?
- 0:35
I think there's so much attention on, on them, but just what's not so obvious is how much time is actually spent, uh, on the system prompts, uh, for those building these coding agents.
- 0:47
So here's actually a look. Um, this is a tweet that went viral about the whole system prompt, uh, of Claude that's been leaked. I'm sure, you know, they've changed it since then.
- 0:57
Um, but you can actually see that Claude, there's Cursor, there's Cline, um, and just the length of the actual system prompt, um, for each one of these. And I think what's not as obvious is these actually aren't just static, they are repeatedly iterated on, and it's such an important piece of context that actually goes into making these
- 1:16
coding agents the most successful agents out there. Um, it's not just us talking about it. Karpathy talks about it a lot. Um, and this was a viral tweet that, that he posted, which was, there's this paradigm around iterating on these prompts that he, he's kind of coined it system prompt learning.
- 1:35
And what he said is that it almost feels like humans learning because they take back English feedback, uh, and use that to actually iterate on what they should do differently the next time.
- 1:48
And I think he wrote something like it's almost like that movie Memento, where the guy forgets, uh, what, you know, what he learns and then he starts writing it down and then uses that to actually kind of go through his next day.
- 2:00
And so this is a little bit of the concept behind system prompt learning. And what we wanted to do was show you guys a little bit of how that works and then put that to test on two of the most popular coding agents, uh, Claude and Cline today.
- 2:14
So first off, how does prompt learning actually work? So for those of you who are familiar with RL, what I thought we'd do is just do a little analogy, compare how does RL work versus system prompt learning.
- 2:24
For RL, you know, if we just took an analogy of a student who's trying to improve their exam scores, they take an exam, you know, somebody grades the exam.
- 2:32
You have a scalar reward, which is like, you know, they got a seventy percent, an eighty percent, ninety percent, and then they have to figure out almost blindly just with that score, how to actually improve their score on the next exam.
- 2:46
And I think this is actually one of the flaws of-- I, I mean, RL works, don't get me wrong, amazing in so many concepts and domains, but it can be y-you know, a long path to actually figure out what the right solution is.
- 3:00
And I think some of the things that we've noticed is that it can be sample inefficient. It takes a lot of data to get what you want. It's time-intensive, it's data-hungry.
- 3:09
You need to have a whole data science team to do this, and it just might be overkill for teams who are trying to build agents because LLMs are already so good.
- 3:17
So if you're a team who's actually trying to build an agent, maybe prompt learning is actually slightly,
- 3:23
m-might be slightly more of an interesting paradigm for you. So in this scenario, same, same analogy. You have a student who's taking an exam, there's some exam score. Except in this case, what actually gets outputted isn't just the score.
- 3:36
They got a seventy, they got an eighty. But you also get back some kind of English feedback. Why did they get this answer right? What did they mess up on?
- 3:43
Here's concepts that they missed on. What do they need to go study? And then they use this information to actually go and, and prepare on what to do next, um, to, to get a better score.
- 3:55
This is basically the, the concept that we applied to coding agents, and we ran this kind of test on both Claude as well as Cline. Um, both of these, as you know, start off with some kind of, uh, system prompt, which in Claude Code, this is kind of a snippet of it, and they both kind of come
- 4:14
with something that you can append rules to. So Cline has rules, Claude MD has the CLAUDE.md file, and it starts off empty. You can go in and, uh, add whatever is important for your repo.
- 4:24
So what we did was actually took, you know, just benchmark both Cline and Claude Code on SWE-bench. I'm gonna kind of run through the exam-- this entire example at SWE-bench, but this entire thing we also ran on BBH and a ton of other, uh, software engineering datasets.
- 4:40
But you can see here, just on vanilla Cline, vanilla Claude Code, um, nothing added to the Claude MD or the Cline rules. Um, they had, y-you know, about, I think with Cline, somewhere on, you know, Claude Sonnet 4.5 was about thirty percent of the GitHub issues actually resolved.
- 4:56
Uh, Claude Code was about forty percent of the GitHub issues resolved. So we took this as kind of our starting benchmark, and the thesis is, is could we actually use prompt learning to see if we can improve the system prompt and see if, um, it was able to, with the new system prompt, actually, you know, give us
- 5:14
a better, uh, score on these benchmarks. We didn't do anything on fine-tuning. We didn't change the models, anything like that. It was just focused on the system prompt. Um, and so the process that we went through, we took the coding agent, uh, we had it actually write some code.
- 5:29
Um, we ran unit tests, and then, um, we then passed that through to some kind of, um, model that was doing the LLM-as-a-judge evals. And I'll show you guys what that looks like.
- 5:40
But the LLM-as-a-judge eval actually gave back, why did it fail? Did it fail because of this? Uh, can you give some examples of, you know, what were common scenarios that it didn't do good on?
- 5:51
And then it actually used those kind of evals to then go back and add it to a meta prompt to come back with kind of the, the system prompt rules that we're gonna append to.
- 6:02
So let's talk through kind of the process. So first, we had kind of the SWE-bench dataset. Uh, SWE-bench in this scenario is just 150 examples. Uh, we did this for both Cline and Claude Code, where we took the original prompt, which had no rules.
- 6:16
We gave it kind of the software engineering problem, and then it generated some kind of patch to actually solve that, and then we ran the generated solution through the unit test.
- 6:27
Then whatever the unit test came back with, whether it was right or wrong, we then passed this into an LLM-as-a-judge eval. And this is kind of the most important part, because this actually generated the explanation for us.
- 6:40
So we passed in the problem statement, we passed in what the coding agent solution was, the unit tests, and then the actual solution that it came up with. Uh, passed that in, and this that you're looking at in the center here is actually the LLM-as-a-judge eval.
- 6:54
And these evals, we're gonna talk into this, uh, talk a bit about this, but eval engineering is a whole kind of concept that y- you know, we spend a lot of time on, and writing really good evals is, I think, um, how you get the best kind of insight into what you could do to improve your agents.
- 7:12
So in this scenario, what we did was we wrote a good LLM-as-a-judge eval prompt. It outputted whether it failed or passed. And then, this is the key part, we actually asked for an explanation.
- 7:22
Why did it actually mess up? Um, you know, for specific libraries in the SWE-bench Lite test, um, you know, it was parsing errors, or it was not handling, um, uh, uh, there, there's all sorts of actually different categories of errors.
- 7:36
But we went through and we, we kind of looked at the explanation of what went wrong in each scenario. We then passed into a huge meta prompt. So this is actually what's helping us iterate on our system prompt.
- 7:48
We passed in the original Claude or Cline system prompt. We passed in the original rules, which for us started off empty. Um, and then we passed in, here was the input, here was the LLM-as-a-judge eval, and then here was the actual explanation from that eval.
- 8:04
Passed that all into the meta prompt, and then we did kind of a diff comparing, you know, the old world. So just for, you know, just to remember, the old world had the original Claude system prompt, no rules kind of added or appended to it, and then the new world where it generated this entire rules of what
- 8:23
to avoid or what to, um, you know, what it had learned essentially from all those mistakes that it actually made.
- 8:29
And then we ran this basically on the entire SWE-bench Lite again. Um, and what we saw was that, you know, on 150 examples, we were able to get Claude Code up by 5%, more GitHub issues resolved.
- 8:43
Cline, um, you know, 15%. And this was literally on, I think the key thing is, like, 150 examples of just training data that was used, um, on the most kind of powerful coding agents that are out there.
- 9:00
Um, and so just think about kind of the impact that could have for your agents. Many of you guys in this room might be thinking, "Okay, well, prompt learning is cool, but how does that compare to GEPA?"
- 9:09
If you're familiar with DSPy and you've kind of seen, I don't know if it's GEPA or GEPA, I've heard both. Um, but y- you know, you guys might be asking, "Well, how is this different?"
- 9:19
Um, so GEPA, just, just in case you guys aren't familiar, it's a prompt optimizer from DSPy that is essentially very, very similar to what we're talking about, which is taking English feedback, using that English feedback inside of the actual prompt.
- 9:35
Um, and what we did was actually run a side-by-side benchmark against GEPA, where we compared kind of our prompt learning against GEPA. And, um, I think what we saw was that GEPA required many, many loops and rollouts compared to, um, kind of a, a fraction of that, which was our approach.
- 9:56
And I think the key difference here, I mean, the underlying approach around using English feedback is the same, but I think the key thing that was really different here was we spent a lot of time actually developing and iterating on the evals.
- 10:08
And the eval prompts really mattered to making sure that you gave really good explanations back to the agent. Um, and so evals make all the difference. This was super critical for us to be able to get this to work.
- 10:21
Um, and if you guys are curious about learning more or reading more about kind of what we do, um, check out kind of our blog. We write a lot about evals, prompt optimization, and, uh, we're actively hiring, so come check us out.
- 10:37
Awesome. [audience applauds] [upbeat music]