The Unreasonable Effectiveness of Prompt Learning

How much of a coding agent’s performance comes from the model, and how much comes from the instructions surrounding it? Aparna Dhinakaran opens with a purportedly leaked Claude system prompt, noting that it has probably changed since publication. Alongside it are examples from Cursor and Cline: substantial instruction sets that developers repeatedly revise as they discover what makes their agents work. The system prompt is an actively maintained part of the product.

Dhinakaran attributes the idea of system prompt learning to Andrej Karpathy: take feedback written in English and turn it into instructions that improve the next attempt. The analogy is Memento, whose protagonist writes things down because he cannot otherwise retain them. An agent can similarly carry lessons forward in its prompt without changing the underlying model. The experiment that follows applies this idea to Claude Code and Cline.

Consider a student trying to improve an exam score. In Dhinakaran’s analogy for reinforcement learning, the student takes an exam and receives a scalar reward: perhaps 70%, 80%, or 90%. That number indicates how well the attempt went, but does not itself explain which concept the student misunderstood. Discovering a better strategy from that signal can require many attempts.

Reinforcement learning works well in many domains. Dhinakaran’s concern is its cost for teams building agents on top of already capable LLMs: it can consume substantial data, time, and specialist effort. Prompt learning offers a different intervention. The student still gets a score, but also learns what was correct, what went wrong, which concepts were missed, and what to study next.

The explanation supplies a proposed direction for the next attempt. For a coding agent, the corresponding question is whether evaluations can produce sufficiently specific lessons to improve its persistent instructions.

Both agents provide a place to add persistent guidance: Cline rules and Claude Code’s CLAUDE.md. These additions sit alongside the agent’s existing instructions. In this experiment, the added rules start empty; that is an experimental starting condition, not a claim that the agents have no system prompt.

The worked example uses SWE-bench Lite, where an agent attempts to resolve software issues by modifying a repository. Dhinakaran says the team also tried the approach on BBH and other software-engineering datasets, but does not present those results here. Her spoken baselines are approximate:

The baseline slide also contains a second Cline configuration, so the Cline results should not be read as a single model-independent number.

The intervention is deliberately narrow: no fine-tuning and no model changes. The team asks whether feedback can improve the instructions supplied to the same coding agent, then measures whether the resulting agent resolves more issues.

The feedback loop begins with code and tests. The coding agent writes a patch, unit tests evaluate it, and an LLM-as-a-judge examines the attempt. The judge’s job extends beyond recording failure: it identifies why the attempt failed and what recurring scenarios caused trouble. Those explanations become inputs to a later prompt that generates new rules.

Dhinakaran describes using 150 examples for learning rules for each agent. Those examples are not the entire SWE-bench Lite dataset, which contains 300 instances. The per-example procedure is:

1. Give the agent the software-engineering problem with no added rules.
2. Collect the patch it generates.
3. Run the generated solution through the unit tests.
4. Pass the outcome to the judge, whether the tests passed or failed.

Keeping successful attempts in the evaluation supplies evidence about what worked as well as what needs correction.

The judge receives the problem statement, the coding-agent solution, and the unit-test evidence. Dhinakaran also mentions an “actual solution,” without clearly distinguishing a separate reference patch in the spoken description. The related Cline experiment report explicitly includes a ground-truth patch; that detail belongs to the published experiment rather than an unambiguous description in the recording.

The important output is pass/fail plus an explanation. Writing the judge prompt is itself engineering work: a generic verdict gives the next stage little to act on, while an explanation can identify parsing errors or other library-specific failure categories. A compact evaluation prompt can make that output contract explicit:

Review the supplied problem statement, generated patch,
unit tests, and test outcome.

Return:
- Verdict: pass or fail.
- Explanation: what the patch did correctly or incorrectly.
- Evidence: the relevant patch behavior and test outcome.
- Failure category: the kind of mistake, if any.
- Lesson: what the coding agent should do differently next time.

Separate conclusions supported by the evidence from hypotheses.

The useful part is the connection between an observed behavior and a corrective lesson. Merely restating that a test failed leaves the rule generator with essentially the original score.

The next stage collects the explanations into a meta prompt: a prompt whose task is to improve the agent’s instructions. It receives the original Claude or Cline system prompt, the existing rules—which are initially empty—and the task inputs, judge evaluations, and explanations. Supplying the existing instructions gives the optimizer the context in which its new rules will operate.

The output is a proposed ruleset. Dhinakaran describes inspecting a diff between the original configuration, with no appended rules, and the new configuration containing lessons from the evaluated attempts. The base agent instructions remain the starting point; the added rules describe what to avoid or do differently. At this stage, the rules encode proposed improvements. Their effect still has to be measured by running the agent again.

After generating the rules, the team reruns SWE-bench Lite. Dhinakaran reports gains of approximately 5% for Claude Code and 15% for Cline in GitHub issues resolved, using 150 training examples to learn the rules. The spoken percentages are not defined as relative gains or percentage-point increases, and the recording does not establish how the training examples relate to the rerun evaluation set.

The optimized-rules slide presents multiple Cline configurations, so the spoken 15% gain should not be attached specifically to the earlier Sonnet 4.5 baseline. The companion Cline report gives a smaller Sonnet-specific test gain. A separate Claude Code report describes evaluation with training and testing repositories separated, but it does not establish that the recording describes that exact run. The result to carry forward is the reported improvement from changing instructions alone, with the precise experimental split and model-specific gains requiring the corresponding run details.

How does this compare with GEPA, the prompt optimizer available through DSPy? Both approaches use natural-language feedback to revise prompts. English feedback is therefore the shared mechanism, not the distinguishing feature of this method.

Dhinakaran reports that GEPA required many more loops and rollouts than Arize’s approach in a side-by-side comparison. This is a separate comparison from the coding-agent experiment: the displayed charts show HotpotQA and HoVer with GPT-4.1 Mini, comparing GEPA, MIPROv2, and Prompt Learning. The published comparison covers language tasks rather than SWE-bench. The recording does not supply numerical rollout counts, matched budgets, or enough configuration detail to quantify the efficiency advantage.

Dhinakaran attributes the difference to the work invested in developing and iterating on the evaluation prompts. If an evaluation returns a useful explanation, the next prompt can translate it into a concrete instruction. If it returns only a verdict or a vague criticism, the optimizer has much less information to work with. Prompt learning depends on evaluations that explain how to improve. The substantive engineering work is making those explanations specific enough that an agent can carry them into its next attempt.