Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse

Does the code compile? A coding agent has at least one concrete test of progress. What is the equivalent for a healthcare assistant, a medical-compliance workflow or a chatbot? Annabell Schäfer, a growth engineer at Langfuse, places that question at the center of application design: before automating improvement, define what improvement means.

In the talk’s June 2026 framing, loops have become the dominant conversation. Schäfer points to Boris Cherny and Peter Steinberger’s emphasis on designing loops, alongside Karpathy’s autoresearch work. Much of that enthusiasm comes from developers, whose tools already provide useful feedback. Compilation does not establish that the features are correct or the code is good, but it supplies a definite pass/fail signal that a broader evaluation process can build on.

Outside coding, the target is often harder to specify. An agent can follow its initial objective faithfully while moving away from what the application actually needs. The winding-path illustration captures that problem: the original goal and the optimal destination are different places. Discovering the difference takes work, especially when success cannot be reduced to a simple yes or no.

Schäfer reports that the teams Langfuse sees improving consistently invest in this middle layer: capturing the intended behavior and building evaluators that recognize it. Those evaluators give changes a meaningful acceptance test, making it easier to keep improving the application and ship with confidence.

To isolate the role of the target function, the experiment starts with single-label classification. Suppose an item’s expected label is Order, and the available labels include Order, Complaint and Inquiry. The evaluator compares the predicted label with the expected label. Agreement earns a correct result; disagreement earns an incorrect one. Accuracy is the fraction of items with matching labels.

The actual task classifies arXiv papers from their titles and abstracts, using each author’s selected primary category as the expected output. The experiment uses 200 fit items, 100 validation items and 300 test items. Keeping these datasets separate allows prompt development and final evaluation to serve different purposes.

The classifier is GPT-5.4 nano with a simple prompt and a flat list of labels. Schäfer chooses a small, inexpensive model to explore how much automated improvement can extract from it. A separate optimizer runs through Claude Code using Claude Opus 4.8. It proposes prompt changes with a GPT-5.4 prompting guide and a task Markdown file describing the loop as context.

In the Langfuse dataset view, each row has an input containing the paper’s title and abstract, plus an expected output containing its category. The three datasets have distinct roles:

The optimizer learns from fit-set error clusters. Validation determines whether a proposed change generalizes beyond those examples. The test set stays outside prompt development until the loop ends.

The baseline deliberately supplies very little guidance: classify the paper using one label from the flat list. There are no label descriptions or additional prose explaining how to choose. This leaves room to observe what guidance the optimizer introduces on its own.

The task Markdown file turns optimization into a specific procedure:

1. Run the baseline prompt on the fit and validation datasets, recording per-item results and overall accuracy.
2. Inspect the fit results: which categories work, which label pairs are confused, and what the underlying titles and abstracts have in common.
3. Form a hypothesis targeting the largest error category, then publish a candidate prompt update.
4. Rerun evaluation and accept the candidate only if validation accuracy improves.
5. Stop after 15 runs or on reaching 92% accuracy, then evaluate the final prompt on the test set. Run that final test if the loop stops for another reason, too.

The acceptance rule matters: an improvement on the examples used to diagnose the problem is not enough.

A compact task-file version of that contract is:

# Prompt optimization

- Establish baseline fit and validation accuracy.
- Score every item by predicted-label equality with expected label.
- Inspect fit errors and recurring label confusions.
- Propose a prompt change for the largest error cluster.
- Accept the candidate only if validation accuracy improves.
- Stop after 15 runs or at 92% accuracy.
- After stopping, evaluate the final prompt on the test dataset.
- Keep the test dataset unavailable during prompt optimization.

The demonstrated file points the optimizer to its working datasets but deliberately omits the test dataset link. That separation keeps the final examples out of the optimizer’s inspection process.

Schäfer reports accuracy rising from 68% initially to 83% on the fourth iteration, a gain of 15 percentage points. Subsequent results plateau around 80%. The loop improves the classifier, but it does not reach the intended stopping target.

Inspecting the data reveals a complication in the supposedly clear target: authors have discretion when selecting a primary category. A classifier can make a plausible subject judgment and still disagree with the recorded label. Exact label agreement is easy to measure; it does not make the underlying categorization uniquely correct. The labels omit the author’s reasons for choosing one relevant category over another.

Schäfer reports 80.2% final accuracy on the 300-item unseen test set. The aggregation is not explained: that percentage does not correspond to an unweighted exact-match count over exactly 300 items. The first prompt update supplies most of the observed improvement, with relatively little movement afterward, making that update the most useful place to inspect the mechanism.

The prompt comparison shows the flat label list becoming a set of classification instructions. The additions explain how to make decisions, rather than merely expanding the vocabulary:

• Classification approach: Explain the goal of choosing a label and how to approach the paper.
• Boundary rules: Distinguish similar classes and address recurring confusion patterns.
• Specificity: Prefer a specific label over a broad label where the category structure contains implicit nesting.
• Examples: Show cases involving label pairs the classifier frequently misclassifies.

These changes turn observed errors into guidance for the next prediction.

Schäfer had expected to add descriptions to the labels. Instead, the optimizer, working with the prompting guide, chose rules and examples. That distinction is useful: knowing what each category means is different from knowing which category should win when several appear relevant.

The first update moves accuracy from 68% to 78%, a gain of 10 percentage points. The displayed Opus analysis starts from 64 errors, identifies dominant patterns, flags an answer sink—a category attracting too many predictions—and finds the most confused label pair. It then forms a hypothesis intended to address that concentration of mistakes.

Two properties make that diagnosis possible. First, the failure signal is explicit: each prediction either matches the expected label or does not. Second, the 200-item fit dataset spans only 10 labels, supplying repeated examples from which to identify patterns. Assembling comparable coverage can take time in a real application, but here it gives the optimizer enough evidence to propose a targeted change.

Later iterations find a better prompt, but the largest jump comes from that first clear failure signal. Schäfer suggests the team could have stopped after the first update and already had a strong baseline. The value came from understanding a repeated mistake, not simply from allowing the loop to continue.

Most domain applications cannot use exact label agreement as their entire target function. An LLM judge may give different evaluations on repeated runs. For a specialized domain agent or a medical-device requirements-extraction workflow, the challenge is to recover the classification experiment’s clear feedback without assuming the task has a deterministic answer.

Broad evaluators such as correctness, helpfulness and hallucination were useful starting points, but they can provide weak guidance for automated improvement. A score on a 0–1, 1–10 or 1–5 scale needs a defined meaning: what criteria earn each value, and in what context? Without those definitions, the judge can change its interpretation between examples or runs. A numerical output alone does not tell the optimizer what to fix.

Start by naming the quality criteria that matter for the application. Instead of asking only whether an answer is correct, ask whether its internal-information claims are supported by the knowledge base. The evaluator can inspect an answer’s snippet and check it against the context retrieved in the preceding step. This gives the judgment a concrete reference.

Other checks can focus on narrower, recurring problems:

• Brand spelling: Did the answer write the company’s name correctly?
• Name preservation: Did an answer in Spanish accidentally translate an English company name?
• Known failures: Which of five recognized failure types occurred?

These checks need not capture all of quality individually. They identify specific behaviors that matter and produce feedback an optimizer can act on. The criteria come from inspecting actual data and understanding what domain experts consider good or bad.

Clear criteria also need enough examples. The paper experiment could have drawn on thousands of records, but its smaller fit, validation and test datasets already supplied useful feedback. That is not a fixed dataset recipe for other applications: the necessary volume depends on the complexity of the task and the steps in the workflow. Coverage and signal quality have to develop together.

Working with experts makes their knowledge available to the loop in concrete form:

1. Ask them to create examples and specify the desired outputs.
2. Review sample runs together.
3. Compare cases that receive different decisions and ask why the distinction matters.
4. Capture the resulting quality criteria and failure modes in evaluators.

The comparison step is especially valuable. Experts often treat a decision rule as obvious and leave it unstated. Asking why one case goes one way and another case goes the other way exposes that implicit knowledge, giving the system more than a collection of unexplained labels.

Once the application has a baseline and enters production, review its data as a human, not only through a coding agent. Compare the intended scope with what people actually do. Usage can shift, and the failure modes can change with it.

Unexpected use is evidence in both directions: it can reveal a missing feature or a failure the original evaluation never covered. Understanding how the application works and fails in production gives the team a way to update its examples and catch recurring problems before customers do.

The examples should represent the conditions the system is expected to encounter in production. As in traditional machine learning, validation must check whether improvements generalize rather than merely fit the examples already seen. That requires an actual validation mechanism, supported by instructions that tell the loop how to use it.

The loop also needs an escape hatch. If progress stalls, continuing for hours can consume tokens without improving the application. Explicit stopping rules make room for a human to revisit the target, the examples or the failure analysis. The intended system keeps humans and agents collaborating: experts help define and revise what good means, while the agent uses that feedback to make measurable progress.