AI Engineer World's Fair 2026
Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse
Read the talk
Stop Burning Tokens: Define What an Improvement Loop Should Improve
A paper-classification experiment shows how clear failure signals drive prompt improvements—and why domain expertise must shape the evaluators that guide the loop.
From a talk by Annabell Schäfer
Before you start: Familiarity with prompts and basic classification accuracy is helpful; no Langfuse experience is required.
What counts as a better result?
Does the code compile? A coding agent has at least one concrete test of progress. What is the equivalent for a healthcare assistant, a medical-compliance workflow or a chatbot? Annabell Schäfer, a growth engineer at Langfuse, places that question at the center of application design: before automating improvement, define what improvement means.
In the talk’s June 2026 framing, loops have become the dominant conversation. Schäfer points to Boris Cherny and Peter Steinberger’s emphasis on designing loops, alongside Karpathy’s autoresearch work. Much of that enthusiasm comes from developers, whose tools already provide useful feedback. Compilation does not establish that the features are correct or the code is good, but it supplies a definite pass/fail signal that a broader evaluation process can build on.
Outside coding, the target is often harder to specify. An agent can follow its initial objective faithfully while moving away from what the application actually needs. The winding-path illustration captures that problem: the original goal and the optimal destination are different places. Discovering the difference takes work, especially when success cannot be reduced to a simple yes or no.
Schäfer reports that the teams Langfuse sees improving consistently invest in this middle layer: capturing the intended behavior and building evaluators that recognize it. Those evaluators give changes a meaningful acceptance test, making it easier to keep improving the application and ship with confidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with a target that can be scored exactly
To isolate the role of the target function, the experiment starts with single-label classification. Suppose an item’s expected label is Order, and the available labels include Order, Complaint and Inquiry. The evaluator compares the predicted label with the expected label. Agreement earns a correct result; disagreement earns an incorrect one. Accuracy is the fraction of items with matching labels.
The actual task classifies arXiv papers from their titles and abstracts, using each author’s selected primary category as the expected output. The experiment uses 200 fit items, 100 validation items and 300 test items. Keeping these datasets separate allows prompt development and final evaluation to serve different purposes.
The classifier is GPT-5.4 nano with a simple prompt and a flat list of labels. Schäfer chooses a small, inexpensive model to explore how much automated improvement can extract from it. A separate optimizer runs through Claude Code using Claude Opus 4.8. It proposes prompt changes with a GPT-5.4 prompting guide and a task Markdown file describing the loop as context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate diagnosis, acceptance and final testing
In the Langfuse dataset view, each row has an input containing the paper’s title and abstract, plus an expected output containing its category. The three datasets have distinct roles:
| Dataset | Role in the loop |
|---|---|
| Fit | Inspect errors and formulate prompt hypotheses |
| Validation | Check whether proposed changes improve unseen examples |
| Test | Evaluate the final prompt on untouched examples |
The optimizer learns from fit-set error clusters. Validation determines whether a proposed change generalizes beyond those examples. The test set stays outside prompt development until the loop ends.
The baseline deliberately supplies very little guidance: classify the paper using one label from the flat list. There are no label descriptions or additional prose explaining how to choose. This leaves room to observe what guidance the optimizer introduces on its own.
The task Markdown file turns optimization into a specific procedure:
- Run the baseline prompt on the fit and validation datasets, recording per-item results and overall accuracy.
- Inspect the fit results: which categories work, which label pairs are confused, and what the underlying titles and abstracts have in common.
- Form a hypothesis targeting the largest error category, then publish a candidate prompt update.
- Rerun evaluation and accept the candidate only if validation accuracy improves.
- Stop after 15 runs or on reaching 92% accuracy, then evaluate the final prompt on the test set. Run that final test if the loop stops for another reason, too.
The acceptance rule matters: an improvement on the examples used to diagnose the problem is not enough.
A compact task-file version of that contract is:
markdown
# Prompt optimization
- Establish baseline fit and validation accuracy.
- Score every item by predicted-label equality with expected label.
- Inspect fit errors and recurring label confusions.
- Propose a prompt change for the largest error cluster.
- Accept the candidate only if validation accuracy improves.
- Stop after 15 runs or at 92% accuracy.
- After stopping, evaluate the final prompt on the test dataset.
- Keep the test dataset unavailable during prompt optimization.
The demonstrated file points the optimizer to its working datasets but deliberately omits the test dataset link. That separation keeps the final examples out of the optimizer’s inspection process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Early gains, then an ambiguous ceiling
Schäfer reports accuracy rising from 68% initially to 83% on the fourth iteration, a gain of 15 percentage points. Subsequent results plateau around 80%. The loop improves the classifier, but it does not reach the intended stopping target.
Inspecting the data reveals a complication in the supposedly clear target: authors have discretion when selecting a primary category. A classifier can make a plausible subject judgment and still disagree with the recorded label. Exact label agreement is easy to measure; it does not make the underlying categorization uniquely correct. The labels omit the author’s reasons for choosing one relevant category over another.
Schäfer reports 80.2% final accuracy on the 300-item unseen test set. The aggregation is not explained: that percentage does not correspond to an unweighted exact-match count over exactly 300 items. The first prompt update supplies most of the observed improvement, with relatively little movement afterward, making that update the most useful place to inspect the mechanism.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The optimizer adds decision rules and examples
The prompt comparison shows the flat label list becoming a set of classification instructions. The additions explain how to make decisions, rather than merely expanding the vocabulary:
- Classification approach: Explain the goal of choosing a label and how to approach the paper.
- Boundary rules: Distinguish similar classes and address recurring confusion patterns.
- Specificity: Prefer a specific label over a broad label where the category structure contains implicit nesting.
- Examples: Show cases involving label pairs the classifier frequently misclassifies.
These changes turn observed errors into guidance for the next prediction.
Schäfer had expected to add descriptions to the labels. Instead, the optimizer, working with the prompting guide, chose rules and examples. That distinction is useful: knowing what each category means is different from knowing which category should win when several appear relevant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why the first update had so much to work with
The first update moves accuracy from 68% to 78%, a gain of 10 percentage points. The displayed Opus analysis starts from 64 errors, identifies dominant patterns, flags an answer sink—a category attracting too many predictions—and finds the most confused label pair. It then forms a hypothesis intended to address that concentration of mistakes.
Two properties make that diagnosis possible. First, the failure signal is explicit: each prediction either matches the expected label or does not. Second, the 200-item fit dataset spans only 10 labels, supplying repeated examples from which to identify patterns. Assembling comparable coverage can take time in a real application, but here it gives the optimizer enough evidence to propose a targeted change.
Later iterations find a better prompt, but the largest jump comes from that first clear failure signal. Schäfer suggests the team could have stopped after the first update and already had a strong baseline. The value came from understanding a repeated mistake, not simply from allowing the loop to continue.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A number is not yet a useful quality signal
Most domain applications cannot use exact label agreement as their entire target function. An LLM judge may give different evaluations on repeated runs. For a specialized domain agent or a medical-device requirements-extraction workflow, the challenge is to recover the classification experiment’s clear feedback without assuming the task has a deterministic answer.
Broad evaluators such as correctness, helpfulness and hallucination were useful starting points, but they can provide weak guidance for automated improvement. A score on a 0–1, 1–10 or 1–5 scale needs a defined meaning: what criteria earn each value, and in what context? Without those definitions, the judge can change its interpretation between examples or runs. A numerical output alone does not tell the optimizer what to fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make domain quality observable
Start by naming the quality criteria that matter for the application. Instead of asking only whether an answer is correct, ask whether its internal-information claims are supported by the knowledge base. The evaluator can inspect an answer’s snippet and check it against the context retrieved in the preceding step. This gives the judgment a concrete reference.
Other checks can focus on narrower, recurring problems:
- Brand spelling: Did the answer write the company’s name correctly?
- Name preservation: Did an answer in Spanish accidentally translate an English company name?
- Known failures: Which of five recognized failure types occurred?
These checks need not capture all of quality individually. They identify specific behaviors that matter and produce feedback an optimizer can act on. The criteria come from inspecting actual data and understanding what domain experts consider good or bad.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn expert judgment into examples
Clear criteria also need enough examples. The paper experiment could have drawn on thousands of records, but its smaller fit, validation and test datasets already supplied useful feedback. That is not a fixed dataset recipe for other applications: the necessary volume depends on the complexity of the task and the steps in the workflow. Coverage and signal quality have to develop together.
Working with experts makes their knowledge available to the loop in concrete form:
- Ask them to create examples and specify the desired outputs.
- Review sample runs together.
- Compare cases that receive different decisions and ask why the distinction matters.
- Capture the resulting quality criteria and failure modes in evaluators.
The comparison step is especially valuable. Experts often treat a decision rule as obvious and leave it unstated. Asking why one case goes one way and another case goes the other way exposes that implicit knowledge, giving the system more than a collection of unexplained labels.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep humans looking at production behavior
Once the application has a baseline and enters production, review its data as a human, not only through a coding agent. Compare the intended scope with what people actually do. Usage can shift, and the failure modes can change with it.
Unexpected use is evidence in both directions: it can reveal a missing feature or a failure the original evaluation never covered. Understanding how the application works and fails in production gives the team a way to update its examples and catch recurring problems before customers do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build validation and a way to stop into the system
The examples should represent the conditions the system is expected to encounter in production. As in traditional machine learning, validation must check whether improvements generalize rather than merely fit the examples already seen. That requires an actual validation mechanism, supported by instructions that tell the loop how to use it.
The loop also needs an escape hatch. If progress stalls, continuing for hours can consume tokens without improving the application. Explicit stopping rules make room for a human to revisit the target, the examples or the failure analysis. The intended system keeps humans and agents collaborating: experts help define and revise what good means, while the agent uses that feedback to make measurable progress.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Create evaluation datasets with inputs and expected outputs, import examples from production traces, and use them in experiments.
OpenAI's launch announcement describes GPT-5.4 nano's classification use case, availability and original pricing.
Anthropic's announcement introduces the optimizer model named in the talk and its availability in Claude workflows.
Further reading
- How arXiv crosslisting worksDocumentation
Explains how papers can appear in additional subject categories beyond their original submission category.
Read the complete timestamped transcript
- 0:00
Hi, everyone. My name is Annabell. I'm a growth engineer at Langfuse, and we're the largest open source observability and evaluation platform for your AI system. And today, I'm gonna share about how you should stop burning your tokens and why you should start building in domain expertise early in into your loop design, but also into your overall, um,
- 0:18
application design, uh, to make sure you're continuously improving and updating your application.
- 0:24
It is June 2026, and the whole internet is about, uh, loops right now. So we have Boris Journey saying he doesn't write any prompts anymore. He has loops, and Peter Steinberger being like, "You should be designing loops and not prompt your agents."
- 0:37
And I mean, the whole capacity auto research and auto-improvement topic blew up earlier this year already. But all of them are coming a little bit from the devel-developer perspective and, um, coding is one for, for good reason, one of the cases where this whole, um,
- 0:53
automatically reaching a goal, uh, worked quite well because they've always had, uh, at least one, um, target function that was qui-quite clear, and this was, uh, does the code compile or not?
- 1:02
And of course, just because code compiles doesn't mean it's great and doesn't mean, um, that all the features are exactly how you wanted them, but you at least know that you shipped something that worked.
- 1:11
And of course, this was then expanded over time to, to expand this target function. But essentially, it's, uh, does it work or does it not? In all other fields, and especially if you're building AI applications for some kind of domains like medicinal compliance or healthcare or all kinds of chatbots, these target functions are, uh, not nearly as
- 1:33
clear. And a target that you give an agent is actually also always incomplete. So, um, you might initially think that you're trying to head down here, uh, but actually, your optimized land destination is up there.
- 1:45
And to get there and to understand this takes quite some time and to figure this out. And it's... Yeah, it's just inherently a, a difficult problem, especially if you're in a field where a clear yes or no, like does the code compile, um, doesn't really, uh, cut it.
- 2:00
So it is very difficult, but at the same time, we at length, you see that the teams who are investing heavily here in the middle, so, uh, making sure they capture what they actually want to work, so the target function, uh, and build this out and make sure they have good evaluators that are evaluating this, are the
- 2:15
ones who manage to continuously upgrade and improve their application over time, and also ship with confidence. Because if you know, um, it's working as you intend it to, uh, then you can also sleep, um, well at night in case you push a code change.
- 2:28
Okay. So now we know it's super important, and you should be doing it, but at the same kind, uh, time, it's almost impossible to actually, uh, get it right.
- 2:37
Uh, so we were setting out and wondering, "What's the clearest cut target function we can find, um, for an agent to use?" And also, if you run an auto optimization on it, uh, what can we learn about the role of target functions from, from running it on it?
- 2:51
And the clearest cut target function we could find was a single label classification task that has a, has a very clear cut yes or no. So for example, let's say we have an item to categorize.
- 3:01
There's a true label, for example, oh, it's an order, and then we have a set of available labels, order, complaint, inquiry, and our classifier then assigns, um, one label, and you can very clearly say, is the true label equal to the predicted label?
- 3:13
And if yes, um, then this one is right, and the next one might be wrong. And overall, you can calculate, uh, an accuracy value and get a very clear signal from here.
- 3:23
So we put this target function together with an agent and also with an optimizer, and how this minimal loop looked like, I'm gonna share with you now. So here's our minimal self-optimization loop.
- 3:36
Up there in the first two rows, we can see our target function. In our case, we opted for classified, um, archive papers. So it's a set of papers that based on their primary, um, uh, based on their title and abstract, uh, got a primary label from the author, um, to categorize it for the other, uh, people in
- 3:54
research. And, um, yeah, we're gonna, uh... We have the, the ground truth here, and we have it in 200, um, items in a fit dataset, 100 in a validate dataset, and 300 in a test dataset, um, just to also make sure we're not overfitting.
- 4:07
Then we have our agent, which is more or less just a, a simple prompt, um, based on GPT-5.4-Nano, because we wanted to see how a, a very small and cheap model performs on the auto-improvement, because the good ones actually got really, really good but also very expensive, um, over time.
- 4:23
And we have this flat list of labels, and then we have our optimization process that runs through, uh, Claude Code, uh, leveraging Claude Opus 4.8, so one of the frontier models, and it proposes the prompt updates and has this context reference, the GPT-5.4 prompting guide, as well as a task MD that is describing the loop.
- 4:40
Looking a little bit closer on how this looks like, here we can see our target function, in this case, especially our, uh, fit dataset, uh, within length view. So we see an input column where the title and the abstracts are inside, as well as an expected output column, so what kind of, uh, label should be applied.
- 4:56
And, um, on the, those three different datasets, the, the idea is on the fit dataset, you run it, you look for the errors and especially error clusters, and, um, Opus should then formulate hypothesis for prompt updates.
- 5:09
In the validation set, uh, we then check if those prompt updates actually also generalize to unseen data. And then finally, when we're, like, reaching a plateau or our stopping criteria, um, then, uh, we're running it on a test set to see how well we actually, um, generalize for untouched data that was not part of the training process.
- 5:28
Our base prompt is a very flat list of labels and just a simple task, classify this paper with a label. Um, of course, if, uh, if we would write the prompt, we would probably add some pros, how to think about it and all of this, but we just wanted to see what happens if we use the very
- 5:42
base version, um, of this prompt, uh, and how the, the system is dealing with it.
- 5:48
And our leap, uh, loop- Is a step-by-step instructed, uh, loop, um, done through a task, uh, markdown file. Uh, so the overall idea of the loop is first you run the base prompt on the fit and then the validate set to get, like, a baseline accuracy.
- 6:02
You score it per item and overall. Then on the fit dataset, you do this error analysis. So you really look into what kind of categories frequently are done right, what kind of, um, uh, item pairs are maybe often confused, and also what's then the, uh, underlying abstract and title, um, belonging to it so we can maybe find
- 6:20
patterns there. Then the, uh, then Claude, um, proposes an update, uh, for the biggest error category and pr- publishes a new prompt change, and then reruns and only ac- um, accepts if this, uh, also improves on the validation set.
- 6:34
And we have two stopping criteria, either it's 15 runs, uh, completed or we have 92% accuracy reached. And as soon as this is happening or, um, it stops for some other reason, then we perform a final, uh, run on a test dataset to see it.
- 6:48
So yeah, here on the right you can see all of this is described in a markdown. We point at the datasets and we not even give it the test dataset link, um, so it, uh, will not look into it before we want it to.
- 7:00
So what happened? Um, yeah, I mean, overall we can see here that there's a, a tendency to increase, uh, but also we can see that, um, yeah, we reached a 15% uptick, uh, from our baseline, which is actually quite solid.
- 7:12
And the first run, uh, ended at, uh, 68, uh, percent accuracy. And on our fourth, fourth iteration, um, we went, uh, all the way to 83%, which then also afterwards, um, plateaued a bit.
- 7:25
Overall it kept this level around 80% and we're wondering, okay, actually we wanted the perfect clear-cut target function. Uh, but then looking deeper into the data and how those labels are chosen, uh, we realized, okay, there's, um, some creative freedom for the authors to choose what kind of label they want.
- 7:42
Uh, so even though our description might make sense, uh, they might have chosen in that moment, uh, that they actually wanna go for a different label and that's, uh, then also of course not, uh, recorded in here.
- 7:52
And we also saw this, uh, improvement then generalized to the, um, test dataset on a, yeah, to 80.2%. Um, so the generalization was also on the 300, uh, item unseen data, um, quite good and, uh, in a way that you could say, "Okay, this actually worked."
- 8:10
And, um, the most interesting thing, uh, we thought is that the first iteration immediately gained 10% and then it was only a little bit movement. So, uh, it somehow got a lot of information from this very first run already and made a, made a big uptick, uh, that we can, can see here, um, up until the, yeah,
- 8:27
very close to the final result also. And we, um, wanted to look closer into what it actually did here. So here we can see on the left side the, um, flat label list prompt.
- 8:37
Uh, so here on the left, again, a screenshot from our platform where we, we managed the prompt and on the right side we can see what happened. So the very first thing it, um, added is a general classification approach.
- 8:50
So how should the model think about, um, the behavior of, uh, choosing one of these labels? So when-- W- what's, what's the main goal of doing so? Then there's some information on how to decide between two very similar classes and then also some, um, related confused patterns.
- 9:07
So it's, uh, basically telling it, um, when to choose which and under which conditions and how to go about it, and to rather go for the more specific label than for the more broad label, because there's some implicit nesting in the label structure as well.
- 9:21
And then it also added some, um, examples for the, the item pairs that it, um, yeah, frequently missed or frequently misclassified. So this is what, uh, we can see here.
- 9:32
Uh, so overall the approach was, um, improvement loop, added rules and examples. And, uh, I would've probably, I was surprised, I would've probably, um, spontaneously added descriptions to the labels.
- 9:42
Uh, but the prompting guide or the model together with the prompting guide, um, decided that actually the right approach is here and I mean it worked. So, uh, I'm not gonna, gonna interfere here.
- 9:53
Um, yeah. What we wanted to dive deeper a bit more then is what actually happened in this one 10% uptick, uh, that we have. So let me remove myself a little bit because we need the bottom right corner.
- 10:05
So this is the, um, the reasoning step that Claude Opus 4.8 took, uh, in this one jump from 68% to 78%. And what we can see here is that it has a very clear, "Okay, we have 64 errors.
- 10:18
These are the dominating patterns. These are some number one answer sync, and this is the biggest confusion." So also what are the two labels that are most frequently confused and where there's, like, um, issues coming up.
- 10:30
And then it's forming a hypothesis and saying, "Okay, this loop will extract, uh, exactly address, uh, this pattern." So it got a very clear cut, um, clearly quantifiable and reliable failure mode, uh, because they were just wrong, uh, on a yes/no, right/wrong basis.
- 10:47
Um, they also had, it had a lot of data to look at. We had a 200 item, um, fit dataset, which probably in most real cases, um, yeah, it takes some time to get there and to also, like, make sure that it was only 10 labels, so each of them got covered, like, um, enough times that, that
- 11:04
it can get a signal from it. Um, yeah, and like this it could form a data-backed, uh, hypothesis and ended up, uh, returning a 10% gain on the first, uh, run.
- 11:14
And then, I mean, it was a bit of movement and we also found a better version then, uh, but the biggest jump was, uh, just the very first one from a very clear-cut signal.
- 11:21
We could have probably stopped there and already have very good baseline.
- 11:27
So, um, given this right/wrong high signal feedback works really, really well on this classification test, but also knowing that this is not, um, the case for all others, like there's barely any deterministic yes/no target functions.
- 11:39
In most cases you will run it this time and next time you get a different, um, next time you run the same evaluator you get a different answer from the same kind of evaluation r- you ran.
- 11:50
Um, just because, yeah, LLM-as-a-judge is also non-deterministic. We were wondering how can we translate this right, wrong, high signal feedback into other applications? So how can we try to shape this in a way it also works for your, um, vertical domain expert AI agent or for your, uh, automation of requirements extraction in medicine device, um, compliance or
- 12:12
whatever your use case might be. And, um, for this, um, we overall see the approach working that, um, while the market for, for good reason likes those, um, evaluators like correctness or helpfulness or hallucination, which like early in the days, um, were, were a good, um, thing to go for.
- 12:32
Um, this is actually if you're trying to, um, im- auto-improve against this, a rather low signal, especially also because it's, uh, put on a, often on a scale out of like between zero and ones, one and 10, one and five.
- 12:45
And for this to really work properly, you need to define each of those numbers. What does it mean? In which context would you need which number? Uh, which kind of criteria need to be met?
- 12:54
Uh, but this is most of the time also not done, so it's just choosing a number between zero and one, and, um, depending on the, uh, context, it might just, uh, totally change perspective and is therefore a rather low signal and also probably inconsistent, um, across runs.
- 13:09
What we see, uh, working instead is, uh, really looking into, um, what are the quality criteria that you want to work, uh, for your application. So what does good mean?
- 13:20
Uh, so for example, instead of correctness, the answer is based on a knowledge base, yes/no. So for example, if there's a snippet about internal information in the answer, and you can check, is it actually also in the retieve- retrieved context we did in the previous step?
- 13:34
Or also, if you're frequently struggling with cor- using your correct brand voice, you could look into, okay, um, is it this time correct? Did we make sure that our name was written correctly?
- 13:45
I've also seen companies that have English names that are checking for, did we not accidentally translate our name to Spanish? Things like this. And also, um, uh, one thing that works well because it's also this categorization, yes/no, is maybe some known failure mode.
- 14:01
So which out of these five types of failures, uh, happened here, and, um, can you categorize it? And these are only a few examples. Um, we usually see that these are created by looking into the data, understanding what good means, what are the things that are important to you and especially to your domain experts, and creating these
- 14:21
high signal, um, feedback loops. Um, because if you can't do code compiles, you need to wrap your head a little bit differently around, um, what is good and what is not.
- 14:31
So, um, this is the one side. The other part is, of course, um, the volume that you need. We had a, like, uh, luckily we could have used thousands of examples.
- 14:39
There's so many papers out there. Uh, we opted for like 200 train, 100, uh, validate, um, 300 test, um, just because we wanted to see that it properly generalizes.
- 14:48
Um, but we made the experience that this already go- gave back a lot of, um, high signal feedback. Um, so depending on the complexity of your application and the steps in there, of course it might differ.
- 14:58
Um, but having the volume there and having the high signal feedback, um, loops are the foundation. So how do you get there? So the most important thing we see is working with your experts.
- 15:08
And this is like repeated advice and everyone's saying it and everyone's saying, "Talk to your customers." Um, but here it's something you can actually start to encode the domain expertise into very concrete examples.
- 15:21
So use them to create examples. What should come out of it? Look at those samples run, sample runs together with them and also ask, "Okay, but why is it like this here and here, um, the other way?"
- 15:31
Uh, because like this, you can often also get information out of them that they implicitly think is clear and also, um, really understand, um, how the decisions are taken there and like this identify failure modes of your application and also define what good m- means because you probably need both, and then use them to find these high
- 15:48
signal evaluators. Then, um, as soon as you have this baseline and, um, you might hit production, um, review this data and don't review it only with your coding agents but review it as a human.
- 15:59
Look through it, understand like what is the, the scope initially and, um, uh, what is then being done with this application. This can shift over time. The failure modes can change.
- 16:11
Um, people might try to do other things and its potential for feature, um, expansion and to really understand how the system works and fails in production, um, is necessary to, to cover the typical failure modes and catch them, uh, before your customers do.
- 16:26
And then, um, finally, um, think about the whole thing as a system that generalizes. So you're trying to build up examples that are res- representative of what you might want to hit, uh, in production at some point.
- 16:38
And, um, for this to properly generalize, you need to, uh, bake in mechanisms for validation. Uh, so like in traditional machine learning context, um, the validation approach is actually quite typical.
- 16:49
So people, um, all the time make sure that they're actually validating if, uh, things don't overfit. And, um, this combined with like a real mechanism, combined with like the instructions in your loop and giving the system an escape hatch instead of having it work for hours and hours, uh, hitting a wall and, uh, burning tokens, um, yeah,
- 17:09
are very important levers, um, to actually, uh, make sure that you're not, um, just burning tokens but actually looping towards, um, a system where a human and your agent together collaborate on an improving system.
- 17:23
Perfect. Thank you so much for, uh, listening to me. I'm very excited that you took the time and if you want to know more about Langfuse and also how we think about these topics, you can find more on langfuse.com and I'm super excited to meet some of you at some point in person.