AI Engineer World's Fair 2026
Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse
Read the talk
Continuously Improving Agents with Langfuse
Lotte Verheyden and Annabell Schäfer use Specs, a dad IT support agent, to turn production conversations into targeted evaluations, investigate failures with a coding agent and build better regression tests.
From a talk by Lotte Verheyden and Annabell Schäfer
At a glance
Ideas worth remembering
Record nested model and tool observations with useful inputs and metadata; incomplete traces limit both monitoring and future test coverage.
Choose evaluator inputs deliberately: disagreement needs conversation context and the latest user message, while scope checking needs the system prompt and the request.
Use scores to find conversations worth reviewing, then inspect their traces to understand causes. A frustration signal alone cannot diagnose retrieval failures or retry costs.
Maintain evaluation coverage, criteria and representative datasets over time, and test proposed changes for regressions before deploying them.
A healthy system can still give the wrong answer
An agent can be running normally while giving bad advice. Requests complete, infrastructure looks healthy, and the response still fails the person asking for help. Lotte Verheyden and Annabell Schäfer begin their Langfuse workshop with this gap between operational health and answer quality. Because agent behavior is nondeterministic, monitoring whether the software runs cannot establish whether its responses or actions are useful.
The improvement process has two working environments:
- Online: tracing and monitoring. Instrument the application, observe production interactions, score relevant behavior and review user feedback.
- Offline: datasets, experiments and evaluation. Turn observed cases into structured tests, run proposed changes against them and check for improvements and regressions before deployment.
For teams outside high-risk settings, the presenters recommend starting with tracing and monitoring: find a problem, make a simple correction and observe the next production interactions. More mature or consequential applications need the structured testing of the offline process.
Everything downstream depends on what the trace records. A trace contains observations of individual steps, including model calls and tool calls, which can be nested. Useful model-call data includes duration, token usage, cost, model identity and prompt version. Tool observations need information appropriate to the operation: a web search, for example, should preserve its query. Missing this context limits what monitoring can discover and which failures later enter the test dataset.
Monitoring serves several purposes at once. Aggregate metrics reveal changes in cost and evaluated quality over time; user feedback helps locate individual conversations worth investigating. Automatic evaluators can score large volumes of traces, but they tend to look for problems someone has already anticipated. Manual review and human annotation scale less easily, yet help discover new or nuanced failures. The workshop combines these sources rather than expecting one score to explain the whole application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow a Bluetooth question through Specs
Specs: The Dad IT Support Agent gives this monitoring work a concrete purpose. It grew out of Schäfer's father's WhatsApp questions about phones, printers and computers; the workshop uses a small web interface. Specs can retrieve support context—such as the phone, operating system and installed apps—and search a help library of frequently asked questions. It uses the OpenAI API to generate its replies. Those tools give it ways to ground advice in the user's circumstances.
The workshop repository contains a TypeScript application using the OpenAI Agents SDK. Setup consists of cloning the repository, copying .env.example, adding the application's API credentials, installing dependencies and starting it with npm run dev. The Langfuse account setup creates a project and supplies API keys. Tracing is already wired into this sample, so the demonstration can concentrate on interpreting observations rather than implementing instrumentation.
The first request is ordinary: “How do I turn on my Bluetooth on my iPhone?” Specs uses tools and returns step-by-step instructions. What happened between the question and that reply? The trace view provides the useful comparison: the interaction is rooted in an agent observation, with generations and tool calls beneath it. Their nesting connects the overall request to the model activity and context retrieval that produced the answer. This is the level of detail needed to investigate a plausible reply that later proves unhelpful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the disagreement judge the right messages
The first monitoring question comes from the support task itself: did Dad disagree with the advice? A follow-up such as “Where's the button?” may reveal that the instructions were unclear or insufficiently detailed. Switching to checkpoint/04-monitoring brings the sample to the evaluator setup. Langfuse offers code evaluators for deterministic checks and LLM-as-a-judge evaluators for judgments that depend on the meaning of a conversation.
The disagreement template needs a model connection. Schäfer connects OpenAI, while noting that other models or an existing proxy gateway can also be used. Next comes a separate decision: which observations should the evaluator inspect? This check targets the root agent observation and its input, rather than an individual generation or tool call. The desired signal concerns the user's response to the interaction.
What context lets a judge distinguish a fresh question from disagreement? The evaluator prompt receives two mapped inputs: the conversation history and the latest user message. JSONPath mappings select the relevant parts of the observation; the history mapping pulls the whole message array, including the question, tool activity and generated reply. The latest-message mapping singles out what the user is saying now. Together, these let the judge relate a complaint about a missing Bluetooth button to the earlier instructions instead of assessing the complaint in isolation.
Template variable Object field JsonPath
{{conversation_history}} Input $.messages
{{last_user_message}} Input $.messages[-1:].content
Executing the evaluator makes it available for incoming observations. Its purpose is to surface conversations for investigation: perhaps an instruction names a button the user cannot find, or assumes knowledge the user does not have. The example of a handful of troublesome questions among hundreds is illustrative, not a measured failure rate. The practical aim is to find those exceptions without reading every successful exchange.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use code for an observable frustration signal
The second evaluator starts with a familiar habit: when a chat agent becomes frustrating, the user starts typing in all caps. This is a concrete text feature, so the workshop uses a code evaluator. Its Python function extracts the relevant user message through JSONPath, checks for all caps and returns a Boolean score. A true score flags the capitalization pattern as a possible frustration signal; capitalization alone does not establish the user's emotional state.
The choice between code and a judge follows the question being asked:
- Disagreement: interpret a user message in the context of earlier advice.
- All caps: detect a specified property of the message text.
- Other explicit text rules: check selected words or brand spelling. Schäfer gives the example of catching an incorrectly capitalized F in Langfuse.
The code evaluator supports Python and TypeScript. These checks are useful because their meaning can be stated narrowly, while the more contextual disagreement judgment requires interpretation.
The all-caps evaluator again targets live agent observations. Before enabling it, a small test run on an ordinary Bluetooth question returns false, matching the expected negative case. The same evaluator can also run against offline experiments; here, Execute enables it for production observations so the next conversation can generate a score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a failed instruction into a filterable trace
The live test returns to Bluetooth. Specs answers the initial question, then receives frustrated feedback and a direct objection: “No, there is no such Bluetooth sign.” It acknowledges the user and offers another approach. This develops the earlier example into a useful failure case: an answer was delivered, but the user could not apply its instructions.
Where does that failure signal land? The root agent observation receives a user-disagreement score of true. The trace retains the exchange and the response offering another way forward, while the attached score makes the interaction searchable. Filtering for disagreement or all-caps frustration can now bring likely problem conversations to the front of the review queue. The score identifies a case worth examining; the conversation explains what happened.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare the user's request with the system's scope
A third monitor asks whether the request belongs in this application at all. Specs is currently scoped to phone IT support, even though Dad also has a printer and computer. A printer question can therefore reveal a mismatch between what the user wants and what the application promises. That is a different problem from a bad answer to an in-scope question.
The out-of-scope judge needs the application's instructions as well as the user's request. This time the evaluator targets generation observations, where the system prompt is available. The intended mappings select the first message as the system prompt and the latest user message as the request, then ask the judge to compare them. During setup, Schäfer notes that the variables are not mapping at that moment; the recording proceeds to Execute without demonstrating a scored out-of-scope request. The intended comparison is clear, but its successful operation is not established by this demonstration.
An out-of-scope signal supports two different product decisions. Repeated requests may justify expanding the application to cover more devices, or they may show that users need a clearer explanation of its limits. Choosing between those responses requires reading the cases. Evaluator templates provide a starting point, but the monitoring plan still has to reflect the application's actual failure modes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let a coding agent investigate beyond the configured scores
A seed script adds a collection of traces and scores directly to the Langfuse instance, producing multiple pages of data for investigation. These are seeded examples for the workshop, rather than newly accumulated user traffic. They make it possible to explore patterns across conversations and identify cases that could later become offline tests.
The Langfuse CLI exposes access to traces, scores, datasets and experiment runs; the accompanying skill helps a coding agent work with that access. After demonstrating installation with Claude, Schäfer switches to Codex and asks it to find issues in the traces. The investigation includes configured scores such as disagreement and frustration, but also tool behavior and latency. Access to the underlying observations lets the agent inspect problems for which no dedicated evaluator exists yet.
What can this broader inspection find that the initial monitors missed? Schäfer first opens a prepared report—joking that she is not “p-hacking”—then compares it with the new run. The prepared analysis separates three findings:
- Retrieval quality: the search help library is silently failing to retrieve useful support information, a failure the existing evaluators do not target.
- Retry costs: retrieval misses and repeated tool calls add latency and cost.
- Evaluation coverage: only a subset of recent traces has evaluation signals, leaving part of the traffic outside the monitoring picture.
These are findings presented from the seeded examples, rather than a quantified production performance study.
The new Codex report also flags a prompt-version issue, the latency cost of retries and a monitoring coverage split. The overlap gives the presenters concrete issues to pursue. It also exposes a limitation of the original scores: disagreement and all caps can highlight unhappy users without explaining whether retrieval, retry behavior or something else caused the trouble. Inspecting tool observations supplies another level of diagnosis.
Schäfer chooses to pursue removal of the search help library tool because it appears not to be helping. With repository context as well as trace access, the coding agent can propose changes to application code, not just Langfuse configuration. It begins that work while the workshop moves back to the improvement process. No completed patch, regression result or measured improvement is demonstrated, so removal remains a proposed intervention whose consequences need testing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep production examples and evaluation criteria current
Finding a suspicious tool and asking for a code change does not finish the improvement work. Evaluation criteria themselves need maintenance: a score may overemphasize one behavior or stop telling the team something useful. The dataset needs the same attention. Initial test cases reflect what developers expect people to do; production interactions reveal edge cases those expectations missed.
Useful production examples should therefore become offline evaluation cases. For a proposed change such as removing a tool, those cases provide a way to check whether the agent still handles representative requests and whether a correction introduces regressions elsewhere. Experiments run against the maintained dataset can become part of CI/CD. The dataset's value depends on continuing to represent real use, rather than remaining a frozen collection from the application's first version.
The closing recommendation starts small: add tracing to the repository, manually review interactions to understand what users actually do, then build production evaluators around the failures that matter. Use those signals to develop the offline tests and experiments. The accompanying eight-step workshop extends into prompt management and dataset setup. Continuous improvement requires “putting in the reps”: revisiting both the criteria and the examples as usage changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Develops the optimization side of this workflow: using execution traces and diagnostic feedback to change prompts, agent programs and evaluators.
Read the complete timestamped transcript
- 0:13
Hello, everyone. Welcome to the workshop on Continuously Improving Agents with Langfuse. Um, you can already go to langfuse.com/workshop. There are steps there that we will go over today, and that way you can also later at home, when you want to revisit it, go over it yourself. And there are also some extra steps that we won't cover today that you can find then as well. We'll show this QR code later in the presentation, uh, again. So Continuously Improving Agents,
- 0:43
what does it mean? We'll be talking a lot about, um, different processes that teams use to then, uh, to improve their agents, uh, or to detect changes and improve their agents in different ways. And you can change these together into a continuous loop. And as we know in San Francisco, we really like loops. And so this process and the set of... these set of processes together, you can... it's also often called AI engineering.
- 1:14
So what is AI engineering and why do we need it, and how is it different from traditional engineering? The main difference is that AI agents are not determi-deterministic. So your system, your status of your system can look healthy while the responses or the actions your agent is taking are wrong. Sometimes it's actually wrong, sometimes it's also not qualitative enough.
- 1:44
And so, um, this is what AI engineering solves. For the people who are not familiar, um, Langfuse offers a platform to actually execute on these processes and, uh, we are open source, model agnostic, framework agnostic, so you can actually use this with whatever setup you already have. There are some large companies that already use this day-to-day, but we also have a large community of smaller individual users who really help us because they're working on many different
- 2:14
setups and agent projects, and this is definitely one of the, um, ways we keep up to date with where this very fast-moving field is going. By the way, uh, this workshop is given by Annabell and me today, and Mark, our founder, is also somewhere in the room. And there he is, and we will also be walking around to help you, um, during the workshop if you need anything. So AI engineering, um,
- 2:44
we like to show or visualize the set of processes that, that people do in this graph. Uh, this is what we call the AI engineering loop. The exact setup depends a bit on the setup also of companies, um, it depends on different needs, but high level, you can categorize them in these five steps. And the main thing here to, um, remember is we have two big parts. We have an online part, tracing and monitoring, and an offline part,
- 3:14
building datasets, experimenting, and evaluating, not in production. And
- 3:21
we are today mainly going to focus on the online part here, tracing and monitoring. So tracing, you instrument your application, you deploy your app to production, and you will see traces coming in of what your agent is doing, how it's interacting with users. Based on that, you will be monitoring these traces. And monitoring is, on the one hand, automatic with evaluators and with dashboards, but also the human feedback part cannot be underestimated here. You will also be
- 3:51
collecting implicit or explici-explicit user feedback and also manually be going through some samples of these traces. This together, I don't know if you can see... Oh, the yellow, um, box is actually quite clear. Great. Um, this together, the online part of, uh, the AI engineering loop is already a lot. A lot of teams that are not in high-risk industries can actually just start with these two and based on what they find during monitoring of their traces, can already,
- 4:21
um, make simple fixes or improvements to their agents, deploy these to production, and see their traces coming in. This is a great first step, and it will already get you very far. Um, and it, it's a great way to iterate on your agents fast. Once your product matures a bit, your setup matures, or you are in a high-risk environment where any mistake has bad consequences, um, it makes sense to also work on the offline part. So that's really for structured testing. So your datasets are your structured test cases.
- 4:51
These will be heavily inspired, by the way, by what you see during monitoring. And when you want to make a change, you run an experiment on your datasets, you evaluate the results of the experiments, and only when the results are better, you didn't introduce any regressions, that's when you deploy to production. Before we go into the workshop, we very quickly want to focus on the two parts in the online loop, um, so that's tracing and monitoring, because this is really important. Everything starts with your trace.
- 5:21
So because if your trace is bad, you will not be catching everything during the monitoring step. If you don't catch everything during the monitoring steps, your datasets will be not as good as they could be. You will not make the best-informed decisions on what to evaluate, et cetera. So it's really important to take enough time to make sure your traces are set up well. So what's... how do you build a trace? A trace exists, um, consists of observations, and observations are different steps an agent is taking while
- 5:51
creating an answer for the user. These consist of LLM calls, or these can also be tool calls or other things. These can also be nested. Um, each observation also has a lot of data attached, and this depends on the kind of observation. For an LLM call, this is usually the duration, token cost, um, token usage, model that was used, a prompt version that was used. Everything you can think of that will actually help you assess the quality, um, of your LLM call. And
- 6:21
for tool calls, this can be completely different. For example, if you do a web search, one data point can be the query that was used to do th-this web sea-search. So this is really important so that you can then do monitoring well. Monitoring, there are a couple of different functions of monitoring. One is you want to have aggregate metrics to then see overall continuously over time how is the health of your system, both things like the cost of your system, for example, but also the
- 6:51
quality. So if you set up evaluators, uh, to measure the quality of your responses, uh, you want to make sure this is not going down over time. A second very important one is signal detection via user feedback. You will have, as your application scales, a lot of traces, and it will be impossible to see, um, or go through everything, and user feedback is a great way to surface the ones that actually are important to look at. Third one is feedback and evals. So you will set up e-automatic evaluators
- 7:21
and, uh, also you will do some human annotation to make sure, um, you catch new things. One important thing here, so you have the automatic evaluators. These, on the one hand, are very scalable. You can keep on doing them on all your traces. On the other hand, you will-- you're less likely to catch very new or nuanced cases that you didn't think about on-- by yourself. Um, but user feedback and human annotation might scale a bit less well,
- 7:51
but these are where you will detect really things you haven't thought about yourself. So this is what we'll focus on today, and my conne- colleague Annabell will go through a practical example, and, uh, Mark and I will, will be walking around. If you need help, just raise your hands.
- 8:07
So today's focus, as Lotte already mentioned, is all about how to effectively working with the trace of your application. Also, how to decide what you actually want to monitor, so what is interesting for you in your application and not, like, generally. And then also, um, how you can leverage coding agents to, uh, ten-x yourself a little bit, but also maybe where you should, uh, be in the loop and not let the coding agent do everything on its own. So, uh, let's dive into it. And, um, to do so, uh, we first wanna meet our sample application for today. And
- 8:37
we brought for you Specs: The Dad IT Support Agent. So this, uh, mini repo came out of a, a mini project, um, I did, uh, for my dad, uh, because my dad always asks me questions on WhatsApp, uh, about how, how to set up his phone and also how to set up all kinds of things with his printer or his computer. Um, so I built this little, uh, agent that is replying to him, um, in the production version via WhatsApp. In this case, we now have a, a nice little UI. So how you can imagine working it is we have a user, my dad or whatever dad,
- 9:07
and, um, the dad is asking Specs a question, and Specs has a few tools available. So we can get support context or some, um, information about, uh, what kind of phone is Dad using, and what kind of operating system is it on, what kind of apps does he have. And we also have a search help library, so some frequently asked questions where there's some context on how to go about solving certain issues. And, um, yeah, it can use these tools. And then we also use the, uh, use OpenAI, the API endpoint, uh, to generate replies and get a
- 9:36
reply back to our, uh, dad in this case. So this application, we're gonna first, um, look at trace together, but then also think about what is actually interesting that we wanna know about, uh, to make sure Dad gets the support he actually wants to have. So, um, yeah, you can scan the QR code here, and th- you will find the workshop repository, and, um, I'll get, uh, this set up together with you. So no worries, and we're gonna go through step by step. There's also documentation in case you get lost. You could also, uh, reference there and if you wanna explore more afterwards.
- 10:07
Uh, there's also be gonna, uh, gonna be some more, um, modules available. So let's go to langfuse.com/workshop. And we can see that, um, yeah, this is a, a Markdown page here. Uh, but if you click here on the Langfuse/Langfuse workshop, we actually get to Git repository. And right here on, um, yeah, you can clone, uh, the repo into your, um, IDE. I already did that, um, so it's prepared. And we can see here that the first thing,
- 10:37
um, we need to get set up is, of course, some, uh, env variables. And for this, um, let's create a org together on langfuse.com.
- 10:49
We can go to Launch App. I already have an organization, um, but you can go to, uh, Launch App or also if you're on the page here and then, uh, select a region. It doesn't matter which one you're taking. And then, um, you can create an account and as soon as you're in there, uh, what you will see is this page. So there will be a project auto-created for you, and you will land, um, on this page. And there you can get your, um, API keys by, uh, clicking on, uh, Create New API Keys. Um, so maybe I'll give you just a second, uh, to get to our
- 11:19
website and get this, um, get this set up, and this will be the, the very first thing we have to do together.
- 11:30
And I'm gonna show also here the
- 11:34
steps that you will also find on our, um, workshop website.
- 11:43
Yes. So quickly signing up, creating a new account. And for everyone who now did that already, um, we brought, uh, for today an OpenAI key. So you can scan it. Uh, we're gonna rotate it afterwards, of course. Um, but for today, um, this is your API key to use, and you can add this into your env, uh, file, and then we'll, uh, get started.
- 12:11
Yeah, so just copying the .env.example and then getting there.
- 12:28
Okay. So when we have the, uh, repo cloned, the first thing we have to do is, um, we have to quickly install the dependencies. So we're running here a TypeScript application that is using the OpenAI, um, agent SDK. And, uh, yeah, like this, we can quickly install all the dependencies, um, it is available. And then we can also start it by running NPM run dev.
- 12:55
And I'll pull this out here. It's opening here directly. I'm also gonna open it here. And we can see our, um, yeah, our sample application. And by the way, if I'm a little too fast, like here are the steps, uh, in the setup page on Langfuse.com/workshop, um, how you can also get there, um, step by step by adjusting the, um, .env file and then running, uh, the application. So let's see what we have here. We have our, um, dad support agent, and it's already greeting us, uh, with certain questions. If I
- 13:25
have maybe questions, I also have some, um, sample questions that dad might ask. Um, so we can say here, "How do I turn on my Bluetooth on my iPhone?" And then Specs is taking a second and using in the background the tools, um, that we're like, uh, needing, um, to get there. And then we reply to our dad in this case, uh, now with the step-by-step instr-instructions on how to, um, yeah, how to turn on Bluetooth on their device. And, um, what happened now in the background, that is now, uh, the interesting part,
- 13:55
um, that we wanna look into. And if we now click back into tracing, and this is now also what should happen for you if you run the application and ask the single question, then, uh, we see the first trace popping up. So every row here is an observation, and we can also see that they have little different icons, and these different icons mean that they are different things. So if we click into one of them, we can see that the whole thing is rooted in our, um, in our agent observation. And then below we have a few generations, but also tool calls
- 14:25
where certain context is pulled. Um, so this is now already a fully instrumented application, and there's also a step-by-step, um, guide on, on how to get this actually set up. Uh, but we also have a Langfuse skill that you can use to automatically get it set up. And for today, we actually wanna focus on working with those traces instead of step-by-step, uh, setting them up. So, um, yeah, here we have our trace. And if we now, uh, get back to our, um, Langfuse.com/workshop page, we can go into the monitoring workshop, uh, module, which is the part
- 14:55
that we actually wanna check out together today, as Lotte already mentioned. So, um, in our context of our dad support app, what does monitoring actually mean? So whenever dad is interacting with the application, we of course don't see what he does, uh, because we don't sit next to him. That's the whole point about the agent. Uh, but we actually wanna track, uh, what's going on. And we don't only want to track, um, what kind of things the agent is doing, but we actually wanna make sense of it. So we thought about what are actually things that can go wrong in a support application. And
- 15:25
the first one is maybe the dad is disagreeing with the application. So for example, um, if I'm asking a question, "How do I set up Bluetooth?" Um, then I'm wondering, okay, um, how does it actually work? Where's the button? And dad is, um, asking this, and we might wanna catch this because there was something maybe not clear enough, not detailed enough, um, for the question to actually be properly, um, done. So, um, let's get this set up together. We're, uh, starting with, uh, um, yeah, with copying this, uh, Git command
- 15:55
here. Git checkout checkpoint/04-monitoring. And then we can get back into our, um, IDE
- 16:03
and run this command. And we can see now that we're swapping or switching to the, uh, monitoring s- uh, step of our workshop, so it's a checkpoint. And now we want to wire up those, um, those evaluators in our application. So, um, if we go down a little bit, we can see that the first thing, um, that we have to do together is configuring the Langfuse evaluator model. So I'm gonna jump into our, uh, application here. I'm gonna close the tracing part, and I'm
- 16:33
gonna hop into our evaluator tab. So here I can see I can get started with my evaluators, and there's two different possibilities. I can either set up, uh, code-based evaluators for more deterministic checks, uh, about my application. And I can also get set up with some LLM-as-a-judge evaluators, um, which are like more for the implicit things that you might wanna get out of the conversation history, uh, and actually wanna find out there. And we can also see that we have some, uh, templates here, and that our, uh,
- 17:03
user disagreement one, um, is actually one template that we are, uh, providing here. So if we click on here, we can see, okay, we need to get up, uh, set up with the, um, LLM connection. So I'm gonna jump over here and very quickly stop sharing because you shouldn't see the API key.
- 17:24
Laptop died.
- 17:29
So this is in the end, um, a possibility to use a certain model to, uh, evaluate, um, what's
- 17:36
going on and to...
- 17:41
Mm-hmm. Give me one second.
- 17:50
Okay. Yeah. So what I now did is I, um, set up a new LLM connection, and in this case, I'm using again the OpenAI API key.
- 18:01
We're back. Yeah. Um, so I connected the OpenAI API key now that we, um, set up together. And, um, in-instead you can also connect all kinds of other, uh, models of course, and you can also, um, use your own proxy in case you have a, a, have a gateway configured yourself. And now we're gonna, um, jump back into our evaluator page. We're gonna click on Create Evaluator and now select our template again.
- 18:26
And select the model. And what we can see here now is that the first thing we can do together is we can filter for the actual, um, observations that we wanna target. And in this case, uh, we wanna go, um, not for the generation, um, but for the agent that is the root, um, of the whole, uh, trace that we're having here. And then you can also see that there are some other options, so you can also target tool calls or generations. But in our specific case, we wanna look at the, um, input only, so we wanna see, uh, the, the message of the
- 18:56
user, um, being looked at. So
- 19:04
looking at our instructions here again, um, we have a certain way to target those different l- different parts, um, of the observation. So we're taking the agent. We have our evaluator prompt here, and down here we can see that, um, via, um, br- curled brackets, we're actually pulling in, um, context here. And to target the part of the trace that is actually relevant for us, uh, we wanna set up this JSON path here and make sure that we're targeting the part, um, that is, that is correct in
- 19:34
this case. So for the conversation history, we actually wanna target the whole message array. So we're gonna copy this here,
- 19:43
and we're packing it in here. And what we can see down here now is that it's mapping in the message history of your trace. So it's pulling together what was the interaction, what did dad ask, what do the tools, um, do, and then also what kind of reply was generated. And as a second part of, um, detecting if a user disagrees, um, we of course, um, need to also target the last, uh, message that is coming from the user and like this also get information here.
- 20:14
And we can see that down here now we are mapping how do I turn on Bluetooth on my phone into this, um, evaluator, and if we now execute this, we have our first live running evaluator. So you can imagine if now a trace is coming in where, um, dad is saying, "Okay, how do I turn on Bluetooth?" And then he asks as a follow-up, "Oh, but this button is actually not there. I don't know where to find it, and, um, it's not true what you're saying." Then we would catch this because this is actually an interesting event, a signal for us to understand, okay, where are
- 20:44
things going wrong? And probably like in the first five hundred questions, there... It's not like all five hundred fail, but maybe there's five or six or seven different ones, uh, where we're not clear enough, and dad might need some more details to actually go deeper into it. So user disagreement is our, uh, first example. Let's look at what also might be interesting. And in our case, we now have one, um, where we're catching with a code-based evaluator if maybe there's some all caps used. So if I think about how I'm interacting with Claude or
- 21:13
Codex sometimes, if I'm getting frustrated, I might just use all caps to get my point across, and I think many of you have also done this in chat applications. And we see especially in chat applications that this is actually a good way of catching some frustration from the user. And, um, in this case it's all caps, but as an alternative, this could also be maybe cuss words or, um, yeah, certain terms that you should maybe not usually use in a chat, um, but when it's an agent you might tend to rather, um, go for it. And, uh, for this we now wanna,
- 21:43
uh, set up a code-based evaluator so we can actually make sure, uh, we get dad, uh, when he's frustrated. So, um, as we can see here, we now have a little, um, Python snippet here. And if we go down a little bit, um, we can see, uh, that here the actual evaluator function is defined. And, um, it's targeting again a part of the JSON path. So it's like extracting, uh, the certain user message, and then it's checking if there's all caps in there. And in case it's doing all caps, then we return a, um, a Boolean score
- 22:13
true. So let's copy this here so we can get this wired up. And I'm only copying the bottom part because the upper part is actually static. And I'm setting up an additional evaluator, in this case now a code-based one. And, um, we also support TypeScript code, uh, so in case you're li- rather operating on TypeScript, this is possible. Um, but we now wanna go for, um, Python code and, um, we call it all caps
- 22:44
frustrated. And since we're go- going for a Boolean score, let's also add a little if, um, so we always know if it's true, then it's, um, he is actually frustrated, uh, with all caps. And then I'm gonna also down here replace this part, um, with our...
- 23:03
Of course.
- 23:10
Ta-da. Copying it again.
- 23:22
And of course down here you can, um, write all kinds of custom code, um, that you wanna go for. So whatever you wanna evaluate, if it's certain words you're looking for or, uh, for example, also if you have some brand name that always should be written the right wa- uh, way. So for example, for Langfuse also sometimes the F is written big. We would usually write it small. So you could also catch these kind of things to see where maybe things are going wrong. And yeah, let's get this set up. So we have our second evaluator created. But then of course we also need to think about, uh, what
- 23:52
kind of observations do we want to target. So let's look again here. Um, we say, okay, of course we wanna run it on live observations, so we wanna know what's going on in production. And we're again targeting the, um, the observation type agent.
- 24:09
And here I can select this, going for the agent one again. And re- on the right side we can also thoo the... see that we can do a small little test run on it. And here we see that in this case it's, um, returning a false because we're here only asking the questions about the turning on and off Bluetooth. And as soon as we're happy with it, uh, we can then, uh, click on Execute. Uh, just as a short note here, like right now we're running on observations, but this of course is also possible to run on all kinds of
- 24:39
experiments that you might, might wanna do in an offline improvement process instead of running it on live production data. So yeah, right now we're happy with, uh, just running it on live observations, and let's also click on Execute here. Nice. Uh, so let's check how this actually turned out, uh, because we of course wanna know if it also works. And for this, I am jumping back into our IT support agent. Let's kick it off first with, uh, "How do I turn on Bluetooth again?"
- 25:10
We're taking a second, and then we're getting our replies. Zooming in. Yeah, I can try. It's also small for me. I was early. I, like, should have brought more glasses. And yeah, now we can see it. Perfect. Um, so let's get frustrated.
- 25:34
This one first, and of course our agent is somehow dealing with it because we have some specifications in the system prompt. And I might also, um, need of course to reply first. So let's see. We get some more details, and then I can maybe ask, "No, there is no such Bluetooth sign,"
- 26:01
just to disagree a little bit with the agent.
- 26:13
And he says that I'm right. So let's see how this turns out in our traces. Okay, we see we have a few more, um, now here coming in.
- 26:26
And let's check maybe on this one. Okay, we also have some evaluators coming in already. Let's dive in a bit deeper into what's actually going on here. So, uh, we can see that this is now not the... Yeah, we have our user message, "How do I turn on Bluetooth?" We get a reply. And then we can see that this is the output we're getting back. "Okay, Dad, no worries. Let's try the other, um, way first." So here we had some, um, disagreement going on, and up here we can see actually this
- 26:56
little score. So on the top left, it's probably still small. Zooming in. Yeah. We can see now that our root observation, like the overall agent, is actually tagged with a user disagreement, um, true, uh, because here the, the user actually said, "Okay, it's actually, um, in this case also all, all caps, um, it's actually not true, and it didn't work out." And, uh, we can see this here now and have, like this, also signals. So we could in theory now filter for the observations that are
- 27:26
having a, um, maybe user disagreement true or also a, um, a all caps frustration, uh, true, and like this find the ones that are actually interesting for us to work with.
- 27:39
Okay. So these are the ones where we get something out of the more or less, um, production behavior, um, of the people. There's also one additional thing that is often interesting to monitor in a production application, mainly being, um, if someone is trying to do something that is actually not supposed to be done, um, with the respective, uh, application. So we call this a out-of-scope request. So in our system prompt, um, we are specifying here that the agent is supposed to help with all kinds of phone IT support questions. Uh, but of course, uh, Dad also has different kinds of
- 28:09
devices, so maybe a printer, also a computer that he wants to, um, get, uh, get questions answered with. But right now we're scoped, um, to this, um, specific scope. So for this we can also, um, set up these kinds of evaluators, and we are calling them, in this case now, a out-of-scope request. And here we have a different, uh, dynamic. We're not looking at only the last user message or the message history, um, but here we're actually also looking at the system prompt. And because we are defining all of this via this
- 28:39
JSON path, we have the possibility to target this also specifically. So, um, one final time, let's get this evaluator, uh, set up.
- 28:48
And for this, we're jumping into our, um, evaluators page. We can click on Evaluators, and once again, um, we can select our out-of-scope request template.
- 29:02
We now wanna target, um, generations because we also wanna reference the system prompt.
- 29:11
And let's now map our variables again. So we are taking the system prompt, which in this case is the very first message that is sent in this interaction.
- 29:29
And then we're also taking our last user message that is coming up.
- 29:39
And in this prompt now we define, okay, we also wanna... we wanna always compare, like, what the system is actually supposed to do and what the agent is compared to do, uh, supposed to do, um, to what the user's actually trying to do. So this is now wired in.
- 30:02
Not mapping the variables right now.
- 30:10
And let's execute this one as well. Okay, so whenever now Dad would try to, uh, do something with the application that he's actually not supposed to, um, then we can know about this, and then we can, based on this information, take an informed decision. Do we either wanna fix this, so I'll expand the scope of our application, or do we maybe have an issue in communicating to our users what they're actually supposed to do with our application? And in this case, of course, it's only the IT support, but maybe in bigger scope application there's sometimes things people are trying to do and really misunderstand how the system is supposed to use,
- 30:40
uh, supposed to be used. And like this, we can, uh, yeah, catch these kind of things, um, that are going on and go into more, um, detail here. Perfect. So this is how we can set up evaluators and monitors, um, for our application, and I think what you can, uh, clearly see here is that this is not about, um, copying just what's out there and what can use. And of course, like many of these things, you could now reuse for chat application, but it's really about thinking what are, what are the typical failure modes we will,
- 31:10
um, endure in our kind of application and not about, like, trying to copy-paste, uh, what's out there, but really looking into what's going on. So whenever we would want to work, uh, work with this now, we could, for example, now go for, um...
- 31:26
use, uh, like using these, um, scores now to also filter for certain, um, for certain traces and results. Here we have two, and like this get, like, an insight on how often these things are actually going wrong. Perfect. So, um, like this, we can wire up monitors and see, uh, what's going on. And, um, let's now seed some production, production data in to see what is actually,
- 31:56
um, happening and how this performs in our app. And for this, um, we go down to the end of our, um, modu-monitoring module, and we have a script here to seed some, um, production information, and like this have a bit more data to actually look at. So I'm jumping back into my, um, repository.
- 32:22
And I'm copying our command, and what you can imagine is happening in the background is we have just a set of traces as a data set, and this information, um, also includes some scores, and we're seeding this, um, into our, um, into our application, into our Lang-Langfuse instance, um, directly. So let's see if it already is seeded.
- 32:57
Checking out our score filter again.
- 33:02
And we can see now, yes, there's new things coming in. We actually now have a lot of, um, a lot of data, multiple pages of traces. And now the question is, of course, how can we work with this, and how we can we find out, um, what is actually going wrong or also what is going, um, right in our application? And, um, all of this data is usually then used to inform what you actually want to make sure that it's also working in an, um, offline, uh, improvement
- 33:32
process. So these production traces, and especially the ones that you find that are interesting to look at, uh, can then be used to build up a data set that you can use, um, in your offline improvement process, uh, to, to make a better application. Um, but today, uh, we wanna focus on the, uh, monitoring part and the tracing part. So let's see how we can maybe use our coding agent to find the larger scale issues that are in this application. And for this, um, let's install the Langfuse skill together, uh,
- 34:03
so we can, um, use this.
- 34:07
And for this, we have a little command up here in the setup step.
- 34:15
And I, I usually go actually with just, um, telling my coding agent, uh, to install it for me, but you can equally also just, um, install it separately. So we have a CLI, and we also have our skill, and this can be installed then, in my case now, with Claude.
- 34:35
Trusting my application.
- 34:44
And what is now going on is that it's pulling our CLI, and our CLI is exposing all APIs that we have, so you can always access your traces, your scores, your data sets, experiment runs, but also start them through your, um, yeah, command line directly or also with your coding agent together. And like this work more effectively, um, with Langfuse. And this is actually also when you are trying to instrument your application in the first place and try to get your first traces. Um, this is something you can, uh, use to automatically add tracing to your application and get your first
- 35:14
insights. So here I'm now, um, installing Langfuse skill and, uh, just to see how it's then running. I'm going, uh, over to Codex now. I already installed it in the background. And what I now wanna find out what is actually going wrong in my traces. So what are the kind of information, um, steps that are going wrong? What kind of, um, scores do we actually have in there? And I'm asking my coding agent here now, "Please use the Langfuse skill to find issues
- 35:45
in our traces."
- 35:51
And naturally, Codex will take now some time. Uh, but just to understand what's going on, um, Codex is now accessing, um, the live traces that we have in our application. So it's looking for, uh, what kind of questions are asked, what kind of scores do we find in there, um, what is maybe a lot of, uh, all caps fru-frustration, um, going on there. Also, where did the user disagrees, but also beyond what we officially set up. So it's also checking on is there maybe issues with our latency? Is there issues with our tool calls? Um, anything that has been
- 36:21
going wrong, uh, in terms of, uh, what we didn't want to. Um, so to find where it's silently file-failing in production, uh, even though we don't want it, um, want it to do this, uh, of course. And since this is now gonna, um, just run a little bit, uh, I think this is a good moment to also, um, wait for Codex and just take a few questions, um, in case you have any. Okay, let's see if the, um, if we're there. Yeah. Uh, we actually got a report, and I could save this now as a markdown, but I'm just gonna, um, very briefly- Show you
- 36:51
the one that I prepared.
- 36:55
And I'm swearing I'm not p-hacking. Uh, we can also look at the proper one. Uh, but just wanted to show you how this looks like. So what our coding agent now did is ran... it ran over the, um, production traces that we, we seeded in there. It was like around one hundred and twenty traces, or it was seventy traces. And it actually looked at all the different dimensions where things could be going wrong. And we get a quite, um, detailed report now, um, about like how many traces were there, how many observations did we have. Also, how many of those were tool calls and what we have in there.
- 37:26
And then it also generated us some findings. And the first finding here is actually that, um, we have a silent retrieval quality failure in search help library. So this is actually something we didn't evaluate for, um, but, uh, we also didn't realize before that our, um, lookup of information, uh, tool, uh, where we're looking like what kind of, um, question could, uh, what kind of answer could be the right answer, that is actually, um, silently failing and not actually, uh, doing what it's supposed to do. So that's the
- 37:55
first one we see. Um, but we also see that, um, the retrieval misses are increasing latency and costs. So we have some issue in like tool retry calls, um, where we are, uh, detecting some kind of, um, problems. And we can also see that there's, um, an evaluated coverage gap that, uh, is across like recent traffic. And, um, we, we actually don't, uh, evaluate everything, but we only get, um, signal from a subset of the traces, um, that are going on there.
- 38:25
And just briefly verifying, um, if Codex now also came up with the same. Okay, it also detected that the prompt version is, um, is off. And then it also says about the latency tax for the retries. And we have the monitoring coverage split. Yeah, and like this, we can find, um, those issues in our application, but we can then also make sure that we find a way to, um, to fix them. And, uh, like this, um, yeah, improve the, the application over time and make sure, um,
- 38:56
that we, yeah, that we fix the issues that are actually coming up. Okay. So we get the proposal here to save this. Um, but actually, let's see if we can get also a fix in here. So what is the most interesting part? Um, I actually wanna get rid of the search help library one, because if it's not really helping us to use those tools, then we can also, um, remove this. Um, and for this, we can also use our skill.
- 39:34
Our search help library tool. And because we now have the repository context, it actually can also help us, um, fix, um, issues in the code and not only, um, in our Langfuse setup. And yeah, now it's getting, um, it's getting back to work and, um, fixing this, uh, for us in our application. But let's now take a moment to zoom out and understand what actually needs to happen, um, with this. And for this, um, we're coming back
- 40:04
into our, um, slides here again. Uh, so what we now focused on today is actually the part of catching signals from our production traces. So we initially set up our tracing and made sure that we have a step-by-step view into what is, um, actually going on, what kind of tools is our agent using, what's the inputs and outputs, and, uh, get a signal from this. But what does need to happen with those kind of signals to make a continuously improving agent over time? And, uh, for this, like, uh, on the one hand, we need to
- 40:33
maintain our evaluation criteria. So, um, we had this question, for example, on how can we make sure that it's like not over-indexing on something, um, or like not really telling us anything. And, um, yeah, from this signal, we need to make sure that over time, we actually have a good, um, setup of our evaluators and get the information we need to get. Um, but most importantly, we also need to maintain our data set. So when you initially build an application, um, you might think you know what kind of things the people are trying to do. Um, but over time, you will realize that there's a lot of edge
- 41:03
cases that you did not account for and not cover, um, beforehand. And that's why you need, um, to over time, uh, also maintain your data set, um, create good production examples, um, add them to your offline evals, and then make sure that you also can iterate and whenever you make a change, also ship, um, with confidence. So you can make it part of your CI/CD and, uh, make sure that it's, um, yeah, actually not regressing whenever you introduce a change.
- 41:33
So, um, we today only covered the part of the AI engineering loop, uh, where we are looking into the online evaluations. Uh, but actually, of course, um, there's more to it. So it's all about, uh, making sure you know what's going on, you catch the interesting parts, um, if it's a user disagreement or, uh, all caps or if it's cussing, um, or also some kind of, uh, quality evaluations that you can do. And out of this, um, you're building and especially also maintaining data sets over time. So all
- 42:03
of this is not a one-off. You actually need to make sure that you are like putting in the reps and close the loop continuously to, to also have then a data set that represents what your user, users are doing. And on those data sets, you then want to, um, run experiments and evaluate those experiments, um, so you always have a good, uh, indication of how good the quality of your, of your app is. And, um, whenever you deploy, then also know, um, that, that you can ship with confidence. So
- 42:33
this is the core part of our workshop. I just wanted to briefly mention some things. So today, we only looked into that part. Uh, but on our website, you can see in this Langfuse workshop that this is actually a eight-step, um, uh, workshop where you are guided through every single module. So we can only touch upon this one and this one today, but there's also prompt management to Langfuse. You can set up those data sets. And, um, this is a step-by-step walkthrough where you're guided through, um, uh, all of these steps, and like this, uh, can get it set up
- 43:03
properly. And, um, we of course also recommend to, to, uh, to use the, the skill, um, to get this set up together with your coding agent. And if we, uh, were you and we would have to start tomorrow, um, this is exactly what we would, uh, be doing. So we would take our repository, and we would, um, use the Langfuse skill to get up-- set up with tracing to really know what's happening in your application, um, via this command. We then would review traces manually and really understand, uh, what people are doing.
- 43:33
And then, um, yeah, finally get into the motion of building out our traces and set, um, eval setup for our production traces to then get also into the motion of, um, optimizing offline. All right. Thank you so much, everyone. Um, I'm taking a few more questions, but thank you so much for participating. And Mark is around, Lotte is around, I am around, and we're super happy, um, to discuss anything with you. And we also have a expo booth, so please come around the next few days and, uh, chat with us.