Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse

An agent can be running normally while giving bad advice. Requests complete, infrastructure looks healthy, and the response still fails the person asking for help. Lotte Verheyden and Annabell Schäfer begin their Langfuse workshop with this gap between operational health and answer quality. Because agent behavior is nondeterministic, monitoring whether the software runs cannot establish whether its responses or actions are useful.

The improvement process has two working environments:

• Online: tracing and monitoring. Instrument the application, observe production interactions, score relevant behavior and review user feedback.
• Offline: datasets, experiments and evaluation. Turn observed cases into structured tests, run proposed changes against them and check for improvements and regressions before deployment.

For teams outside high-risk settings, the presenters recommend starting with tracing and monitoring: find a problem, make a simple correction and observe the next production interactions. More mature or consequential applications need the structured testing of the offline process.

Everything downstream depends on what the trace records. A trace contains observations of individual steps, including model calls and tool calls, which can be nested. Useful model-call data includes duration, token usage, cost, model identity and prompt version. Tool observations need information appropriate to the operation: a web search, for example, should preserve its query. Missing this context limits what monitoring can discover and which failures later enter the test dataset.

Monitoring serves several purposes at once. Aggregate metrics reveal changes in cost and evaluated quality over time; user feedback helps locate individual conversations worth investigating. Automatic evaluators can score large volumes of traces, but they tend to look for problems someone has already anticipated. Manual review and human annotation scale less easily, yet help discover new or nuanced failures. The workshop combines these sources rather than expecting one score to explain the whole application.

Specs: The Dad IT Support Agent gives this monitoring work a concrete purpose. It grew out of Schäfer's father's WhatsApp questions about phones, printers and computers; the workshop uses a small web interface. Specs can retrieve support context—such as the phone, operating system and installed apps—and search a help library of frequently asked questions. It uses the OpenAI API to generate its replies. Those tools give it ways to ground advice in the user's circumstances.

The workshop repository contains a TypeScript application using the OpenAI Agents SDK. Setup consists of cloning the repository, copying .env.example, adding the application's API credentials, installing dependencies and starting it with npm run dev. The Langfuse account setup creates a project and supplies API keys. Tracing is already wired into this sample, so the demonstration can concentrate on interpreting observations rather than implementing instrumentation.

The first request is ordinary: “How do I turn on my Bluetooth on my iPhone?” Specs uses tools and returns step-by-step instructions. What happened between the question and that reply? The trace view provides the useful comparison: the interaction is rooted in an agent observation, with generations and tool calls beneath it. Their nesting connects the overall request to the model activity and context retrieval that produced the answer. This is the level of detail needed to investigate a plausible reply that later proves unhelpful.

The first monitoring question comes from the support task itself: did Dad disagree with the advice? A follow-up such as “Where's the button?” may reveal that the instructions were unclear or insufficiently detailed. Switching to checkpoint/04-monitoring brings the sample to the evaluator setup. Langfuse offers code evaluators for deterministic checks and LLM-as-a-judge evaluators for judgments that depend on the meaning of a conversation.

The disagreement template needs a model connection. Schäfer connects OpenAI, while noting that other models or an existing proxy gateway can also be used. Next comes a separate decision: which observations should the evaluator inspect? This check targets the root agent observation and its input, rather than an individual generation or tool call. The desired signal concerns the user's response to the interaction.

What context lets a judge distinguish a fresh question from disagreement? The evaluator prompt receives two mapped inputs: the conversation history and the latest user message. JSONPath mappings select the relevant parts of the observation; the history mapping pulls the whole message array, including the question, tool activity and generated reply. The latest-message mapping singles out what the user is saying now. Together, these let the judge relate a complaint about a missing Bluetooth button to the earlier instructions instead of assessing the complaint in isolation.

Executing the evaluator makes it available for incoming observations. Its purpose is to surface conversations for investigation: perhaps an instruction names a button the user cannot find, or assumes knowledge the user does not have. The example of a handful of troublesome questions among hundreds is illustrative, not a measured failure rate. The practical aim is to find those exceptions without reading every successful exchange.

The second evaluator starts with a familiar habit: when a chat agent becomes frustrating, the user starts typing in all caps. This is a concrete text feature, so the workshop uses a code evaluator. Its Python function extracts the relevant user message through JSONPath, checks for all caps and returns a Boolean score. A true score flags the capitalization pattern as a possible frustration signal; capitalization alone does not establish the user's emotional state.

The choice between code and a judge follows the question being asked:

• Disagreement: interpret a user message in the context of earlier advice.
• All caps: detect a specified property of the message text.
• Other explicit text rules: check selected words or brand spelling. Schäfer gives the example of catching an incorrectly capitalized F in Langfuse.

The code evaluator supports Python and TypeScript. These checks are useful because their meaning can be stated narrowly, while the more contextual disagreement judgment requires interpretation.

The all-caps evaluator again targets live agent observations. Before enabling it, a small test run on an ordinary Bluetooth question returns false, matching the expected negative case. The same evaluator can also run against offline experiments; here, Execute enables it for production observations so the next conversation can generate a score.

The live test returns to Bluetooth. Specs answers the initial question, then receives frustrated feedback and a direct objection: “No, there is no such Bluetooth sign.” It acknowledges the user and offers another approach. This develops the earlier example into a useful failure case: an answer was delivered, but the user could not apply its instructions.

Where does that failure signal land? The root agent observation receives a user-disagreement score of true. The trace retains the exchange and the response offering another way forward, while the attached score makes the interaction searchable. Filtering for disagreement or all-caps frustration can now bring likely problem conversations to the front of the review queue. The score identifies a case worth examining; the conversation explains what happened.

A third monitor asks whether the request belongs in this application at all. Specs is currently scoped to phone IT support, even though Dad also has a printer and computer. A printer question can therefore reveal a mismatch between what the user wants and what the application promises. That is a different problem from a bad answer to an in-scope question.

The out-of-scope judge needs the application's instructions as well as the user's request. This time the evaluator targets generation observations, where the system prompt is available. The intended mappings select the first message as the system prompt and the latest user message as the request, then ask the judge to compare them. During setup, Schäfer notes that the variables are not mapping at that moment; the recording proceeds to Execute without demonstrating a scored out-of-scope request. The intended comparison is clear, but its successful operation is not established by this demonstration.

An out-of-scope signal supports two different product decisions. Repeated requests may justify expanding the application to cover more devices, or they may show that users need a clearer explanation of its limits. Choosing between those responses requires reading the cases. Evaluator templates provide a starting point, but the monitoring plan still has to reflect the application's actual failure modes.

A seed script adds a collection of traces and scores directly to the Langfuse instance, producing multiple pages of data for investigation. These are seeded examples for the workshop, rather than newly accumulated user traffic. They make it possible to explore patterns across conversations and identify cases that could later become offline tests.

The Langfuse CLI exposes access to traces, scores, datasets and experiment runs; the accompanying skill helps a coding agent work with that access. After demonstrating installation with Claude, Schäfer switches to Codex and asks it to find issues in the traces. The investigation includes configured scores such as disagreement and frustration, but also tool behavior and latency. Access to the underlying observations lets the agent inspect problems for which no dedicated evaluator exists yet.

What can this broader inspection find that the initial monitors missed? Schäfer first opens a prepared report—joking that she is not “p-hacking”—then compares it with the new run. The prepared analysis separates three findings:

• Retrieval quality: the search help library is silently failing to retrieve useful support information, a failure the existing evaluators do not target.
• Retry costs: retrieval misses and repeated tool calls add latency and cost.
• Evaluation coverage: only a subset of recent traces has evaluation signals, leaving part of the traffic outside the monitoring picture.

These are findings presented from the seeded examples, rather than a quantified production performance study.

The new Codex report also flags a prompt-version issue, the latency cost of retries and a monitoring coverage split. The overlap gives the presenters concrete issues to pursue. It also exposes a limitation of the original scores: disagreement and all caps can highlight unhappy users without explaining whether retrieval, retry behavior or something else caused the trouble. Inspecting tool observations supplies another level of diagnosis.

Schäfer chooses to pursue removal of the search help library tool because it appears not to be helping. With repository context as well as trace access, the coding agent can propose changes to application code, not just Langfuse configuration. It begins that work while the workshop moves back to the improvement process. No completed patch, regression result or measured improvement is demonstrated, so removal remains a proposed intervention whose consequences need testing.

Finding a suspicious tool and asking for a code change does not finish the improvement work. Evaluation criteria themselves need maintenance: a score may overemphasize one behavior or stop telling the team something useful. The dataset needs the same attention. Initial test cases reflect what developers expect people to do; production interactions reveal edge cases those expectations missed.

Useful production examples should therefore become offline evaluation cases. For a proposed change such as removing a tool, those cases provide a way to check whether the agent still handles representative requests and whether a correction introduces regressions elsewhere. Experiments run against the maintained dataset can become part of CI/CD. The dataset's value depends on continuing to represent real use, rather than remaining a frozen collection from the application's first version.

The closing recommendation starts small: add tracing to the repository, manually review interactions to understand what users actually do, then build production evaluators around the failures that matter. Use those signals to develop the offline tests and experiments. The accompanying eight-step workshop extends into prompt management and dataset setup. Continuous improvement requires “putting in the reps”: revisiting both the criteria and the examples as usage changes.