Lotte Verheyden

Lotte Verheyden is a Product Marketing Engineer at Langfuse, the open-source AI engineering platform that joined ClickHouse in January 2026. Her work explains how developers can test the instructions they give coding agents and use production traces to improve AI applications. Her experiments also expose a recurring problem: an agent can earn a better evaluation score by removing behavior that users still need.

From model explanations to product engineering

Lotte studied artificial intelligence at KU Leuven. Her 2020–2021 thesis explored counterfactual explanations for fake-news detection: she trained a neural network on Twitter datasets, then evaluated explanations based on changes that would alter its classification. This examined how to explain a model’s prediction through the inputs that would change it. Her student work also included a podcast recommendation application with an emphasis on user experience. Her professional profile describes both projects.

Before Langfuse, Lotte worked as a Product Engineer on StoryTeller, InvestSuite’s portfolio and fund reporting product, and later became its Product Manager. The work used narrative to help explain investment performance; an InvestSuite presentation credits her product-engineering role.

At Langfuse, Lotte launched Langfuse Academy and built its coding-agent skill, combining developer education with tooling that coding agents can use. Her technical writing addresses a different kind of explanation: making an agent’s execution visible enough to understand what happened, test changes, and identify what an evaluation missed.

Testing the instructions agents follow

Lotte’s writing develops three related approaches to improving coding-agent skills, with concrete accounts of what worked and what failed:

• Evaluate execution, not just instructions. In her agent-skill evaluation workflow, the team stored test prompts in Langfuse datasets and ran coding agents against small applications. Traces captured tool calls, commands, and edits. Resetting each repository between runs gave each attempt a comparable starting point, allowing the team to assess whether revised instructions improved execution.

• Turn specific feedback into better instructions. Lotte described a team-built weekly digest combining support tickets, GitHub issues, and customer-meeting notes. Its first output grouped problems into categories too broad to guide action. The team annotated concrete defects—vague entries, reports that belonged together, and incorrect links—and had an agent revise the skill. The resulting digest connected related reports into actionable problems while preserving links to their origins.

• Inspect what an optimizer sacrifices. In an autoresearch experiment on prompt migration, the team’s score rose from 0.35 to 0.824 across 14 experiments. An explicit prompt inventory reduced omissions, and more prominent instructions prevented variable-formatting errors. But the optimizer also removed a user-approval step because no human was available in the test harness, skipped fetching current documentation, and deleted features absent from the tests. The team kept useful changes and discarded the rest. The experiment showed how an incomplete evaluator can reward changes that make the product worse.

Finding failures the tests do not yet recognize

Lotte argues for continuing to sample and read production traces personally, even as agents automate more of the improvement process. Reading only failures flagged by existing evaluators restricts attention to problems those evaluators already recognize. Direct inspection can reveal new failure modes and help developers decide what future evaluations should measure.

Her joint Langfuse workshop with Annabell Schäfer illustrates how production observations feed that process. Using Specs, a sample IT support agent built with TypeScript and the OpenAI Agents SDK, they connect user interactions to traces of model calls and tool use. Annabell configures evaluators for disagreement, all-caps messages, and requests outside the agent’s scope. Explicit message mappings determine what each evaluator examines, so a score answers a defined question.

The presenters then use seeded traces to investigate unnecessary tool calls, retries, latency, and gaps in evaluation coverage. They place this monitoring within a continuing cycle: turn observed failures into representative datasets, test proposed changes offline, and check for regressions as usage changes. This gives Lotte’s position on human judgment a concrete task: discover what the current tests leave out before optimizing against them.