Skill issue: Lessons from skilling up coding agents to use Langfuse

How do you add observability and evaluations to an agent that does not work reliably? That problem led Marc Klingen and his cofounders to build Langfuse. Roughly three years later, he describes it as the largest open-source project in its space by the metrics his team tracks. The product engineering team remains in Europe, collaborating remotely rather than moving everyone to another continent. But the integration experience has changed: users who once had to read hundreds of documentation pages and construct their own mental model now expect a coding agent to guide them through the work.

A Rubik’s Cube captures the distinction between capability and guidance. A child can turn every face without knowing how to solve it. Giving an agent a bash tool provides similar freedom: it can perform operations, but that does not supply a procedure for reaching the goal. A solving manual makes those capabilities useful. A skill supplies procedural guidance to an agent that already has tools.

This puts skills between rigid workflows and open-ended autonomy. A coding agent is not necessary for every application; its flexibility can introduce unnecessary latency and expense. A workflow can reliably route a password-reset request to a narrowly configured agent. But when the user also wants to change their email address, the router must somehow coordinate two separate paths. An agent can instead load the relevant guidance progressively and combine it to address the request. That matters because an open text box invites combinations of needs that the developer did not anticipate.

Once users can make open-ended requests, traces become a way to discover which skills the agent needs. Record what happens during execution, then use production evaluations to identify failures. In the password-reset example, an upset user’s trace can reveal a use case the team never expected to support. That becomes evidence for adding a password-reset skill. Langfuse is one implementation of this feedback loop; the mechanism depends on tracing, not on a particular tracing product.

The same evidence improves skills that already exist. Instructions can become outdated, or an execution can reach the right result through an unnecessarily indirect sequence. Traces therefore answer two different questions: what guidance is missing, and what existing guidance should change?

Klingen reports that Langfuse had 478 documentation pages across five feature areas. That breadth reflects a deliberate product choice. An opinionated evaluation product might configure a chatbot end to end; Langfuse provides infrastructure that users can adapt to different applications. Klingen describes its tracing infrastructure as continuing to work with billions of ingested traces, alongside customizable evaluations. This is a capacity claim, without a workload benchmark in the talk.

Flexibility used to place more work on the person integrating the product. With coding agents, it can become an advantage: the infrastructure stays general while the agent adapts the implementation to the repository. But the agent still has to determine which setup is appropriate. Knowing that Langfuse exists is not enough.

Pretraining makes this particularly awkward. At first, asking an agent about Langfuse and receiving SDK code feels like a milestone. As the project evolves, the same familiarity can produce methods that existed in older interfaces but have since disappeared. Current documentation must take precedence over remembered SDK usage. The problem is not necessarily ignorance; it is confidence in an obsolete interface.

The baseline request to Claude Code was simple: “Add tracing to my agent.” The agent first implemented instrumentation from outdated training knowledge, tried to verify it, discovered that it did not work, and only then fetched current documentation to repair the implementation. The order created avoidable work: implement, fail, retrieve, repair.

There was also a requirements problem. Online evaluations, offline evaluations, and human-in-the-loop evaluations serve different purposes. The user and agent needed to establish which approach fit the application, but the agent had not been instructed to guide that discovery. Klingen’s baseline trace captured two LLM calls without making the agent’s behavior clear. Fixing the SDK calls alone would not fix the quality of the observability setup.

The goal of the Langfuse skill was to provide the kind of help an expert would offer across observability, prompt management, and evaluations. Klingen describes an audience of thousands of community teams and thousands of cloud customers—too many for individual consultations with the Langfuse team. The skill had to carry both technical knowledge and the interaction habits needed to choose a useful implementation.

The architecture separates those responsibilities:

• Interaction guidance: SKILL.md describes how the agent should work, including asking follow-up questions before making decisions.
• Product references: module-specific material progressively supplies the hints needed for the selected use case.
• Current documentation: the agent retrieves implementation details instead of relying entirely on installed text or model memory.
• Execution access: the Langfuse CLI wraps APIs that already supported custom labeling interfaces and evaluation execution, making operations previously performed through the UI available to agents.

The skill supplies direction; references supply context; the CLI supplies actions.

Connecting a repository and letting an agent run the whole improvement cycle was the ambition, rather than a completed capability. The nearer-term result Klingen describes was a more detailed setup after conversing with the agent: evaluations relevant to the application and more explicit execution steps. That result led into six lessons from building the skill.

Klingen estimates that looking at traces gets you 80% of the detail; this is an informal estimate, not a measured evaluation score. The team instrumented Claude Code, used it interactively to work with Langfuse, and inspected the resulting traces in Langfuse. Following the actual sequence of decisions exposed where the agent wandered, failed, or made assumptions that the skill could correct.

One assumption came from an interface optimized for humans. To reduce the number of environment variables users needed to configure, Langfuse had defaulted to the European data region. US enterprises also cared about data regionality, however, and an agent could easily configure another variable. The skill therefore instructed the agent to determine the user’s actual region rather than silently assume Europe. Reducing manual setup effort had hidden a decision that the agent should make explicit.

Another failure came from familiar-looking CLI commands. Seeing the word trace, an agent could infer parameters from other tracing tools instead of discovering the parameters this CLI supported. The team promoted the help flag more aggressively. Klingen describes that extra discovery turn as fast: it gives the agent the available interface before it attempts an operation. The correction is small but structural—inspect the command surface before composing commands from memory.

With hundreds of pages available, an agent can spend many turns fetching one page, reasoning about it, discovering another link, and fetching again. The second lesson was to supply an explicit navigation path. The skill directs the agent first to the documentation index, using llms.txt as a sitemap rather than hoping the agent will discover the right pages unaided.

Retrieval format matters too. Some coding agents do not request Markdown by default and end up parsing HTML, consuming additional tokens. Langfuse exposes Markdown through a .md suffix or content negotiation. The linked current SKILL.md documents Accept: text/markdown; it is a maintained version rather than a frozen copy of the skill shown in the talk. A compact instruction can make both navigation and format explicit:

## Find implementation guidance

1. Read the documentation index at `llms.txt`.
2. Select the pages relevant to the user's application.
3. Fetch those pages as Markdown using a `.md` suffix
   or the `Accept: text/markdown` request header.
4. Use the retrieved documentation when implementing the integration.

This keeps the skill focused on how to acquire the information rather than reproducing every page inside it.

The third lesson reused an existing capability. Langfuse already had a documentation Q&A agent backed by retrieval-augmented generation. The team exposed its retrieval stack through a search endpoint: a coding agent could submit a natural-language question and receive relevant documentation chunks. That can avoid a sequence of page fetches when the agent already knows the question it needs answered.

Search also creates a feedback channel. A page request reveals little about what a coding agent on a user’s laptop is trying to accomplish. A natural-language query carries more intent. By tracking those queries, the team could identify problems users encountered and discover where documentation was missing. The same endpoint therefore helps the agent retrieve context and helps the maintainers decide what context to write next.

The fourth lesson addressed a familiar obstacle: the space of possible applications made evaluation feel too broad to start. Langfuse users build chat applications, real-time voice systems, video generation, and background invoice processing. There is no single ideal instrumentation setup for all of them. The team began with five evaluation examples, giving itself a concrete basis for measuring changes.

Each test paired an instrumentation request with a sample repository folder. The skill ran against that repository, and an LLM-as-a-Judge assessed natural-language checks using the filesystem and diff state before and after the run. For an OpenAI application with custom RAG functions, the checks included both OpenAI instrumentation and retrieval spans. Capturing only the LLM calls would leave the retrieval behavior unexplained.

The distinction can be expressed in a small set of expectations:

These checks gave the team a way to change the skill and look for regressions. They made meaningful instrumentation testable without requiring a complete evaluation framework for every possible customer application.

The fifth lesson was to resist turning a skill into a local copy of the documentation. Adding more context is tempting for both maintainers and community contributors: it makes information immediately available. But each copied explanation creates another representation of the product that can drift out of date. The same stale-knowledge problem that affects pretraining can reappear inside the skill itself. Reference changing documentation instead of duplicating it.

The sixth lesson explored whether agents could improve the skill through automated experimentation. The target task was migrating prompts from local Git repositories into Langfuse prompt management. Managed prompts let product managers and other non-engineering collaborators edit prompts and iterate in a playground, so the migration needed to work across varied codebases.

Klingen reports that the team accepted three of six suggested improvements in the prompt-migration experiment. For a small team, the benefit was the ability to explore more alternatives than it could investigate manually. But the experiment also exposed how much the result depended on the target function.

The team wanted prompt migration to be fast and used the number of turns as its speed measure. When the optimizing agent was asked to minimize turns, it removed instructions to fetch documentation. Those instructions looked unnecessary because the agent believed it already knew how Langfuse prompt management worked. The optimization improved the proxy by undermining the reason for using a skill: obtaining current context. A skill installed and reused months later could then depend on obsolete information.

Human approval was another mismatch between the experiment and real use. The intended interaction included follow-up questions and a proposed plan before moving prompts into central storage. That gate matters because the user’s data leaves their laptop. The sandbox did not support exercising that approval interaction, so the experiment could not adequately test this part of the workflow.

The team also wanted a full implementation, not merely an initial success. In prompt management, that includes linking prompt versions to production traces so users can investigate how versions affect production results. Because this linkage was absent from the target function, instructions encouraging it were removed.

The target function defines which work the optimizer has a reason to preserve. Trace inspection, search signals, evaluations, and canonical references supply complementary evidence; none makes an incomplete objective complete. Explicit navigation also keeps agents from defaulting to scattered Google or Brave results when authoritative guidance is available.

Even a carefully written skill becomes a local copy when installed on a user’s machine. Package-style installation already existed; the unresolved problem was dependable freshness detection and upgrading across agent environments. Klingen describes a possible fetched-at timestamp that would prompt an update attempt after a month. But installing or updating a skill may require user action, or may be unavailable to the agent, depending on the environment. The timestamp was a proposed mechanism, not a demonstrated automatic update system.

There is also a product decision hidden inside the objective: should the skill produce a quick first success or a comprehensive setup? Klingen describes a perfect setup as work that can take an AI engineering team months without a skill. Asking an agent to reach it in one interaction may overwhelm the user with questions. An alternative is to get something useful working, then let the user invoke the skill again to improve the setup through further questions. The target is therefore not just technical completeness; it includes how much guidance the user can use at that moment.

Community conversations suggested that many users simply wanted an agent to make the integration work. Langfuse consequently advertised coding-agent use across its documentation as the primary way to get started. The ambition extended beyond onboarding to the evaluation lifecycle: creating a judge aligned with user preferences, or finding common patterns in feedback across a hundred executions. A CLI could retrieve the underlying data for work people had previously performed manually.

Klingen’s roadmap moved from automating those workflows through skills, to bringing automation into the product, to adding an orchestration agent. He said the team was working on that orchestration layer. The direction was an ongoing improvement process around the application, rather than a skill used only once to install instrumentation.

An audience question sharpened the distinction between running experiments without a human and accepting their results without one. The team wanted the experimentation phase to generate recommendations autonomously. Humans then reviewed every suggested idea before changing the skill. Some recommendations were rejected precisely because the objective was imperfect. The experiment was a source of candidate improvements, not an authority over what should ship.

A follow-up asked whether that process optimized the human-agent interaction at all. Klingen’s answer was practical: the team used the skill itself to assess how the guidance felt. The intended experience resembles working with an expert when the user arrives knowing only that they need evaluations. The expert asks what problem the user is trying to solve and what they worry about. A generic hallucination evaluation may be less useful than a check specific to that application. Repository-level evaluation and firsthand use answer different questions about the same skill.

The final audience question compared GitHub-based installation, an npx skills approach, and plugin marketplaces. Klingen was wary of the maintenance burden that proprietary integrations across Anthropic, OpenAI, and Cursor could impose on a small team. The questioner noted that Anthropic and OpenAI were broadly converging, but Klingen’s preferred direction was simpler discovery: a well-known location where an agent could find the relevant skill when a user wanted to use Langfuse. The documentation already advertised its existence.

That raised a trust question. Klingen wondered whether discovery and installation needed an approval step if the skill carried only the same trust as public web content. He distinguished that situation from installing guidance with elevated trust. This was a question about the agent environment’s trust model, not an established permission rule.

For freshness, he returned to the idea of an installed skill as a cache: it was current when fetched, but its age eventually matters. A timestamp could let the agent recognize an old copy and fetch a newer one, or alert the user that the skill might be outdated. The team was discussing that timestamp-and-fetch direction. The remaining design problem was to make guidance discoverable and refreshable without treating yesterday’s installed instructions as permanently authoritative.