# Skill issue: Lessons from skilling up coding agents to use Langfuse

Marc Klingen, Langfuse, part of ClickHouse

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/vNCY9kXXyDQ-skill-issue-lessons-from-skilling-up-coding) · [Video](https://www.youtube.com/watch?v=vNCY9kXXyDQ)

## Teaching coding agents to use Langfuse

A useful integration skill needs more than SDK instructions: it must guide decisions, retrieve current documentation, expose runtime failures, and survive optimization without losing essential safeguards.

Basic familiarity with coding agents, API integrations, and execution traces will help; no prior Langfuse experience is required.

### From reading documentation to following a guide

[0:16](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=16s)

How do you add observability and evaluations to an agent that does not work reliably? That problem led Marc Klingen and his cofounders to build Langfuse. Roughly three years later, he describes it as the largest open-source project in its space by the metrics his team tracks. The product engineering team remains in Europe, collaborating remotely rather than moving everyone to another continent. But the integration experience has changed: users who once had to read hundreds of documentation pages and construct their own mental model now expect a coding agent to guide them through the work.

A Rubik’s Cube captures the distinction between capability and guidance. A child can turn every face without knowing how to solve it. Giving an agent a bash tool provides similar freedom: it can perform operations, but that does not supply a procedure for reaching the goal. A solving manual makes those capabilities useful. **A skill supplies procedural guidance to an agent that already has tools.**

![A scrambled Rubik’s Cube and a sheet titled “How To Solve A Rubik’s Cube,” with the speaker inset below.](https://ai.engineer/api/talk-essay-media/2cb6f33cd67c08800e1d0c28c25ade5d4519e3e6c85472ddd0563400110b8975)

A Rubik’s Cube beside an illustrated solving guide.

This puts skills between rigid workflows and open-ended autonomy. A coding agent is not necessary for every application; its flexibility can introduce unnecessary latency and expense. A workflow can reliably route a password-reset request to a narrowly configured agent. But when the user also wants to change their email address, the router must somehow coordinate two separate paths. An agent can instead load the relevant guidance progressively and combine it to address the request. That matters because an open text box invites combinations of needs that the developer did not anticipate.

### Let runtime behavior reveal the missing guidance

[3:09](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=189s)

Once users can make open-ended requests, traces become a way to discover which skills the agent needs. Record what happens during execution, then use production evaluations to identify failures. In the password-reset example, an upset user’s trace can reveal a use case the team never expected to support. That becomes evidence for adding a password-reset skill. Langfuse is one implementation of this feedback loop; the mechanism depends on tracing, not on a particular tracing product.

The same evidence improves skills that already exist. Instructions can become outdated, or an execution can reach the right result through an unnecessarily indirect sequence. Traces therefore answer two different questions: what guidance is missing, and what existing guidance should change?

### Flexible infrastructure makes integration a reasoning task

[4:01](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=241s)

Klingen reports that Langfuse had 478 documentation pages across five feature areas. That breadth reflects a deliberate product choice. An opinionated evaluation product might configure a chatbot end to end; Langfuse provides infrastructure that users can adapt to different applications. Klingen describes its tracing infrastructure as continuing to work with billions of ingested traces, alongside customizable evaluations. This is a capacity claim, without a workload benchmark in the talk.

Flexibility used to place more work on the person integrating the product. With coding agents, it can become an advantage: the infrastructure stays general while the agent adapts the implementation to the repository. But the agent still has to determine which setup is appropriate. Knowing that Langfuse exists is not enough.

Pretraining makes this particularly awkward. At first, asking an agent about Langfuse and receiving SDK code feels like a milestone. As the project evolves, the same familiarity can produce methods that existed in older interfaces but have since disappeared. **Current documentation must take precedence over remembered SDK usage.** The problem is not necessarily ignorance; it is confidence in an obsolete interface.

![Slide listing 478 documentation pages, five feature areas, implementation flexibility, and major updates, beside a Langfuse documentation screenshot.](https://ai.engineer/api/talk-essay-media/ec11f06f2a4dacb9d86814ec7368d70785f09d69c825302af6a74cb0ced13722)

Langfuse’s documentation spans 478 content-dense Markdown files.

### A working integration can still explain too little

[5:49](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=349s)

The baseline request to **Claude Code** was simple: “Add tracing to my agent.” The agent first implemented instrumentation from outdated training knowledge, tried to verify it, discovered that it did not work, and only then fetched current documentation to repair the implementation. The order created avoidable work: implement, fail, retrieve, repair.

There was also a requirements problem. Online evaluations, offline evaluations, and human-in-the-loop evaluations serve different purposes. The user and agent needed to establish which approach fit the application, but the agent had not been instructed to guide that discovery. Klingen’s baseline trace captured two LLM calls without making the agent’s behavior clear. Fixing the SDK calls alone would not fix the quality of the observability setup.

The goal of the [Langfuse skill](https://github.com/langfuse/skills) was to provide the kind of help an expert would offer across observability, prompt management, and evaluations. Klingen describes an audience of thousands of community teams and thousands of cloud customers—too many for individual consultations with the Langfuse team. The skill had to carry both technical knowledge and the interaction habits needed to choose a useful implementation.

The architecture separates those responsibilities:

- **Interaction guidance:** [`SKILL.md`](https://github.com/langfuse/skills/blob/main/skills/langfuse/SKILL.md) describes how the agent should work, including asking follow-up questions before making decisions.
- **Product references:** module-specific material progressively supplies the hints needed for the selected use case.
- **Current documentation:** the agent retrieves implementation details instead of relying entirely on installed text or model memory.
- **Execution access:** the [Langfuse CLI](https://github.com/langfuse/langfuse-cli) wraps APIs that already supported custom labeling interfaces and evaluation execution, making operations previously performed through the UI available to agents.

The skill supplies direction; references supply context; the CLI supplies actions.

Connecting a repository and letting an agent run the whole improvement cycle was the ambition, rather than a completed capability. The nearer-term result Klingen describes was a more detailed setup after conversing with the agent: evaluations relevant to the application and more explicit execution steps. That result led into six lessons from building the skill.

### Inspect traces before making evaluation elaborate

[9:04](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=544s)

Klingen estimates that looking at traces gets you 80% of the detail; this is an informal estimate, not a measured evaluation score. The team instrumented Claude Code, used it interactively to work with Langfuse, and inspected the resulting traces in Langfuse. Following the actual sequence of decisions exposed where the agent wandered, failed, or made assumptions that the skill could correct.

One assumption came from an interface optimized for humans. To reduce the number of environment variables users needed to configure, Langfuse had defaulted to the European data region. US enterprises also cared about data regionality, however, and an agent could easily configure another variable. The skill therefore instructed the agent to determine the user’s actual region rather than silently assume Europe. Reducing manual setup effort had hidden a decision that the agent should make explicit.

Another failure came from familiar-looking CLI commands. Seeing the word `trace`, an agent could infer parameters from other tracing tools instead of discovering the parameters this CLI supported. The team promoted the help flag more aggressively. Klingen describes that extra discovery turn as fast: it gives the agent the available interface before it attempts an operation. The correction is small but structural—inspect the command surface before composing commands from memory.

### Make documentation easy to navigate—and learn from searches

[10:46](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=646s)

With hundreds of pages available, an agent can spend many turns fetching one page, reasoning about it, discovering another link, and fetching again. The second lesson was to supply an explicit navigation path. The skill directs the agent first to the [documentation index](https://langfuse.com/llms.txt), using `llms.txt` as a sitemap rather than hoping the agent will discover the right pages unaided.

Retrieval format matters too. Some coding agents do not request Markdown by default and end up parsing HTML, consuming additional tokens. Langfuse exposes Markdown through a `.md` suffix or content negotiation. The linked current `SKILL.md` documents `Accept: text/markdown`; it is a maintained version rather than a frozen copy of the skill shown in the talk. A compact instruction can make both navigation and format explicit:

```markdown
## Find implementation guidance

1. Read the documentation index at `llms.txt`.
2. Select the pages relevant to the user's application.
3. Fetch those pages as Markdown using a `.md` suffix
   or the `Accept: text/markdown` request header.
4. Use the retrieved documentation when implementing the integration.
```

This keeps the skill focused on how to acquire the information rather than reproducing every page inside it.

The third lesson reused an existing capability. Langfuse already had a documentation Q&A agent backed by retrieval-augmented generation. The team exposed its retrieval stack through a search endpoint: a coding agent could submit a natural-language question and receive relevant documentation chunks. That can avoid a sequence of page fetches when the agent already knows the question it needs answered.

Search also creates a feedback channel. A page request reveals little about what a coding agent on a user’s laptop is trying to accomplish. A natural-language query carries more intent. By tracking those queries, the team could identify problems users encountered and discover where documentation was missing. The same endpoint therefore helps the agent retrieve context and helps the maintainers decide what context to write next.

### Start with a small evaluation suite

[12:29](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=749s)

The fourth lesson addressed a familiar obstacle: the space of possible applications made evaluation feel too broad to start. Langfuse users build chat applications, real-time voice systems, video generation, and background invoice processing. There is no single ideal instrumentation setup for all of them. The team began with five evaluation examples, giving itself a concrete basis for measuring changes.

Each test paired an instrumentation request with a sample repository folder. The skill ran against that repository, and an **LLM-as-a-Judge** assessed natural-language checks using the filesystem and diff state before and after the run. For an OpenAI application with custom RAG functions, the checks included both OpenAI instrumentation and retrieval spans. Capturing only the LLM calls would leave the retrieval behavior unexplained.

![Slide titled “#4 - A basic eval setup is better than none,” showing an input and expected-output example alongside setup and result bullets.](https://ai.engineer/api/talk-essay-media/ddd4c0928a94edaf1581220003615d2eb43dd6163818613e7719efa4a06fd1e6)

A basic evaluation setup uses five sample repositories and an LLM judge.

The distinction can be expressed in a small set of expectations:

| Check | What it establishes |
|---|---|
| OpenAI instrumentation added | Model calls are covered |
| Retrieval spans added | Retrieval work is represented |
| Before/after changes inspected | The skill’s repository edits can be judged |

These checks gave the team a way to change the skill and look for regressions. They made meaningful instrumentation testable without requiring a complete evaluation framework for every possible customer application.

### Keep changing knowledge outside the skill

[13:52](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=832s)

The fifth lesson was to resist turning a skill into a local copy of the documentation. Adding more context is tempting for both maintainers and community contributors: it makes information immediately available. But each copied explanation creates another representation of the product that can drift out of date. The same stale-knowledge problem that affects pretraining can reappear inside the skill itself. Reference changing documentation instead of duplicating it.

The sixth lesson explored whether agents could improve the skill through automated experimentation. The target task was migrating prompts from local Git repositories into Langfuse prompt management. Managed prompts let product managers and other non-engineering collaborators edit prompts and iterate in a playground, so the migration needed to work across varied codebases.

Klingen reports that the team accepted three of six suggested improvements in the prompt-migration experiment. For a small team, the benefit was the ability to explore more alternatives than it could investigate manually. But the experiment also exposed how much the result depended on the target function.

### An incomplete objective removes valuable work

[15:13](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=913s)

The team wanted prompt migration to be fast and used the number of turns as its speed measure. When the optimizing agent was asked to minimize turns, it removed instructions to fetch documentation. Those instructions looked unnecessary because the agent believed it already knew how Langfuse prompt management worked. The optimization improved the proxy by undermining the reason for using a skill: obtaining current context. A skill installed and reused months later could then depend on obsolete information.

Human approval was another mismatch between the experiment and real use. The intended interaction included follow-up questions and a proposed plan before moving prompts into central storage. That gate matters because the user’s data leaves their laptop. The sandbox did not support exercising that approval interaction, so the experiment could not adequately test this part of the workflow.

The team also wanted a full implementation, not merely an initial success. In prompt management, that includes linking prompt versions to production traces so users can investigate how versions affect production results. Because this linkage was absent from the target function, instructions encouraging it were removed.

| Intended property | What the experiment missed |
|---|---|
| Current implementation knowledge | Turn minimization discouraged documentation retrieval |
| Approval before uploading prompts | The sandbox could not exercise the interaction |
| Prompt versions linked to traces | The objective did not reward the linkage |

**The target function defines which work the optimizer has a reason to preserve.** Trace inspection, search signals, evaluations, and canonical references supply complementary evidence; none makes an incomplete objective complete. Explicit navigation also keeps agents from defaulting to scattered Google or Brave results when authoritative guidance is available.

### Choose what success means after installation

[17:23](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1043s)

Even a carefully written skill becomes a local copy when installed on a user’s machine. Package-style installation already existed; the unresolved problem was dependable freshness detection and upgrading across agent environments. Klingen describes a possible fetched-at timestamp that would prompt an update attempt after a month. But installing or updating a skill may require user action, or may be unavailable to the agent, depending on the environment. The timestamp was a proposed mechanism, not a demonstrated automatic update system.

There is also a product decision hidden inside the objective: should the skill produce a quick first success or a comprehensive setup? Klingen describes a perfect setup as work that can take an AI engineering team months without a skill. Asking an agent to reach it in one interaction may overwhelm the user with questions. An alternative is to get something useful working, then let the user invoke the skill again to improve the setup through further questions. The target is therefore not just technical completeness; it includes how much guidance the user can use at that moment.

Community conversations suggested that many users simply wanted an agent to make the integration work. Langfuse consequently advertised coding-agent use across its documentation as the primary way to get started. The ambition extended beyond onboarding to the evaluation lifecycle: creating a judge aligned with user preferences, or finding common patterns in feedback across a hundred executions. A CLI could retrieve the underlying data for work people had previously performed manually.

Klingen’s roadmap moved from automating those workflows through skills, to bringing automation into the product, to adding an orchestration agent. He said the team was working on that orchestration layer. The direction was an ongoing improvement process around the application, rather than a skill used only once to install instrumentation.

### Keep human review separate from autonomous experimentation

[20:15](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1215s)

An audience question sharpened the distinction between running experiments without a human and accepting their results without one. The team wanted the experimentation phase to generate recommendations autonomously. Humans then reviewed every suggested idea before changing the skill. Some recommendations were rejected precisely because the objective was imperfect. The experiment was a source of candidate improvements, not an authority over what should ship.

A follow-up asked whether that process optimized the human-agent interaction at all. Klingen’s answer was practical: the team used the skill itself to assess how the guidance felt. The intended experience resembles working with an expert when the user arrives knowing only that they need evaluations. The expert asks what problem the user is trying to solve and what they worry about. A generic hallucination evaluation may be less useful than a check specific to that application. Repository-level evaluation and firsthand use answer different questions about the same skill.

### Discovery, trust, and the installed cache

[22:06](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1326s)

The final audience question compared GitHub-based installation, an `npx skills` approach, and plugin marketplaces. Klingen was wary of the maintenance burden that proprietary integrations across Anthropic, OpenAI, and Cursor could impose on a small team. The questioner noted that Anthropic and OpenAI were broadly converging, but Klingen’s preferred direction was simpler discovery: a well-known location where an agent could find the relevant skill when a user wanted to use Langfuse. The documentation already advertised its existence.

That raised a trust question. Klingen wondered whether discovery and installation needed an approval step if the skill carried only the same trust as public web content. He distinguished that situation from installing guidance with elevated trust. This was a question about the agent environment’s trust model, not an established permission rule.

For freshness, he returned to the idea of an installed skill as a cache: it was current when fetched, but its age eventually matters. A timestamp could let the agent recognize an old copy and fetch a newer one, or alert the user that the skill might be outdated. The team was discussing that timestamp-and-fetch direction. The remaining design problem was to make guidance discoverable and refreshable without treating yesterday’s installed instructions as permanently authoritative.

## Resources

- [Langfuse Skills](https://github.com/langfuse/skills) — Installable guidance for coding agents working with Langfuse tracing, prompts, datasets, and evaluations.
- [Langfuse CLI](https://github.com/langfuse/langfuse-cli) — Command-line access to the Langfuse API for agent and developer workflows.
- [Langfuse documentation index](https://langfuse.com/llms.txt) — A text index for discovering Langfuse documentation.
- [Evaluating AI Agent Skills](https://langfuse.com/blog/2026-02-26-evaluate-ai-agent-skills) — The team's evaluation setup and concrete fixes for missing server configuration, invented CLI arguments, and skill invocation failures.
- [We Used Autoresearch on Our AI Skill, It Taught Us to Write Better Tests](https://langfuse.com/blog/2026-03-24-optimizing-ai-skill-with-autoresearch) — A companion account of automated prompt-migration experiments and the shortcomings of their scoring objective.
- [Langfuse March Update](https://langfuse.com/blog/2026-03-31-langfuse-march-update) — Marc Klingen's contemporaneous overview of the agent skill, CLI, and evaluation work.

## Slides

### AI Engineer Europe

[0:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=0s) [AI Engineer Europe logo in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/24e440bf5dfd4ecccd4fffdc9ef9243be07fff10b16e87c6b80fbc594828e3ea.webp)

AI Engineer
EUROPE

### Presenting sponsor

Google DeepMind is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=5s) [Google DeepMind presenting sponsor card.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a43b90b83b6eca77e82f00387f025461fb2cd84d99ed7f55e0f7cc2bbdddc7d.webp)

PRESENTING SPONSOR

Google DeepMind

### Platinum sponsors

[0:10](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=10s) [Platinum sponsors Braintrust, WorkOS and OpenAI.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca79c819b98dbfef493658afec17bf27a16183cff6cb3cb11ee6b32234d09353.webp)

PLATINUM SPONSORS

Braintrust
WorkOS
OpenAI

### Workflow vs Agent

A workflow follows a fixed sequence of LLM calls. An agent acts on an environment, receives feedback, and can continue or stop.

[1:38](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=98s) [A linear workflow is compared with an agent that acts on an environment and receives feedback.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f6c0c701ba48b012cca11ca79a1964fa9207c06e125e504120c8befd5b5d616.webp)

Workflow vs Agent

Workflow:
Input
LLM call
LLM call
Output

Formalized paths to achieve a goal

Agent:
I/O
agent
stop
feedback
action
environment

Freedom for agent to find and verify the way

AIE
Engineering the future of AI

### Agent with formalized paths

Skills give an agent formalized shortcuts as it loops toward a goal and uses a Bash tool in a sandbox. They can supply context across tasks that once needed separate workflows.

[2:14](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=134s) [An agent connects to input and output, a sandbox through a Bash tool, and skills presented as formalized shortcuts.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f356e635070f4faf8764b43a0fc499a24c13d55d1331d0597ba8b99a09c6a03.webp)

...with formalized paths to achieve a goal

I/O
Agent looping to achieve a goal
Agent
Access to multi-purpose tools
Bash tool
Sandbox
Formalized shortcuts/“workflows”
Skill 1
Skill n

github.com/langfuse
Langfuse

AIE
AI Engineer
EUROPE

### Closing the loop with an improvement engine

Teams trace agent runs to identify new skill needs and improve outdated or inefficient skills. The diagram shows Langfuse collecting I/O, skill invocations and tool usage to inform that loop.

[3:09](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=189s) [An agent runtime sends traces to Langfuse, which informs new skills and improvements to existing skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/48301f83970136d12d0c28ac83e09ad16cff6b2a324b25da9d290a5dbe621255.webp)

...and close the loop with an improvement engine

Agent Runtime
I/O
Agent looping to achieve a goal
Agent
Bash tool
Sandbox
Access to multi-purpose tools
Skill 1
Skill n
Formalized shortcuts/“workflows”

Trace
Langfuse
Full trace of I/O, skill invocation, and tool usage
Informs new skill creation
Allows improving existing skills
Improve

github.com/langfuse
Langfuse

Interface text:
AIE

[3:13](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=193s) [An agent runtime sends traces to Langfuse, which informs creation and improvement of skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/22353372f9624505bfcdb488c1141dfc2a549af9751914c5fa12261adaa2a285.webp)

...and close the loop with an improvement engine

Agent Runtime
I/O
Agent looping to achieve a goal
Agent
Bash tool
Sandbox
Access to multi-purpose tools
Skill 1
Skill n
Formalized shortcuts/“workflows”

Trace
Langfuse
Full trace of I/O, skill invocation, and tool usage
Informs new skill creation
Allows improving existing skills
Improve

github.com/langfuse
Langfuse

Interface text:
AIE
Engineering the future of AI

### Skilling up Coding Agents to use Langfuse

The talk turns to lessons from building a skill that helps customers add Langfuse to their projects.

[4:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=240s) [Section title: Skilling up Coding Agents to use Langfuse.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d2bdb8b6832362fe291d2713f78efdd50cb2c057bd9ec7a9cb088feec962b66.webp)

Skilling up Coding Agents to use Langfuse

AIE

Engineering the future of AI

### The Langfuse docs are composed of 478 content dense markdown files

Langfuse spans 478 docs pages and five feature areas. Three major updates left different versions in training data, so agents may suggest obsolete methods when choosing an implementation.

[4:08](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=248s) [Slide describing 478 documentation pages, five feature areas, implementation flexibility, and the need to guide agents toward the right path.](https://assets.ai.engineer/sites/aiecode2025/sha256/aab7b697641330618f942c8f7bd3a6da54cae9876e1b7d27568d0ec723e78bab.webp)

The Langfuse docs are composed of 478 content dense markdown files

• 478 docs pages
• 5 feature areas
• High implementation flexibility - developer in driver seat
• Langfuse went through three major updates - resulting in a variety of “Langfuses” in model training data

→ Need to support agents in choosing the right path

github.com/langfuse
Langfuse
AIE
Engineering the future of AI

Interface text:
Langfuse Overview
Observability
Prompts
Evaluation
Platform
[illegible]

[5:28](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=328s) [Slide describes 478 documentation pages, implementation flexibility, and the need to guide agents through changes to Langfuse.](https://assets.ai.engineer/sites/aiecode2025/sha256/d002bd13653c47b7d994e77caa5eb0ee745045974fb4bf4237a0d80d99efbfaf.webp)

The Langfuse docs are composed of 478 content dense markdown files

• 478 docs pages
• 5 feature areas
• High implementation flexibility - developer in driver seat
• Langfuse went through three major updates - resulting in a variety of “Langfuses” in model training data

→ Need to support agents in choosing the right path

github.com/langfuse
Langfuse
AIE
Engineering the future of AI

Interface text:
Langfuse Overview
Observability
Prompts
Evaluation
Platform
[illegible]

### Coding agents before skills

Claude Code first instruments tracing using outdated training data, then fetches current information after an error. The human and agent work out best practices over multiple turns, potentially leaving a non-optimal setup.

[6:10](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=370s) [A user requests Langfuse tracing; the agent encounters an error, then fetches current information and arrives at a feasible implementation.](https://assets.ai.engineer/sites/aiecode2025/sha256/11515aaf90f23f0c3280ac39a8d974c60f9944cf9cbc226365a63df1a1e6ab33.webp)

Coding agents were able to implement Langfuse before skills

User
Coding Agent

User Asks Coding Agent to add Langfuse tracing

Implements instrumentation based on model capabilities, runs into an error when trying

Fetches current information on Langfuse implementation from web, analyses and at some point arrives at a feasible implementation

github.com/langfuse
Langfuse
AIE
Engineering the future of AI

[6:37](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=397s) [A tracing workflow highlights outdated training data, a non-optimal setup, and slow, multiple turns.](https://assets.ai.engineer/sites/aiecode2025/sha256/595bcaefee75dae7c4383ceae1b6921c8d024ce23bac85fc62d1fe6c06def09b.webp)

Coding agents were able to implement Langfuse before skills

User
Coding Agent

User Asks Coding Agent to add Langfuse tracing

Implements instrumentation based on model capabilities, runs into an error when trying

! Outdated training data

Fetches current information on Langfuse implementation from web, analyses and at some point arrives at a feasible implementation

Human and coding agent over time figure out best practices for tracing

! Non optimal setup
! Slow, multiple turns

github.com/langfuse
Langfuse
AIE
AI Engineer EUROPE

### Claude Code trace example

Claude Code’s trace captured two LLM calls, but the speaker says it did not reveal what the agent was actually doing.

[6:52](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=412s) [A Langfuse trace view shows two agent observations, captioned Claude Code only.](https://assets.ai.engineer/sites/aiecode2025/sha256/579a7a7cc9e1955338cacc76a80f0960388136bb3d6e323fe98508539992d48b.webp)

Claude Code only

github.com/langfuse
Langfuse
AIE
AI Engineer EUROPE

Interface text:
Trace
Observation ID: [illegible]
Search
Timeline
scenario:single_no_moose_no_guardrails
single_agent:initial
2,321 → 569 (Σ 2,890)
single_agent:tool-result
6,148 → 871 (Σ 7,019)

### Goal of the Langfuse skill

The skill aims to give users Langfuse expertise for observability, prompt management and evals, scaling guidance beyond what the team can provide one-on-one.

[7:13](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=433s) [The skill aims to help users quickly set up observability, prompt management, and evaluations using best practices and current documentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/d198cacdab2fe2ebdd43496deb3a4dd0fb385eff0b2b00f2895291c274ce0188.webp)

Goal of the Langfuse skill

Give Users a Langfuse expert to quickly set up observability, prompt management and evals in line with best practices and up to date docs.

AIE

github.com/langfuse
Langfuse

Engineering the future of AI

### How the Langfuse skill works

The skill guides a coding agent with SKILL.md and product references, then connects it to Langfuse documentation via curl and Langfuse Cloud via the CLI.

[7:34](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=454s) [A coding agent uses skill instructions and references to access Langfuse documentation and its CLI.](https://assets.ai.engineer/sites/aiecode2025/sha256/33666c7023be483b99567c1ef0ddafee096e50cb7552300dcabab4f65b3d87d0.webp)

The Langfuse Skill is structured to help agents effectively navigate up to date docs, API specs & best practices

Schematic Illustration

User
Coding Agent

"Instrument this application with Langfuse tracing following best practices."

Langfuse Skill
SKILL.md
/reference/observability.md
/reference/langfuse-cli.md
/reference/prompt-migration.md
curl
CLI

langfuse.com/docs
cloud.langfuse.com

Embedded screenshots: [illegible]

github.com/langfuse
AIE
Engineering the future of AI

### Claude Code trace comparison

The Claude Code-only trace shows two agent steps. After a similar task with the Langfuse skill, the comparison shows a more detailed trace with evaluations and execution steps.

[8:44](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=524s) [Two trace views compare Claude Code alone with Claude Code using the Langfuse skill; the latter shows more detailed steps and evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/c6b6ba0dd5cbde23b41f69e37af3ddee18ddea03a2c88e2a0855ac3022e25f97.webp)

Claude Code only
Claude Code + Langfuse Skill

github.com/langfuse
Langfuse
AIE
Braintrust
WorkOS
OpenAI

Interface text:
Left trace:
Trace
Observation ID: [illegible]
Search
Timeline
scenario:single_no_moose_no_guardrails
single_agent:initial
2,321 → 569 (Σ 2,890)
single_agent:tool-result
6,148 → 871 (Σ 7,019)

Right trace:
Trace
Observation ID: [illegible]
Search
Timeline
fleetpulse-supervisor:ai.generateText
1m 6s  Σ $0.090859
accuracy: 2.00
completeness: 4.00
grounding: 1.00
grounding_p... 0.00
helpfulness: 3.00
overall_qualit... 2.00
fleetpulse-supervisor:ai.generateText
2.36s  Σ $0.001745
fleetpulse-supervisor:ai.generateText.doGenerate
2.33s  910 → 167 (Σ 1,077)  $0.001745
NYC-Hackto... True
nyc_asks_fol... False
fleetpulse-specialist-retrieval:ai.generateText
27.56s  Σ $0.015012
fleetpulse-specialist-retrieval:ai.generateText.doGenerate
3.91s  2,071 → 569 (Σ 2,640)  $0.004916
NYC-Hackto... True
nyc_asks_fol... False
fleet_zone_intel
13.59s
fleet_hourly_ops
12.81s
fleet_anomaly_scan
13.13s
fleet_trend_compare
12.81s

### Six lessons from building the skill

The team’s lessons span trace review, production signals and evals. Autoresearch can remove useful work if its target function omits the goal: the team saw this with links between prompt versions and production traces.

[9:05](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=545s) [Six takeaways cover trace review, production signals, information navigation, evaluations, dynamic references, and target functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/e69e32660737ce7a6f4ba8a2e8ced910efa179cfd988718db26fbd0c7933f9bf.webp)

Towards auto-researching an agent skills

#1 Looking at traces qualitatively gets you 80% of the way
#2 Capture interesting production signals (e.g., docs search api)
#3 Help the agent understand how to navigate information
#4 Even a basic eval setup boosts confidence to iterate on Skill
#5 Dynamic content should be referenced not copied
#6 Autoresearch quality is bounded by its target function

github.com/langfuse
Langfuse
AIE
AI Engineer EUROPE

### Qualitative trace review

The team used instrumented Claude Code and inspected Langfuse traces to find agent errors and improve the skill. The trace shows where the skill ran, bash tool usage, and context behind decisions.

[9:14](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=554s) [A Langfuse trace illustrates skill invocation, shell tool use, and context used in agent decisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/75dbe0368f837dfe44138b37fc681c72acac62b41a62bc1326bddb0d75963b7a.webp)

#1 - Looking at traces qualitatively gets you 80% of the way

Tracing Claude Code in local development
CLAUDE CODE × Langfuse

Allows easy view on:
• Skill invocation position
• Usage of bash tools
• Context incorporated in agent decisions and intermediate results

github.com/langfuse
Langfuse
AIE
AI Engineer EUROPE

Interface text:
Trace [illegible]
Dataset run: cli-tests-4
claude.conversation
claude.assistant.turn
Skill
Bash
Add to datasets
Annotate
Add comment
2026-02-20 15:29:36.050
Latency: 21.01s
Env: default
$0.013251
2,472 prompt → 389 completion (Σ 2,861)
Preview
Log View
Scores
Formatted
JSON
Input
A user reported an error. Check trace [illegible] and tell me what happened.
Output
The trace ID [illegible] doesn't appear in any Plain support tickets.
This could mean:
• The error hasn't been reported to support yet

### Learning from qualitative trace review

Trace review exposed two skill errors: assuming a Langfuse region and hallucinating CLI parameters. The instructions now require a host and show how to discover available commands.

[9:49](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=589s) [Slide showing examples of an unspecified Langfuse URL and hallucinated CLI parameters.](https://assets.ai.engineer/sites/aiecode2025/sha256/be14b6626db0ef6d5a5ddce0edd74d17a9ec1269159d5e9066e671c0d2b45717.webp)

#1 - Looking at traces qualitatively gets you 80% of the way

Error 1: Not enforcing Langfuse URL

Set environment variables before making calls:

export LANGFUSE_PUBLIC_KEY=[redacted]
export LANGFUSE_SECRET_KEY=[redacted]
- export LANGFUSE_HOST=https://cloud.langfuse.com # optional, default is EU
+ export LANGFUSE_HOST=https://cloud.langfuse.com # example for EU cloud. For US cloud it's us.cloud.langfuse.com, and can also be a self-hosted URL. The server must always be specified in order to access Langfuse.

Error 2: Hallucinated CLI parameters

# Langfuse
## 1. Langfuse API via CLI

Start by discovering the schema and available arguments:

# Discover all available resources
npx langfuse-cli api __schema

# List actions for a resource
npx langfuse-cli api <resource> --help

# Show args/options for a specific action
npx langfuse-cli api <resource> <action> --help

github.com/langfuse
Langfuse
AIE
AI Engineer EUROPE

Interface text:
skills/langfuse/SKILL.md

### Helping agents navigate available information

The skill directs agents to an llms.txt index first, then to individual documentation pages in Markdown. This helps them find relevant information without undirected web queries.

[10:45](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=645s) [Slide describing a documentation index and individual Markdown pages, alongside the intended information-gathering outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/108af6fb64e47dc7ed52bf04300e3fb22890ac1f30e8093070b7aac22a601c50.webp)

#2 - Help the agent understand how to navigate available information

2. Langfuse Documentation

Three methods to access Langfuse docs, in order of preference. Always prefer your application's native web fetch and search tools (e.g., WebFetch, WebSearch, mcp_fetch, etc.) over curl when available. The URLs and patterns below work with any fetching method — the curl examples are just illustrative.

2a. Documentation Index (llms.txt)

Fetch the full index of all documentation pages:
curl -s https://langfuse.com/llms.txt

Returns a structured list of every doc page with titles and URLs. Use this to discover the right page for a topic, then fetch that page directly.

Alternatively, you can start on https://langfuse.com/docs and explore the site to find the page you need.

2b. Fetch Individual Pages as Markdown

Any page listed in llms.txt can be fetched as markdown by appending .md to its path or by using Accept: text/markdown in the request headers. Use this when you know which page contains the information needed. Returns clean markdown with code examples and configuration details.

curl -s "https://langfuse.com/docs/observability/overview.md"
curl -s "https://langfuse.com/docs/observability/overview" -H "Accept: text/markdown"

Setup:
• Skill describes how agent can query an overview of all available information
• Skill describes how agent can access specific pages if interesting

Outcomes:
• Efficient information gathering
• Avoids undirected web queries

github.com/langfuse
Langfuse
AIE
Engineering the future of AI

### Capture interesting prod signals

Langfuse tracks documentation search queries from coding agents to identify problems users encounter, potential new skills, and gaps in its docs.

[11:37](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=697s) [Slide showing documentation search queries and explaining how their patterns reveal interest in new skills and gaps in documentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/33567f91d86a17090d0b9d1468b3ee1f03ae0aee8fe3e1273ff8403a0f469ca5.webp)

#3 - Capture interesting prod signals

Tracking Agent docs queries for real usage data

curl -s "https://langfuse.com/api/search-docs?query=How+do+I+trace+LangGraph+agents"

Detected patterns in queries:
• Shows areas of interest for new skills
  e.g. Users are trying to get support in evaluating their applications
• Reveals ‘gaps’/issues in documentation
  e.g. Agents are inquiring about the different score types and how to set them up

github.com/langfuse
Langfuse

Interface text:
EVENT | QUERY
docs_search:query | BasicTracerProvider sdk-trace-base LangfuseSpanProcessor without NodeSDK
docs_search:query | trace input output empty root observation
docs_search:query | langfuse datasets llm evaluation
docs_search:query | typescript parentTraceId generation same trace id
docs_search:query | propagate_attributes metadata input output trace update
docs_search:query | export traces filtering environment date range
docs_search:query | trace metadata
docs_search:query | start_as_current_generation deprecated start_as_current_observation Python SDK v3
docs_search:query | trace_context CallbackHandler site:langfuse.com langchain
AIE
AI Engineer EUROPE

[12:12](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=732s) [Slide showing documentation search queries and how their patterns reveal skill interests and documentation gaps.](https://assets.ai.engineer/sites/aiecode2025/sha256/68b71a638529fc1311bd00c8f44ea08b22d6cda6d124270d41cd5950f1dd25e5.webp)

#3 - Capture interesting prod signals

Tracking Agent docs queries for real usage data

curl -s "https://langfuse.com/api/search-docs?query=How+do+I+trace+LangGraph+agents"

Detected patterns in queries:
• Shows areas of interest for new skills
  e.g. Users are trying to get support in evaluating their applications
• Reveals ‘gaps’/issues in documentation
  e.g. Agents are inquiring about the different score types and how to set them up

github.com/langfuse
Langfuse

Interface text:
EVENT | QUERY
docs_search:query | BasicTracerProvider sdk-trace-base LangfuseSpanProcessor without NodeSDK
docs_search:query | trace input output empty root observation
docs_search:query | langfuse datasets llm evaluation
docs_search:query | typescript parentTraceId generation same trace id
docs_search:query | propagate_attributes metadata input output trace update
docs_search:query | export traces filtering environment date range
docs_search:query | trace metadata
docs_search:query | start_as_current_generation deprecated start_as_current_observation Python SDK v3
docs_search:query | trace_context CallbackHandler site:langfuse.com langchain
AIE
Engineering the future of AI

### A basic eval setup is better than none

Langfuse runs its skill on five sample repositories, then uses an LLM judge to check natural-language expectations against the filesystem and before-and-after diff. This helps test improvements and regressions.

[12:32](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=752s) [Slide describing five sample repositories, skill runs, and LLM-as-a-judge evaluation to detect improvements and regressions.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e9658626aff75841bc3c451046809fea258337bc3ccd3d3cd857041d3f7e90f.webp)

#4 - A basic eval setup is better than none

Setup:
• 5 sample repos with varying complexity in instrumentation setup
• Skill is run per sample repo
• LLM-as-a-judge is used to evaluate instrumentation accuracy

Result:
• Confidence for releasing new versions of the skill as improvements and regressions can be tested

Input:
prompt: "instrument my application with Langfuse"
sample-code-folder: "openai-custom-functions-rag"

Expected Output:
checks:
0: "expected instrumentation method: from langfuse.openai import openai + @observe() decorator"
1: "One trace with nested spans: parent @observe function → child retrieval span → child generation span; all auto-linked"

github.com/langfuse
Langfuse
AIE
Engineering the future of AI

### Dynamic content should be referenced

The Langfuse skill points to framework integration docs maintained in the regular documentation release cycle, avoiding duplicated, potentially conflicting content while letting readers drill down as needed.

[13:51](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=831s) [Slide showing links to framework integration documentation and explaining that references reduce conflicting sources and support progressive disclosure.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe1ff1b847b2d3078311eee53715f565a009634f94287491e7d253aba1f6f71c.webp)

#5 - Dynamic content should be referenced

Framework Integrations

Prefer these over manual instrumentation:

Framework | Integration | Docs
OpenAI SDK | Drop-in replacement | https://langfuse.com/docs/integrations/openai
LangChain | Callback handler | https://langfuse.com/docs/integrations/langchain
LlamaIndex | Callback handler | https://langfuse.com/docs/integrations/llama-index
Vercel AI SDK | OpenTelemetry exporter | https://langfuse.com/docs/integrations/vercel-ai-sdk
LiteLLM | Callback or proxy | https://langfuse.com/docs/integrations/litellm

Full list: https://langfuse.com/docs/integrations

Setup:
• Relevant info for implementation is explicitly referenced in skill
• Info is part of regular content/docs release cycle, hence does not go out of date

Result:
• Avoids creating multiple (potentially conflicting) sources of truth
• Nests multiple levels of progressive disclosure

github.com/langfuse
Langfuse
AIE
AI Engineer

Interface text:
main
skills / skills / langfuse / references / instrumentation.md
Preview  Code  Blame

[14:11](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=851s) [A framework integration reference links to documentation; bullets explain how references avoid conflicting sources and support progressive disclosure.](https://assets.ai.engineer/sites/aiecode2025/sha256/c475c1194707605c9d1c37e6b0c048417e1ddd4a85303ab64b5770b1643ab0f0.webp)

#5 - Dynamic content should be referenced

Framework Integrations

Prefer these over manual instrumentation:

Framework | Integration | Docs
OpenAI SDK | Drop-in replacement | https://langfuse.com/docs/integrations/openai
LangChain | Callback handler | https://langfuse.com/docs/integrations/langchain
LlamaIndex | Callback handler | https://langfuse.com/docs/integrations/llama-index
Vercel AI SDK | OpenTelemetry exporter | https://langfuse.com/docs/integrations/vercel-ai-sdk
LiteLLM | Callback or proxy | https://langfuse.com/docs/integrations/litellm

Full list: https://langfuse.com/docs/integrations

Setup:
• Relevant info for implementation is explicitly referenced in skill
• Info is part of regular content/docs release cycle, hence does not go out of date

Result:
• Avoids creating multiple (potentially conflicting) sources of truth
• Nests multiple levels of progressive disclosure

github.com/langfuse
Langfuse
AIE
Engineering the future of AI

Interface text:
main
skills / skills / langfuse / references / instrumentation.md
Preview  Code  Blame

## Full transcript

[0:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=0s) [upbeat music] Okay, that was quick.

[0:16](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=16s) Hi, everyone. Super excited to be here. I'm Marc, one of the founders of Langfuse. Uh, when we started Langfuse, like, three years ago when everything was-- felt quite early, uh, building agents that didn't work, uh, and then realized, okay, there needs to be, like, some evaluation tracing built.

[0:30](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=30s) Like, uh, Langfuse is the open source project in the space by now. I mean, by the metrics that we track, uh, we seem to be the largest one in the space.

[0:36](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=36s) We do all product engineering out of Europe. Thus, I'm very excited that this conference is coming to Europe because there are so many great people here. Uh, and we always need to resist the urge to, like, ship the whole team to another continent because actually being here is very nice, um, and you can just travel and hang

[0:49](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=49s) with people on Discord and Zoom. So yeah, very excited to be here in person. And, um, what I wanna talk about today is how, um, well, the lessons we made from skilling up coding agents and actually adding Langfuse to an application because, like, back in the days when you added, like, observability or evals, you needed to read,

[1:04](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=64s) like, hundreds of pages of docs, like, figure out your own mental model, and now you kind of expect, like, to be hand-holded by a custom agent to do this.

[1:11](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=71s) And, uh, we've come a long way, uh, in achieving this kind of vision, and I just wanna explain what we learned on the way. Um, first of all, like, I'll start very conceptual, very easy, then a bit more conceptual deeper, and then I'll do-- go to the learnings.

[1:22](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=82s) So, uh, my mental model for skills is just like, I get this Rubik's Cube when I was a kid. I d- had no idea what to actually do, but I mean, I basically have a bash tool.

[1:29](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=89s) I can do whatever I want with this Rubik's Cube, but it just looks colorful in different ways and no idea how to solve it. So skills was, like, a great way.

[1:35](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=95s) Uh, once you get the manual, it's easy. You just need to follow the manual and you can, you can solve a Rubik's Cube. And I feel like the same thing now applies to, to agents, where, uh, like there was this whole debate of, like, workflow versus, uh, fully autonomous agent, and was, like, huge fight on X of,

[1:49](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=109s) like, what is the best way to build an application? Everyone was kind of like, "Yes, you kind of like need both." And I think Marta this morning had, like, a good note where, like, the surface area of deploying agents is so broad that, um, like for some, you don't need, like, a coding agent, even if it's, like,

[2:02](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=122s) the best way you can build agents at the frontier today. Like, you don't need this for every application. It makes things slow, expensive. And, um, like, there's this kind of, like, balance between workflow being very reliable and agent now having kind of, like, unlimited capabilities.

[2:14](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=134s) And I think what's very exciting is that then skills are kind of like a formalized shortcut to make things more reliable, where you historically would've built a workflow. Like, I don't know, you have a customer res-- uh, like, customer support agent, and someone asks for, like, a password reset, then historically you would've built, like, a workflow that's

[2:29](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=149s) very reliable of, like, ah, you have a router that routes to, like, an agent that can only do password resets, and that agent has, like, the context to do password resets well.

[2:36](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=156s) Um, that was great, but also if the user then wants to do password reset but also change the email address at the same time, then, like, the router is kind of like, "Okay, I have this email router, I have this password reset router.

[2:46](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=166s) What, what do I even do?" And I think what's very exciting now is that an agent can just, like, progressively get the context needed to then solve, like, a problem that's multi-domain.

[2:54](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=174s) Like, that would've historically been in multiple workflows. So that's very exciting. However, uh, it has always been hard to kind of like how to, how to build an agent, what kind of use cases even exist because you have, like, this open-ended text input box often or, like, open world context, so you didn't really know.

[3:09](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=189s) So what we now see is what many teams do is they have the agent runtime, they trace everything. I mean, I'm building Langfuse. Thus, I put my logo here.

[3:16](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=196s) But you can also use whatever you want. In the end, it's more about, like, the concept of tracing is what you need to, uh, like, identify what happens at runtime when a trace-- when, like, an agent's executed.

[3:25](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=205s) Because then it helps you learn two things. One, uh, like, new kind of, like, use cases that you didn't expect, uh, users to do because you might have expected that nobody ever wants to change a password, but now they want to change a password.

[3:36](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=216s) Thus, you need to see the execution trace of someone being upset via, like, a production eval to then derive that you need to add a skill, um, for handling password resets well.

[3:44](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=224s) Uh, and two, once you have these, uh, skills, they can get out of date, or you can realize they're not the most efficient way of actually solving for this use case.

[3:51](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=231s) So it's, like, the second thing of, like, helping you improve skills that you already have in your agent. So this is very conceptual, but this is what we, what we now see, see most teams do, uh, that use Langfuse.

[4:01](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=241s) Now more towards, like, the learnings, uh, we made, uh, when building a skill to help, uh, customers add Langfuse to their project. So what was the lay of the land before we got started with this?

[4:11](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=251s) Um, like, four hundred and seventy-eight pages of documentation. Whenever I see, like, the thing deploying, I'm like, "Who wrote all of this?" So apparently, if you build a project over three years, it just grows in complexity.

[4:21](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=261s) People can do all of these different things, but then you need to read all of these things, and nobody has the time. Across, like, five different f- uh, feature areas and a lot of implementation flexibility because there have always been, like, projects in, like, the eval space that were very, I'd say, opinionated.

[4:35](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=275s) So they were like, "Oh, you have a chatbot? Then add this project, and it'll just solve this for you opinionated end to end." We always were like, "No, no, we are infrastructure.

[4:43](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=283s) We do, like, tracing well. Like, if you ingest, like, billions of traces, we'll still work. If you wanna customize your evals, it'll still work." So we're always, like, more on the unopinionated side, uh, which was always, I'd say, uh, a weakness compared to, uh, projects that are more opinionated, but now I think it's a strength because in

[4:57](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=297s) the end, what, what, what, what do you need, uh, when agents do all of this? You only need the infrastructure piece if agents can then customize for different workflows.

[5:03](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=303s) Um, so but there's the problem if people wanna add Langfuse to a project. How to do it correctly for a project is to be figured out by an agent.

[5:12](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=312s) Um, and, uh, interestingly, uh, when we first got into model pre-training context as a project, and you can just ask an agent how to add Langfuse, and it'll spit out, like, Langfuse SDK logic.

[5:22](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=322s) The first time this happens, it's amazing. But then if you're, like, two years in, like, the project evolves, like, interfaces change. Uh, and now being in pre-training context might even be, like, a disadvantage if, uh, you don't fetch up-to-date information.

[5:34](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=334s) So we're just like, "Oh, we get, like, all of these hallucinations of methods that have been, uh, available in the past, but they are not, uh, available there today."

[5:41](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=341s) So yeah, we, we felt like when skills launched, this is, like, the exact pattern that we need in order to help teams achieve this. Um, so I- I'll use an example.

[5:49](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=349s) Um, when you just ask Claude Code to add, uh, Langfuse to a project, it just works, but it was not working in the best way possible. So for example, user asks, "Add tracing to my agent," and, um, then Claude Code kind of, like, implements the instrumentation based on the outdated pre-training context, then tries to verify whether the

[6:08](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=368s) tracing works, then realize, oh, it doesn't work, and then only in the second step fetches up-to-date information to then correct the issue. Um, at the same time, how you add tracing or evaluation to a project, they can evaluate in, like, like, gazillion different ways, like online evals, offline evals, human-in-the-loop.

[6:24](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=384s) There are so many different things, and often the question is like, what even is relevant for your application? Uh, so human and agent kind of need to figure it out on the way, but agent is not tasked to help you, uh, like, figure out what's the best thing for your, for your, um, application.

[6:37](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=397s) So main problems, outdated training data, the non-optimal setup because the agent wasn't really, like, primed to help you discover what to do for your, for your app, and it's very slow because you first add instrumentation of wrong way, then you figure out it's wrong, and then you need to fetch more documentation to, to fix the, the issues.

[6:53](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=413s) So what did we do? Uh, yeah, this is how the, uh, this is how, uh, just mental, mental note. This is how the trace looked like when we just tried it with Claude Code.

[7:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=420s) So like just tracks two LLM calls in an agent, but like you still don't know what the agent's actually doing. So, um, what was the goal of our skill?

[7:07](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=427s) Uh, like, give every, uh, Langfuse user, like, they have thousands of teams in the community, thousands of customers on our cloud product, give them all, like, a Langfuse expert to help them quickly set up observability, prompt management, evals in line with, like, best practices and up-to-date docs and rep- uh, like, references.

[7:21](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=441s) Because, uh, all of you, I mean, uh, there's Anabel from the team here as well. Like, if you have questions regarding observability evals, you can talk to us, but in the end, that doesn't scale to thousands of people, uh, to basically talk through your problems and figure out what the best strategy is for you.

[7:33](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=453s) So we are like, "Okay, skills is the, is the way to go." And this is, like, very conceptual how our skill works, where, um, like user comes in, asks coding agent to do something, and then the skill kind of has like a reference of like ...

[7:45](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=465s) The SKILL.md is more like, well, what kind of style do we want, uh, in order to implement, um, Langfuse? So for example, ask follow-up questions before making a decision because there's so much you can be doing, and then references for the different product modules to kind of like progressively disclose additional hints that the agent need, might

[8:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=480s) need to have. And then, uh, it can call, um, the documentation. And interesting enough, as we start open source and tr- like, saw ourselves as, like, independent infrastructure, we always had like APIs for everything because teams built their own, I don't know, own labeling UIs on top of it, own evaluation, uh, execution logic on top of our,

[8:16](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=496s) our, our backend. And we'd had APIs for everything. Now we've wrapped it in a CLI, and now an agent can just do everything humans needed to do in the UI in the past, um, which is very cool, uh, because, uh, like so many teams spend so many hours every week in clicking around in our UI to evaluate

[8:30](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=510s) and improve the application. And in the end, how will this look like end of year? It'll probably just be like, uh, like connect repository to Langfuse, and then agent just does the whole thing, uh, like autoregressively.

[8:39](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=519s) I mean, that's what we are building towards. That's what everyone is building towards, and I think that's a cool step in the right direction. So, um, to, to shortcut to the end result, okay, after, like, conversing with the agent now for the, uh, for a similar thing, it looks, uh, way more detailed, like, um, like detailed evals

[8:55](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=535s) that are relevant and detailed steps, uh, regarding to execution. Um, so, so just like there, there's a stark difference. And yeah, what did we learn on the way? Um, six main things.

[9:04](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=544s) Uh, I'll go through every single one of them, um, of, uh, what were basically our realizations, uh, when we're building the skill. Um, and, uh, like one, looking at traces still, uh, like gets you to like 80% of the detail.

[9:15](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=555s) This was always what we kind of like tried to preach regarding like evals, where many people try to complicate things right away while they haven't digged through, um, through like, just like what did the agent actually do at runtime themselves a couple of times.

[9:26](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=566s) So, so, uh, what do we do? We, uh, have instrumentation for Claude Code and just ourselves interactively, um, like tried to use Langfuse with Claude Code and then looked through, uh, traces in Langfuse to, uh, understand where did the, um, agent error, how can we improve the skill to make it, like, uh, straight, uh, shooting at the

[9:45](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=585s) goal instead of like wandering in, in different ways to the, to the target. And that was really helpful, too. Um, there were like some of the, uh, interesting learnings here.

[9:53](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=593s) Um, for example, for humans, we tried to cut down on the number of environment variables that you need to set in order to set up Langfuse. So for example, we just auto assumed, um, a data region.

[10:01](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=601s) Like Langfuse is available in Europe, it's available in the US. Uh, fun anecdote, we assumed that only Europeans, uh, like care about data regions. Thus, we made Europe the default.

[10:09](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=609s) Then we learned some US enterprises also care about data regionality. Now we have a US data region, so many different other data regions. Um, so we always defaulted to Europe, and now we kind of like for an agent, like adding another environment variable, like they don't care.

[10:20](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=620s) Like it's not effort for them. Thus, uh, we always prompt for figure out what data region the user's actually in and don't assume Europe, for example. Uh, two, hallucinated hal- uh, CLI parameters, uh, because it just, oh, uh, this like includes the word trace.

[10:34](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=634s) I've seen tracing CLIs before. I just assume what, what we could be doing here, and we just advertise the, the help flag more aggressively. Uh, it takes another turn, but it's fast, and thereby, uh, like it directly knows, uh, what the CLI can do.

[10:46](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=646s) Um, two, um, we try to help the agent to understand how to navigate available information because, like, I mean, five hundred documentation pages, how to find the right one instead of like looping through, I don't know, fetching one, then learning something, then fetching another one, always with like thought process in the meantime.

[11:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=660s) Um, so what did we do? We always had this llms.txt, which was very hyped when it launched but never actually used. I think what's now cool is we have this agent sitemap kind of that we just expose to, um, like a coding agent via the skill of like, go there first in order to learn what kind of

[11:14](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=674s) like documentation is available. And, uh, two, I mean, there's like this whole content negotiation that if you send like an, a request header that you want Markdown, that you get Markdown back from the docs, but some coding agents don't do this by default.

[11:26](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=686s) So we just advertise this because otherwise some coding agents might try to parse the HTML, which just adds additional tokens. So, uh, like for Langfuse for example, you can just add like a .md to any documentation page, or you can request Markdown, and you'll get a Markdown page.

[11:38](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=698s) Uh, three, um, I think that was, uh, like one of the things I was most excited about. Uh, we always had this, uh, like docs Q&A agent that was able to answer questions about Langfuse more like interactively.

[11:48](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=708s) Therefore, we built like a RAG stack, and now we just surface this, uh, RAG stack again via like a search endpoint. So like a coding agent can just ask whatever natural language query about Langfuse, and we'll get back documentation chunks for this query.

[12:02](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=722s) Why is this exciting? One, you don't need to fetch five different docs page where you can just ask a question, get something back that's relevant directly, uh, solve for the problem.

[12:09](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=729s) And two, we get to track these search parameters, um, because, uh, like if a coding agent fetches documentation, it's very difficult to understand what did like Claude Code on our user's laptop do.

[12:18](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=738s) But if they, um, ask questions about Langfuse to our search endpoint, we can track the searches and thereby understand what problems do they run into, where do we need to add more documentation pages, because maybe we didn't expect this kind of problem to happen.

[12:29](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=749s) So yeah, so adding a search endpoint was like, was really cool to, um, capture more data. Um, then, uh, basic eval setup is better than none because we initially struggled to get this done because it's like so broad.

[12:41](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=761s) Some Langfuse users build like chat applications, real-time voice, video generation, like batch pro- Processing of like invoices in the background of some kind of like text, uh, software. Like so many different use cases where then the question is, what's even like a good evaluation, uh, like setup?

[12:55](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=775s) And we just created like five, uh, different ones, and this was already helpful, um, because otherwise, uh, it's really hard to, uh, to kind of like measure, measure anything.

[13:04](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=784s) And what we did here, um... I don't know, can I zoom in? No. Um, so basically we have this, uh, like just like a prompt instrument application with Langfuse and then like a sample, a repository folder.

[13:13](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=793s) So for example, like an OpenAI custom function RAG, whatever, uh, application. And our checks are just natural language statements that we then, by LLM-as-a-Judge try to, um, like evaluate on top of the file system and diff state before and after, um, running the skill.

[13:27](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=807s) Uh, so for example, we ex- uh, we expect that our OpenAI instrumentation was added because it's like an OpenAI example, and we, because it's RAG, we, um, expect like some retrieval spans to show up in our, uh, trace because if they, if there are no retrieval spans, then probably we only capture, for example, LLM, um, calls.

[13:43](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=823s) This was already helpful because that way we were able to make changes and see that we didn't break anything. The whole thing that, uh, why, why we even built Langfuse for like building AI agents now also applies here.

[13:52](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=832s) Five, uh, dynamic content should be referenced because there's a huge, I'd say, incentive for like developers on the team, but also for users in the community to just contribute a lot of context to the skill because then you're like, "Ah, it's kind of like a local cache of the documentation that's immediately available."

[14:06](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=846s) However, then the same thing applies, uh, that applies to pre-training context. It's kind of, uh, like it goes out of date, and now we have the documentation, and now we have yet another representation of what Langfuse is.

[14:17](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=857s) So we'd rather try to point just to straight to the reference of documentation and, um, because otherwise you just duplicate all, all content. And six, uh, we applied, um, of like auto research, uh, to, uh, the skill of, okay, if we have a target function, how can like agents help us improve the agent?

[14:33](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=873s) Um, because there are so many like different patterns that we can explore. Um, so we set up a target function mostly geared towards, uh, like, uh, our experiment really here was help teams move prompts from their local Git repository into Langfuse prompt management, which is used by like larger teams to collaborate on prompts with their non-engineering counterparts

[14:48](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=888s) because then like PMs can make changes to prompts, iterate on a playground, like all of this kind of like collaborative stuff. Um, and the task was, okay, how do we improve the skill to migrate prompts out of any kind of like code base into our managed prompt system?

[15:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=900s) And, um, in the end, we accepted three out of the six, uh, improvements that were suggested, which I think is a success. Uh, uh, but, uh, it allowed us to experiment much more than we could have explored manually, uh, with the time that we have as we are like a very small team.

[15:13](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=913s) Uh, learnings, uh, like the target function really matters. Uh, like, uh, I think it's-- it sounds obvious, but, um, for us, defining like the right target function was very hard for this, um, because we, we assumed like a prompt migration should be fast.

[15:25](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=925s) Fast, we measured in like the number of turns, but if we, uh, basically asked to minimize the number of turns, then like our, uh, like the agent that tried to optimize the skill just took out all of the notes that we had to, um, like fetch documentation because it was like, "I know how to-- how Langfuse prompt

[15:39](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=939s) management works. I don't need this. I'll just try it myself." Uh, which then negates the whole thing of we wanna fetch up-to-date context, because otherwise, if you use the skill-- install the skill once, wait three month, then you'll have like wrong context, um, because, uh, we duplicate information.

[15:53](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=953s) Two, um, like we had like an approval gate usually, uh, where we wanna suggest a plan or ask follow-up questions, suggest plan to a user before doing anything because we kind of like push their prompt to like a central repository, and it's kind of like their data leaving their laptop somewhere else.

[16:07](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=967s) Um, but the sandbox didn't have this, so we didn't really-- we weren't really able to, uh, to, to try for this. And, um, like Langfuse, like the, the whole feature set, like usually we try to make it easy to get going with something, but then it's very deep of how to do it in a good way, and we

[16:19](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=979s) want agents to directly go for the good way. Like figure out with the user what they wanna achieve and then have like a very full implementation, not start with something and then like two month later go deeper.

[16:29](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=989s) However, if the target function does not include, uh, like we want like, uh, linking prompt versions to prod traces so then you can see how like different prompt versions impact like, for example, production results.

[16:39](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=999s) Um, like we didn't have this in the target function. This, uh, like e- everything that y- like nudged towards this was kind of like removed because it's kind of like, it's just like, like garbage on the way that we don't need to achieve the goal.

[16:49](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1009s) So again, the target function really matters. High level, these were like the, the six, uh, main takeaways. Looking at traces gets you eighty percent of the way. The, uh, production signals really help.

[16:58](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1018s) So the search endpoint, uh, was really helpful for our documentation. Um, help agent to navigate, uh, the information because otherwise it just searches with like Google, Brave, whatever search and finds all sorts of different things on the internet.

[17:11](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1031s) Um, even a basic s- eval setup helped. It wasn't that hard to set up. The dynamic content should be referenced, otherwise you have just duplicates, and the auto research was, uh, very helpful to explore things, but, um, it's bound by the target function.

[17:23](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1043s) Um, topics basically on our minds here are, uh, it's so powerful, but at the same time, you kind of then duplicate stuff into like user space, kind of like somewhere on, on like a machine.

[17:33](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1053s) Um, like there's no like package management for this, which then like a year later like tells the user this is outdated. Um, like we could, uh, we, we, we thought about just adding like a timestamp of, uh, this is the current date where it was fetched, uh, the skill, uh, and then just, uh, oh, if this like older

[17:48](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1068s) than a month, then try to update. But then we go to second problem of skill distribution and like installing into like the agent environment. Usually this is kind of like gated or not possible for the agent, like depending on, uh, what you use.

[18:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1080s) Um, this user needs to do something to install the skill. This also upgrading doesn't, uh, auto-upgrading doesn't really, uh, really work, but it really depends on the coding agent that you use.

[18:08](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1088s) And, uh, like the target function is interesting for us because like we can either go for user needs to get to like an initial aha of like, oh, this works, or do we want to directly straight shoot for this is the perfect setup of how you would do evals for this use case?

[18:22](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1102s) But this is-- I mean, without a skill, it takes like usually like an AI engineering team, it takes like month to get to a perfect setup. Do we now aim for an agent to do this in a single shot and overload the user with lots and lots of questions, or do we just try to get to something

[18:34](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1114s) and then you can still invoke it again of like improve my setup, ask, and then it can ask questions to improve it? So it's kind of like, well, what's, what's the target for the skill?

[18:42](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1122s) That was, uh, very interesting for us. Uh, yep. I would invite you to try it and give us feedback, um, because that'd be really interesting. Um, and, uh, like, uh, we do lots of, lots of calls with people from the community every week and, uh, like, like Uh, I think it's not a surprise that I think nobody

[18:59](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1139s) reads documentation themselves, and everyone is just like, "Yeah, just add this to my-- Like, I just want this to work. Like, just add it." Um, so yeah, the skill is the primary way of how things get done.

[19:07](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1147s) This is also, like, now the advertised way across all of our documentation, um, that you just should ask your coding agent to do whatever you try to do right now.

[19:15](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1155s) Um, I'm very excited that it works really well, uh, but also I'm, I'm excited to see what, what comes next. Um, for us as a project roadmap-wise, we see the skill right now, like our users use this when getting started with the, our project, but also, um, to drive a lot of automation around the, like, evaluation life

[19:30](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1170s) cycle of, oh, I now wanna create like an LLM-as-a-Judge that's aligned with user preferences, or, um, like I got user feedback on a hundred different ex-executions. What do they have in common?

[19:40](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1180s) Um, do you just then fetch this via the CLIs? So many of these workflows that people needed to do manually now coding agents do for them. Um, we'll bring this in product via, like-- We, we'll help automate this via skills one, bring this in product two, and then three, I, I feel like we just need this orchestration

[19:52](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1192s) agent, then that's what the team is doing right now. So yeah, I'm very excited for our roadmap to, to automate all of this. But yeah, if you have any feedback, uh, I'm around, Anibal is around.

[20:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1200s) I would love to talk to you. Uh, and yeah, thanks so much for your time. [audience clapping]

[20:09](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1209s) I don't know if... Do we have time for a question?

[20:11](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1211s) Yeah.

[20:11](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1211s) Okay. Yep.

[20:15](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1215s) I may have missed it. Uh, you say like when you were tuning the skill, the human was completely out of the loop. So you were like-- It was basically like Claude Code directly trying to implement using the skill, the right implementation and then you measured this or?

[20:32](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1232s) Uh, yeah, it was kind of, um... Like, I mean, you kind of wanna be out of the loop for the experimentation and then just review the suggested changes. Uh, so it was kind of, um, like experimental things, um, give us like all sorts of different recommendations and then human review the suggestions.

[20:47](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1247s) Like, we didn't accept all because many didn't make sense because our target function wasn't perfect. It was really difficult to get to like a very perfect target function, but it's good at just creating ideas and then, uh, we human reviewed all of the ideas to make the changes to the skill.

[21:00](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1260s) But so at the end, the skill might not be optimizing for the human AI interaction. Let's say I work with like Claude Code and to try to set up like the trace for whichever project.

[21:13](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1273s) Then the skill right now might be optimizing for something different and not necessarily what you needed or I don't know.

[21:19](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1279s) Yeah, that's what we try to, uh, kind of like you need to try it yourself to just get a sense of how it feels to use the skill to then, um, like, like add Langfuse to an application.

[21:27](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1287s) So we just use it ourselves to get a sense for the feeling because it should... Like, where, where you wanna go is it should feel like, um, like an expert user trying to guide you through what you need for your problem, where usually someone comes in with just like, "I need evals because I read about it online,"

[21:43](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1303s) uh, but I don't know what actually I need for my application and it kind of like it needs guidance of where you wanna go. Uh, like, "Well, what is your problem?

[21:50](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1310s) I don't know. What do you, what do you worry about? You probably don't need like a, I don't know, hallucination eval, but probably you need something that's very specific to your application."

[21:56](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1316s) And, um, yeah, that's what we wanna achieve with the skill that you get like some, some like professional guidance.

[22:03](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1323s) Yep.

[22:05](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1325s) Time for a question.

[22:06](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1326s) Uh, I really resonated with your last point about, uh, skills distribution seems practically insane, uh, right now of just like you just install it, do whatever, GitHub repo link is on day one.

[22:17](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1337s) Uh, what are your thoughts on sort of like the treating skills as packages, like an npx skills kind of approach or like going full all in on, you know, plug-in marketplaces instead?

[22:26](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1346s) Like what, what do you think is, you know, able to be adopted by the community versus, uh, sustainable long term if you have like problems with stability?

[22:35](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1355s) Hmm.

[22:36](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1356s) Sure.

[22:36](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1356s) Uh, as a small team, I'm not that excited about plug-in marketplaces because then you need, now need to kind of like maintain all of these proprietary integrations, update them in, I don't know, tell Anthropic, tell OpenAI, tell Cursor whenever you're making-

[22:47](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1367s) Well, Anthropic and OpenAI are now agreeing on sort of like broadly on it.

[22:51](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1371s) Yeah. S-still, I mean, like for the skill, uh, I think it would be cool if we just had like a well-known skill or something and like whenever someone is like, "Oh, I wanna, for example, use Langfuse," like the agent can just auto discover that it exists.

[23:03](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1383s) Uh, like we have it across all of our docs, so I think it would be enough if agent can kind of like ask user, "I wanna install skill." Question is, do you even need to ask?

[23:10](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1390s) Like I think you only need to ask if the skill is kind of like more trusted than the public web. If it's like same trust level, then why even bother asking?

[23:17](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1397s) Um, and then two, it's kind of like if I have this installed, it's kind of like a cache of something that was up to date when it, when I installed it, but then the question is how do I know whether it's out of date?

[23:27](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1407s) So I think so just like timestamping it is enough. So when you use the skill, that agent can be like, "Oh, this seems old. I should probably like fetch a new one."

[23:35](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1415s) Um, I think this would already go a long way. Um, but yeah, I'm, I'm excited to see. Like, uh, we are going more the timestamp fetch route or alert user of this might be out of date.

[23:44](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1424s) Uh, that's at least like what we discussed now. Um, but yeah, I'm excited to see what everyone's shipping in this space. Yeah, I, I'm around. Uh, thanks so much.

[23:52](https://www.youtube.com/watch?v=vNCY9kXXyDQ&t=1432s) Bye-bye. [audience clapping] [outro music]
