Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

All right, let's get started. Welcome everybody to our workshop on evaluating AI applications with Langfuse. My name is Annabell. I'm a, um, AI engineer, uh, AI solutions engineer at Langfuse. And I'm today joined by my, um, colleague Doneyli, who is load- leading AI solution architecture at Langfuse. So working with our biggest customers on making sure they are actually shipping reliable agents to production. I myself, uh, was in my previous life, how I like to call it because it feels like a lifetime, in a company building AI agents for medical device

compliance. And in this industry, it was super important to make sure we are running evals. We're making sure we have ground truth dataset that is validated by our customers, and we also ran a lot of human-in-the-loop systems. And a lot of my time I actually spent in Langfuse. And at some point I decided, okay, why not swap sides and also help other teams that are shipping agents, especially also in regulated industries, and, uh, joined the Langfuse team, uh, roughly seven months ago. So but before we dive right into it, we wanted to get a little bit of a

feeling of who is here today and who we are, um, yeah, coming together with. So, uh, we were wondering who of you has brought AI agents to production before? And with that I mean been part of a team doing it, not necessarily, yes, someone in the company did it. So actually, like, been part of a process to bring it to production. Okay, that's good. Amazing. It's a lot of people, and we hoped, uh, to have this also. And who of you has worked with Langfuse before? Okay, quite some. And any

other kind of observability and evals tool? Okay, yeah, also a few. Uh, amazing. And then finally, who has solved evals for AI applications?

For the confident hands, I would love you to come by at our booth and explain us how. Um, because actually it is quite hard, and I think, um, all of you have experienced it if you're working with AI or also just chatting with ChatGPT or Claude. Uh, it is quite tough, uh, because AI agents can fail while the system metrics themselves look green. So if you, for example, look at this example here. We have a user support agent, and the user is asking, "Can I get a refund?" The agent takes in the request, looks up some

policies, and then is like, "Yeah, sure, I issued your refund." And the user's like, "Thank you." And it will take some days until the user sees if it actually worked out. And the status is green. Nothing went wrong in traditional observability terms. There's also no latency issues. But if we dig deeper into what actually happened, the agent never did refund anything. So the tool was never called, it was never executed, and the agent lied about doing it to the user. And we will only know about it in a few weeks when the user come backs and is like, "Hey, by the

way, I didn't forget... I've never gotten my refund." And this is the problem that we have with those non-determinig-- deterministic systems. They might tell us they do things, but they actually don't. And this is why you have to find an approach and a way to continuously mine for those signals where things are going wrong and make sure they don't happen again. Or maybe not even at all in the first place. And this is the problem our founders also faced three years ago when they were in the Y Combinator batch, and they realized, okay, there's nothing like a unit test for AI applications, and we need a solution

for it. So they built Langfuse, which is by now an open source platform to continuously trace, monitor, and improve your AI systems. And while we started out as a dev tool mostly, by now we also have domain experts as a target group, so collaborating with AI engineers to make sure the system does what it actually should be doing. Uh, but also a new user group for the past probably eight to twelve months, uh, which are coding agents that are helping our engineers also to work with AI applications and improve them.

Langfuse is by now adopted by more than hundred and fifty thousand, uh, developers globally, also largely driven by our open source, um, uh, setup, of course. Twenty-one of the Fortune fifty have us as part of their core tech stack when shipping AI applications. And every month, more than ninety billion observations are ingested into, uh, Langfuse instances. Uh, of course, increasing tendency as agents are taking more, more and more steps and also taking over more and more tasks. Companies like Ramp, Canva, Khan Academy, and SumUp, uh, work with us. And over the last three

years, we of course spent a lot of time with them. And one thing we realized working with those companies is that the ones that successfully managed to ship reliable agents and also continuously improve them follow something that we call, like, some form of this, um, the AI engineering loop. And it is about a set of practices that help you to continuously improve your system by taking a systematic approach. The foundation to this is always tracing. Tracing is about getting all the steps

of what your agent is doing into a more or less log, but ideally also in a way you can inspect it a- as a human. Monitoring is about finding the interesting ones. So what is going wrong? What is going right? Where should we look deeper into? But also understanding how is it performing on average. Building datasets is about finding a representative set of examples that you want your application to be able to handle and also how you want it to handle it. And then you wanna run experiments on it, so try different variations of your system to

check, um, how it is performing, how it's going, and then of course, you want to evaluate how it's going. And this is not something you do once, but you continuously close this loop and want to improve, uh, those A- AI agents and learn from production and get, get it back into datasets. Uh, but let's dive in each of those steps a little bit more into detail.

So for traces, the most important goal is that you want to get deep insights into what your agent is actually doing. On the right side of the screen, we can see a trace that is also a screenshot from Langfuse, and in this case it's a dad IT support agent. So an agent that is answering questions from your parents, so you don't have to do it. And we can see that there's first, like, the root trace, and we can see aggregate, um, time and also, uh, cost. Of this step, but we then also get a breakdown of each of the individual steps that a agent is taking,

taking. So both the generations, which are the OpenAI chat ones, and also the tool calls, which we can see with the little tool in front of it. So we have a detailed view of what's going on, and if the screen would be wider, we would also see that we can see the inputs and outputs and all the metadata involved.

This is then the foundation to track the cost, the latency, the tokens involved, and also have a log to inspect for you as a human to really dig into what's going on, but also for your coding agents to work with it and look at it at scale and just have this, this inspectable log of what's actually been going on.

Monitoring is then about making sense of it at scale because especially now with increasing, uh, automation levels, longer traces, and, like, a lot, a lot of use cases, it's just not feasible anymore for you as a human to look through those logs completely manually. But you actually want a way to get aggregate metrics on what's actually going on and also, uh, detect signals where things are going wrong. So for example, cost and latency metrics, but maybe also quality signals like are our compliance guardrails, um, kept true to? Uh, also, like,

how many users are actually frustrated with our system? How many are writing all caps messages because they are maybe a bit upset with what, uh, the agent just did? And, um, yeah. All of this aggregating and then, um, of course, also finding the interesting stuff that you wanna dig, dig deeper into.

What you get from production through this monitoring is then also one of the foundations, um, that you can use to build out your datasets. And datasets are, in the end, a representation of what you want your agent to handle in production. So all the cases that it might be getting, also maybe out-of-scope requests that you want to, like, correctly also be deferred and make sure that, like, the agent's not trying to give investment advice or also, uh, answering what the weather in New York's gonna be like if it's actually just a dad IT support agent. And those, um, items you

can then have, like, the input, but also what you expect, um, it to have as the output, which can both be, like, a sample reply or also expected behavior that you want to see or keywords you want to have covered. And this can then be used as a stable benchmark either for, like, improving and iterating offline or also as part of your CI/CD.

Those datasets are then used to run experiments on. And experiments are, um, if you maybe look back into university and statistics or anything that you might have done, it's in the end this idea of varying something about your setup and looking what happens then. So for example, changing a prompt, changing a model, changing the context or the tool description and to see if it's then better dealing what you act- what you, with what you actually want it to deal with, and ideally isolating concerns. So if you wanna check if the same model performs better with a different prompt, you ideally only change the prompt and

not five other things because otherwise it's gonna be hard for you to interpret what you actually see. And then finally, you need evals or evaluation approaches to make sure you can also measure what's happening in those experiments because, again, you don't wanna look at everything yourself, uh, which you, by the way, should, should probably sometimes do. Um, but still, you also want the scalable approaches, which are, for example, LLM-as-a-judge, so using another LLM to check what the LLM did, um, or also code-based evals for things like, um, classification tasks where you

can check for an exact match or, um, also checking behavior can be a, a deterministic check or if you are checking for all caps type, um, things.

So this is the AI engineering loop. And what we observe with, um, many of our customers is that there's, depending on your use case, a few different entry points into this. So if you're, for example, in a low-stakes environment and you're only building your internal chat agent, you might first not have a observability tool in place at all, and then you maybe get started with tracing and add some monitors to just check where people are getting frustrated. But you don't care about the evals, offline evals part too much in the first place. You just build it out over

time and just, like, add those test cases as you progress, but you can go to production directly. However, if you're building a case that is customer-facing, and especially if you're building a case that is in regulated environments, you probably don't wanna risk the agent doing something that it's actually not supposed to do. So for those regulated industries, we've seen a few approaches emerging, and the foundation's always tracing. So you always want to know what is actually happening, what steps is it taking, what inputs and outputs is it seeing, and

also what tools are being called. And then on top of this, um, those teams in regulated industries found a few approaches on how to go about building out these ground truth datasets that you can then also loop on, improve on. And also, you probably all heard about auto-improvement. Um, all of this definitely needs the foundation of a ground truth dataset, and you, yeah, have a few approaches to get there. For example, expert curation, so working with your domain experts within the company to make sure you're building out the ground truth dataset.

Then, for example, shadow runs. So you run the process continuously manually, but in parallel you also have the agent running, and you can always compare how would the agent have handled it, how would the agent have gone about it. And then, like, this, uh, measure the accuracy, but also build out your ground truth. And then finally, um, like we did it also in my previous company, run a fully human-in-the-loop system, and over time slice the data into parts where you are confident to automate versus things where you always want a human in the loop.

So, so much on the introduction. Uh, we now wanna go in a more practical example, and for this I'm also welcoming Doneyli again on stage. And we're gonna look together into how to trace your agent execution into Langfuse, how to evaluate and improve your system, and how to continuously close this loop.

Hi, everybody. Like Annabell said, I'm Doneyli, but you can call me Don to keep it easy for everybody. Only my mom calls me Doneyli when I'm in trouble. Um, yeah, that... Okay. That's great. A few giggles. So, you know, nice ice- a nice icebreaker. So that's the first eval. It went perfectly. Um, so now what I'm gonna show you is an example that we built for a few financials institutions that we're working with because they're in the- in, in kinda this process of they build their first prototype, they build the, their

first demo, but then it- the question comes from leadership or maybe from the teams, "How do we, we evaluate this customer-facing agentic solution that we're building?" How many of you here are in financial services or in finance? Yeah, we're in New York, so of course, you all are. Um, any other industries others than financial services? Regulated industries like healthcare and stuff like that. Okay, cool. Awesome. So I think this is gonna- this is gonna speak to you very much. Um, so basically what I have here is, you know, a

typical, you know, banking assistant. Everybody, anybody of us that has banking services probably have seen this in, in each of, of, of your banks, right? However, the... all the stuff that happens behind is, you know, very important for the bank itself or for the institution to provide you with very accurate information because there is, like, financial liability, there's legal liability. There's a bunch of, you know, risk for the bank itself to put out these type of solutions when they are

powered by LLMs, which as we all know, are inherently non-deterministic. So we want to make sure that we reduce the unpredictability, and that we can then reproduce the outcomes as much as we can and figure out where are the failure modes and where is the agent giving us the results that we don't want so that then we can, you know, improve upon those. In this example, I have a few things that I, I would like to show you. The first thing is that I can impersonate, you know, a few different type of clients

because obviously a bank is always going to have c-customers that are, like, you know, your everyday Joe like myself. I only have a couple of bucks in the bank. But then you're gonna have your ultra-high net worth individuals and all that. So we can have here impersonation of different kinds. And this is important because we wanna be able to know if the agent is able to respect any guardrails that we have put in place when it comes to those type of, uh, personas that we have. The other thing is, if you notice here—maybe let me blow it up—we also

have PII identification, uh, enabled, and any sort of, uh, data that is d-deemed as, uh, PII is automatically masked before it gets persisted into Langfuse. Again, this is another very big requirement for espicif- specifically financials institutions, that they don't want to, you know, persist in an observability system their credit card information or Social Security numbers for the clients. They wanna keep that on the systems of record and as much as possible

in their systems, uh, that they own. So that is also enabled, and I'll show you a couple examples on how we can, uh, get that to work. And then we're also going to see if the agent itself is respecting some of the controls that we have put in place and how it's being scored, and if the quality gates that we have defined are also being enforced. So we're gonna m-measure all of that. I'm going to show you a few examples and then take you behind the scenes in the product itself to show you how you can set

that up and then how of... an end-to-end flow looks like. So let's say that I am, you know, in this case, I'm coming in as Ana. I'm asking about international wire fees. Very simple question. Just a knowledge base. And then in a few seconds, it's gonna come back and give me the results from our knowledge base, and here you go. You see in here I'm surfacing information that obviously I wouldn't surface this to a, to a customer, but I'm doing it here for the purposes of the demo. But I'm calling out here the tools that, uh, are being

called, the, the data sources that are being called as well, and some of the details as well that make up the configuration of this agent when it comes to giving back this answer. If I were to then say, "Okay, look, I, I lost my card. This is my number. Is it blocked?" I'm putting my credit card number in clear text, right? So that is obviously not, not, not great. So you wanna make sure that even though that the user, you cannot control that. You cannot control what your uses- your users input in the system.

You wanna make sure that you catch that whenever you are instrumenting and whenever you are, uh, building your evals. So now we'll see in the trace itself that the credit card will be indeed redacted. If I come here to Langfuse and I open my trace, you'll see that now the question is, "My card number is," and then it landed in Langfuse completely redacted. That is because I have a masking function that automatically any sort of PII information that I have defined will be immediately masked and then is not persisted in

plain text in my system. So just to show you, um, in terms of what a, a ty- a typical interaction looks like, and I can see from the top here, this was the, the prompt that the user sent. But then as you all know, behind the scenes, there's a lot of stuff going on that we don't necessarily surface to our end users but are critical for us as the platform teams, as the developers, to understand how the agent is, is behaving, performing, and the quality of the outputs as well. I can see the entire tree

of all the steps that it took. You know, here the, the, the Anthropic LLM call that it made, the tools that are available. Here it called a list of tools, it listed the accounts, and, and so forth, and these are the outputs. But then more importantly, here on the top, I can see all the scores that I have defined for this particular use case. You see that they're, they're automatically calculated whenever there is an, a new interaction or a new trace being captured by the system. And I can s- I can

capture things e- you know, down from if the information that I gave to the customer is compliant with banking information, meaning that the customer's asking about the weather. I'm not gonna give that information. It's not compliant. Um, if there's any sort of failure modes, um, if there's any guardrails that we're, that we're blocking any, any answers that we're giving back to the customer. One example of that is, for instance, I can come... someone can, like, come on and ask- Can I put my savings in Bitcoin? Right?

So that is something that the bank itself shouldn't be providing information about that because that is, you know, that has some ramifications and the agent, you know, correctly does that. But at the same time, it is flagging it as investment advice, and this requires, you know, for the person to either talk to an advisor or, or something else, and we are using a score to do that. And if I go again to my trace, this is how I see it. I have the input guardrail here that tells

me that this is the primary risk is investment advice, and in the score here, I can see that the intent of the customer on this score was indeed investment advice, so that way I can very easily go out and filter for those in particular. But anyways, I wanted to show you kinda a simple flow of this. The last one before getting into the end-to-end on how you build a dataset and how you build, uh, an evaluation flow is that the fact that we can also support

multimodal. So let's say that instead of you having the customers writing via chatbot, they're calling your contact center. We can come here, we can say, "I wanna process this call," and this gonna do a speech-to-text, and then the text is going to be evaluated to f- to figure out if there's any violation on, on here. And I can come then, in here it says that the customer was asking about blocking his card, and I can indeed see the trace here, and I can even see the actual link to

the media that the customer was sent, uh, in terms of that audio, uh, or, or that call that the customer made, and also the transcript and what the assistant gave back. So that is in terms of the tracing, how can I capture all of those activities? Now, when I am trying to then think about, okay, how do I then put together or put in place a framework on how to evaluate these interactions that my customers are going to be sending through my system? Because as we all

know, customers are unpredictable. We might design a system or a solution with, uh, a, a g- specific intention, but our customers are always gonna find very clever ways to use the system in a different way. And we also d- cannot anticipate every single outcome, uh, specifically with LLM-powered solutions, that are going to happen when we, we, we release this into the wild for customers to use. So that is when we start relying on building our evaluation criteria.

And for, for us, the anchor of that is building datasets. So what is a dataset? A dataset is basically... And let's go to this one that is the, the generic one. A dataset is basically your set of questions and expected outputs that allow you to then measure the quality of the outputs of your LLM or AI solution. If you come from a software development background, the way that I like to think about this is that these are your test cases. And I don't think anybody here that has shipped software into

production will ship anything without passing it to QA, unless you merge into main or prod, which I'm pretty sure most of you have done it, but it's not a best practice. Um, I know I h- I know I have, but whatever. Um, but at the same time, this is the way that you can then have the evidence that you passed the agent through a series of steps or requirements that measure all the things that you wanna capture. And one thing to note here, that the expected output doesn't

necessarily need to be the answer that the, uh, the agent is going to give. It could also... You want to measure maybe the intent of the agent, you want to measure what the agent is supposed to do, you want to measure behavior. There are different ways that you can measure the quality of the output, and that is why you typically end up with different sets of datasets to measure a specific things or dif- or, or specific areas of quality in your agent fleet. You see here how I have one that is, like, basically

red teaming my agent and looking for prompt injection attacks and any sort of, uh, attack that will allow someone to come in and circumvent the guardrails that we have put in place to then provide information that they shouldn't get. The other one I have here is, like, very specifically for a use case that I want to know which, uh, c- uh, which cases have been disputed and where the dis- the, that dispute has not been solved. So the point here is that I can

come and c- and curate a specific datasets that are very specific to not only the use case, but also the technical requirements that I have for that agent. This is where I would say that, not unfortunately, because I think, I still think humans, we are very important. So this is where you still need your domain experts in the loop working with the development teams, defining what are these. There are two main ways that you can basically create your golden datasets. You either have a team...

And I've seen variations of both. You either have a team that already has a spreadsheet somewhere, these are all the things that we wanna test on my agent, and just upload it. And that's pretty easy. You can come here and say, "Okay, perfect. I wanna create a test dataset, and then I want to upload a CSV." And you can upload the CSV, and it will exactly bring up everything that you, you just had before. The other thing I'd like to mention here, everything that I'm showing, you can do it as well identically through our CLI or MCP, in our APIs. It's just that

I think it's ver- it's better for me to show it in a UI rather than telling Claude to do it end-to-end, make no mistakes. Okay? Um, so that is that on that. But then I could also come here and say that I want to add, um, you know, items into my dataset, for instance, that are already as part of my production traces. So let's say that you saw how I was already collecting a lot of activity from my application. I can come

here and say, "I want to add all of these into my dataset," and then select the dataset that I want to select. So this way you can kind of dog food, in a way, the dataset that you're going to use by looking at the production traces that your agent is producing as long as you have determined that, yes, these are good examples of what I want future outputs or traces to look like. Now, an important thing here to do is

a process that I always like to call out, which is error analysis, right? So error analysis is basically this idea that I want to look at my traces, and I think Annabell kind of talked about this, uh, very eloquently as well when she was on the other side as a user of, of Langfuse, that she had to like... A- and we were talking earlier today, she spent, like, days, like, literally looking at traces. Like, that was her job, like, looking at the trace, the structure, what was... what were the outputs and all of that.

And that is still a very important process because when you look at the trace, you f- you first start understanding how the agent is behaving, if it's taking all the steps that you need it to take, if it's calling all the tools that it needs to call, and if it's giving you the output that you want. That's, you know, very, very important. But then how do you simplify that process? Because obviously it is not feasible for me as a human to go out and look at thousands of traces once the system is in production.

So what we typically do is we wanna curate a sample that is representative of our production system on, of traces for a human expert to come look and give me feedback, right? It's the same as like, "Hey, I have all this activity. I'm just gonna take 10% or 20%," and I can say, "I just wanna focus on the ones that the customer is talking about," you know, disputes or credit card information, whatever filter that you want, you can do that. And then you can build

for them what we call an annotation queue, right? In this annotation queue, effectively what you're doing, you're telling a domain expert, in this case let's say me, that, "Hey, you are the risk expert. I need you to take a look at this output that the agent is giving us or is giving our clients and give me your input." And the thing is here, you wanna make it very, very simple for the domain expert because obviously they have a, their day job. They cannot spend just hours

looking at this. So the, the typical things that I recommend is you can give them, like, a couple scores that they can give you, you know, yes, no, true, false, a scale of one to 10, whatever that is, and then they can also come here and give you a comment. So let's say here the SME failure mode, let's just say for, for, for the sake of argument that in this case it's missing a citation, and then I'm gonna... I'm going to put a comment.

"Citation is not there. Needs to be included in all these cases." Right? You're going to go through this process. And, and look how, you know, it was simple for me. It took me, what, 10 seconds maybe. So let's say I have, I don't know, 100 of these and it takes me 30 seconds each. That is, what, 50 minutes? No. No, not even, right? I'm, I'm, I'm bad at math today. But anyways, you, you get the idea, right? So it's... You, you c- you can tell your domain expert, "Hey, give me

an hour of your time. I'm going to curate a bunch of activity for you. Just give me your feedback. Just give me a few comments." And now with Wispr Flow or something, they can just dictate it. They don't even need to type it, right? So make it very simple. The second thing that you're going to do with this, I'm going to mark this as completed. Now this human annotation lives with the trace as evidence. But then the second step that I want to do is take all this information, all this annotation that they gave us, and then actually use an

LLM to find all the failure modes or categories that are patterns across all of the comments, right? So it... There might be you're not asking the, the, the, the SME to invent categories. You're just telling them, "Give me your unfiltered raw commentary on these traces," and then you're gonna use an LLM to then figure out the categories because that's what they are great at. They are great at pattern matching. They are great at extracting

information from text. So that way you keep it simple for the domain expert and use LLMs for what, for what they are great at. Now, once you have all those categories, that is where we can then get into our failure modes, and those failure modes, if I come here and look at my experiments, that is what you're going to see. Whenever I run an evaluation, you see all of these things that I'm measuring against. Maybe let me bring it up here. So now after I

have my domain expert give me the, the, the, the inputs, I extracted what are the most common categories that appear in my corpus of data based on my domain expert. I can then define what are my vectors or dimensions of, you know, quality, let's just call it. We just call it scores, right? But I like to come up with fancy words to, to say things. But you see here how e- one of them was cites expert's expected source, correctness, formal,

formal register, meaning that, you know, in some languages you want to address your customers with a more formal tone, uh, if the language matches what the customer came in with, um, must include non-solicited and so forth. So now that I have that foundation of the categories that I'm going to measure on, the quality of my outputs, then I can decide, okay, cool, so now how am I gonna grade that, right? What type of score is that? What type of evaluator is that?

And for that, I can either do... come here and build my kinda library of evaluators, right? So that those as, as, you know, traces coming in, they're automatically calculating those scores for me. But at the same time, I can have the scores being calculated directly From the client application whenever something happens online in real time, so that then I can surface it back to the system. But basically, I want to have,

um, a, a way of me measuring the outputs of the agent across both LLM-as-a-judge scores or evaluators, but also deterministic evaluators. Another lesson that we've, that we've seen working with these clients is that the more that you can have a mix, um, a balanced mix that is, depending on the type of categories that you identify, between LLM-as-a-judge evaluators and deterministic evaluators, the better. Because what that's

gonna do is if you rely 100% just on LLM-as-a-judge, there's a non-zero chance that some of the outputs are going... are not going to be accurate because that's just the nature of the LLMs. But if you factor in some deterministic checks that are always going to be exactly the same for the same inputs, irrespective of what happens, then you can have a blended score or a blended assessment on the quality of your agent. So that's what we

typically would like to see. And you see here when I'm def- trying to define a new evaluator, I can come and say, "I want it to be a new LLM-as-a-judge evaluator," or, "I want it to be a code evaluator," and I can define all the details on as to how I'm going to evaluate that particular agent activity. So that's that. And then if I wanna have a code evaluator, you see this is just code. It's a function. It's code.

This is very simple. It's just a regex. This is doing a- an exact match. But this is going to give me the same output 100% of the time. This is not gonna be, well, 95% or 80%. It's always gonna be 100% the same output for the same input. So again, the lesson here, you wanna have a mix of both. So that's on the evaluation side. So now we've talked about, like Annabell said, and maybe let me go back here to the AI engineering loop. So we've talked about here

building the datasets. We've, we've touched on how we can then operationalize the measurement of the quality of those agents running experiments. And then maybe let me go back here. So basically an experiment is anchored on a dataset. That's my test cases. Those are my tests that I'm going to run against my agents. And now I'm going to say, "You know what? This agent is running today with this system prompt, this model, these tools. What

happens if I change the model? What happens if I make a change on the system prompt? What happens if I now all of a sudden I give it access to new tools to access new data?" So you wanna be able to kinda prototype that before, you know, promoting into production and just running on vibes, which is pretty much what all- we've all been doing, right? It's like, "Ah, yeah, it looks kinda good. Let me just push it into prod." But this is a, a systematic way of you doing that. And in a few minutes, I'll show

you how you can incorporate this as part of your CI/CD pipeline so you don't have to be in the loop. So that any time a change hits on your agent, then a CI job is gonna kick off to run the experiment against the dataset. It's going to evaluate your agent with a new configuration, and it's gonna give you a pass-fail gate effectively across all of the scores and evaluators that I mentioned before. So you have a consistent way on how you evaluate and measure the outputs of these agents. Make

sense? Okay. Perfect. So that's experiments. That's great. So we touched on experiments. We touched on datasets and human annotation to bring those humans and their expertise into the loop. So now how do I bring it all together, right? If I come here and I have my prompt, this, my system prompts that I'm versioning here in Langfuse. You can do it either here in Langfuse. You know, we don't, we don't have... It's okay. We don't, we don't even charge for this, so that's fine.

A lot of our customers, what they do is they have another version in system. They have a version in GitHub, GitLab, whatever that is. But you should look at this as your trigger for then kicking off the automation to do your eval completely automatically. So the way that this happens is I already have defined an automation here. And if you see the configuration, it's triggering on any prompt changes that I have here. And then the action is it's going to launch a CI job in

my GitHub, which is this guy right here. Hopefully, I don't-- I'm not gonna, I'm not getting the unicorn from, from GitHub. So it's this guy right here. So I have already my CI set it up, set up waiting for any changes in the config of my agent. Let me bring it closer here. And then I can come and see my prompt. Let's just see this one. I have five versions of the prompt. You see how the prompt has

evolved over time. Version one, it used to have this, and then version two, a slight change. Then version three, it was, y- you know, we tried something different. This is for development. Then version four, we added more details and also passing different parameters into the configuration of the agent. But what I can do now is I can say, "You know what? This is the production system prompt that I have right now. I wanna test again with version four because I saw some regressions." So I can come

here and do Promote into production. And again, I'm showing you this as a human, you know, like a caveman, like clicking through. All of this can be done via the API and the CLI, so you don't even have to do this by hand. You can just call the API and say, "Promote this prompt version with a production, uh, tag," and it will kick it off for you. I have the production tag. I'm gonna save and promo- promote to production, which is the new configuration. In a few seconds, we should have a new CI job that is going,

is going to launch here that then is going to kick off the experiment to run my evaluation run. Okay? There you go. Thank you, GitHub. Thank you so much. So perfect. So now I have this job that is running automatically. So any time that I push any changes in the config of my agent, it's going to hit the CI, it's going to run the eval gates to tell me if I'm passing or failing any of the evaluators that we have defined from before. And then with

this way, I can have the evidence and also the assurance if I can go into production, you know, safely with this new configuration. If it's actually the agent is degrading or improving in quality or maybe even violating any sort of guardrails that I have put in place. That's gonna run for a few minutes, but let me show you one that I ran earlier and to show you the result. I can come here and see what the output is, and it's telling me that this was the prompt that was sent, the version, the labels, and

then it's telling me that the gate passed, right? So perfect. And these are the evaluators that I measured my agent against. I have here the must include, the source recall, the language match, the correctness, and no unsolicited upsell. And I can also define minimums. I can say, "This is the minimum threshold I have for this specific score. If it doesn't clear that bar, it's not going to be considered as a pass, uh, to go into production." Now, obviously,

in this case, I'm not crazy enough to just say, "Okay, cool. You pass. Go merge it and push into pro- into main." I'm just letting it sit there. But admittedly, you could have a full end-to-end process that it will just create a PR for you, and it will just wait for someone to review the PR. Or maybe if you are, you know, like, like everybody's doing today, just pushing the PR into main straight up, uh, if it's clearing the bar. But at least you have the evidence here, right? So you're not pushing a

PR just running on vibes or just running on what you think that the agent is good at, at doing. You actually have the, the paper trail, so to speak, and the evidence on why you make that decision for this new configuration of the agent. Perfect. So now I want to show you one couple more things as an actual, like, use case, right? Because we've been talking a lot about the technical aspects of, of Langfuse,

how you can set up all the entire end to end. But I also understand that more and more, you know, organizations are focused a lot on tokenomics, right? I think twenty twenty-five, in beginning twenty twenty-six, everybody was token maxing. Now everybody is, like, token valuing or whatever they're calling it now. So now everybody's, like, trying to extract as much value as they can of their tokens. So now let me show you a couple of ways that you can

then show your bosses or your leaders or whoever, or maybe you want to build this for your business stakeholders, for them to feel a little bit more reassured that whatever agentic solution that you're building is actually bringing benefits to the business. This is not just like, oh, the cool thing that we're building now. I have here a dashboard that I'm trying to put together a few metrics that I can report on to the business based on the data that I've been collecting from my agent, right?

The first one is, and this is where you might argue with me in the logic behind this, but, you know, every organization is different. But what I'm trying to show here is that I've had fourteen hundred interactions of this agent over the last seven days that didn't need to go to a human agent, right? So that's great. So that means that I'm saving, you know, human capital from spending time to talking to customers. A-and,

and do the math then, how much has that cost me? It has cost me less than seven dollars, right, over the last seven days. If you take any contacts... I used to be a contact center employee back in the day, twenty years ago. You know what you're gonna pay that person, fifteen, twenty dollars an hour, right? So that person is not gonna handle fourteen hundred calls in one hour. So already you're showing at least some directional business benefit by having this solution in place, right? So that's the first

thing. The second thing is you could also start pinpointing what are the areas where your agent is tripping up, right? If you remember, we defined the categories or the failure modes, as we call them in the industry, where the agent is not giving me the results or the answers that I want. And these are the ones that I'm capturing in order of occurrence over the last seven days. You see here that now out of scope,

informal register, and dispute not resolved are, are my top three, and they've been creeping up over time. So now, instead of me having to go through thousands or hundreds of thousands of traces trying to figure out which one is the activity from the agent that is causing the most errors or the most trouble, I can come here and scope it down. And I know already that this failure mode is my top one, the out of scope, the informal register, and the dispute

not resolved. So now I can come here in my tracing and do a very simple filter and say, "Dispute not resolved." I can save it as a view, so I don't even need to every time filter it. You see how it's filtering up top for me. And even on the side, I can come here and say, "Dispute not resolved equals zero." So that shows all the instances where the customer had a dispute or was asking for a dispute of a product and it hasn't been

resolved by the agent. So it either had to go to a human agent or it just lingered and the session closed and nothing happened. But now I can then spend my time, instead of going through thousands, I can scope it down to maybe dozens or hundreds, and now I can do a, um, an investigation that is more targeted, and now I can also find where are the areas where my agent is actually, uh, tripping itself up. So that's, that's one. The second one is that if I want

to come here in the dashboards, I want to then be able to say, "Okay, what are the number of times where my customers are giving me either thumbs up, thumbs down?" I can ca- I can capture that as a score very easily. You see here every time I can come here, thumbs up, thumbs down, and I can just show it here in a dashboard, very easy to consume. Any business person, any user, any human, actually, is gonna be able to see this and say, "Okay, most of them are

thumbs up. That's cool. But what about the thumbs down? Let's focus on that, you know, eighteen percent of them. What's been happening?" And then the other one is around the security mix. Mo-- Uh, in here, good news, most of our interactions from the agent is not catching any sort of risk in the interaction that it's having with the customers. But we see here that in eighty-nine cases over the last seven days, it actually... the customer was asking for investment

advice, and the agent might have given or might have tried to give investment advice. So we can then go out and focus on those. And again, same thing, I can come here and say... Let me go back to my default view. And now I can say, instead of the numeric score, I can come and say the categorical scores add a condition, and then say, what is the intent or the security risk? And then the security risk, let's say, investment advice.

And now I can see what are the specific examples where investment advice was identified as a reason for being flagging that the, that the agent shouldn't engage in that with the customer. And you see here, even from the previous example, um, that I was showing you. So that's kinda what I wanted to show in terms of, first, the, the nature of going end-to-end on a solution that you're putting customer-facing first, obviously,

which has a lot of critical implications in terms of liability, you know, financial implications for, and reputational as well, because you don't want to be putting out information that is not accurate or is hallucinated in front of your customers. And how you can follow a systematic process so that then you can show evidence and validate that the outputs of these agents is actually following the guardrails and the controls that you have put in place. And with that, I would like to leave you

with Annabell again, so we can close, and then maybe we can take a couple of questions at the end. Thank you.

So what we see some companies trying to do or some teams trying to do is this. "Please fix my agent and make no mistakes." And there's probably a few things that we can recommend less than this. Even though your agent might be hooked up via MCP or CLI with your data in Langfuse or in whichever tool you might be using, um, this is something that we just see producing rather slop than anything, um, that is actually helping you. But of course, we're also very much, uh, in favor of working with your coding agents to

continuously improve your applications. Um, so we wanted to share this just as a picture, how you can maybe memorize it. So we have two of those axes here. We have one, uh, being the time invest, so a lot of time spent or very little time spent, and then also a low-quality system or a high-quality system. And you don't wanna be down there. If you fully automate it and just let your agent go for it and just be like, "Fix everything, do it for me,"

um, this is not gonna turn out well. You will end up with low quality, and yes, you don't have to do anything, but also your customers will not be super satisfied. At the same time, you also don't wanna go, go the full manual way anymore. You don't wanna instrument your app yourself anymore. This is something that can be automated. You don't wanna catch all the failure modes yourself and re-review all the data yourself. This can be automated. Um, but at the same time, for some things, you should get your taste in, you should get your domain experts in the room to build out the datasets, to curate what is actually, um,

what actually is the problem that your application should be solving and in which direction you want it to go. And for this, um, you want to land somewhere here. So where the ideal balance is between, like, um, you reviewing data, bringing in your taste, bringing in your domain expertise, but also working with your coding agent for all the things that you can automate, like instrumenting it, finding failure modes, running the evals, maybe ma-making also twenty experiments instead of only one, um, that you would come up with. And yeah, just, like, avoid getting down

there in the left, uh, down corner, but try to be the, the tasteful AI engineer that is bringing in the taste, making sure the, the evals are great, and then you can also more and more automate this loop over time. If we were you and we would have to get started tomorrow, uh, this is what we would be doing. We would install the Langfuse skill, which is a, yeah, it's an agent skill, so it's a set of Markdown files that is helping your coding agent to wrap its head around how to work with Langfuse and how to implement those best practices. It is hooked up with our

documentation, with all our integration pages. Um, it is hooked up with our academy, where we ingrain the best practices around datasets and how to instrument and how to write evaluators in all of this. So it's a, a great starting point, um, to do the things, uh, automatically, um, that you should be doing automatically, but also to maybe curate a set, um, of data that you should look manually at. Uh, you should then start to collect, um, also ground truth that you can evaluate your agent against, maybe from historical data, also from, uh, production or from

some samples that you're running with your domain experts. And if you're in a regulated space, uh, define your hard limits in a way to make sure these are the things that definitely shouldn't happen. For financial services, for example, it should never give investment advice. So make sure that you're ingraining this also in your evaluation set, um, that is ideally never happening, but at least you will know about it, um, when it's gonna happen. And we have a few resources for you. So the repository, uh, Doneyli showed you and used, um, is also available

online. Uh, you can check it out yourself. You can take everything that you want from it and also use it for your own use cases. It's of course fictional data and not, um, not something that actually, um, was taken from our customers. And I think also this session will be recorded, so you can take a look again. Uh, we then also recommend the Langfuse Academy, which is like a larger writeup of, uh, how to go about AI engineering, um, the whole steps, um, uh, described again. And we also have a, a self-serve workshop that is basically guiding you through all the

steps so you understood it once and then, um, yeah, can go ahead with your own application. There are some opportunities to bump into us over the next days. Uh, one is, um, Doneyli is giving a talk also on building deployment gates, uh, for AI agents on Wednesday. So check it out if you wanna understand how you can make sure, uh, that AI models that go to production are actually, um, clearing quality gates. Uh, we're also having a booth, a booth nine. Um, I haven't checked out where it is, but I'm sure you will find it. Um, you can meet the team there, but also you can get custom stickers.

Yeah.

So you take a picture of your face, and then it, uh, AI generates stickers of your face and Langfuse with, uh, maybe some jokes also included. Um, and you can take them home and sticker it on everything you have. And then finally, we also have a community run, so if you wanna step out the conference center and, uh, meet the team. We also have two of our founders here, um, and a few more team members, and, uh, we're all big runners. So, uh, we thought we just might invite everyone else as well to join us, um, for this. So yeah, please come for this one as well. And we're super happy to use the, uh, remaining

minutes for any kind of questions. I also have some non-custom stickers here. It's, uh, just a Langfuse sticker, but I think it's still pretty cool.

Yeah.

So if you want some, you can come up, uh, and get them. We're happy to take any questions, and ple- please meet us and, uh, don't be a stranger. I'm handing this to Doneyli.

Yeah. Well, the Langfuse per se has a hard dependency for their stores with ClickHouse. Um, that is just because, you know, in the evolution of the product, it started with Postgres, uh, but then Postgres couldn't handle the scale that we achieved in, like, billions of billions of events per day. That's when we, we moved it over to ClickHouse. So in terms of that data plane, yes, in ClickHouse. But maybe the follow-up for you, what is the, the need for you in terms of that data stores, because we also offer the ability for you to export the data. You can export it to

S3. You can use the A-

API.

Yeah, you can use the API, the CLI, or the MCP if you want to consume the data from Langfuse. So that is totally up to you. Yeah.

I wanna host it on Amazon. So like-

Yeah

... that's my question.

Yeah. You, you can host it 100% in Amazon. Um, it's-- you know, we have a self-hosted deployment as well. Um, you can either use our cloud offering that is fully SaaS, turnkey. You don't need to do anything. It runs on Amazon. But if you wanna run the infra yourself because you have, I don't know, some v- you know, EC2 instances lying around there you wanna use, we also have a deployment model for that, and it comes prepackaged with all the components.

Yeah, super important question. So the question was about inter-rater reliability, um, and also probably just consistency of LLM-as-a-judge evaluators to make sure it's actually catching, um, what you want it to catch. Um, that's actually also part of the Langfuse skill is handling this as well and helping you to, um... Uh, there's a term for it. Um, it fell out of my head, but it's i- in the end about the process of making sure the LLM is judging like a human would judge. And, uh, what you do basically is you judge a sample, um, ideally a representative sample yourself,

and then you run your LLM-as-a-judge, and you then optimize for the LLM-as-a-judge being coherent with what your human j- uh, human raters do. And ideally, you have like a few human raters, and that's aca- academic approach to it. So probably the practical approach to it is have one human label some data and then optimize against this ground truth. And then ideally, you just check it over time to make sure your set stays representative. And especially if you are using it for, um, running on production data, it's in the end, ideally you just start with having a f- a few too many

false positives. So you catch a little more than necessary, but don't miss anything, and then you just tune it down over time. Yeah.