AI Engineer World's Fair 2026
Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI
Read the talk
Let Your Agent Cook: Using Skills to Evaluate and Improve Your App
Ankur Duggal connects a coding agent to traces, evaluations and application code, then follows a flawed financial agent from confident answers to specific, testable repairs.
From a talk by Ankur Duggal
At a glance
Ideas worth remembering
Traces reveal the tool choices and intermediate work hidden behind a plausible final answer, making failures specific enough to investigate.
Test proposed repairs on the cases that originally failed. In the international-stock example, country normalization still failed until an outdated tool was updated; the later iteration reached about 90% success on the collected dataset.
Prompts and skills change behavior; evaluations detect whether that behavior meets the intended standard and whether it later regresses.
Review both the repair and its evaluator. The demonstrated code-based evaluator failed to set up, and the live application repair remained unfinished at the final check.
Choose judge models, sampling and evaluation criteria around the application’s failures, quality needs and operating budget.
A plausible answer can hide the wrong decision
Checking an agent’s input and output can get a project moving. Ankur Duggal of Arize AI calls this shipping on “vibes”: read the answer, decide whether it looks reasonable, and keep going. His opening audience poll separates that habit from having evaluations—and separates having evaluations from trusting their coverage. The difficult part is knowing which behaviors deserve a test before production exposes them.
Traces supply the missing execution history. In a conventional submit handler, the code makes the branches visible: validate the input, return an error if it fails, otherwise continue. An agent may enter through something as small as agent.run(query), while the consequential choices happen afterward. It can call tools, retrieve documents or perform analysis. Reading the entry point does not tell you which path a particular request took; tracing records that path through model calls and tool activity.
Consider a customer asking where their refund is. The agent calls tool.getOrderStatus and replies that the refund was issued. That answer may repeat a true order status while leaving the customer’s question unanswered: why has the money not arrived? The useful thing to inspect in the trace is the tool choice. It exposes a mismatch between the user’s intent and the information the agent sought, giving the investigation a more specific starting point than “the answer was bad.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the coding agent the evidence behind the failure
Finding the wrong tool call still leaves several possible repairs. The prompt may need clearer instructions. A guardrail may need to constrain behavior. The tool itself may be broken. Collecting a sample of similar failures creates a dataset on which those explanations can be tested. An evaluator then checks the intended behavior—for example, whether a guardrail actually prevents information from leaking—before the changed application returns to production.
Duggal’s proposal is to put the evidence and the means of repair in the same coding-agent workspace. Each input contributes something different:
- Agent traces reveal the model calls, retrievals and tools used for a request.
- Evaluations identify behaviors that failed an existing quality check.
- Human annotations add judgments about what should have happened.
- APM traces help distinguish application behavior from infrastructure problems, such as a service being down.
- Application code gives the coding agent somewhere to implement a change once it identifies the relevant failure.
The interface is a CLI plus skills that teach the coding agent how to use it. The available operations span instrumentation, downloading traces, creating datasets, running experiments and evaluating results. Claude Code is the client in this demonstration, but the proposed arrangement also accommodates Cursor or Codex. The useful capability is access to runtime evidence alongside the repository.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with an investment answer that knows too little
The live example is a financial agent that answered an investment question with a recommendation to invest. Duggal objects to that behavior because neither he nor the agent is registered to give financial advice. Audience questions expose additional problems: the agent has not asked enough follow-up questions, does not know the user’s financial background, and has no established ground truth for validating the answer. These are separate defects; a more detailed response would not, by itself, resolve all of them.
Setup has three parts: install the CLI, create a profile pointing to the Arize data, and install the skills for the coding agent. Duggal uses Claude Code’s marketplace for the last step. The profile connects the investigation to stored traces, evaluations and annotations. He also explicitly discourages copying his demo’s permission-skipping configuration, which he uses to avoid repeated approval prompts onstage.
The investigation request asks Claude to retrieve the financial agent’s traces, inspect existing evaluations and annotations, perform its own analysis, and suggest evaluations that would catch the discovered problems in the future. That leaves room for several kinds of repair: asking for more context, changing prompts, or adding a guardrail that limits unwarranted financial advice. It does not assume that the audience’s first hypothesis is the root cause.
While that investigation runs, a similar question about investing in Nvidia provides a manual tracing example. An earlier run exposes the query, final output, supervisor agent, financial data agent, model calls and tool calls. The trace answers a practical question: what work actually produced this recommendation? It lets a reviewer move beneath the final prose and inspect the participating agents and their actions.
The skills operate through the CLI underneath them. Some operations download traces or datasets into the local system; others make API requests or create evaluations on the platform. This explains how a local coding session can both inspect production evidence and configure ongoing monitoring without requiring the developer to perform each platform operation manually.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate bad summaries from silent tool failures
The investigation returns several distinct findings. Existing evaluations flag about 44% of the analyzed agent interactions as producing a bad summary, alongside fabricated answers and redundant tool calls. That percentage describes the interactions assessed by the existing summary evaluator; it is not a universal measure of the application’s accuracy. The findings also carry different costs: an unsupported answer damages reliability, while repeated tool calls consume time and money even when the final answer happens to be useful.
Claude’s additional trace analysis identifies a more specific failure mechanism: a tool can appear to succeed while returning no actual response, and the financial agent can fill the gap with a hallucinated answer instead of disclosing the failure. This connects two layers of behavior. Checking that a call reported success is insufficient if the returned content is unusable; checking the final answer is also necessary because the agent may conceal the missing evidence with fluent prose.
Duggal chooses two targets for the next repair:
- Silent tool failure: build an evaluation that detects the apparent-success-without-a-useful-response case, and change the agent’s instructions so it does not invent an answer when a tool fails.
- Numeric groundedness: make numerical claims refer back to actual supporting information rather than allowing the agent to manufacture numbers.
The request includes a concrete test condition: use the spans that originally failed as the basis for a dataset, then run experiments against those cases. Dataset, evaluator and experiment skills connect the proposed edits to an observable result. The experiment should establish whether the changed behavior survives the input that exposed the defect, rather than merely whether Claude completed an edit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Country context cannot repair an outdated tool
While the live repair continues, an earlier experiment shows why testing intermediate fixes matters. Users had asked about stocks outside the United States, and the displayed failure cases concerned South Korea. Claude collected the failing cases into a dataset so the same requests could be replayed after changes. The goal was to support questions across markets, but the demonstrated evidence concerns those collected cases.
The buggy baseline passed 0% of that dataset. Claude’s first correction normalized the query by adding the country name. It still failed. The missing country context was only part of the problem: the underlying tool was outdated and needed an update to its stable version. A subsequent iteration addressed that tool problem and reached about a 90% success rate.
What changed between attempts? The comparison separates an input correction from a tool correction:
| Attempt | Change | Reported result |
|---|---|---|
| Baseline | Original buggy behavior | 0% passing |
| First fix | Add country context during normalization | Still failed |
| Later iteration | Update the outdated tool to its stable version | About 90% success |
The failed middle attempt is the useful lesson. Improving the information supplied to a tool cannot make an unsuitable tool implementation work. The final result is substantial progress on this dataset, with remaining failures still unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The experiment and its judge can run in different places
An experiment does not have to execute inside Arize. In this demonstration, the application experiment runs on Duggal’s local machine; a cloud environment is another option. Evaluations can then run on Arize. Separating those locations lets the application execute where its code and dependencies are available while the platform performs the quality checks.
Evaluator creation is itself fallible. Claude attempted both a code-based evaluator and an LLM-based evaluator, but failed to set up the code-based one successfully. The final comparison used the LLM judge it created on the platform. Two evaluator entries therefore did not mean two working, independent checks.
Judge-model selection introduces a cost and quality decision. The platform supports configurable providers, including local open-source models. Some teams prefer cheaper judges; others use a stronger model for evaluation than for the production agent. The production model and the evaluator need not be the same, because generating an answer and assessing it serve different purposes.
A smaller custom repair agent could use the same approach. The added value of the skills is access to evidence beyond the code: execution paths, evaluations, annotations, APM logs and potentially issue tickets. Duggal also identifies Phoenix as an open-source option for connecting this kind of information to a coding agent, while describing easier online-evaluation setup and additional dashboarding as enterprise conveniences.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A good-summary skill still needs a summary evaluator
The earlier “bad summary” finding came from a previously configured evaluator, rather than an unexplained judgment invented during the investigation. Its inputs include the information gathered across the agent’s work and the context supplied to the summary agent. Its checks ask whether the summary references the appropriate points and answers the user’s question. Inspecting that evaluator answers what the label means: summary quality is being judged against the available context and the task.
Why not install a skill that produces good summaries in the first place? That is a useful application change: improve prompts, skills and coding workflows so the agent performs better. Evaluation has a separate job. A skill that works today may produce worse results after a model regression or a change in how users phrase questions. Duggal’s hypothetical two-month example illustrates why the quality check remains useful after the initial repair: it watches for the return of a behavior the team already knows is harmful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the repair focused—and account for the judges
An audience member raises a familiar coding-agent failure: it spends its effort adding linting instead of solving the actual problem. Runtime evidence gives the task a more specific focus. A trace showing the wrong tool call points toward tool selection or tool behavior; lint cleanup does not explain that failure. Duggal reports seeing less of this distraction with the skills, while retaining human review as the check that the proposed change addresses the real defect.
Continuous LLM evaluation also adds operating cost. Evaluation traces expose what the judges did and include cost information. Sampling controls let a team evaluate a selected fraction of traces or focus checks on particular areas instead of judging every call. That makes evaluation coverage a deliberate budget decision, configurable through either the platform or the coding agent.
The proposed automation can gather failures, build datasets, edit the application and test the result. The human still needs to inspect whether those changes improve the agent rather than merely making the repository look tidier. At the final live check, the current repair is still running after about 14 minutes. The recording therefore demonstrates its investigation and repair request, but does not establish a completed fix for the live numeric-grounding and silent-failure problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose evaluations for the failures your application actually has
The closing questions clarify who supplied the quality standard. Duggal and his team had already defined the evaluations behind the initial results table, covering broad concerns such as summaries, agent health and tool-call correctness. Claude then used the actual traces to identify more specific problems occurring in the application. Those are complementary contributions: existing checks organize known risks, while trace analysis can propose additional checks for failures the original suite missed.
There is no universal judge-model recommendation offered here. A benchmark can indicate general model capability, but selecting an evaluator requires analysis of the application’s own task and domain. Finance may need a different choice from a workflow combining law and finance. Duggal’s closing warning is pointed: “out-of-the-box evals” yield “out-of-the-box results.” The practical next step is to use the failures users actually encounter to decide what the evaluator must recognize—and whether its judgments are useful enough to guide the next repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explores a complementary mechanism: using execution traces and diagnostic feedback to search over better prompts, agent programs, repository skills and evaluators.
Read the complete timestamped transcript
- 0:12
Good afternoon, everyone. My name's Ankur. Uh, I'm gonna be teaching you how to make your agent cook. Again, not food related. I don't know. Uh, but yeah, we're gonna get into it. Um, so what I mean by that is... Actually, before we get into that, let's go ahead and start with a quick question and a poll with everyone. Um, I'm sure a lot of you are building agents or have even shipped agents. I'm just curious, how many of you currently are shipping based off vibes? And when I say vibes, it's not a bad thing. I mean, like, hey, you're checking your answers, you're looking at it, you're making sure that it's
- 0:42
potentially doing the right thing just based on the input and output, and you're making sure that it's correct or not. How many of y'all are doing that? Okay. Pretty good, reasonable amount of you. And then how many of you guys actually have, like, set eval set up? Like, you know, like, oh, you have LLM as a judge or code-based evals or some sort of eval set up.
- 1:03
Okay, a little less. Not too much less, though. That's pretty good, actually. And then how many of you that actually have your eval set up trust it enough that, you know, you're good enough, you have a full coverage of your evals, and you're ready to-- you're continuously ready to keep going and moving forward? Wow, that's, that's pretty impressive. That was a solid hand up. Which is kinda what I expected. I mean, it's very hard to really trust your evals because they're hard to build. You don't really know what you're looking for. It's not a deterministic use case, and it's very hard to determine
- 1:33
what we need to actually evaluate to make sure our agents are working properly. And that's kinda one of the biggest issues, is a lot of people are gonna start off in a very good spot, which is based off vibes. It gets you going, it makes sure that your agent's doing what you expect it to. But realistically, you can't really ship based on this feeling of, "Yeah, it's correct," because when you're in production, you're not gonna know where things are broken, what's going wrong, unless you have actual eval set up to find out where things are breaking and how you can actually directly fix it.
- 2:02
So before we get into evals per se, what I wanna kinda take is a whole step back and talk about how you can, like, what your evals would be on top of, and that would be on top of traces. And why we need traces is if you've ever used traditional software engineering, you have something that you do c- how-- call traces. It's for APM monitoring. It's making sure your systems are reliable and are working. But realistically, what you ne- you can actually go ahead and debug those, and it's very easy to do because you have a
- 2:32
very deterministic flow, right? You can even see right here, this is very easy to follow, and this is more or less what, like, code looks like. You have a handle submit method. You check if it's valid. If it doesn't, it errors out. If it is, then it goes down the next list and continues down that process. So it's very easy to figure out where things are break- are breaking based on a, just based on that sort of, uh, path history. But an agent, unfortunately, with your code, really all you have is that one line of code, which is just gonna be agent.run and then the query. And
- 3:02
ultimately, what it's gonna do is kind of up to it. Will it call a tool? Will it go ahead and do some analysis? Will it do some internal thinking? What is its next best step to do and its actual best course of action? We don't know until we actually see the traces. You won't know what your agent's doing. So that trace kinda helps you define that. And so as a kind of a decent example as to what a trace would look like would be this. You have your agent call. You go ahead and see the LLM
- 3:32
calls. You have your documents you're retrieving, your tools that you're retrieving. You basically have every step along your journey that you would see of what your agent's doing. And here you can even go ahead and see, hey, we have, for example, this tool.getOrderStatus. It's the wrong tool it called. The user asked us, "Hey, I was supposed to get a refund here. Where's my refund?" If we respond back, "Hey, your refund was issued," and they know that that was their status based on their order status, that's not what they were looking for. They
- 4:02
were looking to understand why hasn't their refund arrived. So we need to potentially call different tools, do different actions, and there's so many different varieties, even just knowing that there's an issue here. But we'll dive into that in a little bit. So when you actually go ahead and understand, okay, we have a trace, now I wanna go ahead and debug what's going wrong with my agent, we're gonna start with where-- First, you're gonna identify those issues, right? Those production failures, like we just did in that last slide. We found out that tool was wrong. But what's your
- 4:32
next step? You wanna go ahead and find out, okay, not only where did it go wrong, can I get a nice sample size of that sort of information that I can then use for experimentation and iteration to improve the agent when I actually wanna go back and test against those cases? Then, what actually was the issue, right? Sure, the tool call was wrong, but should the LLM prompt change? Should we add in guardrails? Should we change up the tool? Was the tool just doing the wrong things, right? There's so many different areas that we need to test and go against,
- 5:02
and so we can go and test against those, build out evaluations to continuously monitor those use cases. So for example, if you find out, hey, we need to make sure that we set up a guardrail here. How are we gonna do that? We wanna-- And then we wanna make sure that guardrail's doing its job, so we set up a evaluator on top of that guardrail, making sure, is that guardrail doing its job by prohibiting certain information from leaking? Finally, you actually make those changes, run that experiment that I was talking about in the
- 5:32
beginning, and finally ship it to production, and do that whole loop over and over again continuously. Now, this is your standard go-to process for actually improving your agent. It's pretty long, pretty manual, and pretty tedious. It does, it does work, but we wanna kind of iterate and improve that. How can we do that? We wanna make sure that we're not spending hours, days finding the issue, finding the trace, finding all these different issue points, coming up with a
- 6:02
potential solution, going through that full eight-step process manually. We wanna pull all of that together and bring it into minutes. The best way to do that would be right here. We're gonna pull in all the data from all our different sources into one specific area. We're gonna start with, you know, pulling our, our ap-- our Arize traces, so you can get all the information of what your agent did. You're gonna pull in your evaluations to see where things actually went wrong. You're gonna bring in your annotations from any
- 6:31
humans that could theoretically tell you a little bit more information as to, you know, where it could go right and how they can improve it. You bring in your actual APM traces, which could be, hey, is the system up or down? You bring in your code base so you can understand, all right, based on this specific area where it failed the tool call, what changes can I make in my code base so that we're ca-- we're going down the right pathways, we're calling the right tools, we're fixing our tools that are broken?
- 7:02
And so you'll bring in all of these into one final space, which is ca-- which is our coding agent. Whether you use Claude Code, Cursor, Codex, it doesn't matter. All of those, you can connect all that data into, and it can automatically do all of that work for you. So I know this may sound really complicated, and I promise this also probably doesn't help to make it look simpler, but it is very simple to get it set up. It's gonna be just a couple steps of just, you know, setting up your CLI and skills, and you're ready to go.
- 7:32
You get access to a bunch of informati-- a bunch of different tools that your agent can then do, which could be, for example, here, which are like, hey, you can get your code instrumented. What that means is you can get all the traces from your agent into whatever platform you choose, like Arize. You could go ahead and pull down those traces, build out data sets, run experiments, run evaluations. You have the full gamut of the experience all from within your coding agent.
- 8:01
So what I'm gonna do here is we're gonna go ahead and look at a specific issue, and I want a little bit of kind of, uh, voices from this side of the room, uh, where we can get some idea of like, "Hey, what do you guys think could be the actual issue?" So I'll give you a little bit of background. This agent right here is a simple financial agent. I've asked it, "Should I invest in this company?" I think I chose Tata Motors. Um, anyways. And ultimately, it gave me a
- 8:30
response which said, "Yes, you should." I don't think it's directly in the screenshot, but I promise you it told me that. Um, and ultimately, that's not really what we wanna do because I'm not registered to give any financial advice, nor is this agent, and I don't wanna get in legal trouble. If you guys can help me figure out maybe, hey, what do you guys think could be the issue in this agent? So what she said was, "Hey, maybe the agent needs to ask a lot more questions back to get a little bit more information before it can give a solid, concrete answer." That's a
- 9:00
very good guess. Anyone else have any ideas? Okay. And how do you suggest we potentially improve that? By the way, he said it should ask, again, pretty similar, it should ask more questions before making that recommendation. But how do you suggest maybe we can go ahead and do that?
- 9:14
More context, more asking questions, more details, context.
- 9:17
Okay. So maybe some prompt changes to add in some context to get more context. One more.
- 9:24
Where's validation?
- 9:26
Sorry?
- 9:26
Where's validation?
- 9:27
Where's validation? Yeah, that's true. We don't have any sort of ground truth to compare against. We don't know if any of this is correct, so there's no validation that any of the answers are even correct. You're right. There could be so many different issues, and truthfully-- Oh, yeah, go ahead.
- 9:40
Quick question. Does the agent know the user's financial background before giving this answer?
- 9:46
That's a great question. In this case, no, it does not. And yeah, it should definitely get a lot more information into that. Uh, he asked, by the way, "Should the user-- Uh, does the, does the agent know any information about the user's financial background?" Um, and yeah, it definitely does not, and there's a lot of areas that this agent could be fixed, and there's genuinely, genuinely not one correct answer, right? But-- And I genuinely don't even know what the right answer is to this because this agent is still messed up right now, and we're gonna go ahead and try to fix it today in this demo.
- 10:15
Um, and so what we're gonna do is we're gonna go ahead and set up the CLI and skills, get that set up so we can actually get our coding agent to go ahead and understand what's going wrong with our, uh, to go ahead and pull down the traces, understand what's going wrong with our agent, look at the evaluations we already have run, look at the annotations that exist, maybe even do some of its own analysis, and potentially come back with some future suggestions, just like many of you have done today, for how to actually improve this agent so
- 10:45
that, hey, maybe we could have it ask more questions. Maybe we could add a guardrail so it only responds to certain questions and doesn't give financial advice where it's not warranted. So there's lots of different ways that it can come to do those fixes. To get it started, what I'm gonna do is I'm gonna go ahead and pull open... Let's see. Do I have it here? Yes. Let's go ahead and pull open a new session. Um, okay, cool. I'm in the right space. All right.
- 11:16
So generally, what you'd wanna do whenever you wanna set up the CLI and skills, super easy, would just be a couple steps to get that set up. Um, I apologize. Oh, here we go. I know what I'm doing, guys. Uh, here we go. So we're gonna start by setting up the ski-- CLI. You would just go ahead and copy this one line of code to get that set up. So I'd go into, back into my terminal and install that. Of course, I already had that installed right before this, but just goes, just very simple and quick to set up. Then what you'd do is you would create your
- 11:46
actual profile, which would help you connect it to wherever your data is stored. In this case, it would be Arize, and you create that profile so it can go ahead and pull in those traces and visual-- and get that sort of information from those evals and annotations. Once you have that profile created, you'd go ahead and set up your skills. You could do this in a multitude of ways, depending on your coding agent. For today, we're using Claude Code, so I use the marketplace to set it up, but it's pretty easy to set up however you want.
- 12:17
From here, let's go ahead and we're just gonna ask it-- And again, uh, just as a warning, I am using dangerously skip permissions. Please don't do this. This is not actually good practice. This is just for a demo, so I don't have to keep pressing yes. Um, but we're gonna go ahead and talk to Claude and have it go ahead and take a look at these traces. So I'm gonna go ahead and see if I can find the prompt I asked earlier, which was just use this skill and pull down the traces from that
- 12:47
specific agent, find out what issues exist that we can determine from any-- Like I said earlier, right? What-- An-any issues that exist from any of the evals, any of the annotations that we've done. Do your own analysis and tell me what evals maybe I can build out in the future to make sure that I don't run into any of these issues again. And so this is gonna take a couple minutes. Um, in the meantime, I'm gonna go ahead and first go ahe-- and show you kind of how those traces look and how you can actually start triaging it if you want to do it
- 13:17
manually a little bit. But while this is running in the background, let's go back into the actual pl-- uh, run into the actual agent that has those sort of queries that we've asked. So we'll go ahead and ask it a pretty similar question. Um, you know, tell me if I should invest into, let's say, Nvidia this time. It's a little bit more of a well-known stock. And we're gonna get that, and it's gonna try to tell us some sort of information. And as that data comes in,
- 13:47
looks like I've run this earlier, so-- And you can see it's coming in. Um, but from an earlier run, we can kinda see what it's done already, where you can see that query we asked, you can see the output that it'll finally generate, which we'll see how similar it is in a minute from this previous run, um, and you can see all the steps that actually happened in your agent, right? So you can understand, hey, we had a supervisor agent, a financial data agent. You can see the LLM calls. You can see all your different tool calls. You get a lot of information right away that you can start using, just
- 14:17
like we'd done in the initial slide to see where those issues could lie. Now, let's see. Did Claude finish up? Looks like it's still running. It's doing its thing. We'll let it cook, you know. It's kind of the point of this. Um, while it's cooking, anyone have any questions as to like, you know, how you can set up any sort of formative evals, any sort of traces, anything along those lines anyone wanted to learn a little bit more about?
- 14:45
Yeah.
- 14:47
Uh, can you share a little bit more about how the traces works in the system? Like, are we posting it up on you guys? Is it driving from JSON logs?
- 14:57
Great question. Yeah. So you would go ahe-- So what it does, it actually goes ahead and downloads a lot of those traces, um, into the-- into your local system. So what it'll do is-- Actually, let me go ahead and show you what it-- how it actually goes ahead and accesses the system itself. So when you give it access to these skills, what you're actually giving it access to is the CLI underneath. And what I mean by that is you're gonna have all these different CLIs that it can integrate with. So you can go ahead and set up and find the datasets. You can
- 15:27
pull down datasets. You can have it interact with the-- essentially interact with the UI, where it can set up evaluations on the platform on your behalf. So you can monitor, continuously monitor when your agents are failing, build out datasets, run experiments. It can basically do anything that you, that you would be able to do in the platform. Sometimes it'll download, sometimes it'll make API requests, but that's generally the, uh, idea here. Great question.
- 15:55
Any other questions in regards to CLI, tracing?
- 16:02
Okay, let's go ahead and see what Claude's done. Oh, still crunching away. Um, so as we can see here, let's, let's walk through what Claude's done so far. So we can see that we've gone ahead and gotten the spans. It's digging into the evaluations to figure out where those issues are. Um, it looks like it's not quite-- There's-- definitely found one very big issue with some numeric groundedness. I know someone mentioned that earlier as a potential issue or something related to that. Um, we're also seeing some
- 16:32
tool calls directly fail. Um, and let's see. We also see... Yeah, so we're getting a lot of different errors. Oh, looks like it just finished up. And so you can see here, we get a full list as to like, "Hey, what's going wrong with our agent and what we can do to kinda fix it?" So while it's finishing up that, you can see here, one of the biggest a-alarming things right out of the gate is about forty-four percent of our agent interactions are resulting in a bad summary.
- 17:03
That's really bad. I did not know that. So we definitely need to fix that one. Another big issue is-- Oh, did not mean to scroll there. Another big issue is, hey, we're also just getting a lot of fabricated answers, which is just as bad as the s-- the summarization being wrong. So it sounds like it may be related, it may not be. We can kinda debug that. We're also seeing a relatively high rate of us just redundantly calling tools. It's
- 17:33
a waste of our time, our resources, and our money. So clearly we're doing a lot of bad things that we need to go ahead and fix, and that's just from the evals we already have set up. From the e-- from Claude's own analysis, it's able to even go ahead and figure out, hey, there's other issues that are coming along. We're getting silent tool failures. It's telling us it succeeded, but ultimately it's not giving us any actual responses. We're getting the agent hallucinates when tool fails instead of disclosing it. So again, related,
- 18:03
but hey, that means maybe we need to not just do, uh, you know, some evals built on top of this to understand when the tool is failing, but we also need to make some prompt-based changes to make sure that the t-- the agent doesn't hallucinate here. And you have a multitude of, like, different things that it'll go ahead and tell you, as well as the evals it can build out. So what we could do is we can actually go ahead and tell it Hey, let's go ahead and build number one, which was the tool sex, uh, success silent failure eval, and fix numeric
- 18:33
groundedness so that it'll make sure that whenever it does respond, it's always grounded to some sort of actual reference point, and it's not making up these numbers. But we're also going to head and make sure you test this against a dataset, uh, against the spans that caused the failure initially using... And then we're gonna go ahead and use another Arize skill, which would be here, the Arize, uh,
- 19:03
dataset skill, the Arize... Oh, there we go. Arize, um, evaluator skill, as well as the Arize experiment skill. So here it's gonna be able to-- Let me just go ahead and run, let this run and build out evals. All right. So again, I'm being pretty vague here, but essentially I'm just telling it, "Go ahead and make the changes that you recommended, as well as go ahead and run these experiments for us." So now we can actually compare against
- 19:34
where it failed to make sure it passes against that, right? And so in this scenario, we can see, okay, it's gonna go through and take its time. Um, I'm gonna go ahead and show you a previous run because, again, this will take some time, and while we wanna let it cook, I don't wanna have you guys staring at a screen where Claude's just discombobulating or whatever it does in its free time. Um, so let's go ahead and look at where I've actually gone ahead and asked Alex, or, uh, sorry, Claude Code to go ahead and answer some of our, uh,
- 20:03
previous issue that I ran into. And one of those was related to, hey, I've had some times where I'm trying to get some answers about, uh, stocks that aren't necessarily from the United States. And so we're having errors where we're talking about, like, South Korea or... Looks like they're all South Korea-based. But let's assume there's other answers here, maybe Japan or China and whatever, right? And you wanna see and you wanna be able to understand, hey, I wanna be able to compare stocks
- 20:33
across different markets, and clearly people are asking that, but we don't support it. And so what Claude did, it went ahead, found those areas where that issue existed. Like I said, it looks like it was all South Korea here. But it found those areas, it built out a dataset for me, which I can then go ahead and test against to make sure it works, and then it went ahead and ran experiments. And it made a couple changes. So it started with the initial one, which was the baseline, which was buggy, which as we expected, would
- 21:03
result in 0%, uh, passing rate, which is what happened when we initially got the actual production-based traces. It also made a quick fix where it normalized it by adding in that, uh, country name. It didn't work. Why? Because the tool I was calling was also outdated and I needed to update to the stable version. So then it made a third iteration, and you can see that on this chart here as well, where we were able to see, hey, the first iteration didn't work.
- 21:33
It went back, made a fix, and finally got us to the final iteration, where we're getting about a ninety percent success rate. Clearly, still not a hundred percent. We still need to make some changes. But hey, we're doing a lot better than we were earlier. So this is, like, a great way to get you going so you can actually start from, "Hey, I know there's an issue," to, "Here's a fix." Now, I know we have quite a bit more time. I wanna check in. How is everyone doing? How do you guys feel? Do you guys have any questions
- 22:03
about how you can go ahead and get this started? Yeah.
- 22:06
I'm curious how iterations really happen. Like, Claude makes the code changes, then push for an experiment. Where did the experiment run? Where did the traces go? Where, how many runs, experiments? How, how, how actually it works.
- 22:23
Great question. So his question was, how do-- where do the experiments actually run? Where do the evaluations run? Where does everything run? Um, so you do have a little bit of customization there. In this scenario, uh, what I did is I had the experiment run on my local machine, which is, I think, what will be the case most of the time. And then what you'd-- what I did afterwards was I went... Or you can host it on a cloud or wherever you'd like to experiment on, and then you can go ahead and actually run the evaluations on Arize itself.
- 22:53
So in this case, I built out an LLM as a judge, which would then go ahead and... Let's see if I can see the evaluator here itself. So you can see we built out two different evaluators, a code-based and an LLM-based. I actually had Claude go ahead and do this. It built it out for me in the platform, and this was the final LLM as a judge because it actually kinda failed in getting the code-based eval to set up. So that's why we have two. And so you can see the LLM as a judge. It went ahead and built it out for us with this prompt and got it going so that we can
- 23:23
actually compare, hey, did the agent work properly?
- 23:28
Any other questions? That was a really good question, by the way. Yeah.
- 23:31
So do you use certain Claude models out of the box for LLM as a judge, or do you also like to configure them?
- 23:40
Very, very good question. Um, so evals can be set up however you like. We have, of course, cu- people can, people can definitely use whatever sort of models they want. No one wants to be limited to just Claude or just OpenAI, especially in today's modern age, where you can see AI models literally ship every other day, and then you wanna be like, "Okay, I wanna use that one now." Right? And also, some people have a preference for a slightly cheaper model. Some people have a preference for making sure their evals are top of the line because
- 24:10
ultimately they know their agent's only gonna be as good as their evals. So they use the highest end models for their evals, even if they're not necessarily using those in production. So it kinda varies. You have full customizable control over that. And so what you can see here, for example, you have a full list of whatever sort of model provider you wanna use, and you can connect those, even if you want to use any sort of local, um, open source models as well.
- 24:35
Yeah.
- 24:37
Good question. So her question was, if we have-- uh, if I spin up a smaller agent to go ahead and make some of these bug fixes, how does this compare to that? Realistically, you don't have to use Claude Code, Codex. You can use whatever sort of agent you want. The benefit of what we've provided is a skill that you're able to go ahead and pull down the information from your traces to make those fixes. So you're not making those fixes just based on, "Hey, I know this is wrong, and the output's incorrect." You're
- 25:07
making those fixes based on, "Hey, I know this is wrong. Here's where it failed. Here's why it failed. Here's how we can go ahead and fix it." Because you have not just the information of your code base, you have the information of any of your other APM tickets, whatever, or APM logs, any of your other tickets that exist in your Jira systems, et cetera, as well as your actual Arize traces, which would be like, "Here's the pathway the agent took. Here's where it messed up," your evals, your annotations, et cetera.
- 25:37
Any other questions? Yes.
- 25:38
Are these features also available in Phoenix?
- 25:41
Great question. Yes. Uh, we do have... So he asked, is-- are these features available in Phoenix? Phoenix is Arize's open source version. We do have, I believe, skills and MCP as well in Phoenix, which you can go ahead and hook up to your agent, and it can go ahead and pull down the same sort of information. Of course, there are gonna be some nice-to-haves in the Enterprise edition where you have access to these online evals in a certain way that are much easier to format and set up. You have a little bit more dashboarding and a few other features. But
- 26:11
yes, the, the skills in CLI, you can connect to your coding agent. Yes.
- 26:17
Uh, in your presentation that you said, the agent said that that's like a bad summary. So how would you classify something as, like, a bad summary? How does the agent know that's a good summary or a bad summary?
- 26:30
Very good question. So her question was, in what I showed earlier, there was a eval-- or there was a response from Claude that said, "Hey, there was a bad summary." How do we know what is a bad summary or a good summary? In that case, what I actually had set up, it was actually based on one of the evals that I had previously set up. And so we can actually go ahead and look at that evaluator here. We can go ahead and look at, let's see, some...
- 26:58
I actually don't remember which one, but I'm gonna go ahead and pull open this one and pr- hope this is the right one. Um, so as you can see here, this is basically the summary eval that I'd created. It's basically taking a look at the information we've gathered throughout our entire agent, all the sort of context we've given to that summary agent, and making sure that it's referencing the right points, it's answering the question accordingly, et cetera. Um, and so that's how we've gotten to determine whether or not it was a good summary or not.
- 27:28
What if we just, like, set up a skill in the first place to give us a good summary?
- 27:33
Uh, what if we set up a skill in the first place to see if it can do a good summary?
- 27:38
Just like, give it, give it, give us a good summary all the time. Like, what if just, like, if we set up a skill instead, instead of, like, running the evaluators?
- 27:48
Yeah, great question. So she's asking, why can't we just set up skills that, you know, ensure that it gives a good summary? And yes, absolutely. That is what you wanna do. You wanna make sure that you update your prompts, update your skills, update your coding workflows so that your agents will give the best summaries. The evals are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat, right? And so you set up that skill, great. It works for the next two months. Two months later, because there's a model regression,
- 28:18
because people start asking questions in a different way, you end up getting different results. And you don't want your e- you don't want your performance to degrade because of that. So you wanna be able to monitor and observe that. That's the point of the evals. Did that help? Beautiful. Um, I don't know for how much time we have left, but go ahead.
- 28:38
I have a question for later, but, um, in most of these, uh, evaluators that I've worked with, uh, the agents are linting obsessed. Um, and so I end up with a ton more linting on top of processes. I'm just curious if you've run into that with any other-
- 28:56
So your question was, if I've run into issues with linting with the agents?
- 29:00
With the agents kind of piling linting on top of things as a way of...
- 29:07
It seems like they, they spend more time on linting than doing solving the actual problem.
- 29:11
Good question. Yeah. So he's basically saying, "Hey, um, have you seen the issue where, hey, you've given access Cla-Claude to all these skills and all these sort of traces and all this sort of information, but ultimately, most of the time I've seen with Claude Code or any of these other sort of coding agents, they end up focusing on linting rather than any other sort of issue. Am I correct?" Yeah, that's a very good understanding. And that's kinda why we built out these skills, which kinda focus on ensuring that they've hyper fixate on the
- 29:41
actual issues that have been given to them, right? So they'll get all these trace information, so they'll understand, hey, it's not gonna be a linting-based issue because you can see that it's calling the wrong tool. And that's not gonna happen because of linting. And it does have some reasoning capability to understand that. Um, and so genuinely, I haven't seen that so much when using the skills.
- 30:03
Awesome. So as it-- Oh, yeah.
- 30:07
Sorry, I have a question here. Um, so with the evals, I love the idea that it runs continuously on your, on your live data. Does your system make it easy to understand how much it's costing each of those-
- 30:21
Does, does the system make it easy to understand? Sorry.
- 30:24
How much it's costing you to run those evals continuously?
- 30:27
Great question. Yeah. So how-- So his question was, does it make it easy to understand how much it costs to run those evals? Absolutely. So one of the best parts of running evals is, in the platform, is you're gonna be able to get full trace history of what your evals did, which also comes with the added benefit- Of, you can see right here, a nice little cost. And so, you'll be able to understand how much your evals are actually costing you. Because yes, of course, anything, anytime you're running another LLM, it's gonna be additional
- 30:57
cost to you. So you wanna make sure it's not. And of course, what's really nice about building out evals, whether it's in the UI or via the coding agent, you can have those set up basic requirements like, hey, we want a sample specific amount so we don't run it across 100% of our traces and 100% of our calls. Um, you can make it focus on specific areas, and of course, you can always dictate that in the coding agent or on the platform itself. Um, so as a
- 31:27
kind of refresher as to everything we kinda covered today, again, it w- oh, that's the wrong one. Oh, I think I have it over here.
- 31:35
Um,
- 31:38
so
- 31:40
essentially what I really wanted to give you as a takeaway, these AI agents, they're great. They will do basically everything you ask them to, but again, they're not magic. Like he said, they can definitely focus on linting. They can definitely get caught up in small little areas, but the best part of them is they will automate your full workflow from beginning to end. So whether it's that starting from observing where the issues lie,
- 32:07
finding those issues, being able to actually pull it down, understand where the issues lie, create those datasets, build out exact fixes, test them to make sure that they're working, and finally, you can then go ahead and be the manual human to make sure that, hey, it didn't just go back and fix the linting changes, right? And we can make sure that it actually wa- made changes that actually help improve the agent and not go for degradation.
- 32:39
That's kind of the general gist of what I wanted to show today. Um, any other questions on this before, you know, I can maybe show one last thing of seeing... Oh, still finishing up what it was trying to do earlier. I was hoping it might be done after this. Took about 14 minutes. But,
- 32:55
um, while that's finishing up, any other questions on how we can use our agents to automate our workflows in improving your agents? Yeah.
- 33:05
Are we primarily relying on Claude to follow the analysis directly from the data? Like, it showed that the minimal decrease percentage is better than
- 33:19
Great question. So his question was, are we primarily relying on Claude or any of these coding agents to actually go ahead and give us the evaluation if things are working well or not? Um, yes, you can definitely rely on Claude to build out everything. You can also manually do it. You can also ask Claude to do the specific evaluations you want. So in this case, when you saw that nice clean table, that was actually defined by the evaluations me and my team had set up prior to this. Um, we'd gone ahead and looked at, hey, these
- 33:49
are specific areas that we know are potential limitations in most agents that work in this workflow, and we set up those as generic ones, which is why we got a bad summary. We had some, hey, is the agent healthy? Is it giving the right-- Are we getting tool calls correct, right? We know those are general areas. But what Claude was able to do was look at our actual traces, understand based on those traces, what are the actual issues we're falling into today, right? We have our live agent in production. I'm not gonna be able to go through all of
- 34:19
those manually and find those, but Claude can. Or Claude, Cursor, Codex, whatever, right? And that's where, like, one of the biggest superpowers that you're able to give these coding agents to be able to do that for you. I saw another question here. Good question. So his question was, is there any sort of benchmark dataset that we can see or any sort of, like, visualization that can tell us, hey, this is a higher quality LLM to use for evaluations, more expensive, less expensive? Um, realistically, no.
- 34:50
Uh, that would be sort of something you'd have to do for your own analysis. You'd have to figure it out because ultimately, one of the things that I like to say is if you're gonna use out-of-the-box evals to judge your agent, you're gonna get out-of-the-box results, and that's not what you want when you're building custom-tailored AI agents which are built to be personal for you. Um, and the same goes for benchmarks, right? Benchmarks are great. They'll show you, hey, this, this is a good agent. This is a good-- Sorry, this is a good LLM. But realistically, what you want is, is it a good LLM for
- 35:20
me? And that's gonna be tailored to your use case. So if you're building for finance, maybe you'll use a specific one. If you're building for, you know, law and finance, it might be even a slightly different model, right? So it really depends on what you're building it for.
- 35:37
Cool. I think that's more or less it, more or less it from me. Um, if anyone has any other questions, I'm happy to answer them. But yeah. Thank you, guys.