Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Checking an agent’s input and output can get a project moving. Ankur Duggal of Arize AI calls this shipping on “vibes”: read the answer, decide whether it looks reasonable, and keep going. His opening audience poll separates that habit from having evaluations—and separates having evaluations from trusting their coverage. The difficult part is knowing which behaviors deserve a test before production exposes them.

Traces supply the missing execution history. In a conventional submit handler, the code makes the branches visible: validate the input, return an error if it fails, otherwise continue. An agent may enter through something as small as agent.run(query), while the consequential choices happen afterward. It can call tools, retrieve documents or perform analysis. Reading the entry point does not tell you which path a particular request took; tracing records that path through model calls and tool activity.

Consider a customer asking where their refund is. The agent calls tool.getOrderStatus and replies that the refund was issued. That answer may repeat a true order status while leaving the customer’s question unanswered: why has the money not arrived? The useful thing to inspect in the trace is the tool choice. It exposes a mismatch between the user’s intent and the information the agent sought, giving the investigation a more specific starting point than “the answer was bad.”

Finding the wrong tool call still leaves several possible repairs. The prompt may need clearer instructions. A guardrail may need to constrain behavior. The tool itself may be broken. Collecting a sample of similar failures creates a dataset on which those explanations can be tested. An evaluator then checks the intended behavior—for example, whether a guardrail actually prevents information from leaking—before the changed application returns to production.

Duggal’s proposal is to put the evidence and the means of repair in the same coding-agent workspace. Each input contributes something different:

• Agent traces reveal the model calls, retrievals and tools used for a request.
• Evaluations identify behaviors that failed an existing quality check.
• Human annotations add judgments about what should have happened.
• APM traces help distinguish application behavior from infrastructure problems, such as a service being down.
• Application code gives the coding agent somewhere to implement a change once it identifies the relevant failure.

The interface is a CLI plus skills that teach the coding agent how to use it. The available operations span instrumentation, downloading traces, creating datasets, running experiments and evaluating results. Claude Code is the client in this demonstration, but the proposed arrangement also accommodates Cursor or Codex. The useful capability is access to runtime evidence alongside the repository.

The live example is a financial agent that answered an investment question with a recommendation to invest. Duggal objects to that behavior because neither he nor the agent is registered to give financial advice. Audience questions expose additional problems: the agent has not asked enough follow-up questions, does not know the user’s financial background, and has no established ground truth for validating the answer. These are separate defects; a more detailed response would not, by itself, resolve all of them.

Setup has three parts: install the CLI, create a profile pointing to the Arize data, and install the skills for the coding agent. Duggal uses Claude Code’s marketplace for the last step. The profile connects the investigation to stored traces, evaluations and annotations. He also explicitly discourages copying his demo’s permission-skipping configuration, which he uses to avoid repeated approval prompts onstage.

The investigation request asks Claude to retrieve the financial agent’s traces, inspect existing evaluations and annotations, perform its own analysis, and suggest evaluations that would catch the discovered problems in the future. That leaves room for several kinds of repair: asking for more context, changing prompts, or adding a guardrail that limits unwarranted financial advice. It does not assume that the audience’s first hypothesis is the root cause.

While that investigation runs, a similar question about investing in Nvidia provides a manual tracing example. An earlier run exposes the query, final output, supervisor agent, financial data agent, model calls and tool calls. The trace answers a practical question: what work actually produced this recommendation? It lets a reviewer move beneath the final prose and inspect the participating agents and their actions.

The skills operate through the CLI underneath them. Some operations download traces or datasets into the local system; others make API requests or create evaluations on the platform. This explains how a local coding session can both inspect production evidence and configure ongoing monitoring without requiring the developer to perform each platform operation manually.

The investigation returns several distinct findings. Existing evaluations flag about 44% of the analyzed agent interactions as producing a bad summary, alongside fabricated answers and redundant tool calls. That percentage describes the interactions assessed by the existing summary evaluator; it is not a universal measure of the application’s accuracy. The findings also carry different costs: an unsupported answer damages reliability, while repeated tool calls consume time and money even when the final answer happens to be useful.

Claude’s additional trace analysis identifies a more specific failure mechanism: a tool can appear to succeed while returning no actual response, and the financial agent can fill the gap with a hallucinated answer instead of disclosing the failure. This connects two layers of behavior. Checking that a call reported success is insufficient if the returned content is unusable; checking the final answer is also necessary because the agent may conceal the missing evidence with fluent prose.

Duggal chooses two targets for the next repair:

• Silent tool failure: build an evaluation that detects the apparent-success-without-a-useful-response case, and change the agent’s instructions so it does not invent an answer when a tool fails.
• Numeric groundedness: make numerical claims refer back to actual supporting information rather than allowing the agent to manufacture numbers.

The request includes a concrete test condition: use the spans that originally failed as the basis for a dataset, then run experiments against those cases. Dataset, evaluator and experiment skills connect the proposed edits to an observable result. The experiment should establish whether the changed behavior survives the input that exposed the defect, rather than merely whether Claude completed an edit.

While the live repair continues, an earlier experiment shows why testing intermediate fixes matters. Users had asked about stocks outside the United States, and the displayed failure cases concerned South Korea. Claude collected the failing cases into a dataset so the same requests could be replayed after changes. The goal was to support questions across markets, but the demonstrated evidence concerns those collected cases.

The buggy baseline passed 0% of that dataset. Claude’s first correction normalized the query by adding the country name. It still failed. The missing country context was only part of the problem: the underlying tool was outdated and needed an update to its stable version. A subsequent iteration addressed that tool problem and reached about a 90% success rate.

What changed between attempts? The comparison separates an input correction from a tool correction:

The failed middle attempt is the useful lesson. Improving the information supplied to a tool cannot make an unsuitable tool implementation work. The final result is substantial progress on this dataset, with remaining failures still unresolved.

An experiment does not have to execute inside Arize. In this demonstration, the application experiment runs on Duggal’s local machine; a cloud environment is another option. Evaluations can then run on Arize. Separating those locations lets the application execute where its code and dependencies are available while the platform performs the quality checks.

Evaluator creation is itself fallible. Claude attempted both a code-based evaluator and an LLM-based evaluator, but failed to set up the code-based one successfully. The final comparison used the LLM judge it created on the platform. Two evaluator entries therefore did not mean two working, independent checks.

Judge-model selection introduces a cost and quality decision. The platform supports configurable providers, including local open-source models. Some teams prefer cheaper judges; others use a stronger model for evaluation than for the production agent. The production model and the evaluator need not be the same, because generating an answer and assessing it serve different purposes.

A smaller custom repair agent could use the same approach. The added value of the skills is access to evidence beyond the code: execution paths, evaluations, annotations, APM logs and potentially issue tickets. Duggal also identifies Phoenix as an open-source option for connecting this kind of information to a coding agent, while describing easier online-evaluation setup and additional dashboarding as enterprise conveniences.

The earlier “bad summary” finding came from a previously configured evaluator, rather than an unexplained judgment invented during the investigation. Its inputs include the information gathered across the agent’s work and the context supplied to the summary agent. Its checks ask whether the summary references the appropriate points and answers the user’s question. Inspecting that evaluator answers what the label means: summary quality is being judged against the available context and the task.

Why not install a skill that produces good summaries in the first place? That is a useful application change: improve prompts, skills and coding workflows so the agent performs better. Evaluation has a separate job. A skill that works today may produce worse results after a model regression or a change in how users phrase questions. Duggal’s hypothetical two-month example illustrates why the quality check remains useful after the initial repair: it watches for the return of a behavior the team already knows is harmful.

An audience member raises a familiar coding-agent failure: it spends its effort adding linting instead of solving the actual problem. Runtime evidence gives the task a more specific focus. A trace showing the wrong tool call points toward tool selection or tool behavior; lint cleanup does not explain that failure. Duggal reports seeing less of this distraction with the skills, while retaining human review as the check that the proposed change addresses the real defect.

Continuous LLM evaluation also adds operating cost. Evaluation traces expose what the judges did and include cost information. Sampling controls let a team evaluate a selected fraction of traces or focus checks on particular areas instead of judging every call. That makes evaluation coverage a deliberate budget decision, configurable through either the platform or the coding agent.

The proposed automation can gather failures, build datasets, edit the application and test the result. The human still needs to inspect whether those changes improve the agent rather than merely making the repository look tidier. At the final live check, the current repair is still running after about 14 minutes. The recording therefore demonstrates its investigation and repair request, but does not establish a completed fix for the live numeric-grounding and silent-failure problems.

The closing questions clarify who supplied the quality standard. Duggal and his team had already defined the evaluations behind the initial results table, covering broad concerns such as summaries, agent health and tool-call correctness. Claude then used the actual traces to identify more specific problems occurring in the application. Those are complementary contributions: existing checks organize known risks, while trace analysis can propose additional checks for failures the original suite missed.

There is no universal judge-model recommendation offered here. A benchmark can indicate general model capability, but selecting an evaluator requires analysis of the application’s own task and domain. Finance may need a different choice from a workflow combining law and finance. Duggal’s closing warning is pointed: “out-of-the-box evals” yield “out-of-the-box results.” The practical next step is to use the failures users actually encounter to decide what the evaluator must recognize—and whether its judgments are useful enough to guide the next repair.