Agentic Excellence: Mastering Evaluation of AI Agents with Azure AI Evaluation SDK

How do you know an agent behaves well enough to release? Ordinary dataset evaluation addresses that question alongside red teaming. Red teaming deliberately supplies inputs intended to provoke harmful content or behavior; ordinary evaluation tests the agent against a collection of expected tasks. Here, Cedric Vidal, Principal AI Advocate at Microsoft, develops the latter workflow.

Model and SDK improvements can make an agent more capable almost overnight—something Vidal noticed even while preparing this demonstration. But greater independence also increases the consequences of mistakes. Trying a couple of prompts, liking the responses, and putting the agent into production leaves too much unexamined. A successful spot check is the beginning of evaluation, not a release criterion.

Evaluation should therefore begin when the project begins. It also needs to cover more than the foundation model. Vidal separates the system into four layers:

Vidal describes the first two layers as built into Azure. The remaining layers depend on application design: platform protections do not supply the instructions, grounding, and interaction choices that make a particular agent safe and useful.

Manual model evaluation comes first because aggregate scores hide individual behavior. A high average cannot tell you exactly what happened on a particular prompt. Before running a large dataset, inspect selected examples closely enough to understand what a good answer looks like.

The demonstration uses AI Toolkit for VS Code, bringing comparisons into the development environment instead of switching among websites or using GitHub Models separately. The recording uses the historical AI Toolkit name and interface; current documentation calls the product Foundry Toolkit. Vidal starts with a concrete request: a panna cotta recipe with salted caramel butter.

GPT-4.1 supplies the initial recipe. Vidal then submits the request to GPT-4o, displaying GPT-4.1 on the left and GPT-4o on the right. Vidal observes faster output from GPT-4.1 on this recipe prompt and prefers its answer after inspecting it before the conference; no timing measurements or controlled benchmark conditions are supplied. This is still a comparison of uncustomized foundation models. There are no agent instructions or tools to evaluate yet.

Once a model is selected, the evaluation boundary expands to the whole system. Vidal's application is an assistant for his work as an advocate: it retrieves event information from websites, including agendas, talks, speakers, and attendance information. AI Toolkit provides a short path from that requirement to a working browser agent.

The setup proceeds through a built-in sample and then a customized agent:

1. Select the web-scraper sample. It generates a system prompt describing a web-exploration assistant and configures an MCP server.
2. Run the sample. It starts Playwright MCP and explores an example domain.
3. Customize the sample for event extraction. Vidal uses GPT-4.1 and requests the event name, date, location, and attendee count in a specific output format.
4. Supply a Luma event page and run the agent. Playwright navigates the page, and the agent returns the extracted information.

The customized instructions narrow a general browser assistant into an application with identifiable fields to check.

The demonstrated response identifies AI Agents and Startup Talks at GitHub, at GitHub headquarters in San Francisco, on June 11. It reports 269 registered people at demonstration time. Vidal co-organizes the event, giving him a familiar example against which to inspect the result. The agent has now been built, customized, and spot-checked on one input.

The Evaluation tab runs the configured agent against multiple inputs. Selecting Run all executes those inputs and populates a response column; the demonstration already has earlier responses available to inspect while the new run proceeds.

One response contains an attendee count even though the supplied Reactor event page does not. That is a reason to investigate, not immediately label the field a hallucination. Vidal explains that the agent found a link to Luma on the Reactor page, followed it, and retrieved the count there. It combined information from both pages to complete the requested record. Evaluating an agent means checking how it obtained the answer as well as what it returned. Vidal judges the displayed answers positively using the available thumbs-up or thumbs-down ratings.

The next handoff has two distinct parts. The evaluation dataset can be exported as JSON Lines for use in a more automated system, although Vidal does not perform that export. Separately, View code generates an implementation for a chosen framework, with OpenAI Agents offered as the example. That generated code includes the MCP server configuration and boilerplate to run the agent; it is not itself the automated evaluation suite. At this point, the workflow covers individual checks and small local batches.

Before the automated evaluation walkthrough, an audience member asks for the presentation. Vidal agrees to share it; the distribution channel is settled in the closing exchange.

Scaling beyond visual inspection requires repeatable checks over a wider range of inputs. Azure AI Foundry offers several families of evaluators:

These checks can run through the Foundry portal or through code. The choice of measures should follow the application's purpose rather than treating every available score as equally useful.

Vidal next opens a Python notebook that connects to an Azure AI Foundry project. He walks through existing code and results without executing the notebook live because of the time required, and promises to share its link. The central operation is evaluate(), supplied with a dataset and evaluators for relevance, coherence, groundedness, fluency, and similarity.

The Azure AI Evaluation SDK distinguishes scoring stored responses from running an application to collect new ones: evaluate() can read responses directly from a dataset, while its optional target parameter invokes an application first. The following Python pattern scores stored responses using the five quality evaluators discussed. Its JSONL rows need the evaluator inputs, including query, response, context, and ground_truth.

import os

from azure.ai.evaluation import (
    CoherenceEvaluator,
    FluencyEvaluator,
    GroundednessEvaluator,
    RelevanceEvaluator,
    SimilarityEvaluator,
    evaluate,
)

model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
    "api_key": os.environ["AZURE_OPENAI_API_KEY"],
}

evaluators = {
    "relevance": RelevanceEvaluator(model_config=model_config),
    "coherence": CoherenceEvaluator(model_config=model_config),
    "groundedness": GroundednessEvaluator(model_config=model_config),
    "fluency": FluencyEvaluator(model_config=model_config),
    "similarity": SimilarityEvaluator(model_config=model_config),
}

results = evaluate(
    data="responses.jsonl",
    evaluators=evaluators,
)
print(results["metrics"])

This separates the response-producing application from the models judging its output, so an evaluator run need not trigger another browser session for every row.

The notebook's examples include France's capital, the most weatherproof tent, and a camping table. Vidal describes the displayed quality scores as ranging from 1 to 5, with configurable thresholds, and shows a passing case. Per-question results make it possible to inspect failures rather than relying only on an aggregate. His threshold discussion then moves to product expectations: a game that permits violent content and an application for children would make different acceptance decisions.

The final technical example extends evaluation to text and images. Vidal also mentions multi-turn conversation support, without working through a conversation example. Those are separate capabilities: current classic SDK documentation supports text conversations but limits multimodal evaluation to one user message and one assistant message.

For the image example, Vidal needed something violent enough to test but still suitable for a public conference. The evaluator's explanation describes a character with numerous pins or nails protruding from its head. The displayed image-safety result assigns violence a score of 4 and fails it against a threshold of 3. The same output also shows passing sexual-content and self-harm results.

Vidal describes the violence score as below a maximum of 5. Current risk and safety evaluator documentation instead specifies a severity scale of 0–7, with 4–5 representing medium severity; the exact historical evaluator is unidentified. Under the documented rule, a score passes when it is at or below the configured threshold.

For a violent-video-game context, Vidal proposes accepting the score of 4. Raising the threshold to 4 would change this result from failing to passing under that rule, without changing the image or its assessed severity. He connects this choice to allowing such imagery in a game, but an evaluation threshold changes the acceptance judgment, not an image generator's safety policy or capabilities. The useful design decision is to make the application's tolerance explicit and test against it consistently.

Vidal closes by pointing to Azure AI Foundry discussions on GitHub and the Foundry Discord for questions about building agents and using the evaluation SDK. His contact information appears on the closing slide. In the final audience exchange, he specifies that he will post the presentation slides to the Discord server shown in the middle of that slide.