# Agentic Excellence: Mastering Evaluation of AI Agents with Azure AI Evaluation SDK

Cedric Vidal, Microsoft

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/J4vPq2i0QzE-agentic-excellence-mastering-evaluation-ai-agents) · [Video](https://www.youtube.com/watch?v=J4vPq2i0QzE)

## Agentic Excellence: Mastering Evaluation of AI Agents with Azure AI Evaluation SDK

Move from inspecting individual model responses to testing a browser agent, then scale evaluation with datasets, Python evaluators, and application-specific acceptance thresholds.

Familiarity with Python, model prompts, and agents that call tools will help you follow the examples.

### What would justify shipping this agent?

[0:18](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=18s)

How do you know an agent behaves well enough to release? Ordinary dataset evaluation addresses that question alongside red teaming. Red teaming deliberately supplies inputs intended to provoke harmful content or behavior; ordinary evaluation tests the agent against a collection of expected tasks. Here, Cedric Vidal, Principal AI Advocate at Microsoft, develops the latter workflow.

Model and SDK improvements can make an agent more capable almost overnight—something Vidal noticed even while preparing this demonstration. But greater independence also increases the consequences of mistakes. Trying a couple of prompts, liking the responses, and putting the agent into production leaves too much unexamined. **A successful spot check is the beginning of evaluation, not a release criterion.**

![Slide showing a giant robot above city buildings and asking how to keep an AI agent from wrecking havoc.](https://ai.engineer/api/talk-essay-media/3c36cad2bffaf5be8173dc6ef3402fac0c0b61dd1568d96b094546f0486c029e)

How do you make sure your AI agent is not wrecking havoc?

Evaluation should therefore begin when the project begins. It also needs to cover more than the foundation model. Vidal separates the system into four layers:

| Layer | Responsibility in the demonstration |
| --- | --- |
| Model | Platform-level model behavior |
| Safety system | Platform-level protections |
| System message and grounding | Application instructions and supporting information |
| User experience | Application interactions and mitigations |

Vidal describes the first two layers as built into Azure. The remaining layers depend on application design: platform protections do not supply the instructions, grounding, and interaction choices that make a particular agent safe and useful.

### Inspect an answer before averaging many answers

[4:16](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=256s)

Manual model evaluation comes first because aggregate scores hide individual behavior. A high average cannot tell you exactly what happened on a particular prompt. Before running a large dataset, inspect selected examples closely enough to understand what a good answer looks like.

The demonstration uses [AI Toolkit for VS Code](https://code.visualstudio.com/docs/intelligentapps/evaluation), bringing comparisons into the development environment instead of switching among websites or using GitHub Models separately. The recording uses the historical AI Toolkit name and interface; current documentation calls the product Foundry Toolkit. Vidal starts with a concrete request: a panna cotta recipe with salted caramel butter.

![VS Code displays a panna cotta recipe response with salted caramel butter instructions, serving suggestions, and model preferences alongside it.](https://ai.engineer/api/talk-essay-media/ab7cf78a9dff403b7a575d8df198cce82f85fd83b5546e9d4d1927cafc73f286)

A recipe response in the VS Code AI Toolkit playground.

**GPT-4.1** supplies the initial recipe. Vidal then submits the request to **GPT-4o**, displaying GPT-4.1 on the left and GPT-4o on the right. Vidal observes faster output from GPT-4.1 on this recipe prompt and prefers its answer after inspecting it before the conference; no timing measurements or controlled benchmark conditions are supplied. This is still a comparison of uncustomized foundation models. There are no agent instructions or tools to evaluate yet.

### Turn a model into an event-extraction agent

[7:12](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=432s)

Once a model is selected, the evaluation boundary expands to the whole system. Vidal's application is an assistant for his work as an advocate: it retrieves event information from websites, including agendas, talks, speakers, and attendance information. AI Toolkit provides a short path from that requirement to a working browser agent.

The setup proceeds through a built-in sample and then a customized agent:

1. Select the web-scraper sample. It generates a system prompt describing a web-exploration assistant and configures an MCP server.
2. Run the sample. It starts [Playwright MCP](https://github.com/microsoft/playwright-mcp) and explores an example domain.
3. Customize the sample for event extraction. Vidal uses GPT-4.1 and requests the event name, date, location, and attendee count in a specific output format.
4. Supply a Luma event page and run the agent. Playwright navigates the page, and the agent returns the extracted information.

The customized instructions narrow a general browser assistant into an application with identifiable fields to check.

The demonstrated response identifies AI Agents and Startup Talks at GitHub, at GitHub headquarters in San Francisco, on June 11. It reports 269 registered people at demonstration time. Vidal co-organizes the event, giving him a familiar example against which to inspect the result. The agent has now been built, customized, and spot-checked on one input.

### Inspect the route behind a batch result

[10:39](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=639s)

The Evaluation tab runs the configured agent against multiple inputs. Selecting `Run all` executes those inputs and populates a response column; the demonstration already has earlier responses available to inspect while the new run proceeds.

One response contains an attendee count even though the supplied Reactor event page does not. That is a reason to investigate, not immediately label the field a hallucination. Vidal explains that the agent found a link to Luma on the Reactor page, followed it, and retrieved the count there. It combined information from both pages to complete the requested record. **Evaluating an agent means checking how it obtained the answer as well as what it returned.** Vidal judges the displayed answers positively using the available thumbs-up or thumbs-down ratings.

The next handoff has two distinct parts. The evaluation dataset can be exported as JSON Lines for use in a more automated system, although Vidal does not perform that export. Separately, `View code` generates an implementation for a chosen framework, with OpenAI Agents offered as the example. That generated code includes the MCP server configuration and boilerplate to run the agent; it is not itself the automated evaluation suite. At this point, the workflow covers individual checks and small local batches.

Before the automated evaluation walkthrough, an audience member asks for the presentation. Vidal agrees to share it; the distribution channel is settled in the closing exchange.

### Choose measures, then run them across a dataset

[13:40](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=820s)

Scaling beyond visual inspection requires repeatable checks over a wider range of inputs. Azure AI Foundry offers several families of evaluators:

| Family | Examples | Purpose |
| --- | --- | --- |
| AI-assisted quality | Groundedness, fluency, coherence | Assess response quality |
| Traditional NLP | F1, BLEU, ROUGE | Compare against reference answers |
| AI-assisted risk and safety | Content-risk checks | Assess harmful content |
| Custom | Application-defined evaluators | Measure task-specific requirements |

These checks can run through the Foundry portal or through code. The choice of measures should follow the application's purpose rather than treating every available score as equally useful.

Vidal next opens a Python notebook that connects to an Azure AI Foundry project. He walks through existing code and results without executing the notebook live because of the time required, and promises to share its link. The central operation is `evaluate()`, supplied with a dataset and evaluators for relevance, coherence, groundedness, fluency, and similarity.

The [Azure AI Evaluation SDK](https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk) distinguishes scoring stored responses from running an application to collect new ones: `evaluate()` can read responses directly from a dataset, while its optional `target` parameter invokes an application first. The following Python pattern scores stored responses using the five quality evaluators discussed. Its JSONL rows need the evaluator inputs, including `query`, `response`, `context`, and `ground_truth`.

```python
import os

from azure.ai.evaluation import (
    CoherenceEvaluator,
    FluencyEvaluator,
    GroundednessEvaluator,
    RelevanceEvaluator,
    SimilarityEvaluator,
    evaluate,
)

model_config = {
    "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
    "azure_deployment": os.environ["AZURE_OPENAI_DEPLOYMENT"],
    "api_key": os.environ["AZURE_OPENAI_API_KEY"],
}

evaluators = {
    "relevance": RelevanceEvaluator(model_config=model_config),
    "coherence": CoherenceEvaluator(model_config=model_config),
    "groundedness": GroundednessEvaluator(model_config=model_config),
    "fluency": FluencyEvaluator(model_config=model_config),
    "similarity": SimilarityEvaluator(model_config=model_config),
}

results = evaluate(
    data="responses.jsonl",
    evaluators=evaluators,
)
print(results["metrics"])
```

This separates the response-producing application from the models judging its output, so an evaluator run need not trigger another browser session for every row.

The notebook's examples include France's capital, the most weatherproof tent, and a camping table. Vidal describes the displayed quality scores as ranging from 1 to 5, with configurable thresholds, and shows a passing case. Per-question results make it possible to inspect failures rather than relying only on an aggregate. His threshold discussion then moves to product expectations: a game that permits violent content and an application for children would make different acceptance decisions.

### Separate safety severity from acceptance policy

[17:00](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1020s)

The final technical example extends evaluation to text and images. Vidal also mentions multi-turn conversation support, without working through a conversation example. Those are separate capabilities: current classic SDK documentation supports text conversations but limits multimodal evaluation to one user message and one assistant message.

For the image example, Vidal needed something violent enough to test but still suitable for a public conference. The evaluator's explanation describes a character with numerous pins or nails protruding from its head. The displayed image-safety result assigns violence a score of 4 and fails it against a threshold of 3. The same output also shows passing sexual-content and self-harm results.

![VS Code notebook output shows a violence explanation, a failed violence result, score 4, threshold 3, and passing sexual-content and self-harm results.](https://ai.engineer/api/talk-essay-media/8417051c5cf0db6eaea05be6222cb409070fc893c696a906f99594463bc3affd)

Image safety evaluation reports a violence score of 4 and a failed result.

Vidal describes the violence score as below a maximum of 5. Current [risk and safety evaluator documentation](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/risk-safety-evaluators) instead specifies a severity scale of 0–7, with 4–5 representing medium severity; the exact historical evaluator is unidentified. Under the documented rule, a score passes when it is at or below the configured threshold.

For a violent-video-game context, Vidal proposes accepting the score of 4. Raising the threshold to 4 would change this result from failing to passing under that rule, without changing the image or its assessed severity. He connects this choice to allowing such imagery in a game, but **an evaluation threshold changes the acceptance judgment, not an image generator's safety policy or capabilities**. The useful design decision is to make the application's tolerance explicit and test against it consistently.

### Continue with the SDK and community

[18:51](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1131s)

Vidal closes by pointing to Azure AI Foundry discussions on GitHub and the Foundry Discord for questions about building agents and using the evaluation SDK. His contact information appears on the closing slide. In the final audience exchange, he specifies that he will post the presentation slides to the Discord server shown in the middle of that slide.

## Resources

- [Playwright MCP](https://github.com/microsoft/playwright-mcp) — Browser automation server exposing Playwright through MCP, with setup instructions and tool documentation.
- [Azure AI Evaluation SDK: local evaluation](https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk) — Python examples for dataset evaluation, evaluator inputs, JSONL formatting, application targets, and result reporting in Foundry classic.
- [Evaluate models, prompts, and agents in VS Code](https://code.visualstudio.com/docs/intelligentapps/evaluation) — Current Foundry Toolkit guidance for manual ratings, automated evaluators, and comparing prompt or agent versions.
- [Content-safety scoring and thresholds](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/risk-safety-evaluators) — Definitions of safety categories, severity scores, and pass/fail thresholds in current Microsoft Foundry evaluation.

## Slides

### How do you make sure your AI agent is not wrecking havoc?

A giant robot looms over a city as the speaker links greater agent independence to increased risk and calls for more methodical evaluation than trying a few prompts.

[1:23](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=83s) [A question about preventing AI agents from wrecking havoc appears over an illustration of a robot and city.](https://assets.ai.engineer/sites/aiecode2025/sha256/261ab7a0d8dc1e32661608573c64d1e4bd81fdf4aa7e82c6a421a73d18a18b67.webp)

How do you make sure your AI Agent is not wrecking havoc?

AI
MARRIOTT MARQUIS
AIE
Microsoft
smol ai
World’s Fair

### When should I start doing evaluations?

Start evaluations at the beginning of an AI development project. If you wait until the app is built, the speaker says you have started late.

[2:39](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=159s) [A slide asks when to start doing evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f3576facbccca46304c17f19b4439f8823d29d9870bb499bdbf047da07e9678.webp)

When should I start doing evaluations?

Interface text:
AIE
World’s Fair

### Risk mitigation layers

The model sits inside a safety system, surrounded by system messages, grounding and user experience. The speaker places application design in those outer layers.

[3:29](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=209s) [Four nested layers: model, safety system, system message and grounding, and user experience.](https://assets.ai.engineer/sites/aiecode2025/sha256/929e82e02cf351fa37c950b5caac6e9bce3f82aa20f13d6b94d88c4ea1bd9217.webp)

Risk mitigation layers

User Experience
Design for responsible human-AI interaction

System Message & Grounding
Ground your model and steer its behavior

Safety System
Monitor and protect model inputs and outputs

Model
Choose the right model for your use case

AIE
AI Engineer World's Fair
Microsoft
smol ai

### Manual model evaluation

GPT-4o and Claude 3.7 Sonnet answer the same cake-recipe prompt side by side. Inspecting specific responses can reveal details that an average evaluation score may miss.

[4:23](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=263s) [Tools for manually comparing model outputs, alongside a comparison screenshot.](https://assets.ai.engineer/sites/aiecode2025/sha256/e41548e595a57553ca7ece1920fd593e633586b014912a172ecf1a7d02f4190c.webp)

Manual Model Evaluation

Model Choice

Various tools are available for evaluating models and comparing model output.

• Azure AI Foundry Chat playground
  • ai.azure.com
• GitHub Models
  • github.com/marketplace/models
• AI Toolkit for Visual Studio Code
  • aka.ms/AIToolkit

Interface text:
Compare
OpenAI GPT-4o (via GitHub)
Select this model
Claude 3.7 Sonnet (20250219) (via Anthropic)
Select this model

User: What's a good cake recipe that includes strawberries?

OpenAI GPT-4o
Here's a delicious and easy recipe for Strawberry Shortcake Cake—a moist vanilla cake layered with whipped cream and fresh strawberries. It's a perfect dessert for any occasion!

Strawberry Shortcake Cake Recipe
Servings: 10–12 slices
Prep Time: 25 minutes
Cook Time: 25–30 minutes
Total Time: ~1 hour (plus cooling time)
Ingredients:
For the Cake

User: What's a good cake recipe that includes strawberries?

Claude 3.7 Sonnet (20250219)
Strawberry Cake Recipe
Ingredients
For the cake:
• 2¼ cups all-purpose flour
• 2 cups granulated sugar
• 1 teaspoon baking soda
• ½ teaspoon salt
• 1 cup unsalted butter, softened
• 4 large eggs, room temperature
• 1 cup sour cream
• ½ cup milk

Type a prompt
Thinking

AIE
AI Engineer World's Fair

### AI Toolkit playground

The VS Code playground opens with GPT-4o selected, example prompts, and model settings. The presenter uses it to inspect individual examples before evaluating at scale.

[5:11](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=311s) [VS Code playground with example prompts and model preferences.](https://assets.ai.engineer/sites/aiecode2025/sha256/54bf25e4ad5cde4bba23078c9f5d9e664c9dc1833e759189ec42a201bf275419.webp)

Explore AI Model Capabilities

Try AI models with code, text, or files to see how thoughtfully they respond to different prompts.

Don't forget to review the models responses carefully as AI can make mistakes.

Explore typical scenarios where traditional applications can leverage large language model capabilities, structure the output in markdown format.

Add descriptive captions for the image.

Write a Python function to encode the input text in base64.

MODEL PREFERENCES
Model
OpenAI GPT-4o (via GitHub)
Browse models

Context instructions
Provide instructions with relevant context to guide the system response.

Inference Parameters
Max response length (tokens)
Temperature: 1
Top P: 1

Type a prompt
Web

Interface text:
Playground
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
History
New Playground
Compare
View Code
EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
data.jsonl
model_endpoint.py
image1.jpg
AIE
Microsoft
smol ai
World's Fair

### Playground model response

In VS Code’s AI Toolkit playground, GPT-4.1 returns a panna cotta recipe. The presenter uses this example to inspect a model response before comparing models.

[5:18](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=318s) [VS Code playground showing a model-generated recipe response and model preferences.](https://assets.ai.engineer/sites/aiecode2025/sha256/11b25b39e6c2faba4b03cb5ed872f2b22472a7c55c8dae4645bc15b14a8850fe.webp)

2. Make the salted caramel butter sauce:
1. In a saucepan, combine sugar and water. Cook over medium heat, swirling (not stirring), until sugar dissolves and turns deep amber, about 5–7 minutes.
2. Remove from the heat and carefully whisk in cream (it will bubble a lot).
3. Add butter cubes, whisking until melted and emulsified.
4. Stir in flaky sea salt, to taste. Set aside to cool (sauce will thicken as it stands).

3. Serve:
• Unmold panna cottas onto plates (or serve in glasses).
• Drizzle with generous salted caramel butter sauce just before serving.

Tips:
• You can prepare both components a day ahead.
• If you like, top with toasted nuts or flaky sea salt for extra texture!

Enjoy your silky, salty-sweet treat! Let me know if you have any ingredient or technique questions.

Token used: 0 (2049 characters)

Interface text:
Playground
History  New Playground
what's a good panacotta recipe (OpenAI GPT-4.1)  Compare  View Code
MODEL PREFERENCES
Model
OpenAI GPT-4.1 (via GitHub)
Browse models
Context instructions
Provide instructions with relevant context to guide the system response.
Inference Parameters
Max response length (tokens)
Temperature  1
Top P  1
Type a prompt
Web
EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
data.jsonl
model_endpoint.py
image1.jpg
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
AIE
World's Fair

[5:29](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=329s) [AI Toolkit playground showing the end of a GPT-4.1 recipe response and model settings.](https://assets.ai.engineer/sites/aiecode2025/sha256/12ada6beb9c8282f0e7fc8d4bb779c91180ad487b0904ca4dfba989d57e04658.webp)

2. Make the salted caramel butter sauce:
1. In a saucepan, combine sugar and water. Cook over medium heat, swirling (not stirring), until sugar dissolves and turns deep amber, about 5–7 minutes.
2. Remove from the heat and carefully whisk in cream (it will bubble a lot).
3. Add butter cubes, whisking until melted and emulsified.
4. Stir in flaky sea salt, to taste. Set aside to cool (sauce will thicken as it stands).

3. Serve:
• Unmold panna cottas onto plates (or serve in glasses).
• Drizzle with generous salted caramel butter sauce just before serving.

Tips:
• You can prepare both components a day ahead.
• If you like, top with toasted nuts or flaky sea salt for extra texture!

Enjoy your silky, salty-sweet treat! Let me know if you have any ingredient or technique questions.

Token used: 0 (2049 characters)

Interface text:
Playground
History  New Playground
what's a good panacotta recipe (OpenAI GPT-4.1)  Compare  View Code
MODEL PREFERENCES
Model
OpenAI GPT-4.1 (via GitHub)
Browse models
Context instructions
Provide instructions with relevant context to guide the system response.
Inference Parameters
Max response length (tokens)
Temperature  1
Top P  1
Type a prompt
Web
EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
data.jsonl
model_endpoint.py
image1.jpg
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
AIE
World's Fair
Microsoft
smol ai

### Playground prompt and response

In the AI Toolkit playground, GPT-4.1 answers a salted caramel butter panna cotta prompt with a recipe. The presenter prepares to compare it with GPT-4o.

[5:55](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=355s) [AI Toolkit playground showing a panna cotta prompt and the beginning of the GPT-4.1 response.](https://assets.ai.engineer/sites/aiecode2025/sha256/6eb185a50741df82059cf1725b00b0765ea7e8f936929c59dfdc3a421b2bdc2d.webp)

User
what's a good panacotta recipe with salted caramel butter

OpenAI GPT-4.1
Absolutely! Salted caramel butter is a delicious twist to classic panna cotta (and you can easily make a luscious salted caramel sauce with butter folded in, which will work beautifully). Here's a recipe for Salted Caramel Butter Panna Cotta:

Salted Caramel Butter Panna Cotta

Ingredients

For the panna cotta:
• 2 cups (480 ml) heavy cream
• 1 cup (240 ml) whole milk
• 1/2 cup (100 g) granulated sugar
• 2 tsp (1 envelope) powdered gelatin

Interface text:
Playground
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
History
New Playground
what's a good panacotta recipe (OpenAI GPT-4.1)
Compare
View Code

MODEL PREFERENCES
Model
OpenAI GPT-4.1 (via GitHub)
Browse models
Context instructions
Provide instructions with relevant context to guide the system response.
Inference Parameters
Max response length (tokens)
Temperature 1
Top P 1
Type a prompt
Web

EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
data.jsonl
model_endpoint.py
image1.jpg

AIE
Microsoft
smol ai
World's Fair

### Compare model prompts

The presenter enters the same panna cotta and salted caramel prompt in side-by-side GPT-4.1 and GPT-4o playgrounds to compare their responses.

[6:13](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=373s) [Side-by-side GPT-4.1 and GPT-4o playgrounds with the same recipe prompt entered.](https://assets.ai.engineer/sites/aiecode2025/sha256/f236fe7a68294181ec037ff072bc956b4d3a4d0c05588e2f73a92198effa4c5c.webp)

Compare

Left panel:
OpenAI GPT-4.1 (via GitHub)
Select this model

Explore AI Model Capabilities
Try AI models with code, text, or files to see how thoughtfully they respond to different prompts.
Don't forget to review the models responses carefully as AI can make mistakes.

Explore typical scenarios where traditional applications can leverage large language model capabilities, structure the output in markdown format.
Add descriptive captions for the image.
Write a Python function to encode the input text in base64.

What's a good recipe for panacotta with salted caramel
Web

Right panel:
OpenAI GPT-4o (via GitHub)
Select this model

Explore AI Model Capabilities
Try AI models with code, text, or files to see how thoughtfully they respond to different prompts.
Don't forget to review the models responses carefully as AI can make mistakes.

Explore typical scenarios where traditional applications can leverage large language model capabilities, structure the output in markdown format.
Add descriptive captions for the image.
Write a Python function to encode the input text in base64.

What's a good recipe for panacotta with salted caramel
Web

Each model is assessed according to its own capabilities. If a capability (such as Image Processing) is unique to one model, it will only be displayed for that model.

Interface text:
azureai-samples
EXPLORER
Playground
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
data.jsonl
model_endpoint.py
image1.jpg
AIE
Microsoft
smol ai
World's Fair

### Compare model responses

The playground shows GPT-4.1 and GPT-4o recipes side by side. The speaker prefers the GPT-4.1 answer and says it arrived faster; this spot-check evaluates foundation models before building an agent.

[6:50](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=410s) [Side-by-side GPT-4.1 and GPT-4o recipe responses in the AI Toolkit playground.](https://assets.ai.engineer/sites/aiecode2025/sha256/656a4d36f09c58bda25ee2f98796422724bb0c5538efc89ef95684847577ec3a.webp)

OpenAI GPT-4.1 (via GitHub)

4. Strain & Pour:
Strain mixture through a fine sieve into a pouring jug. Pour evenly into 6 ramekins or glasses (4-6 oz each). Cool slightly, then cover and refrigerate at least 4 hours (or overnight) until set.

SALTED CARAMEL BUTTER SAUCE

Ingredients:
• 1 cup (200 g) granulated sugar
• 6 tbsp (85 g) unsalted butter, cubed
• 1/2 cup (120 ml) heavy cream, at room temperature
• 1/2-3/4 tsp sea salt (to taste)

Instructions:
1. Make the Caramel:
In a medium, heavy-bottomed saucepan, melt sugar over medium heat. Stir constantly with a wooden spoon or heat-safe spatula. The sugar will clump and then melt into a golden-amber liquid; be careful not to burn.

OpenAI GPT-4o (via GitHub)

5. Remove the pan from heat and stir in the sea salt. Let the salted caramel butter sauce cool slightly before serving.

Step 3: Assemble and Serve
1. Once the panna cotta has set, gently run a knife around the edge of each ramekin and invert onto a plate to unmold. (Alternatively, serve directly in the glasses or ramekins.)
2. Drizzle the salted caramel butter sauce generously over the top of each panna cotta.
3. Garnish with a sprinkle of flaky sea salt, a dollop of whipped cream, or fresh berries if desired.

Enjoy this indulgent dessert! The creamy, mildly sweet panna cotta combined with the rich, buttery salted caramel sauce is absolutely irresistible.

Token used: 641 (2684 characters)

Interface text:
Compare
Select this model
Select this model
Each model is assessed according to its own capabilities. If a capability (such as Image Processing) is unique to one model, it will only be displayed for that model.
Type a prompt
Web
Type a prompt
Web
Playground
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
AI_Judge_Evaluator...
data.jsonl
model_endpoint.py
README.md
AI_Judge_Evaluators...
AI_Judge_Evaluator...
AI_Judge_Evaluator...
AI_Judge_Evaluat...
data.jsonl
image1.jpg

### Manual System Evaluation

Human graders spot-check system outputs with thumbs-up or thumbs-down ratings, using a CSV or JSONL dataset or manual entry. The evaluation is limited to 50 input rows.

[7:19](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=439s) [Slide describing human spot-checking, dataset input, ratings, adjustable settings, saved results, and a 50-row limit.](https://assets.ai.engineer/sites/aiecode2025/sha256/7de789efe7d145a63c81d64055ebf356652a4906ef57e0d4c2e621828c275ee1.webp)

Manual System Evaluation

Evaluation

Enable human graders to spot-check output performance for a specific risk on a small dataset.

• Supports a test dataset in CSV or JSONL format, or manual data entry via the UI
• Provide thumbs-up or thumbs-down ratings
• Modify system messages, models, or parameters as needed
• Save evaluation results to share with team members or for future reference
• Note: Limited to 50 input rows

Interface text:
Manual evaluation result
Run | Import test data | Export | Metric evaluation | Save results
Imported dataset: [illegible]
Data rated: 15.38% (2/13)
Thumbs up: 7.69% (1/13)
Thumbs down: 7.69% (1/13)
Input | Expected response | Output
Which tent is the most waterproof? | The Alpine Explorer Tent has the highest rainfly waterproof rating at 3000m | The Alpine Explorer Tent has a rainfly waterproof rating of 3000mm, which indicates a higher level of waterproofing compared to the TrailMaster X4 Tent, which has a rainfly waterproof rating of 2000mm. Therefore, the Alpine Explorer Tent is the most waterproof option among the tents mentioned in the provided documents.
Which camping table holds the most weight? | The Adventure Dining Table has a higher weight capacity than all of the other camping tables mentioned | The Adventure Dining Table can support up to 100 lbs (45 kg) of evenly distributed weight.
How much does TrailWalker Hiking Shoes cost? | $110 | The TrailWalker Hiking Shoes cost $110.

AIE
AI Engineer World's Fair
Microsoft
smol ai

### Event agenda extractor

A prepared agent combines a web exploration system prompt with fields for event name, date, location, and attendees. The speaker uses it to collect agenda details from web pages.

[7:41](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=461s) [Agent builder showing the selected model, system prompt, and event information fields.](https://assets.ai.engineer/sites/aiecode2025/sha256/351b713f004c915971da82756a752fcc10a5eeee3f16fa62d2ad0b453f2d3068.webp)

Event agenda extractor

Model *
OpenAI GPT-4.1 (via GitHub)
Settings

Prompts *
Add Prompt
Click the button below to automatically generate a prompt with your own description, or enter your prompts manually.
Generate system prompt

System prompt
You are a web exploration assistant that can navigate to websites, take screenshots, and provide summaries of web content. You'll help users explore the web by visiting sites and extracting useful information.

User prompt *
name: the name of the event
date: the date of the event
location: the location
attendees: the number of registered attendees

Answer with the following format:
name: <name>
location: <location>

Prompt
Evaluation
Run
View Code

Interface text:
azureai-samples
EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
AI_Judge_Evaluator...
data.jsonl
model_endpoint.py
README.md
AI_Judge_Evaluators...
AI_Judge_Evaluator...
AI_Judge_Evaluator...
AI_Judge_Evaluat...
data.jsonl
image1.jpg
Playground
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
Recent
New Agent
AIE
AI Engineer World's Fair
Microsoft
smol ai

### New agent examples

The presenter starts a new agent and highlights the Web Scraper example, which pairs browser automation with an MCP server.

[8:22](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=502s) [Agent builder showing a new agent and example options including a web scraper.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c74c5b3fe13738de1fda4978c12f626bd14e0b817ebb33acd29d3d8656606ed.webp)

New Agent 2025060413

Model *
Settings
OpenAI GPT-4o (via GitHub)
Loading model...
Cancel

Prompts *
Add Prompt
Click the button below to automatically generate a prompt with your own description, or enter your prompts manually.
Generate system prompt

System prompt
You are a professor in university teaching math

User prompt *
Enter the task or question for the model. Use {{variables}} for placeholders, for example: "Explain to me the {{topic}} in simple terms"

Prompt
Evaluation

Get Started with Examples
Creating, iterating and optimizing your prompts.

Web Scraper
With MCP Server
Navigate to websites, take screenshots, and summarize content using browser automation.

Second-Grade Simplifier
Make complex text easy for young learners to understand.

Dream Interpreter
Offer interpretations and insights into the symbolism of the user's dreams.

Run
View Code

Interface text:
AIE
EXPLORER
AZUREAI-SAMPLES
scenarios
evaluate
Supported_Evaluation...
Agent_Evaluation
AI_Judge_Evaluator...
evaluate_azure_ai_...
Evaluate_Azure_AI_...
evaluation_data.jsonl
log.log
README.md
response_complete...
sample.json
user_functions.py
AI_Judge_Evaluators...
__pycache__
data.jsonl
model_endpoint.py
image1.jpg
Playground
Agent (Prompt) Builder
AI_Judge_Evaluators_Quality.ipynb
AI_Judge_Evaluators_Safety_Risks_Image.ipynb
Recent
New Agent
Microsoft
smol ai
AI Engineer World's Fair

## Full transcript

[0:00](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=0s) [upbeat music] Well, um, welcome everyone.

[0:18](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=18s) Um, happy to be here, uh, today. I'm very excited. It's a very, uh, hot topic. Um, so I am Cedric Vidal, uh, Principal AI Advocate at, uh, Microsoft, and today we are going to talk about how to evaluate agents.

[0:32](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=32s) Uh, so for those of you who were in this very room the session just before, my, uh, colleagues, uh, presented, uh, red teaming, uh, which is how you create, uh, data, uh, that tries to, uh, put your AI in a bad situation and, uh, generate bad, uh, content, and try to verify that it behaves correctly.

[0:53](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=53s) Today, in this s- uh, session, we're going to, uh, look at more traditional, normal, uh, types of, uh, evaluations when you have a data set that you want to evaluate on your, uh, AI agents.

[1:07](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=67s) Um, so we're gonna see, uh, look at a bunch of things, uh, on how to make sure your AIs are safe. So, I see that people are still coming in the room.

[1:17](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=77s) Uh, it's okay. Please come in. Don't be afraid. Um, so

[1:24](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=84s) we don't want that, right? Uh, [laughs] uh, it's, uh, r- AI agents are all the rage. Uh, to be honest, every single day, like even as I was preparing this very presentation and I was trying the latest models and the latest, uh, uh, SDKs, uh, I'm always, uh, uh, amazed at the progress that those agents are making.

[1:47](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=107s) Um, and, uh, but of course, the more, um, agency we give to them, the more independent they become, uh, the more the risk of, uh, creating havoc, uh, increases.

[1:59](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=119s) So let's, uh, see how we can make sure that your AI agents behave correctly, and uh, do not, uh, create tha-that kind, uh, of mess.

[2:09](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=129s) Um, so how do you, um... Nope, sorry. Um, how do you go about, um, uh, evaluating your AI agent? Do you submit like a couple prompts to validate that the models, um, respond correctly and go, "Yeah, well, that checks.

[2:26](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=146s) That should go," and put it, uh, in production, or do you go about a more, uh, methodical approach? Uh, if you are doing the former, then I have some news for you.

[2:36](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=156s) You are, uh, in the right place. Um, you need to change something. Uh, it's not gonna, uh, work. If you're in the, uh, latter, then, um, today I have some, uh, frameworks to show to you on how to-- which might help you improve, uh, your evaluation process.

[2:53](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=173s) So, when should you start doing evaluations? Um, you may be wondering what's the, uh, evaluation, um, or when does it occur? Um, and I mean, if you have already built an app and, uh, you're asking yourself, "Should I evaluate now?"

[3:09](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=189s) well, uh, good news, I mean, or bad news, uh, you're a bit late. You should have started way earlier. Evaluation starts, uh, at the very beginning of your, uh, AI development, uh, project.

[3:21](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=201s) Um, uh, the sooner the better. So to get a b- uh, a sense of how to approach the, uh, subject of a- AI agent evaluation, we distinguish four, uh, layers.

[3:37](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=217s) Um, uh, we... First you have the model and the safety system, uh, which are platform-specific, uh, level protections, and this is built in Azure. You don't have to do, uh, anything about that when using Azure, uh, uh, models on Azure.

[3:52](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=232s) Uh, and then you have system message and grounding. So for that part, and user experience, and for that part, that's where your app design matters the most. Um, the key takeaway, the foundation model is just one part.

[4:06](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=246s) Um, real safety comes from layering, uh, smart mitigations at the application layer, and we're gonna see how to do that.

[4:16](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=256s) The first thing you should do is manu-manual model evaluation. So which model would, do you wanna use for your AI agent? Um, uh, you wanna get a clear sense of how different models would respond to a given prompt, uh, something automatic metrics can sometimes miss.

[4:35](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=275s) When you, uh, launch a batch, uh, of metrics, of evaluations on a data set, you, sometimes you have a, a big average score and you might be left wondering, "Okay, but, uh, I'm not sure exactly how it works specifically for a very specific example."

[4:53](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=293s) Before evaluating at scale, you need first to, um, uh, cherry-pick and look at specific examples. So now I'm going to demo to you how to do that in VS Code.

[5:12](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=312s) So, um, the first thing here, I'm gonna look at my history,

[5:19](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=319s) um, is that you can, uh... So in VS Code, there is a new freshly re-re- relatively new plugin, uh, called AI Toolkit, which was released, uh, at Build, I believe.

[5:30](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=330s) And oh my God, I love that plugin. Uh, before I used to go to, uh, different websites all over the web to, uh, evaluate models and compare. I mean, you had, you had, uh, GitHub Models, uh, but now you can do it right from your development environment, and if you're like me and you like to code, that's

[5:46](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=346s) where I like to do things. Um- AI Toolkit? AI Toolkit, yes. Um, and so you can ask, though, uh, I did ask it, uh, that question already. "What's a good panna cotta recipe with salted caramel butter?"

[6:00](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=360s) Which is my favorite. Uh, and then you get, uh, a pretty good response with 4.1. But what if you wanna compare with 4o, for example? So-

[6:10](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=370s) What, what's a good recipe for, uh, panna cotta, uh, with salted caramel butter.

[6:24](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=384s) And then you can see, uh, side to side how the two models will respond. Uh, 4.1 on the left and 4o on the right. And as you can see, 4.1, uh, is a major improvement in terms of, uh, throughput.

[6:39](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=399s) Uh, you're gonna get the answer much faster. Uh, when it comes to the quality, uh, of the answer, um, uh, so I looked at it ahead of the conference, and to be honest, I prefer the 4.1 answer.

[6:49](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=409s) 4o is not too bad, but I mean, 4.1 is so much faster that usually that's what you're gonna use. Um, so that's for

[7:00](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=420s) spot-checking, uh, the answer of a foundation model, um, without any customization. We, we don't have an AI agent yet. Um, then,

[7:12](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=432s) uh, you want to evaluate the whole system. So that's where, uh, we are gonna actually build an AI agent and evaluate the, the agent, uh, from a systemat- systemic approach as a whole.

[7:24](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=444s) Um, once you have selected the model, it's time to evaluate it end to end. Uh, and um, so let's jump in and sh-- let me show you how that works in VS Code.

[7:36](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=456s) So same. That same AI Toolkit, uh, extension for VS Code. Uh, wow. I mean, to be honest, I love it, uh, because now you can build an AI agent, like, super fast and evaluate it super fast too.

[7:48](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=468s) So here I created ahead, um, uh, I prepared an agent to extract agenda, uh, and event information from web pages. Um, uh, for me as an advocate, I do that kind of talks pretty often, and I need to know, uh, uh, I created, basically, an agent that helps me easily fetch information from the web and pull, um,

[8:11](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=491s) the, the names or list of, uh, of talks, of speakers and number of attendees, that kind of thing. And, um, it's super easy to do. So I'm gonna show you how to create, um, a new agent really quick.

[8:24](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=504s) Uh, and you have an example here with a web scraper, um, and it automatically generates a system prompt saying, "Hey, you are a web exploration assistant that can navigate websites."

[8:35](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=515s) Uh, it's gonna configure an MCP server, um, uh, ready to use. And, um, if I run it,

[8:44](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=524s) uh, it's gonna start, uh, the Playwright MCP server. Um, uh, the-- by default, the, uh, it uses an example domain.

[8:59](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=539s) And will extract... You can see the background, will extract information about, um, the website. Now I'm gonna switch back to the agent that I created, 'cause the one I just showed you is the, the built-in.

[9:12](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=552s) So this one, um, I created, and I'm gonna use a GPT-4.1. Um, and this one is more focused. What I want is to extract, uh, the name, date, location and atten- and number of attendees, uh, in a specific format.

[9:28](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=568s) Um, and, uh, for that website, which is a Luma event page. So run. So what I did is that I took the, the automatically generated, one of the sample, uh, AI agent that was created by, uh, AI, um, uh, Toolkit, and I customized it for my use case.

[9:49](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=589s) And here you can see the AI, uh, agent, uh, working with, uh, and piloting, um, uh, Playwright, going to the web page, extracting the information and giving me the response.

[10:03](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=603s) So the, um, the event is AI Agents and Startup Talks at GitHub. Location is, uh, GitHub headquarter in San Francisco, uh, on June eleventh. Uh, and for now, we have two hundred and sixty-nine, uh, people that registered.

[10:16](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=616s) And I hope that after doing the demo, we're gonna have more [laughs]

[10:20](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=620s) uh, because that's an event that I co-organize, um, uh, in San Francisco. Um, and, um, so now that we have, uh, spot-checked our, uh... We have built, we have customized, we have spot-checked what our agent, AI agent does for a specific, uh, input.

[10:39](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=639s) Let's see how we can evaluate it on multiple, uh, inputs. So you have, uh, uh, a, um, a tab here called, uh, Evaluation, uh, which allows you to, uh, take, uh, that AI agent previously, uh, configured and to execute it for, uh, on a data set.

[10:58](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=658s) So here I, uh... So I can type Run all.

[11:07](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=667s) And in the background, it's gonna run, uh, the agent on those inputs and give us the answer in the response column. As you can see, I had executed it, uh, before, so you can see, uh, what was the previous answer.

[11:20](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=680s) Uh, but what's cool here is that you can take that answer, have a look at it, and as you can see, we can see the, uh, the information correctly, um, extracted.

[11:29](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=689s) Uh, what's interesting is that the web page here, by the way, does not contain the number of attendees. Still, we can see here that we have a, an answer here.

[11:38](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=698s) That's very interesting because it actually went-- well, it went to the, um, Reactor, uh, page, so that event page, found the link to the Luma page, navigated to the Luma page and on the Luma page we have the number of attendees.

[11:52](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=712s) So it pulled in, it mixed the information from the Reactor event page and the Luma page, uh, to co-collect everything I needed to, in order to get my answer.

[12:03](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=723s) Okay, so that was a side note. Um, and I mean, I love it. In both cases, those are good answers. So we can, um, manually evaluate whether it's a thumbs up or a thumbs down, and then we can do a few things.

[12:18](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=738s) We can, uh, export- The data set to a JSON file. Uh, so I'm not gonna do it, but, uh, it's basically a JSON Lines file with, uh, the result of the evaluation that you can then re-inject, um, into, um, a, a, a more automated, uh, system.

[12:35](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=755s) And then once you have your mo- your agent like this, you can type view code, generate, uh, using whichever framework you prefer. Uh, OpenAI Agents is usually the one people wanna use those days.

[12:47](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=767s) Um, and then you have all configured, uh, an agent with, uh, the MCP, uh, server and, uh, boilerplate code to evaluate, uh, uh, to run, sorry, your agent. So let me close that.

[13:01](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=781s) Let me move on. Okay. So we've seen how to build and, um, manually evaluate, uh, our AI agent, uh, on the spot example, and, uh, how to run it on a batch of example locally, so a small batch.

[13:16](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=796s) Then how do you scale beyond a few samples?

[13:21](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=801s) Uh, let me move on to the next slide. Where do we get the PowerPoint presentation? Sorry, what? Do you get the, do we get the PowerPoint presentation? Yeah, sure. [laughs]

[13:32](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=812s) I can share it before. Excuse me. I'm... [laughs] I have five minutes left. Uh, we, I, yes, I will share it. Um, so, um,

[13:40](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=820s) okay, so we've seen AI Toolkit. Okay. So how do we scale, uh, beyond what I just showed? Uh, because, okay, uh, eyeballing, uh, is great to get a sense if it works, but what you wanna do is, uh, uh, go through more thorough, more wide, uh, range of checks, uh, and you want to automate this.

[13:59](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=839s) Um, so, um, well, Azure AI Foundry gives, uh, a wide set of built-in evaluators to automate and scale those evaluations. We've got AI-assisted quality checks like groundedness, fluency, coherence, perfect for measuring how well your agent performs in realistic conversations.

[14:17](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=857s) You also find classic NLP metrics, uh, F1, BLEU, ROUGE for benchmark comparisons, as well as, um, uh, a suite of AI-assisted risk and safety evaluators, and you can also customize and build your own.

[14:35](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=875s) Um, once you've spot checked, uh, the next che- the next step is to scale. Uh, and for that you, uh, you need automated evaluation, um, to measure quality at a bigger scale.

[14:48](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=888s) Uh, you can do it either in Azure AI Foundry portal or via code, and I'm going to show you how to do it via code. Um, it's important because, um, we can define what we want to measure based on our app's, uh, use and goal.

[15:01](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=901s) Um, so now demo. Crazy how twenty minutes goes fast. Um, so here is a notebook, and given the time we have, I'm not gonna execute it because it takes a bit of time.

[15:14](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=914s) Uh, but here you have the, the Python code to, um, and I'm gonna share at the end of the presentation the link to the notebook. Uh, you have the notebook that allows you to, um, programmatically, uh, use, connect to the, an Azure AI Foundry, uh, project, uh, and run those evaluation.

[15:33](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=933s) So the key function here is that you can define... So those are quality, uh, evaluators to evaluate relevance, coherence, groundedness, fluency, and similarity. And you have an evaluate, uh, function code that takes those evaluators, takes the data set that you want to evaluate, uh, and, um, bulk evaluate, um, the AI agent on all those, um, uh,

[15:56](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=956s) metrics. And the result is, uh, here. So, uh, on that data set here, uh, which is about camping, uh, like what is the capital of France, which tent is the most weatherproof, what camping table, what- whatever.

[16:12](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=972s) Um, you can see for each, um, question, uh, here, you can see, um, the result of the evaluation, which also you can configure a threshold. So it's gonna get, um, give you an answer between one and five, and depending on which threshold you configure, because depending on your application, you might, uh, want your AI agent to be

[16:32](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=992s) more or less strict, depending whether you're in the gaming industry, where usually, uh, they accept more, uh, like, um, violent content, or whether you are doing an application for kids.

[16:43](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1003s) Obviously, the threshold is not gonna be the same. Um, and so I'm gonna move on to the next. Um, so the, in this case, this, this was passing. I wa- I wanna just wanna show the next, uh, evaluator that we have.

[17:00](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1020s) Um, uh, also very cool. Now you can evaluate multimodal models mixing text and images, uh, this is very, um, important, and for multi-turn conversations. So here, um, I have an image on purpose.

[17:20](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1040s) Uh, so I tried to find a violent image, and it's hard to find something violent that you can show at a conference publicly, right? So, uh, I, I did what I could, and I spent a lot of time looking, and believe me, when you search for something violent on the web, you see things you don't wanna see.

[17:35](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1055s) Uh, and so I found that, um, and let's go straight to the end and see what it tells us. So, um, the system's response, uh, blah, blah, blah. The image actually depicts a character with numerous pins or nails protruding from the head, which is a graphic and violent depictions.

[17:54](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1074s) Uh, but what's interesting, the score is four. It's not like five. It's not the max. So it's failing, um, but it, um, like for example, if you were doing like a, like I said, a, a video game with violent content, you could increase to four and say, "Hey, four, I'm fine with it."

[18:10](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1090s) Uh, and so in order to be able to generate that kind of image.

[18:15](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1095s) Uh, and at the end, uh, what's interesting,

[18:19](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1099s) uh, and I'm gonna show you on another... Uh, okay, I'm gonna move on. Um,

[18:28](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1108s) I showed you that. Um, okay, I don't have...

[18:35](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1115s) I wanted to show you something else. Okay, you also have an evaluat- evaluator to, uh... Oh, I think I'm on time, sadly. Okay, so here's, um, some links to more information.

[18:51](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1131s) Uh, we have, um, uh, on GitHub, Azure AI Foundry discussions, uh, where you can come and ask questions about, uh, that evaluation SDK and how to build a AI agent and how to evaluate them.

[19:04](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1144s) Uh, you have the Azure AI Foundry Discord too, where you can come and discuss if you prefer Discord. Uh, and then at the very end, you have my, uh, contact, uh, information, uh, if you want to reach out for more, uh, questions.

[19:17](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1157s) Um, so yeah, very packed. Uh, sorry, a lot to say and very little time. So thank you very much. Uh, I'm here if you have more questions. [clapping]

[19:29](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1169s) How are you sharing the slides? Uh, that's a good question. Uh, I'm gonna put them on, uh, the Discord.

[19:38](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1178s) Where? The Discord. Where? On the, on the middle, you have our Discord server, so you can come on the Discord server, and I will post it there.

[19:53](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1193s) Thank you very much. [outro music]
