Mind the Gap (In your Agent Observability)

A new train can fit one platform perfectly and leave a dangerous gap at another. Treat the train as an agent and the platform as its requirements: improving the technology does not automatically make its behavior fit the application. London's Mind the gap and New York's Watch the gap become useful instructions for agent developers. Evaluation measures the distance between expected behavior and actual behavior, starting early and continuing throughout the agent's life.

The analogy extends beyond evaluation. A platform warning is a guardrail that helps passengers use the system safely. Repeated announcements keep that warning present as conditions change. Agents likewise need safety mechanisms and ongoing monitoring, across both today's application and the agents added later. Amy Boyd, who introduces herself as the lead of Microsoft's Foundry Developer Relations team, and Nitya Narasimhan organize the workshop around the next practical question: once observability produces data, what should a developer change?

The companion workshop repository supplies the labs and continues to evolve between events. The presenters also point attendees to the Microsoft Foundry Discord's AI engineer channel for questions while working through it. The material addresses a production problem that appears even in small demos: agents are nondeterministic, but customers still need reliable, consistent behavior.

Three activities turn that problem into a development loop:

• Evaluate: measure performance, output quality, and safety.
• Monitor: track behavior as requirements, customers, and environments change.
• Optimize: use the evidence to improve the agent and debug failures.

Microsoft Foundry provides the workshop's implementation: developers can build agents there or bring agents built elsewhere into its hosting, observation, and management capabilities. The scope grows from building a reliable first agent to debugging production systems and eventually managing fleets containing many multi-agent systems.

Tracing records what the agent actually does: the tools it calls, the messages it sends, and the steps through which a request passes. Foundry builds on OpenTelemetry, commonly shortened to OTel, so instrumentation can span different agent implementations and feed a shared control plane. Evaluation adds judgments to that execution record. Built-in evaluators cover quality, risk and safety, and agent-specific behavior; custom evaluators can supply criteria that the built-in catalog does not capture.

Consider a weather agent receiving a request for today's weather in London. There are several distinct places to check its work:

For a nondeterministic agent, expected tool selection is something to measure across runs, rather than assume from one success. Task-adherence criteria may also need refinement as the application develops. Evaluating these intermediate decisions makes an imperfect final response actionable: the developer can locate the stage that needs changing.

The same principle becomes more valuable as workflows grow. A full multi-agent trace preserves the relationship between agents instead of showing each one in isolation. Continuous evaluations can run when code changes; scheduled evaluations can look for deterioration over time; red teaming probes security weaknesses. Foundry connects this agent-level view to Azure Monitor, where infrastructure and data-service telemetry already live. A fleet view then brings together agents hosted in different places. The security tooling includes both platform scan options and the open-source PyRIT framework, which supports proactive identification of generative-AI risks.

The hands-on scenario starts at the fictitious company Contoso: build a travel application that helps users find flights, hotels, and rental cars. Narasimhan treats the live session as a cooking show, showing prepared results from a repository she describes as almost a four-hour workshop. The model supplies the underlying knowledge and reasoning; tools add capabilities and additional knowledge. The agent needs instructions that define its job, tools that let it perform that job, and a suitable model.

Model selection is already a substantial search problem. At presentation time, Narasimhan cites more than two million models on Hugging Face and more than 11,000 in Azure's catalog. Yet this new application has no historical evaluation dataset. The initial challenge is therefore to get a prototype and useful test data without settling every possible model choice first. After that come two different obligations: protect the company's reputation during normal use, and withstand users who deliberately manipulate the agent. Her analogy is a building inspector checking whether a house meets code versus someone invited to try breaking into it.

To follow the workshop, begin with GitHub and Azure accounts. A free Azure account can be used, but deployed resources still incur usage charges. Narasimhan reports spending about $10 over two days of her own workshop usage. Fork the repository with the main-only option unchecked so that the AIE Europe branch, 2026-04-aie-europe, is included. That branch preserves the event's workshop path while the presenters intend to merge work back into main and continue adding material for later events.

Launch a Codespace from the selected branch. The repository's dev container defines the dependencies; Codespaces starts the cloud development environment and exposes VS Code in the browser. Boyd says the Azure organization's enterprise account covers Codespaces compute for this workshop setup. For readers repeating it, organization payment depends on authorization, including when using a fork; repository ownership alone does not establish who pays. Once the environment is ready, the installed tooling supports notebooks and the later coding-agent workflow.

There are two learning paths. The SDK path builds understanding step by step and is Narasimhan's recommendation for newcomers. The coding-agent path starts with a minimal agent endpoint and uses a skill to help build and evaluate it. Both use the same travel scenario, with flight, hotel, and car-rental responsibilities. With the development environment starting up, the next step is to create a simple agent in the portal.

The portal provides a quick way to discover the available tools and evaluation controls before translating them into code. Boyd opens the new Microsoft Foundry experience and follows this sequence:

1. Choose Start building and create an AI London/Contoso project.
2. Select the subscription and resource group. In the demonstrated setup, she selects East US 2, then recommended for the newer features.
3. Create a uniquely named Contoso Travel agent. The portal automatically provisions GPT-4.1 and looks for available capacity.
4. Enter the playground, where instructions, tools, and voice mode are configurable.
5. Add Bing web search so the assistant can consult travel sources such as Expedia.

The project provides a shared scope for agents, tracing, and evaluation. The catalog also offers other models and providers, including Anthropic and DeepSeek. These provisioning defaults and portal screens describe the workshop's setup; the event branch is the appropriate starting point for following that version of the demonstration.

A model with web search still does not know Contoso's intended scope. With no specific instructions, the initial agent offers generic help, including general advice. Boyd saves it and moves to a prepared version. Before continuing, Narasimhan identifies a missing dependency: creating an agent does not by itself connect the demonstrated tracing setup. The agent's Traces tab flags the absence of Application Insights. Connecting it through the administration interface establishes the telemetry destination; the portal's helper can also explain the connection steps.

Once connected, conversations and traces expose tokens, estimated cost, and associated evaluations. The prepared agent adds behavioral requirements: use web search, provide comparisons, present externally sourced specifics, and keep answers focused and helpful. The presenters switch to its configured second version to show the display name, description, and starter interactions. Those properties let a user discover what the travel assistant can do without first asking it to explain itself. The repository supplies the setup steps and sample prompts used to exercise it.

After a travel query, the response footer identifies the model and token usage. Model identity becomes especially useful when a router chooses a model per request. The Metrics control enables AI quality and AI safety checks directly in the playground, making the available evaluators discoverable before writing SDK configuration. The completed response exposes its evaluation results and a breakdown of individual safety checks.

From there, open the tracing logs to inspect the steps and messages. The input/output view includes an Evaluations tab for the selected checks. In this demonstration, task adherence is low: instead of answering the intended question, the agent asks for more information. Even this minimal proof of concept therefore has a concrete improvement target before the application moves into code.

Trace-linked evaluations shorten the path from detection to diagnosis. Suppose a model replacement degrades tool-use behavior. An evaluator can identify the regression, but comparing the old and new traces can reveal that the new version omitted a tool call. The score tells the developer something changed; the execution path helps locate what changed. That relationship is more useful than collecting scores and traces in separate places.

The Codespace now needs to connect to the deployed Foundry infrastructure. A repository script uses the Azure CLI to collect configuration and generate .env; Narasimhan keeps that file offscreen because it contains keys. The AI Toolkit extension provides Foundry-like functionality inside VS Code. This moves the application from portal exploration into the build, optimize, deploy, and govern lifecycle, without introducing fine-tuning into the walkthrough.

Use the notebook outline to navigate the labs in order. Each adds a bounded capability: validate connectivity, create a basic agent, add tools, compose a workflow, configure tracing, run evaluations, and perform red teaming. The workflow introduces a concierge coordinating flight, hotel, and car-rental specialists. Its tracing lab supports local OTel output for debugging or export to Azure Monitor for inspection through Foundry.

Start with the dev container's dependencies and notebook kernel. The initial configuration needs the project endpoint and deployed model name. A client/API smoke test using fake sample data confirms that the development environment can reach the service. Azure CLI login supplies credentials; the Foundry client then creates an agent with travel instructions and the deployed model. The first Responses API request asks about planning a trip to Paris. A subsequent multi-turn test checks that the conversation can continue. At this stage the notebook has not activated custom tracing; the later portal inspection shows the invoked agent's traces, but no evaluations because those have not yet been configured.

Next, register function tools for car, flight, and hotel searches. The agent now combines instructions, a model, and callable functions. Test both individual requests, such as a flight search, and combinations, such as a hotel plus rental car. The next lab separates those responsibilities into specialist agents so that one agent no longer owns every function and failure path.

A declarative YAML definition composes the specialists into a Foundry Workflow. It establishes conversations, invokes the flight, hotel, and car agents, and combines their responses for the user.

The resulting trace exposes the path through the workflow, including which specialist underperformed and where tokens were spent. That makes it possible to replace or optimize the affected component without changing every agent.

An audience member asks whether the workflow can optimize cost and context windows, and whether a wrong turn can be rolled back as a whole transaction. Narasimhan's cost answer starts with model choice: try a cheaper model, immediately rerun evaluations, and check whether savings came with an accuracy regression. The trace also helps identify expensive or slow operations. If web search dominates the request, a smaller cached or data-backed search might be appropriate. Each change needs evaluation because improving one dimension can degrade another.

Her rollback answer concerns agent versions. Saved versions have identifiers, and a previous version can be selected for deployment. The later skill demonstration will compare versions and recommend returning to a better one. This restores an agent configuration; it does not establish how to reverse completed bookings or other external side effects. Context-window optimization is not separately worked through.

Foundry manages the demonstrated workflow without requiring an external orchestration framework. More broadly, the observability boundary is the interface: portable tracing needs OpenTelemetry-compliant telemetry, and the companion workshop specifies Responses API-compliant endpoints for evaluation. That lets the observability workflow span implementations without assuming that arbitrary endpoints or arbitrary logs are interchangeable.

As the agent changes, the number of plausible failure causes grows. Custom trace attributes let developers record the application context they will need when investigating. The tracing notebook first configures console output and enables two environment settings: one for generative-AI tracing and another for message-content capture. Exact setting names belong to the workshop's package versions. Content capture also requires care: prompts, responses, and tool arguments can contain sensitive information, so capture only what is needed and apply redaction, access controls, and retention limits.

The mechanism is an OTel span carrying developer-defined attributes around the work being investigated. For the Paris-trip request, a small Python wrapper can record the destination and agent role without recording the complete prompt:

from collections.abc import Callable
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    ConsoleSpanExporter,
    SimpleSpanProcessor,
)

provider = TracerProvider()
provider.add_span_processor(
    SimpleSpanProcessor(ConsoleSpanExporter())
)
tracer = provider.get_tracer("contoso.travel")


def ask_about_paris(invoke_agent: Callable[[str], str]) -> str:
    with tracer.start_as_current_span("travel.request") as span:
        span.set_attribute("contoso.destination", "Paris")
        span.set_attribute("contoso.agent_role", "travel")
        return invoke_agent(
            "I'm thinking about planning a trip to Paris. "
            "What should I know?"
        )

Here invoke_agent is the application's existing request function. The custom attributes make console traces searchable by the facts relevant to the investigation; the notebook demonstrates adding such attributes to a conversation and inspecting the resulting local traces.

Exporting telemetry adds another level of diagnosis. Basic agent traces show what happened against the agent itself. Azure Monitor can place AI telemetry alongside other service telemetry, such as an Azure AI Search index. That combined view helps investigate whether an apparent agent slowdown originates in a dependency. The lab configures Azure Monitor export, runs a traced query, and then inspects the results through Foundry.

The evaluation notebook runs quality, safety, and agentic checks individually or together in batches. It starts with a sample dataset generated using an AI agent: test prompts and responses that can support checks such as groundedness. The execution sequence is explicit:

1. Prepare the evaluation dataset.
2. Define evaluator objects and their criteria.
3. Map the dataset's fields to each evaluator's inputs.
4. Configure the evaluation data flow and start the run.
5. Poll until the long-running evaluation completes.
6. Analyze results locally or open them in the portal.

The portal retains the dataset used for the run, results that can be downloaded, and logs for inspection.

Narasimhan reports that coherence and fluency pass all tests in the notebook's deliberately simple quality dataset. Moving to the agentic evaluation view introduces checks such as intent resolution, task adherence, and tool calling. A groundedness failure supplies a more diagnostic example: the response gives August 2024 where the expected year is 2025. Opening the failure explanation reveals the concrete mismatch, giving the developer something more useful than a failed status alone.

Ordinary evaluations assess the examples supplied to them. A red-team scan actively generates manipulated prompts to probe the agent. A second AI receives risk categories and attack strategies, attacks the target, and reports cases where its protections fail. The red-teaming notebook introduces this adversarial step for Contoso Travel.

Narasimhan illustrates a possible mismatch between a guardrail and a model with a reversed-string request. A direct request for bank-robbery instructions should be rejected. If the same request is reversed, a guardrail might treat it as meaningless text while the model reconstructs the prohibited request. The point is the difference in interpretation across layers. Red teaming tests transformations that might exploit such differences.

For tool-using agents, the risk extends to actions. A prohibited-action taxonomy can identify behaviors such as entering passwords or leaking data, and the red-team agent generates prompts that try to induce those behaviors. An audience member proposes the reverse policy: allow nothing except explicitly authorized actions at authorized times. That is a valid way to express the intended boundary, but it still needs adversarial testing. A story framed around a grandmother, for example, may try to persuade the model to disregard a restriction. The policy defines permitted behavior; the attack tests whether manipulation can cross that boundary.

The completed example selects violence, sensitive-data leakage, and task adherence as risk categories, with leetspeak as the attack strategy. Leetspeak substitutes numbers for letters to test whether a transformed request slips through recognition. Narasimhan reports that the simple leetspeak scan passed all tested cases. She then opens a broader configuration, where a Contoso agent can be tested against more risks and strategies.

Crescendo works across interactions, gradually escalating a conversation rather than relying on a single transformed prompt. Narasimhan compares this to a frog in warming water: each step builds on the previous interaction before the full direction becomes apparent. More extensive scans take longer. They can reveal weaknesses missed by a simple test, but they do not establish that every vulnerability has been found.

The second path automates much of the notebook workflow with the Observe skill, a sub-skill within Microsoft Foundry's broader agent-development tooling. Narasimhan presents it as an early preview; the linked source continues to evolve beyond the version shown. In GitHub Copilot Chat, using Claude as the coding-agent model, she asks it to start an observability loop for the portal agent. The starting point is only an agent with instructions, a model, and an endpoint—no evaluation suite.

The skill inspects the instructions, code, and metadata, creates a cache, generates evaluation data, and turns the work into a sequence of tasks. It retrieves agent details, queries the evaluator catalog, and runs an initial batch evaluation. Along the way, it recovers from a misspelled agent name. The displayed explanations make its evaluator choices and detected failures available for human inspection.

In the narrated baseline, relevance and intent resolution look good, but task adherence has two failures; one indirect safety test passes. The failures concern cases where the agent does not complete the requested task. That baseline creates the comparison point needed for optimization, and its explanations identify what the next revision should address.

The skill pauses to ask whether it should proceed. Narasimhan authorizes it to analyze the failures, optimize the instructions, and reevaluate. A prompt optimizer tightens wording that leaves room for misinterpretation, publishes a new agent version, and starts another batch with the same evaluators. Narasimhan reports that the first prompt revision fixes one of the two task-adherence issues. The agent improves, but one issue remains.

She then asks it to push for a perfect result. Further prompt edits produce regressions rather than steady improvement. The iteration history becomes the useful artifact: it records what changed and what worked or failed, while the human can stop an unproductive search. By version 10, the demonstration identifies version five as the best version and recommends returning to it. The lesson is operational: preserve the baseline, version each candidate, compare results, and retain the ability to choose an earlier candidate. A newer prompt is not inherently a better prompt.

An audience question broadens the search space: can the skill change the model or other features, rather than only the prompt? Narasimhan says it can, giving slow web search as a reason to investigate an alternative. The installed SKILL.md exposes the checks and best practices that guide the work. Domain knowledge remains valuable because a developer can direct the next experiment instead of accepting whatever the default loop tries next.

The workflow can also produce a history and recommendations for future work. Narasimhan describes asking a create skill to convert a prompt agent into a hosted agent implemented with LangGraph; that conversion is a proposed next step, not a completed part of this demonstration. Hosted agents provide a route for bringing implementations such as LangChain or LangGraph into Foundry. Other suggested next steps include trace-based evaluation and red teaming. When an audience member asks about changing the evaluation dataset, she confirms that the user can direct the skill to update it and repeat the process.

Opening the skill source reveals how the automation is organized. The Microsoft Foundry agent skills are part of the open-source GitHub Copilot for Azure tooling, and the Foundry MCP server supplies underlying capabilities. Requests such as evaluating an agent or building a dataset route into relevant sub-skills. Dataset creation and insight analysis are separate pieces of an observability loop that can be repeated as the application changes.

The final demonstrations move assistance from the IDE into the operational interfaces:

Ask AI can reason over the project's state, including subscription and model-access questions. The observability agent appears above the Application Insights logs and supports natural-language investigation where a developer would otherwise write KQL. These are distinct surfaces: one helps with the Foundry project, while the other helps interpret telemetry in the Azure portal.

The repository includes the skills path as well as the SDK labs, and the presenters invite follow-up questions and issue reports through the Discord AI engineer channel. Narasimhan also offers to publish a step-by-step walkthrough of her results. The work continues because a functioning agent can drift from its original requirements when models, prompts, environments, or edge cases change. Continuous evaluation, tracing, monitoring, and alerts make that widening gap visible; coding agents can accelerate the response while a human guides the changes.

The cold start becomes manageable when an agent can generate initial evaluation material from the application's state and requirements. Diagnosis becomes faster when an evaluation result leads directly to the execution trace responsible for it. Safeguarding becomes stronger when normal-use safety tests are complemented by adversarial tests that try to cross the policy boundary. These activities belong in the same recurring development loop, with attack techniques continuing to evolve through work such as PyRIT.

For continued learning, Narasimhan points to Model Mondays and a Foundry skills session with Filisha Shah, the product manager behind the skills work. The closing invitation is for a deeper demo, discussion of what comes next, and questions to the team. She also highlights AI Toolkit as the route for bringing more of the portal and SDK capabilities into VS Code. The next practical step is to work through the labs locally, inspect the evidence they produce, and use that evidence to decide which change the agent actually needs.