Production Evals For Agentic AI Systems

Hey everyone. My name is Nishant Gupta, and I'm a software engineering tech lead at Meta working on building the training and inference infrastructure for the Meta Superintelligence Lab and their infrastructure organization.

Today we are going to be talking about production evals for agentic systems.

When most people hear the word evaluation, they think about benchmarks. A model scores ninety percent on a benchmark, a new version scores ninety-two percent, the team celebrates. But agentic systems have fundamentally changed what the evaluation means.

Today, the systems don't simply generate answers. They plan, they call tools, they retrieve information, they execute workflows, they interact with the production infrastructure. The question is no longer: Did the model generate the right answer?

The question is: Did the system behave correctly? Today, I would like to discuss how evaluation is evolving from model benchmarking into production infrastructure.

This is the problem almost every AI organization is encountering today, offline benchmarks continue improving, yet production reliability often remains unpredictable. Why is that? Because benchmarks measure model capability. Production measures system behavior.

A benchmark doesn't capture tool failure, API outage, context changes, user variability, long-running workflows. And as systems become more autonomous, the gap between the benchmark performance and production performance grows.

The result is what many teams experience today. High benchmark scores, as you can see, but unreliable production behavior.

Traditional LLM evaluation focus on outputs, but we should ask the question: Did the model produce the correct answer? Agentic systems force us to ask a different question: Did the system behave correctly?

Behavior includes planning quality, tool usage, execution, workflow execution, recovery from failures, decision-making. In other words, we are moving from evaluating answers to evaluating workflows, and that requires fundamentally different evaluation architectures.

Many teams still think hallucinations are the primary AI failure modes. In production, they are often just one category. Agentic systems introduce an entire hierarchy of failure modes. At the very foundation, the memory failures, retrieval failures, safety failures.

As you go up, you have to think about reasoning mistakes, poor planning, incorrect tool execution. At the highest layer, you have to think about multi-agent coordination failures. And this is why evaluating only model output misses the most production risks we observe.

One of the most useful mindset shifts is to stop thinking like researchers and start thinking like a SRE or a production engineer. SREs don't measure success using accuracy, they measure reliability, availability, latency, cost recovery, and agentic systems require the same approach.

The goal is not maximizing the benchmark scores, the goal is to maximize dependable outcomes. Reliability becomes the North Star metric. Accuracy becomes the only input.

In this pyramid is how I think-- personally think about modern AI evaluation systems. At the bottom, you can see there are benchmarks. They're useful, they're scalable, they're reputable, but their operational value is limited.

In the middle, there are scenario-based evaluations. These simulate realistic workflows. And at the very top, you see production telemetry. This is where the highest value evaluation signals come from.

The surprising insight is that the most evaluation data often comes from real users interacting with real systems.

Now let's talk about offline evals. So offline evaluation still matters, but the methodology changes. Instead of evaluating prompts, we evaluate scenarios. For example, a customer support workflow, a code generation workflow, a research workflow.

The agent operates inside that simulated environment. We measure the task completion rate, tool correctness, planning quality, resource usage, which is-- which becomes exponentially high at high scale. The key takeaway, agent evaluation should be scenario-driven, not prompt-driven.

Once a system reaches production, every interaction becomes a signal. This is one of the biggest shifts in evaluation thinking. Production traffic is no longer just traffic, it becomes evaluation data.

We collect execution traces, user outcomes, escalations, failures, feedback signals. Production is the largest and the most representative evaluation data any organization will ever have.

Now many organizations view humans as fallback systems. I think that's a wrong framing. Humans are the evaluators. They provide signals that automated systems cannot. They assess correctness, trust, usefulness, safety.

These signals become really critical for calibrating evaluation pipelines and identifying blind spots in automated metrics. The most successful systems combine automated evaluation with targeted human review.

Now agent systems drift constantly. Model changes. You have a new version every couple of weeks or months. The prompts can change, tools can change, user behavior can change. The challenge is that no longer a single change appear catastrophic.

Reliability slowly degrades, success rate declines, escalation increases, tool failure rises. Without continuous evaluation, teams often don't discover drift until users complain. Continuous bec-- monitoring becomes essential.

Observability and evaluation are inseparable. Inseparable. To evaluate an agent, we need visibility into the reasoning paths, the tool calls, the memory access, execution timelines, the state transitions, as you can see here in this chart.

Traditional logs are not sufficient. We need detailed traces, just like with any

deep nested microservice architecture for any s- application or service we're talking about. Agent traces become the equivalent of distributed tracing for autonomous workflows. Without observability, evaluation becomes the guesswork.

Let's talk about the continuous evaluation loop because evaluation is an always running service, not a testing phase.

Historically, evaluation always happened before deployment, but now evaluation continues after deployment. Telemetry identifies issues, as you can see in A. Human reviews the edge cases. Feedback improves the datasets.

Offline scenarios validate updates. The loop never stops. Evaluation is no longer just a phase, it's an operational capability.

Now, this is probably the most important slide in this presentation. Every metric shown here maps directly to a business outcome. Task completion measures value delivered. Tool success measures operational reliability.

Escalation rate measures human burden. Safety evaluations measure risk exposure. Latency affects user experience. Cost determines scalability. Recovery rate reflects resilience. And notice-- but notice that accuracy is missing. It's not because accuracy doesn't matter, but because business success depends on much more than just accuracy.

Now, this is the architecture where the industry is heading more or less. Evaluation becomes part of a control plane, not a separate tool, not an offline process. The control plane continuously, which observes the systems, collects telemetry, runs simulations, coordinates human review, and the execution plane performs the work.

The control plane measures and governs the behavior, and this separation is becoming a foundational pattern for production AI systems.

Now let's summarize the key lessons. First, benchmark remains necessary, but they are insufficient. Second, agent systems must be evaluated as workflows, not individual outputs. Third, production telemetry is the most important evaluation signal.

Fourth, reliability ultimately matters more than raw model accuracy. And finally, evaluation is becoming the infrastructure. Not testing, not QA, infrastructure. This is the shift every organization building agentic AI will eventually need to make.