LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize AI

How do you make AI work once you move beyond building an agent to operating it? Dat Ngo approaches that question as an AI architect at Arize AI, working with enterprises on observability, evaluation, and experimentation. Ngo estimates that he used between 100 billion and one trillion tokens in the previous year. That is personal context, not a benchmark; his broader vantage point comes from seeing what teams build, where they struggle, and how they attempt to fix it.

Agent development still relies on familiar software engineering patterns, but nondeterminism complicates the feedback loop. Observability asks what happened inside the agent or harness. Evaluation turns that behavior into a signal about quality. Experimentation tests whether a change improves the system. A change that fixes the example in front of you can introduce regressions elsewhere—Ngo illustrates this with a fix that creates two or three new problems. The improvement process therefore needs evidence about more than the case that motivated the change.

Arize uses OpenTelemetry as a common foundation across models, frameworks, SDKs, and agent harnesses. For supported integrations, Ngo describes auto-instrumentation as adding one line that observes framework or SDK activity and produces traces and spans. That shorthand assumes the surrounding setup; custom instrumentation and telemetry export still need configuration. The resulting records expose the calls an agent actually made.

Runtime telemetry is the audit record of an agent’s chosen execution. Source code describes the system, but it does not tell you which route a nondeterministic agent took on a particular run. A trace makes that route inspectable. The demonstrated trace tree shows Financial_Data_Agent with nested LangGraph, model, and tool calls beside an input/output panel: the hierarchy and the data are available together.

A trace is only one layer. Ngo also points to sessions and mentions Anthropic’s Managed Agents work as related reading. Sessions capture conversational back-and-forth and state across runs. At that level, an enterprise stakeholder may care less about each individual tool call than whether the user was satisfied and whether all their questions were answered. Those outcome questions require a wider view than one invocation.

In Arize AX, the next step is to aggregate executions. A single trace answers what happened once; a distributional view shows the branches and loops taken across observed, instrumented runs. It can reveal what share of traffic follows each branch and whether a component on one branch contributes substantial latency. This is a view of recorded behavior, not an enumeration of every theoretically possible path.

Evaluation signals become more useful when attached to those trajectories. Suppose one route performs well while another consistently produces worse evaluations. Ngo’s example identifies an ordering error: the agent calls B before A, even though B depends on A. The calls themselves are not enough to explain success; their order matters. The proposed correction is to add context telling the model that A must happen before B. That is a change to test, not evidence that the issue has already been resolved.

Customizable real-time analytics complement these detailed views. Teams can inspect the individual execution, the conversation, the distribution of paths, and the operational picture their stakeholders need. Observability means being able to move between those layers rather than treating a trace viewer as the whole system.

Once behavior is visible, evaluation must establish what it means. Ngo separates signal into five sources:

• LLM-as-a-judge: A model assesses quality, though making that assessment dependable can be more complex than the basic idea suggests.
• Human feedback: End users and other reviewers provide evidence about the experience the system delivers.
• Golden datasets: Trusted domain labels establish judgments that a model judge can be tuned to approximate.
• Deterministic checks: Logic tests properties such as valid JSON, schema compliance, and required non-null fields.
• Business metrics: Outcomes track whether the application makes money, saves money, or saves time.

These sources serve different purposes. A model judgment is not a substitute for every structural check or business outcome.

Evaluation brings two groups together. Engineers and developers know how to build frameworks and automate work. Subject matter experts and product managers know what the AI experience should be: which domain expectations matter, how prompts should express them, and which evaluations are meaningful. The division of work should let engineers build while domain experts shape the behavior being assessed.

Arize’s interface supports that collaboration by letting nontechnical users select an evaluation model, start with a template, or customize an evaluator. Technical users can attach and run evaluations programmatically. Both routes expose the evaluation workflow to the people responsible for its criteria and its execution.

An agent or harness contains components that may be invoked deterministically or selected by a model. Choosing an evaluation method is therefore only part of the design. You must also choose the scope of evidence it will inspect.

A multi-span evaluation of agent handoffs needs data from the participating agents; one isolated output cannot show the exchange. A trajectory evaluation examines the sequence of work. A session evaluation zooms out to the evolving conversation. Evaluation flavor and evaluation scope are separate choices.

The goal is not to evaluate everything that can be evaluated. Each additional check has a cost. Ngo recommends finding the minimum set that gives enough signal to determine whether the application behaves as intended. Granularity is useful when it answers a question; exhaustiveness is not a goal by itself.

The next step converts observations into a repeatable test surface:

1. Collect examples. Find traces where the signal is poor or something is missing, then add those cases to a dataset. If traces are unavailable, upload input/output pairs directly.
2. Run experiments over the dataset. Its rows provide the examples against which changes can be assessed.
3. Test changes to the system. Vary prompts, models, orchestration, or configuration through the UI or programmatically.

This gives a proposed fix a broader test than the single example that prompted it—the practical response to the earlier warning about regressions.

Dashboards make the process visible, but teams do not necessarily want to perform every investigation manually. Ngo describes exposing platform primitives through command-line interfaces, tools, and opinionated skills so Claude Code or Codex can work with them from the coding environment. The Arize AX CLI is one entry point; its current command surface should not be treated as an exact reconstruction of the interface shown in the recording. External AI systems can also call Arize’s built-in AI system.

That system, Alyx, accepts an application-level question about whether anything is wrong. With access to telemetry and the platform’s hooks, it can plan and run investigation tasks. Ngo presents this as a step toward automating the full cycle of observability, evaluation, experimentation, and improvement.

The longer-term ambition goes further than automating existing buttons. Ngo wants trace context to inform which evaluations should exist, with changed application behavior prompting new evaluations. During the demonstration, he describes Alyx identifying high latency and detected errors. Those are investigation findings; the recording does not establish a completed autonomous fix-and-retest cycle. Automatic evaluation selection and a system that continually improves itself remain the stated goal.

The walkthrough closes with two product entry points. Arize Phoenix is open source and aimed at engineering-first users. It can run locally in a single container without requiring Kubernetes. Arize AX is positioned for large enterprises; Ngo names Uber, Booking, and Reddit as enterprise examples. Phoenix’s local deployment offers a way to begin working with the observability and evaluation workflow without first introducing a cluster.