Shipping AI That Works: An Evaluation Framework for PMs

How do you get an AI prototype into production once it contains an LLM or an agent? Generating a convincing interface is increasingly easy; knowing whether its behavior deserves users’ trust is a different problem. Aman Khan approaches that problem through experience building evaluation systems at Cruise, working on Spotify’s machine-learning platform, and then developing evaluation tools at Arize. The application changes—from self-driving cars to recommendations to agents—but the need to inspect and test behavior remains.

The confidence curve is familiar: first learn to use AI, then use it to produce a prototype more concrete than a requirements document, then encounter the uncertainty of shipping it. For a product manager, that last step raises the bar for what to bring to an engineering team. A working demonstration needs a definition of acceptable behavior and evidence that the system meets it.

Khan points to OpenAI and Anthropic product leaders acknowledging hallucinations at Lenny’s conference, alongside other industry advocates treating evaluation as a competitive advantage. His accompanying estimate that the two providers represent probably 95% of the LLM market is an informal estimate with no defined market or measurement period. The useful premise is simpler: even the model suppliers tell application builders to evaluate their outputs. Autonomous vehicles such as Waymo make the same challenge visible in the physical world—an AI system must work beyond a convincing demonstration.

Evaluation resembles software testing, but an agent introduces variation in both outputs and execution paths. A deterministic arithmetic test expects 1 + 1 == 2; a suggestible language model might agree with a user insisting that the answer is three. An agent can also choose among tools and intermediate steps. Meanwhile, useful generation requires flexibility, so treating every departure from a fixed answer as failure would eliminate some of the behavior the product needs. Enterprise agents add another dependency: their behavior depends on the organization’s data, which must also shape their evaluations.

For an LLM-as-a-judge evaluation, Khan divides the prompt into four ingredients:

The labels are only useful when their meanings are explicit. A business-specific evaluator follows the same structure, replacing toxicity with a criterion that matters to the application.

The workshop recommends asking for a meaningful text label and mapping it to a number afterward, rather than asking for an undefined score from one to five. Khan attributes unreliable direct scoring to how models represent numbers and mentions supporting research, but the recording does not establish that explanation as a general rule. The practical distinction is between a defined rubric and an ambiguous number. A label such as toxic can have a clear interpretation before the application maps it to a score.

There are three complementary ways to evaluate:

• Human annotation: People, including subject-matter experts, decide whether the output meets the intended standard.
• Code checks: Deterministic logic checks properties that can be expressed directly.
• LLM judges: A model applies a semantic rubric across many outputs.

The workshop concentrates on the third approach because it can extend human-style classification to larger volumes of data. That makes the judge another component whose behavior needs checking.

Fast prototyping with tools such as Bolt or Lovable can end at a familiar judgment: the output looks good enough. Khan calls the next step “thrive coding”—keeping that speed while using data to develop confidence. The workshop itself retains the rough edges of a prototype: its walkthrough and repositories were shared through the workshop-ai-pm Slack channel, and last-minute dependency changes could prevent participants from running the application. The exercise is to build, inspect, and evaluate the system, rather than assume that a generated application already works reliably.

The planner began as a CrewAI tracing notebook. Khan gave it to Cursor and asked for a UI workflow using LangGraph instead. His working definition of an agent is an LLM combined with a tool to perform an action. Here, a form supplies destination, duration, budget, interests, and travel style. Such structured inputs could support retrieval, RAG, or tool calls behind the interface, although those capabilities are motivations for the design rather than demonstrated retrieval behavior.

The live request is concrete: Tokyo for seven days, a $1,000 budget, an interest in food, and an adventurous travel style. The application returns an itinerary organized into morning and afternoon activities. Budget accounting matters because the activities must fit the requested spending limit, and an interest such as sake tasting could alter the recommendations. The visible success is that an itinerary appears. Establishing that its arithmetic, recommendations, and underlying agents are correct requires further inspection.

To look beneath the interface, Khan opens Arize. He also identifies Phoenix as an open-source option with overlapping setup and evaluation workflows, while positioning Arize for scale, security, and support. The latest live trace is unavailable, so the inspection uses an earlier run. A trace records the request’s inputs, outputs, and metadata; its nested work explains how the application arrived at a response.

Instrumentation turns the application’s function calls into an inspectable structure. Khan describes giving Cursor the documentation and asking it to add that instrumentation. The resulting planner has three parallel branches—a budget agent, a local-experiences agent, and a research agent—whose outputs feed a fourth, itinerary-generating agent. The final response therefore depends on intermediate results that a user never sees in the form.

Each span represents a timed unit of work. The view distinguishes agent work, tool work, and LLM calls, so it can answer both what happened and how long each part took. For a PM, the useful questions are architectural: What does our agent actually look like? Where do the outputs of each branch go? Which component should change if the final itinerary is poor? A graph or trace hierarchy makes those questions concrete enough to discuss with engineers.

An earlier itinerary for Marrakesh exposes a product problem: the response is long, difficult to scan, and, to Khan, conspicuously machine-written. He imports the traced LLM call into a prompt playground. The import brings the prompt template together with destination, duration, travel style, upstream agent outputs, and the final generation. This is more useful than copying a prompt into an empty editor because it preserves the context that produced the behavior being investigated.

With data and prompt together, the PM can change a variable or revise an instruction and inspect the new result. That also changes the ownership conversation. If the PM is accountable for the final experience, prompt wording is part of the product specification they should be able to influence. Whether the PM or engineer edits it is a team decision; the workflow should make the behavior accessible to both.

An audience question identifies the boundary: importing an LLM span does not automatically replay the tools surrounding it. Tool selection needs its own evaluation, and a system-level test must include the relevant tool behavior. Khan says the product offers tool-porting and tool-call evaluation workflows, but this example does not exercise them. A successful prompt replay establishes behavior for that call, not for the entire agent.

Khan saves a version named AI Eng prompt, duplicates it, and switches to GPT-4.1 mini from a baseline he calls “4.0”; the exact baseline model identifier is not established. He also changes the instructions: keep the response within 500 characters and offer a discount in exchange for an email address. In a controlled experiment, change one variable at a time. This live exercise deliberately changes several to make the effects visible.

The replay uses an existing run. The output still seems too long, and its discount wording asks the user to email the system for a 10% discount code. It has picked up part of the requirement without necessarily expressing the intended interaction. Looking at one response and calling it good or bad cannot establish that the prompt is suitable for hundreds or thousands of users. The next step is to repeat the comparison across examples.

A dataset turns an isolated failure into a repeatable development input. Khan proposes moving from one example to tens or hundreds, then clarifies the origin of the workshop data: what he has been calling production data is synthetic. A Cursor-written script sent requests through the same server and generated 15 itineraries before the session. He selects itinerary spans and uses Add to dataset to show how recorded calls become reusable examples.

The demonstrated setup uses an Arize API key and space ID from Account Settings for instrumentation, plus an OpenAI key for the playground. Khan then reuses a prepared dataset. The evaluation scope is deliberately narrow: the itinerary generation span, rather than the agent’s router, function calls, or memory. Think of the dataset as a table or CSV containing the examples needed to reproduce that call. Human thumbs-up/down judgments and subject-matter-expert labels can begin in a spreadsheet; the important first step is actually examining the data.

The dataset comparison follows a straightforward procedure:

1. Save the original prompt as a version the team can reference later.
2. Duplicate it and apply the proposed changes.
3. Run both versions against the same dataset.
4. Retain their outputs as experiments for evaluation.

Here, prompt A is unchanged and prompt B contains the edits. The comparison runs across 12 examples with destination, duration, travel style, and upstream agent outputs. It is an offline comparison of itinerary-generation prompts, rather than a randomized experiment with live users.

Changing an upstream agent requires another layer of testing. First evaluate that component against its own requirements; then replay the downstream chain to see how its changed output affects later calls. At the time of the recording, the demonstrated platform workflow handles one prompt, and prompt-chain replay is announced as forthcoming. Reusing stored upstream outputs is useful for isolating the itinerary prompt, but it cannot reveal the consequences of changing the upstream agents themselves.

For the dataset run, Khan tightens a 1,000-character maximum to 500, requests a very friendly tone, and explicitly asks the model to request the user’s email while offering a discount. The new outputs include emojis and casual language. Khan observes that the constrained outputs finish sooner, while the unconstrained comparison averages about 32 seconds in the live run. Because the demonstration changes multiple prompt properties and also explores a model change, this observation does not isolate the character limit’s causal effect on latency.

Latency is only one product property. The planner also needs checks for the behavior the prompt is supposed to produce:

Khan distinguishes context use from correctness: having the right information available does not ensure a correct answer. Off-the-shelf evaluators provide starting templates, but their definitions must be adapted to the application’s requirements. The workshop uses separate checks; it does not demonstrate a formula that combines them into one overall quality score.

The friendliness judge examines the itinerary text and returns friendly or robotic. Its initial definition describes friendliness as upbeat and cheerful. The purpose is to detect whether changing the application prompt changes the resulting tone across the dataset, without requiring someone to grade every row by hand. That definition will soon prove too permissive for Khan’s own expectations.

The judge itself can vary between runs. In response to an audience member reporting inconsistent labels, Khan recommends lowering temperature where the model supports it and rerunning the evaluation to measure the remaining variation. Lower temperature can make responses more repeatable; it does not guarantee identical judgments. Repetition helps distinguish an unstable evaluator from a consistent evaluator applying the wrong definition.

Khan selects both the friendliness and discount evaluators and runs them against the experiments. The experiment numbering is reversed relative to prompt order: experiment 2 is original prompt A, and experiment 1 is modified prompt B. Across the 12-example synthetic itinerary comparison, the discount judge labels none of the original-prompt outputs and 100% of the modified-prompt outputs as offering a discount. The friendliness judge also labels the original responses friendly, which Khan disputes. A clear-looking aggregate can therefore coexist with a poorly calibrated criterion.

The next inspection is of the evaluator’s explanation. The friendliness judge discusses language, tone, and writing style, then concludes that the output is friendly. Khan still considers the original response long and robotic. An explanation makes a judgment inspectable, but it does not make the judgment correct. The disagreement identifies a gap between the evaluator’s rubric and the intended product experience.

Human annotations provide a reference for that gap. A labeling queue lets a PM or subject-matter expert label examples, and those annotations return to the original dataset. Khan prepared labels before the session, marking selected long, LLM-sounding responses as robotic. Keeping the human and judge labels on the same examples makes agreement measurable rather than anecdotal.

Comparing two categorical labels does not require another semantic judgment. A small Python evaluator can express the operation directly:

from typing import Literal

Tone = Literal["friendly", "robotic"]
Match = Literal["match", "no match"]


def compare_labels(judge_label: Tone, human_label: Tone) -> Match:
    return "match" if judge_label == human_label else "no match"


judge_label: Tone = "friendly"
human_label: Tone = "robotic"
comparison = compare_labels(judge_label, human_label)

For this example, comparison is no match. The operation adds an agreement result while preserving both original labels. Khan mentions code evaluators, including string matching, regular expressions, and containment checks, but uses another LLM to return match or no match in the live demonstration. He describes that choice as quick-and-dirty rather than best practice.

The first displayed match result checks the wrong field: discount rather than friendliness. Khan catches the mistake, retracts the interpretation, and reruns the check against the friendly label. During the rerun, he also explains that datasets and evaluations can be handled programmatically, either locally or through code sent to the platform. This makes the same comparison usable outside the UI.

The corrected friendliness comparison shows very little agreement with Khan’s human labels. The settled result view includes the new match-friendly metric; the earlier discount match must not be read as evidence that the friendliness judge is reliable. The action is now clear: improve the evaluator before using it to justify product changes. A judge’s agreement with the intended human standard is a separate evaluation target from the application’s performance.

The development loop has now moved through prompt and context changes, a dataset, output evaluations, human annotations, and a comparison of judge labels with human labels. Together, those artifacts can become executable acceptance criteria. Instead of handing engineers only a prose requirements document, a PM can supply the examples the system must handle and the evaluator that defines acceptable behavior. The team can then test proposed changes against a shared standard.

Khan frames Arize’s product direction around putting observability, evaluation, and development in one workspace for PMs and engineers. After a brief company and conference interlude, the discussion turns to how teams actually adopt that workflow. The first staffing question comes from a head of product deciding whether to hire a specialist, an AI PM, or an engineer. Khan’s advice is to write the first evaluations personally: experience the company-specific difficulties before deciding which skills to hire and which questions to ask in interviews. Accessible dashboards and low-code tools can help a leader make that first pass, but they do not determine the staffing answer.

What should happen after the friendliness judge fails to match the human labels? Apply the same iteration process to its prompt. The initial rubric lacks few-shot examples showing what the team means by friendly and robotic. Khan proposes making the definition stricter and adding examples of both categories, so the evaluator has concrete boundaries rather than only broad adjectives.

He demonstrates asking the product’s copilot to draft a friendliness evaluator, moving that prompt into the playground, and requesting a stricter revision. The useful procedure is to draft the rubric, add examples, rerun it, and compare the new labels with the human annotations. AI assistance can speed up writing the evaluator; the comparison must still determine whether the revision helped. The session shows the revision workflow, not a completed recalibration that successfully matches the human labels.

Evaluation need not use only a standard chat model. Asked about BERT or ALBERT prediction scores, Khan opens a custom-model endpoint configuration alongside provider options for OpenAI, Azure, and Google. The form includes endpoint and model names, a base URL, optional authentication, headers, and request-format controls. He describes chat completions as the default request shape and says other APIs can be adapted. The integration question is whether the endpoint accepts the runner’s request and returns a usable result; entering a classifier’s name alone does not demonstrate that adaptation.

Agreement on a labeled sample raises another question: will it hold across the broader population, especially as the application’s data changes? Khan extends the workflow into production. Evaluations can label incoming data, while people continuously sample outputs and check those judgments. Human–judge disagreements become candidates for the development dataset. Automatic additions to datasets and confidence-based identification of difficult examples are described as planned features in the recording, not completed parts of the demonstration.

A hard example is one whose classification is uncertain or borderline. A response may be neither plainly warm nor plainly robotic, making it useful for clarifying the rubric. The reference dataset should therefore evolve as the team discovers new ambiguities. It is a maintained product asset, not a fixed file created once before launch.

There is no universal number of evaluations that makes a release safe. Quantity and quality are separate concerns. Khan explicitly treats the small development exercise as insufficient to establish statistical confidence. The proposed loop is to curate examples, rerun experiments, and build team confidence before release; once in production, new failures return to development as additional cases.

His Cruise example shows how coverage grows. Early driving required frequent human intervention. Straight-road driving improved, but left turns remained difficult, so the team built a left-turn dataset. Once those turns improved, pedestrians created a more specific challenge, requiring examples of left turns with pedestrians. Each new capability exposed the next missing scenario. A travel planner follows the same pattern at a different risk level, while healthcare or legal applications require a substantially higher release bar.

A custom application may do important work outside its LLM calls and tools. An audience member asks for a runner that invokes the entire system on a dataset, including that surrounding logic. Khan points to some existing tooling but says end-to-end agent capabilities are still being developed and defers the specific integration to a follow-up conversation. The distinction matters: the workshop’s prompt experiments do not constitute a demonstrated runner for arbitrary complete applications.

Faster iteration also changes how a feature moves through a team. Khan reports that his team can sometimes go from an idea to a revised, tested, shipped prompt within a day. In that environment, the PM’s responsibility for the final experience includes keeping evaluation criteria and human-labeled data useful to the development team. Code literacy helps: a PM can interrogate a codebase or ask Cursor to produce a script that generates examples, as Khan did for the itineraries. The ability to make a change independently does not determine whether one should ship it independently; that boundary belongs in an explicit team discussion.

For a PM without code access, Khan suggests building technical influence through working examples. The questioner already uses Cursor for Python data tasks but works in an organization where the PM team cannot inspect the application code. Company-wide AI demos provide one way to make new capabilities visible and invite collaboration. Showing useful work can help change expectations about what PMs contribute without turning that effort into a dispute over job titles.

Asked how staffing ratios might change, a guest introduced as Arize’s CEO, Jason, avoids a firm forecast. He recommends letting PMs use Cursor on the codebase to answer questions and draft requirements from the existing implementation. Khan then extends the idea in both directions: as building becomes cheaper, deciding what to build becomes more valuable. His baseball-card analogy replaces rigid roles with complementary skills—customer conversations and occasional coding on one side, high-quality implementation and responsibility for production failures on another. The goal is to cover the necessary work, rather than assume every person should perform every task.

An audience example makes that skills-based view literal. An accounting agent that lacks information can invoke the CFO as a listed tool: send a Slack question, receive the answer, and continue. The participant describes a partly developed workflow for obtaining context only a human has. They explicitly correct Khan’s first interpretation that this is primarily an approval step. A human supplying missing information and a human judging whether an action is acceptable are different operations, even though both introduce a person into the agent’s execution.

The final implementation questions return to the foundation of the workflow: what must the application send to the observability platform? Existing traces may already record timing and execution information in systems such as Datadog or Grafana. AI observability adds structured information about the model calls and agent work so the platform can render them and make their inputs and outputs available for evaluation. OpenTelemetry supplies the tracing foundation; OpenInference adds AI-oriented conventions and framework-specific instrumentation. Portable telemetry and automatic coverage are separate properties: a supported instrumentor can capture recognized framework operations, while custom application work may need explicit instrumentation.

For LangGraph, Khan names arize-phoenix, arize-otel, and LangChainInstrumentor in an abbreviated setup explanation. The current LangGraph tracing instructions additionally identify the openinference-instrumentation-langchain package, tracer registration, and initialization before application imports. Those current instructions are the setup reference, rather than treating the spoken package list as a complete installation recipe. For functions automatic instrumentation misses, decorators can capture the additional units of work.

A follow-up asks exactly what data must be passed into an evaluator, but the question is deferred before a complete input contract is specified. The discussion does clarify telemetry enrichment: add identifiers such as user_id and session_id, preserve the relationship between human and AI messages, and attach context from the server invocation. These fields let a tool reconstruct a conversation across calls instead of presenting isolated timing records. Application-specific server logic can contribute further metadata. The fields should explain the behavior being evaluated, rather than merely increase the volume of logging.

The closing audience contribution comes from video-game development, where a working prototype can settle an argument about whether a feature is fun more effectively than a document. The same route is available to a PM without production-code access: obtain a small, authorized slice of data, build a working version of the desired experience, and include initial evaluations. The prototype may still have rough implementation details and missing edge cases, but it gives the team something concrete to assess against a user need.

Khan connects this to dataset export. With administrator permission and security-team approval, a PM can retrieve a dataset from Arize and use it in local code to iterate on the example. In the recording, programmatic dataset retrieval is available while a download button is still forthcoming. Production-code access is therefore not always required to contribute a useful experiment. What matters is that the team can work together on the data, prompts, and evaluations that define the experience—and use the results to decide what to change next.