# How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix

Cameron Wolfe, Netflix

[Talk page](https://ai.engineer/talks/zLc8zeGeNaY-netflix-evaluates-ai-when-theres-no-right) · [Video](https://www.youtube.com/watch?v=zLc8zeGeNaY)

## How Netflix evaluates AI when there’s no single right answer

Netflix staff research scientist Cameron Wolfe explains how to turn expert judgment into a repeatable evaluation, measure how much a model’s score can fluctuate, and test whether an offline gain carries through to users.

### One evaluation trial runs an agent and grades its solution

[0:12](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=12s)

Give a creative-writing agent the same assignment twice and it can produce two different pieces. Suppose a new version earns a higher score on that assignment. How much credit should go to the new model, and how much to the particular piece it happened to write? Cameron Wolfe, a staff research scientist at Netflix and author of the Deep (Learning) Focus newsletter, starts with this problem of noise. It arises even on tasks where everyone agrees on the right answer.

An evaluation suite contains a set of tasks. For each task, the agent produces a rollout, the course of one attempt, that eventually yields a proposed solution. A grader takes that solution and returns a result, which in the simplest case is pass or fail. The benchmark score summarizes what happened across those trials. Repeat the exercise with the same agent and a different set of attempts can produce a different pass rate.

![A diagram follows a task from specification through rollout, outcome, and grading. A red brace over Tasks 2 and 3 reads “Small to limit cost,” and a red box marks the model and harness as non-deterministic.](https://ai.engineer/api/talk-essay-media/4396336e4590da109167d830baf954aab87ac05e72db1c8d8765e9ebea243137)

Structure of an evaluation trial — cost and non-determinism

Cost limits how much of that variation a team can average away by testing more tasks. Wolfe puts the cost of running some recent benchmarks at roughly $50,000 to $100,000. A team may consequently have to draw its conclusion from a relatively small set. A handful of tasks on which the agent happened to succeed or fail can then have appreciable influence on the reported difference between models.

### Creative writing adds a second source of variation

[2:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=140s)

Many benchmarks concentrate on verifiable tasks. String matching can check whether a response matches an expected answer. Symbolic verification can check a mathematical solution, including whether two expressions are equivalent. Test cases let a grader execute checks against a proposed solution. In each case, a function implements the decision. An agent might arrive at a different answer on its next attempt, but the grader can apply the same check to either one.

Wolfe places real assignments on a spectrum of verifiability. At the creative-writing end, many quite different pieces could satisfy a prompt, and experienced readers can disagree about which ones succeed. An LLM judge can read a piece and assess its quality, either using general guidelines or checking the requirements of that particular assignment. Now hold the piece fixed and run the evaluation again. The judge itself can return a different assessment. Both the choice of completion and the grading of that completion contribute to the observed score.

![An arrow from Creative Writing at the non-verifiable end of the spectrum points to a sunrise prompt and response. An LLM judge considers originality, emotional impact, and coherence and awards 4 out of 5.](https://ai.engineer/api/talk-essay-media/3e39c0babdc2d029348e3919d2073412ef77da82dc42ec1e62a66734cca0c98d)

Creative writing on the verifiability spectrum — an LLM judge scores a paragraph

Wolfe organizes the work into three parts. Human evaluation establishes what quality means for the task. Offline evaluation gives the team an affordable way to apply that definition across many experiments. Production monitoring supplies evidence from people actually using the system. The order of construction matters. Before a team can decide whether an automatic grader is doing a good job, it needs examples of judgments it has reason to trust.

### Write criteria that two reviewers can apply to the same output

[4:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=260s)

Human evaluation is Wolfe’s North Star for defining, measuring and improving quality. He starts with a small group of subject-matter experts and asks them to identify a few things they care about in the finished solution. For each criterion, the team has to specify what the reviewer should look for and how to record the judgment. A binary rating fits a decision about whether an output meets a requirement. A Likert scale lets a reviewer assign an ordered rating when the degree to which the output meets the criterion matters.

The instructions need enough detail to support the decision a reviewer faces with a piece of work in front of them. Wolfe calls for specific examples of how to rate each criterion. On a scale, a reviewer needs a basis for choosing one rating over another. On a binary check, they need to know what qualifies as meeting the requirement. Examples give the experts a way to make those distinctions concrete before the guidelines go into wider use.

Calibration begins by collecting multiple human ratings for the same data. Holding the output fixed makes a disagreement worth investigating. Ask the reviewers how they reached their decisions. Two readers may agree on what the piece contains yet interpret a criterion differently. Their explanations tell the team where to revise a definition or add a scoring example. Rate common examples again and measure agreement. The result of this iteration is both a clearer guide and an observed level of consistency among the people applying it.

### Size the sample for the improvement you need to detect

[5:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=350s)

Once reviewers can apply the guidelines consistently, the next decision is how much work to send them. Wolfe frames this around a comparison between two models. What difference in performance would matter, and how many examples would the team need to distinguish a difference of that size from ordinary fluctuations? Power analysis gives the team a way to estimate that volume before committing to a round of human evaluation.

To size the evaluation, the team specifies the smallest performance gap it wants to detect and estimates the noise in its scores. Sampling several completions for the same task helps estimate how much results fluctuate between attempts. The team also chooses a significance level, which limits the risk of reporting a difference when none exists, and a target power, the chance of detecting a gap of the chosen size when it exists. With the same noise, significance level and target power, detecting a smaller improvement requires a larger sample.

![An annotated sample-size equation labels the normal percentiles, variance terms, resample counts for models A and B, and minimum detectable effect. A dashed box expands the paired variance term.](https://ai.engineer/api/talk-essay-media/5118d54edcf0d024667ae6faf43cadbe2bffdec0cee5f10f7e3c936b927ba4f3)

Human evaluation sample size and minimum detectable effect

### Profile model attempts to find tasks that distinguish candidates

[7:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=440s)

Which tasks enter the sample is a separate decision. Wolfe looks for diversity and realism together. The evaluation should cover different kinds of work and reflect what the agent will be asked to do in production. A team could collect many examples of one familiar assignment and estimate performance on it quite precisely while learning little about the other assignments its users bring.

Wolfe also looks for tasks that are difficult and discriminative. To assess that property, profile the candidates by sampling rollouts and inspecting which kinds of problems each can solve. A very hard problem that defeats every candidate yields the same result for all of them. A problem that some candidates solve more reliably than others can reveal a capability difference. Difficulty alone gives the benchmark designer little basis for telling those cases apart.

The examples also need to be accurate. Wolfe describes improving the collection over time through a combination of model-based and human filtering. These sampling decisions carry over to automated benchmarks. Replacing human review with a faster grader changes how much experimentation a team can afford. The choice of tasks still determines which capabilities its comparison can reveal.

### Choose between reusable guidelines and a task-specific rubric

[8:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=500s)

Having people evaluate every new agent quickly creates a bottleneck. An offline evaluation keeps the task and the candidate’s solution but hands the quality decision to an automatic proxy. One approach gives an LLM judge high-level guidelines similar to the ones the human reviewers use. For every assignment in the collection, the judge reads the proposed solution and assesses how well it adheres to the same criteria. The work that went into defining quality carries across assignments.

A rubric changes what the judge is asked to do. Wolfe illustrates this with an assignment from what he calls the Dr. Seuss paper: culturally adapt a book into a Netflix film. Many adaptations could satisfy the brief. The rubric lists requirements specific to that assignment, and the LLM judge checks each one independently against the proposed adaptation.

With the general guidelines, an assessment tells the team how the work fared against a shared definition of quality. With the rubric, the team can inspect the judgment on each requirement of this adaptation. That specificity takes work for each new assignment and gives the judge a narrower decision to make on each check. In both approaches, the resulting metric depends on how well the LLM can carry out the judgment it has been given.

![A DR Tulu report about One Hundred Years of Solitude appears beside a training diagram. Successive rubric examples ask for increasingly specific adaptation details, from avoiding a literal replica to using 3D scans for Macondo’s main square.](https://ai.engineer/api/talk-essay-media/6ef6ca90a0da81cd58c139bac3f3998641b471717e68c1d1dd7a10ac313ecd7d)

DR Tulu and search-guided rubrics for a Netflix adaptation question

### Check the judge against human ratings and viewer behavior

[9:49](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=589s)

Wolfe calls the next step meta-evaluation. Take examples from the calibrated human process whose ratings the team trusts and use them as golden data. Run the automatic evaluation on those same examples and compare its decisions with the human assessments. Binary accuracy can express how often a yes-or-no check matches the human decision. For scored outputs, a correlation measure can show whether pieces that humans rate more highly also tend to earn higher scores from the judge.

A Netflix project Wolfe worked on tested a judge against a different kind of evidence. His team established a causal link between the quality the LLM judge assessed and streaming behavior. When the judge rated something poorly, a viewer who began watching the show was more likely to abandon it. Agreement with an expert rating would establish that the judge could reproduce that judgment. Abandonment gave the team a specific consequence for viewers against which to assess the significance of a poor rating.

How much judge–human agreement is good enough depends on the assignment. Calibrated reviewers may still disagree relatively often on a highly subjective task. Wolfe uses the agreement those reviewers achieve with one another as a baseline and aims for a similar level between the judge and the humans. Once the grader meets that standard, the team can build a benchmark around it, applying the same sampling, profiling and filtering decisions it used for human review.

### Repeat both the generation and the grading

[11:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=710s)

A judge that agrees with humans across a golden set can still vary from one run to the next. Wolfe recommends resampling at both places where variation enters the benchmark. Generate several rollouts for a task to see how much the agent’s work varies. Then have the judge assess a fixed solution several times to see how much the grading varies. If every new completion receives only one new judgment, a change in score could have come from either step.

Use power analysis to size the benchmark and report error bars alongside the scores. Repeated samples can reduce the variance of the estimated result. Confidence intervals make the remaining uncertainty visible to the person comparing candidates. That distinction matters when deciding whether to spend human-review time on a new version. A slightly higher average on one run gives the team less reason to advance a candidate when fluctuations of that size are common across repetitions.

At this point the team can run many ideas through the offline benchmark and return the promising candidates to its calibrated human reviewers. This is where the investment in automation pays off. The team gets to explore broadly while reserving close human assessment for changes with evidence of an improvement. A candidate that survives both comparisons is ready for a test of how the change affects users.

### Use an A/B test as a limited deployment

[13:24](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=804s)

Production adds evidence unavailable from inspection of a finished answer. Users can give explicit feedback, such as a thumbs-up, a thumbs-down or a preference between responses in a chatbot. They can also leave implicit signals through their behavior, such as the length of a session or the decision to abandon something they started. The team needs a reason to connect its chosen measure to quality in this product. Session length by itself records elapsed use. Its interpretation depends on what users are trying to accomplish.

An A/B test gives the team a comparison against the system already in place. Expose a portion of users to the candidate and measure the chosen outcomes relative to the existing experience. The statistical question from benchmark design returns here in operational form. Given the size of the difference that matters and the variability of the metric, how long does the experiment need to run before the team can draw a meaningful conclusion?

Wolfe treats the experiment as a soft deployment. The agent gets real use while the decision about a full rollout is still open. An offline gain earned the candidate a human review, and favorable human judgments earned it this production comparison. The A/B result can now inform whether to extend the change to everyone.

### Turn production failures into cases for the next evaluation

[14:53](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=893s)

After rollout, the team can monitor performance trends and inspect individual failures or new uses. Those observations can lead to different repairs. If a production task was missing from the benchmark, add an example so future candidates face it offline. If the existing criteria overlooked what made an answer poor, revise the quality guidelines. If the criterion already covers the failure but people or the judge apply it inconsistently, revisit calibration.

![Three bullets cover usage metrics, A/B testing and monitoring after deployment. Red arrows lead from production monitoring back to both offline evaluation and human evaluation.](https://ai.engineer/api/talk-essay-media/b0b19d9d315a893932b20e4a7a733b4471ac4a9d4ed2046b46209afe6363cf48)

Production Monitoring — Feeding Real-World Data Back Into Evaluation

This gives each stage a role in the next model decision. Offline runs narrow a large field of candidates, human reviewers examine the strongest ones, and a limited deployment tests the result against the current user experience. A case discovered after deployment can travel all the way back to the experts who first defined quality. The next comparison can then measure a failure that every stage of the previous evaluation allowed through.

## Key takeaways

- Collect multiple human ratings on the same output. Investigate the disagreements, revise the scoring examples and use the reviewers’ eventual agreement as a benchmark for the judge.
- Choose evaluation tasks by profiling what competing models can solve. Size the sample around the difference worth detecting and the variation observed across attempts.
- Resample agent rollouts and judge ratings separately, and report uncertainty alongside each benchmark score.
- Advance promising offline results through human review and a limited A/B test. Use failures found after rollout to decide whether the benchmark needs new tasks, clearer criteria or better calibration.

## Slides

### Why evals are difficult — scope, noise, and cost

[1:21](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=81s) [Three columns describe scope, noise, and cost. A red box around Noise and the label “This talk!” mark it as the focus.](https://assets.ai.engineer/sites/aiecode2025/sha256/38a3ca823accfb0a5efde1122688baaec54ed9dd79ed0321adddad093e6e4359.webp)

Motivation
• Evals are difficult… why?

Scope
LLMs / agents have broad capabilities that span diverse tasks and domains

This talk!
Noise
Non-determinism (from several sources) makes the evaluation process variable

Cost
Benchmarks contain many test cases that can be long-running or expensive

### Structure of an evaluation trial — cost and non-determinism

[1:34](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=94s) [A diagram follows a task from specification through rollout, outcome, and grading. A red brace over Tasks 2 and 3 reads “Small to limit cost,” and a red box marks the model and harness as non-deterministic.](https://assets.ai.engineer/sites/aiecode2025/sha256/4396336e4590da109167d830baf954aab87ac05e72db1c8d8765e9ebea243137.webp)

Motivation
• Evals are difficult… why?
• Structure of an evaluation trial

Small to limit cost
Evaluation suite
A collection of many evaluation tasks

Task 1
Specification and context
prompt    instructions    environment
tools    success criteria
Rollout (executed by model + harness)
Trajectory
messages, actions, tool calls, reasoning...
Outcome
final answer / final environment state

Task 2    Task 3    …

Non-deterministic
Model + harness
Runs the task and produces a rollout

Outcome
Final result of rollout
Final answer + environment state
Grader
Heuristic / rule-based check
final result + outcome → correct / incorrect
Pass ✓
Fail ✕

Task = specification + context for an evaluation case
Rollout = one execution producing a trajectory and outcome
Grader = check for correctness using heuristics or rules

### Verifiable tasks — grading a rollout outcome

[2:24](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=144s) [A red outline highlights the grader between a rollout’s outcome and its pass or fail result. The caption lists string matching, symbolic verification, and executable test cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab45b57f7145a5b9002e9241b85b456e08333c046f579cb425bfd677a8ee662e.webp)

Motivation
• Evals are difficult… why?
• Structure of an evaluation trial ← focus on verifiable tasks

Evaluation suite
A collection of many evaluation tasks

Task 1
Specification and context
prompt    instructions    environment
tools    success criteria
Rollout (executed by model + harness)
Trajectory
messages, actions, tool calls, reasoning...
Outcome
final answer / final environment state

Task 2    Task 3    …

Model + harness
Runs the task and produces a rollout

Outcome
Final result of rollout
Final answer + environment state
Grader
Heuristic / rule-based check
final result + outcome → correct / incorrect
Pass ✓
Fail ✕

Task = specification + context for an evaluation case
Rollout = one execution producing a trajectory and outcome
Grader = check for correctness using heuristics or rules

String match, symbolic verification, executable test cases, and more…

### Math on the verifiability spectrum — checking 72 by string match

[2:57](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=177s) [An arrow from Math at the verifiable end of a grayscale spectrum points to a word problem. The model adds 48 and 24, and a string match checks its answer of 72 against the ground truth.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d87e389488049e22ded62360365028e5d0ab4a37c104421c3c6fe4d6aac4738.webp)

Motivation
Evals are
Structure
Real-world tasks lie on a spectrum…

Model Input
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

Model Output
Half of 48 is 24.
48 + 24 = 72.
Natalia sold 72 clips altogether.
### 72

Ground Truth
Final Answer: 72
String Match → ✓

Creative Writing    Deep Research    SWE    Competitive Programming
Non-Verifiable                                      Verifiable
Chat    Workplace Tasks    Instruction Following    Math

### Creative writing on the verifiability spectrum — an LLM judge scores a paragraph

[3:05](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=185s) [An arrow from Creative Writing at the non-verifiable end of the spectrum points to a sunrise prompt and response. An LLM judge considers originality, emotional impact, and coherence and awards 4 out of 5.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e39c0babdc2d029348e3919d2073412ef77da82dc42ec1e62a66734cca0c98d.webp)

Motivation
Evals are
Structure
Real-wor

Model Input
Write a short, evocative paragraph about a quiet town at sunrise.

Model Output
The town woke slowly, with amber light spilling over the rooftops and mist curling above the river. ...

LLM Judge
Evaluate the creative writing response on:
- Originality: unique concepts, unexpected elements
- Emotional impact: how the writing affects the reader
- Coherence: logical flow and narrative structure

Provide a single overall score from 1–5 with a brief rationale.

Evaluation Output
The paragraph is original, evocative, and well structured, with vivid imagery and a calm, consistent tone. ...
Score: 4 / 5

Creative Writing    Deep Research    SWE    Competitive Programming
Non-Verifiable                                      Verifiable
Chat    Workplace Tasks    Instruction Following    Math

### Step One: Human Evaluation — defining quality with subject matter experts

[4:44](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=284s) [Two bullets introduce human evaluation and quality guidelines. Three green checkmarks call for a small set of criteria, instructions for rating them, and specific examples. A clipboard icon appears below.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d1cea615486af73227f6924d9802d0310249f85a14fc6477a089aedbece5316.webp)

Step One: Human Evaluation
• Our North Star for defining, measuring, and improving quality
• Partner with SMEs → clear quality guidelines to minimize subjectivity

✅ Small set of quality criteria
✅ Define how to rate them
✅ Details & specific examples

Human Evaluation

### Human evaluation sample size and minimum detectable effect

[6:28](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=388s) [An annotated sample-size equation labels the normal percentiles, variance terms, resample counts for models A and B, and minimum detectable effect. A dashed box expands the paired variance term.](https://assets.ai.engineer/sites/aiecode2025/sha256/5118d54edcf0d024667ae6faf43cadbe2bffdec0cee5f10f7e3c936b927ba4f3.webp)

Step One: Human Evaluation
⏩ Effect size we want to detect
⏩ Level of noise in eval results
⏩ Desired significance level

Percentiles of Standard Normal Distribution
σ²_A = 𝔼[σ²_{A,i}]
σ²_B = 𝔼[σ²_{B,i}]
Number of Resamples for Models A and B

n = (z_{α/2} + z_β)²(ω² + σ²_A/K_A + σ²_B/K_B)/δ²
ω² = Var(x_A) + Var(x_B) − 2Cov(x_A, x_B)
Minimum Detectable Effect

### Human evaluation guidelines, annotation calibration, and data sampling

[7:52](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=472s) [Bullets describe developing quality guidelines with experts and calibrating annotations. A red sampling guideline separates how much data to collect from which examples to choose.](https://assets.ai.engineer/sites/aiecode2025/sha256/e51b65ef26f7bff163833a35c653bb9a733fa9a7250170d8cbb7380fbfe5ec9c.webp)

Step One: Human Evaluation
• Our North Star for defining, measuring, and improving quality
• Partner with SMEs → clear quality guidelines to minimize subjectivity
• Calibrate the annotation process ↔ revise guidelines
• Sample data properly to draw reliable conclusions:
  ○ Amount of data: power analysis → setup & volume for confident eval results
  ○ Data type / quality: diverse, realistic, difficult, discriminative, accurate

Applicable to all
forms of evaluation!

Human Evaluation

### DR Tulu and search-guided rubrics for a Netflix adaptation question

[8:34](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=514s) [A DR Tulu report about One Hundred Years of Solitude appears beside a training diagram. Successive rubric examples ask for increasingly specific adaptation details, from avoiding a literal replica to using 3D scans for Macondo’s main square.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ef6ca90a0da81cd58c139bac3f3998641b471717e68c1d1dd7a10ac313ecd7d.webp)

Step Two: Offline (Automatic) Evaluation

User
How did Netflix manage to successfully adapt One Hundred Years of Solitude, a notoriously difficult book to bring to the screen?

DR Tulu
Agentic Workflow
Think   Tool 1   Think   Tool 2   ...   Answer
Long-form Report with Citations
Netflix's adaptation avoided a literal replica of Macondo and instead fused real locations with meticulously built sets [1] to honor the novel's essence while giving the show contemporary visual grammar. The production grounded magical realism in front-of-camera practical craft, relying on makeup, special effects.[2]
The location strategy and production design...
Sources
[1] The production team behind Netflix’s adaptation of “One Hundred Years of Solitude,” LA Times
[2] The article discusses Netflix’s adaptation of Gabriel García Márquez’s celebrated novel, NY Times

Training Data
x   Query
Policy π_θ(·)
y₁, y₂, …   Responses
Generate Evolving Rubrics r
Rubric Rewards
S(x, yᵢ) = (1/W) ∑_{k=1}^{K} wₖ JUDGE(rₖ, yᵢ)
Reward

Search-Guided Rubrics Co-Evolve with the Policy π_θ
Mention that Netflix’s adaptation avoided a literal replica.
Mention that Netflix worked closely with García Márquez’s family.
Mention that Netflix used 3D scans of real Colombian towns to design Macondo’s main square.
π_{θ₀} → π_{θ₁} → π_{θ₂} → π_{θ₃} → π_{θ₄}
①   ②   ③   ④
① Precisely describe the adaptation process
Environment
Web Browsing   Paper Search   Google Search
Closed-book Rubric (baseline)
Evolving Rubric (generated during training)
Closed-book knowledge from rubric generator
Explored knowledge during training policy π_θ
(source)

### Checking automated judges against golden samples and human agreement

[10:49](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=649s) [The slide asks how much agreement a judge needs for a given task. A red callout identifies human agreement as a baseline, above arrows connecting offline and human evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d4c8706acf8a84c46ff5cc4f839b984e0bcee29dbfa5644553a047502742916.webp)

Step Two: Offline (Automatic) Evaluation
• Create an efficient (but imperfect!) proxy for human verification
• Meta-evaluation: ensure your judge is accurate!
  ○ Measure agreement of judges / rubrics with golden samples
  ○ What level of agreement is “good”? Depends on the task…

Human agreement is an
informative baseline

Offline Evaluation ↔ Human Evaluation

### Building and revising benchmarks around automatic verifiers

[11:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=680s) [Three indented bullets describe sampling benchmark data, verifying outputs with LLM judges or rubrics, and revising benchmarks. Below, arrows connect offline and human evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/022db42c8f1b4baae07d62e62b47ffba5c76fb67cfdb4c735def2f2f6c389f84.webp)

Step Two: Offline (Automatic) Evaluation
• Create an efficient (but imperfect!) proxy for human verification
• Meta-evaluation: ensure your judge is accurate!
• Create benchmark(s) around these automatic verifiers
  ○ Sample data properly (similar to before)
  ○ Use LLM judges or rubrics to verify outputs
  ○ Continually revise and improve your benchmarks

Offline Evaluation ↔ Human Evaluation

### Handling evaluation noise with power analysis, error bars, and re-sampling

[12:02](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=722s) [A red “Use statistics!” callout introduces three ways to handle noise in benchmarks. Below it, a diagram links offline evaluation and human evaluation in both directions.](https://assets.ai.engineer/sites/aiecode2025/sha256/3216ae5ec91998d4cc1cbc8077a908eb7c06c39c1d7b6d691b360fb2966dd90f.webp)

Step Two: Offline (Automatic) Evaluation
• Create an efficient (but imperfect!) proxy for human verification
• Meta-evaluation: ensure your judge is accurate!
• Create benchmark(s) around these automatic verifiers
• How do we handle noise? Use statistics!
  ○ Power analysis → benchmark size
  ○ Report error bars with evaluation results
  ○ Use re-sampling to reduce variance

Offline Evaluation ↔ Human Evaluation

### Production Monitoring — Explicit and Implicit Usage Metrics

[13:32](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=812s) [The slide groups thumbs up/down and preference feedback under explicit metrics, and session length and abandonment rate under implicit metrics. A red caption sits above the production monitoring chart in a three-stage evaluation diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba743fb7d3c7a9cd206ec78919b230c1277d8d822305cf2c954f421b94dca1ec.webp)

Step Three: Production Monitoring
• Define and track production usage metrics for our agent
  ○ Explicit: thumbs up / down, user preference feedback
  ○ Implicit: session length, abandonment rate

Quality signals as agent
is used in production!

Offline Evaluation
Human Evaluation
Production Monitoring

### Production Monitoring — A/B Testing Before Full Rollout

[14:08](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=848s) [Bullets describe using an A/B test as a “soft” model deployment. In the diagram below, the production monitoring box contains a line chart and two branches labeled A and B.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e348fe9e65b2c35e4417923fa702f385dd24176978e3315e52d77bf7beface3.webp)

Step Three: Production Monitoring
• Define and track production usage metrics for our agent
• Measure these metrics in an A/B test to validate performance
  ○ Use this as a “soft” model deployment
  ○ Vet agent quality in production → full rollout

Offline Evaluation
Human Evaluation
Production Monitoring
A    B

### Production Monitoring — Feeding Real-World Data Back Into Evaluation

[14:58](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=898s) [Three bullets cover usage metrics, A/B testing and monitoring after deployment. Red arrows lead from production monitoring back to both offline evaluation and human evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/b0b19d9d315a893932b20e4a7a733b4471ac4a9d4ed2046b46209afe6363cf48.webp)

Step Three: Production Monitoring
• Define and track production usage metrics for our agent
• Measure these metrics in an A/B test to validate performance
• Continual production monitoring after the agent is deployed

Ground your evaluations with data from the real world!

Offline Evaluation
Human Evaluation
Production Monitoring
A    B

### Repeat — Refining Criteria, Calibration and Evaluation Data

[15:13](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=913s) [A list describes how to improve evaluations over time. Its final item, repeating the process in a new domain, is bold, italicized and underlined. Below, two-way arrows connect the three evaluation stages.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd76dde4287441f2001777011ea63b9e56106fd2879c524dd25fb05eeb8a5dd6.webp)

Step Four: Repeat
• Evaluation is a moving target…
• Continually improve the evaluation process over time:
  ○ Define new or better criteria
  ○ Improve calibration of humans and judges
  ○ Find better data to avoid saturation
  ○ Repeat the entire process in a new domain!

Offline Evaluation
Human Evaluation
Production Monitoring
A    B

### Takeaways — The Roles of Offline Evaluation, Human Evaluation and Production Monitoring

[16:01](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=961s) [Three color-coded captions explain the roadmap. Red pairs offline evaluation with calibrated judges or rubrics, blue pairs human evaluation with an agreed definition of quality, and green pairs production monitoring with metrics and A/B tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/494f384b3a40af66f70eb611cb64399538bb857af5aa2d78a9cb4f7b45232b8d.webp)

Takeaways
• Evaluation is already hard, subjective tasks are even noisier
• Our roadmap

Use calibrated LLM judges
or rubrics as a proxy to
scale experiments
Offline Evaluation

North star for quality, provides
a clear quality definition that
humans agree on
Human Evaluation

Monitor and A/B test key
production metrics for
further evaluation
Production Monitoring
A    B

### Takeaways — Offline → Human → A/B test → Deploy

[16:56](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=1016s) [A red annotation beside “Our roadmap” says to fight noise at every step. The next bullet shows four steps linked by arrows, from offline evaluation through deployment. The three-box evaluation diagram appears below.](https://assets.ai.engineer/sites/aiecode2025/sha256/78b9fa479e4e5ee015bcbadbe4255cde1965fdc520aac699cef3cbcfaea27e1d.webp)

Takeaways
• Evaluation is already hard, subjective tasks are even noisier
• Our roadmap ← fight noise at every step!
• Multi-Step Evaluation: Offline → Human → A/B test → Deploy

Offline Evaluation
Human Evaluation
Production Monitoring
A    B

## Full transcript

[0:12](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=12s) All right. Well, hi everybody. Uh, I'm Cameron Wolfe. I'm a staff research scientist at Netflix. I also write the Deep Learning Focus newsletter, hopefully you read it. And, uh, today I'm gonna be talking about evaluations, and more specifically, trying to evaluate highly subjective tasks. And we're gonna go over a lot of information. Uh, it's supposed to be kind of like an overview. If there's anything that you wanna look into in more depth, you could scan this QR code, it'll take you to the slides. I've also written a lot of stuff that outlines all of these different concepts in

[0:42](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=42s) detail. So plenty opportunity for further reading if there's anything that you're interested in throughout the talk.

[0:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=50s) So we all know evaluations are difficult. We've seen a lot of presentations on this during AIE. And if we think about why this is the case, there's just ob- there's evidence of this everywhere within the AI research community. Uh, people are talking about benchmarksing constantly. There's new evals being released like every day, and different re- revisions to the eval. So we all know that evaluations are difficult. But if we step back and think about why this is the case, there are a couple of different reasons, and one of the primary reasons, and the one that we're going to focus on a lot

[1:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=80s) today, is noise. We're working with, uh, several non-deterministic components when we're trying to evaluate models. This creates a lot of fluctuations in behavior, and it just makes it really difficult to draw reba- reliable conclusions from eval results. And to understand why this is the case, we can just look at the structure of an evaluation trial. So we have our evaluation suite here. It's composed of a bunch of different tasks that we wanna try and solve. We pass these, those tasks to a non-deterministic agent that gives us a rollout, uh, and

[1:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=110s) eventually a solution, and then we take this solution and pass it to a grader, which will give us just some, some kind of pass/fail result. So at the end of the day, we're just trying to get our agents to solve these tasks and seeing if they're passing them or not. And because of the cost of evaluation, you know, some of these recent benchmarks will cost like, you know, fifty, $100,000 to run them. Uh, we can't evaluate over that many tasks, so the set of tasks within our evaluation benchmark typically is going to be relatively small. And this, combined with the fact that we're already working with a non-deterministic

[2:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=140s) agent, it already just makes it difficult to have any sort of like reliance on the eval result. And on top of this, if we look at most of the stuff that people are working on in terms of evals, we're primarily focusing on verifiable tasks. So this grader that we see at the bottom of the screen here, it's implemented as some sort of like deterministic check, whether it be string matching, symbolic verification, test cases, and so on. Uh, so we're mostly focusing on these verifiable tasks where pass/fail is something that we can evaluate with a function.

[2:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=170s) But if we look at the different things that an agent will try and do in the real world, it kind of lies on a spectrum of verifiability. Not all tasks are verifiable. So on one end of the spectrum, we have things like math, where we can verify solutions deterministically with things like string matching or symbolic verification. But on the other end of the spectrum, we have super subjective tasks like creative writing, and here the definition of pass/fail is extremely subjective and open-ended, and it's hard even for humans to evaluate reliably. So usually we're gonna have to draw upon

[3:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=200s) some type of like LLM judge or rubric to help us with this. And what we end up with is that we're taking a component that was previously deterministic, this grader, and we're replacing it with another non-deterministic LLM-based component. So we just end up with a lot of noise and subjectivity within an evaluation process that was already difficult to begin with. Um, so what we end up with with this is just something that looks like this. We're drowning in all this noise. Uh, we're completely blind to if our agent is, like, doing well or not. So the goal of, of the

[3:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=230s) talk today is to try and figure out how to not become this person. So what we're going to do is we're going to outline a repeatable three-part framework for out- evaluating super subjective tasks. This will consist of offline evaluation, human evaluation, and production monitoring. And then we're going to outline the best practices for each of these components, which applies both to subjective tasks and just to evaluation in general. And then along the way, we're gonna cover a bunch of different tricks that we can use to reduce noise, have more statistical, like, confidence in these evaluation results, and just

[4:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=260s) dry the mo- dra- draw the most reliable conclusions that we can from our evaluation results. So let's get into it. So when we're working on a very subjective or open-ended task, the first step is to set up a human evaluation process, and this is going to be the North Star for defining, measuring, and improving the quality of our task over time. But in order to set up human evaluation, we need to first create a clear definition of what quality means. And usually the way that we do this is by partnering with some small number of subject matter experts

[4:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=290s) to create quality guidelines. And within these quality guidelines for our tasks, we define a small set of criteria that we care about in terms of what should be in the final solution. We define how to rate these criteria, usually using some type of like binary or Likert score, and then we just provide a ton of details defining what each of these criteria are, specific examples of how to rate them, and things like this. So we wanna provide humans as much information possible so that we can minimize the subjectivity of this human evaluation process. But even if

[5:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=320s) we write the best possible human evaluation guidelines, we can't just assume that the human evaluation process is going to be reliable, so we need to calibrate it extensively. And what this means concretely is that we take multiple human evaluation samples for the same data, look at the agreement between them, and then update our guidelines based upon the sources of disagreement that we see. So this is kind of an iterative process where you're literally just talking with humans and trying to see, you know, is there some issue in my guideline that makes it unclear how do we evaluate this? And over time, we wanna make our quality

[5:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=350s) guidelines as clear as possible so we can get high agreement between all of these different humans. So if we do this properly, then we end up with a calibrated human evaluation process and very clear quality guidelines. And we can already use this to start performing evaluations. But if we wanna draw some sort of reliable conclusion from these human evaluations, we also need to think about how we sample data. Uh, so we need to think about things like the amount of data that we're going to put into the evaluation process, the type of data that we've, we'll evaluate on, and so forth. And

[6:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=380s) for the amount of data, we can draw upon a statistical concept called power analysis, which will help us understand the amount of data that we need to pass into the evaluation process. And power is a really interesting area of statistics, but for the purposes of this presentation, the high le- level idea is that power analysis allows us to estimate the volume of data that we need to put into the evaluation process to make a meaningful conclusion about the difference in performance between two different models. And we make this estimation by looking at things like the effect size

[6:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=410s) that we wanna detect, the level of noise that we observe in our eval results, you could estimate this with like resampling model completions and so forth, and the desired significance level, so how confident we wanna be that a difference in two models' performance is actually legitimate and not due to noise. So we can use power analysis with these factors to estimate the size of our eval that we need for a reliable conclusion. And then in terms of the type of data that we wanna sample, there are a bunch of different factors, but at a high level, if you read a bunch of different benchmark reports and see what people do, generally

[7:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=440s) they're going to ver-- uh, focus upon diversity and, uh, the realism of the evaluation. So we want all of the data that we evaluate on to accurately reflect the types of tasks that the agent is solving in production. We want it to be difficult and discriminative, meaning that it differentiates between the capabilities of different models. Usually, this is assessed via some form of like profiling, just sampling rollouts and seeing what types of problems the model can actually solve versus not. And then we want it to be as accurate as possible, which is usually accomplished via some combination of

[7:49](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=469s) like model and human-based filtering over time. And these concepts for sampling data properly, they're not only applicable to human evaluation. You can kind of use these anywhere. So we'll also see again in the future when we talk about offline evals, we'll probably use like very similar data sampling techniques. So at this point, we have a reliable human evaluation process, but the problem is if we want to human evaluate every new agent or model that we're looking into, uh, it's quickly going to become a bottleneck, and we can't iterate fast enough.

[8:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=500s) So we need to invest into creating offline or automatic evaluations. And the key idea here is that instead of verifying the solution quality with a human, we need to find some sort of automatic proxy for this, which in this case is usually going to be some LLM judge or rubric. And we won't talk deeply about the differences between LLM judges and rubrics in this talk, but at a high level, basically, LLM judges tend to be given high-level quality guidelines, and it's very similar to how humans evaluate tasks. We give them our quality guidelines, and they're going to evaluate

[8:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=530s) whether the solution adheres to that, and it's the same set of quality guidelines that we look at across all different tasks that we're trying to evaluate. Rubrics, on the other hand, are a little bit more specific. So they define a task-specific checklist of individual items that are each separately verified within an L-- with an LLM judge. So one example of this from the Dr. Seuss paper up here, uh, I like this example 'cause it's both super subjective, and it's also talking about trying to culturally adapt a book into a Nexfl- Netflix film. But if we look at all

[9:19](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=559s) the rubric checks that the model is looking at here, we're not looking at some high level like quality guideline. We're outlining a checklist of specific things that the LLM judge needs to verify within the solution that it's looking at. So it's task specific, and we're looking and verifying each of them independently to make the evaluation way more objective. And what we end up with here is automatr- automatic metrics that we can use to verify solution f- quality in a similar way to how a human would evaluate a solution. But the problem with this is that these are all automatic

[9:49](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=589s) metrics. They're just a proxy, so they're going to be imperfect. So to solve this, we need to do some sort of meta-evaluation to ensure that our judge is accurate. And what this means is that we'll take data from the human evaluation process, golden data. These are going to be evaluation examples from humans that were confident in their quality. We know these are good evaluation examples. And then we want to measure the agreement between what our judge or rubric is telling us and this golden data set. So we basically just wanna look at basic things like correlation metrics, binary

[10:19](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=619s) accuracy, and so forth. But meta-evaluation is like a pretty wide space, and there's a bunch of different ways that you could approach this. One example that I like to pull on from Netflix, a project that I worked on, we actually took an LLM judge and showed that the outputs of this LLM judge were causally correlated with key streaming metrics like abandonment. So if your LLM judge tells you something is bad, it's likely that somebody will abandon the show after trying to watch it. So there's a bunch of different possibilities for how to evaluate judges, and I'd encourage you to look into this blog if you're interested in them.

[10:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=650s) And then finally, before moving on from me- from meta-evaluation, one common question is how to figure out what agreement level is good. And for this, it depends a lot on the task. Like more subjective tasks, the level of agreement that you can achieve is going to be a lot lower. But what we can do is just use human evaluation as a baseline. So we can look at the amount of agreement between different humans who are trying to evaluate this task, and we can shoot for a similar level of agreement of our judge with the humans. So, uh, similar levels of agreement both between judges and humans and mutually between

[11:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=680s) humans. And then from there, we have automatic verifiers that we know work well. So we can just start to create benchmark, benchmarks around all of these verifiers. To do this, it's pretty similar to human evaluation. We just sample data properly using the same techniques we just talked about, and then we use LLM judges or rubrics to verify these outputs and then just try and continually revise and improve the data that we're evaluating on over time. The only point that we still need to talk about is that we now have multiple non-deterministic components that we're working

[11:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=710s) with. In human evaluat-- e-e-evaluations, we only had a non-deterministic LLM. Now we have a non-deterministic model and a non-deterministic evaluator, so we have a lot of noise that we need to figure out how to handle. And in order to do this, we can just apply the kitchen sink of different like statistical methods to try and put confidence intervals on all of this. So similarly to human evaluation, we can use power analysis to tell us what the size of our benchmark should be. We can report error bars along with all of our evaluation results and then use techniques like

[12:20](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=740s) resampling, which basically just means taking multiple rollouts from your LLM or, uh, from the judge as well to reduce variance. And what we end up with is we don't just report evaluation results, we report our evaluation results with clear confidence signals, and then we take interventions to try and make all of these confidence signals as tight as possible. And if we report those together, we can basically more meaningfully detect capability improvements in our model. We know if one score on an eval is better, it's because the model is actually

[12:50](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=770s) better, not because of noise or something like this.

[12:54](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=774s) So now we have a calibrated human evaluation process. We've created offline evaluations and calibrated judges that we can use to run this automatically. So we can just run this a bunch of different times and test out tons of different ideas to scale the experimentation process. And then from there, we can take whatever is working well and pass it back to human evaluation to have humans evaluate whether this is actually going well. And at this point, we might be asking ourselves whether we're ready to deploy this model or not. And the answer could be yes, but if

[13:24](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=804s) we, if we say yes at this point, we'd actually be skipping a super important part of the evaluation process, which is production monitoring. And the idea here is that we can define and track production metrics for our agent. So we're using how the agent is used in production as another mechanism for evaluating whether it's working properly or not. And to do this, you can look at a variety of different metrics. Some of them are explicit, like thumbs up, thumbs down feedback, preference feedback, and so forth. You'll see this done in like chatbots all the time. But it could also be implicit metrics,

[13:54](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=834s) like the length of a session, abandonment rate, and so on. So you can define these different metrics that are some- somehow correlated to whether the agent is performing well or not. And these are basically just quality signals from production that we could use to see if our agent is doing good. And what we could do is we can take these production metrics and measure them in an A/B test to ge- gain a relative sense of how our agent is doing. So seeing compared to what we already have in place, is the current agent doing better than that? And again, we can use a lot of statistical

[14:23](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=863s) techniques to tell us how long we need to run this A/B test and so forth. This is related to power analysis, which we talked about before. And the nice thing about A/B testing is that what we could use-- what we could do is basically treat this as a soft model deployment process. So instead of just sending the model out to everybody, we can vet the quality of this model in production with an A/B test and make sure everything's going well before pursuing a full rollout. And even after we roll out the model, there's a lot of opportunities to just continually production monitor. We can look for any concerning

[14:53](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=893s) trends in performance or even find cases where the agent is failing, new use cases and so forth, and use production to ground the other evaluation steps in the real world. So production monitoring is incredibly useful for extending evals, finding new failure cases, all of these things. We want to find all of these cases and then pass them back to the other parts of the evaluation process. So in this way, we just have to continually work towards improving all of the evaluation components over time. Uh, we've seen all of the different components that we're going to cover in this talk,

[15:24](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=924s) uh, but we can always make all of them better. So we could define new or better quality criteria, we can improve calibration of humans or judges, or we could look at production to find new error examples that we can work into offline evals and so forth. Or we could just take this entire process and repeat it in a new, new domain, so just build an entirely different eval.

[15:45](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=945s) So to sum everything up, evaluation is difficult, but for subjective tasks, it's e- especially difficult because we're dealing with more sources of noise. And we've seen a roadmap and all of these different components and tricks that we can use to reduce the amount of noise as much as possible to draw more confident conclusions. And in this roadmap, human evaluation serves as our North Star for quality. It provides a clear quality definition that humans agree on. And then from this, we could use this definition to calibrate LLM judges or rubrics as a proxy to

[16:15](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=975s) human evaluation, and this allows us to scale the experimentation process. Then anything that goes well, we can pass it to human evaluation and eventually deploy it. And when we deploy, we can use A/B testing. Production monitoring is another evaluation mechanism to make sure everything's going well in the real world. And this basically is something where we have to fight noise at every step. So because we're working with subjective evaluations, we know that they're going to be noisy. We need to be very skeptical of evaluation results and try and use statistics to give us like the best

[16:44](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=1004s) picture of how our agent is performing. So we can use techniques like power analysis, confidence intervals, resampling, and more to reduce the variance of the evaluation process as much as possible and provide clearer confidence signals. And if we put all of this together, we basically just end up with a multi-step evaluation process. We run tons of offline evals to give us an idea of what's going to go well. We pass the best ones to human evaluation to give us a higher fidelity view of how something is doing, and then we A/B test it to kind of test the waters in terms of well--

[17:15](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=1035s) whether it's doing well in production, and eventually deploy it. And once we deploy, we can continue to monitor for failure cases that we then work back into all of the other steps of the process. All right, so I know that's a lot of information, uh, but for any areas that were confusing or you wanna read more deeply into, I created this index here. So this links out to further reading on pretty much everything that we talked about, uh, both from myself and other people. So I encourage you to look into all of these details if it's interesting to you. Thank you so much, and I hope it was

[17:44](https://www.youtube.com/watch?v=zLc8zeGeNaY&t=1064s) helpful.
