How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix

Run a creative-writing model twice on the same prompt and it can give you two different pieces of writing. Give the same piece to an LLM judge twice and its rating can change, too. When a new version of the writer earns a higher average score, an engineer has to work out how much of that difference to credit to the model. A small, expensive benchmark gives chance variation in either component room to influence the result.

Wolfe frames this as a spectrum of verifiability. A symbolic check can establish whether a proposed math answer is equivalent to the expected one. Creative writing allows many acceptable answers, and even experienced readers can disagree about their quality. Wolfe’s approach gives each part of the evaluation a distinct job. People define what counts as good work, an automated grader makes it practical to compare many candidates, and a production experiment tests whether a promising change improves the experience people actually have.

Wolfe starts with a small group of subject-matter experts. They choose a few criteria that matter in the finished work, specify how to score each one, and provide examples showing how to assign a rating. A criterion might call for a binary decision or a point on a rating scale. In either case, the examples give reviewers a concrete basis for deciding which score a piece deserves.

The first test of those instructions is to have several people independently rate the same outputs. When their scores diverge, ask how each reviewer reached the decision. Their explanations can reveal an ambiguity that survives a careful reading of the guidelines. Revise the relevant criterion or example, then measure agreement again. The agreement people achieve after calibration also gives the team a reference point for judging an automated grader later.

Benchmark size depends in part on the improvement you want to detect. Wolfe recommends power analysis to estimate how many examples a comparison needs. Inputs include the size of the difference you care about, the variability in observed results, and the significance level you choose. Sampling several completions for the same task can help estimate that variability before the team commits to a large round of human review.

Which examples earn a place matters just as much. Wolfe looks for diversity, resemblance to production work, and difficulty that distinguishes models. To assess that last property, sample model runs and inspect which problems the models solve. A difficult task that every candidate fails tells you little about their relative capabilities. Human and model-based filtering can improve the accuracy of the examples over time. The same selection process applies when an automated judge takes over the grading.

Once human review becomes the bottleneck, one option is to give an LLM judge the same general quality guidelines for every task. Another is to write a checklist specific to each assignment and have the judge evaluate each item separately. Wolfe points to a rubric for culturally adapting a Dr. Seuss book into a Netflix film. That assignment admits many creative choices, yet a reviewer can still check individual requirements of the adaptation. The tradeoff is in what carries over to the next task. General guidelines can be reused, while the checklist spells out what matters for this particular assignment.

To test either approach, keep a set of examples whose human ratings the team trusts. Compare the judge’s ratings with those labels using a measure suited to the scale, such as accuracy for binary decisions or correlation for scored outputs. Human-to-human agreement helps interpret the result. On a particularly subjective task, matching the agreement that calibrated reviewers achieve with one another is a more grounded target than choosing a universal cutoff.

A separate question is whether the judge picks up on something users experience. Wolfe reports that on one Netflix project his team established a causal link between the quality their LLM judge assessed and streaming behavior. When the judge rated something poorly, viewers were more likely to abandon the show after starting it. That result ties one judge to a specific outcome, beyond showing that it could reproduce a reviewer’s rating.

Even a well-calibrated judge introduces variation on each run. Sampling several outputs for the same prompt captures variation in the model’s work. Having the judge rate a fixed output several times captures variation in the grading. Repeating only the first step leaves open how much a different judgment of the same work might change the comparison.

Wolfe recommends reporting error bars alongside benchmark scores and using repeated samples to reduce the variance of the estimate. The uncertainty matters when deciding which candidates merit expensive human attention. Offline evaluation lets a team try many ideas quickly. A gain large enough to distinguish from run-to-run variation gives the team a reason to send that candidate back to calibrated human reviewers for a closer look.

A candidate that does well with both the judge and human reviewers can enter an A/B test against the system already in use. The choice of outcome depends on the product. Users can express a preference directly, as with a thumbs-up or thumbs-down in a chatbot, or leave an implicit signal such as session length or abandonment. The team needs a reason to believe the chosen behavior reflects quality. Power analysis can then help determine how long to run the comparison before deciding on a wider rollout.

After rollout, individual failures and new uses can be more informative than the aggregate trend alone. A production failure can supply a missing case for the offline benchmark. It can also prompt the team to revise a quality criterion and recalibrate the reviewers and judge who use it. That gives the next model comparison a way to detect a problem that the previous version of the evaluation let through.