Evals

What an evaluation establishes

Look beneath the sushi in the original photograph on the left. There is sauce under the food. In the candidate enhancement on the right, that sauce has disappeared. Uber calls the criterion completeness: did the edit retain the details of the dish? Chopra marked the edit as a failure. Looking only at the enhanced photograph, a reviewer could judge its appearance and never know anything was missing. [4][5]

At Uber, product, design, policy and legal staff helped decide what a better photo should preserve. The team assesses completeness and naturalness separately and allows an unsure judgment. That distinction tells an editor why even a natural-looking plate might need another edit. [6]

A single pair of photographs is enough to reject this edit. To compare two versions of the editor, run both on the same dishes. To find out whether the editor helps merchants, look beyond the photograph to the work a merchant does, including corrections. The choice of scorer is a separate decision. A person, an executable check or a calibrated model can score a case when it can observe the property you care about. [4][7][8][9]

Intended work and comparison baselines

A baseline is the alternative against which a change will be judged. For replacing a service, use the deployed service. For introducing assistance, use the workflow people already have. For deciding whether a learned system adds value, a simple rule can make a revealing baseline. Give each alternative the information and help it would have in the proposed use. Define who is eligible, which mistakes have serious consequences and where the clock stops. Time to generate a patch and time to finish a review-ready change have different finish lines. [11][12][8]

In early 2025, METR recruited 16 experienced developers who submitted 246 issues from familiar open-source repositories. Random assignment determined whether AI assistance was allowed for each issue. Developers worked until they considered the change ready for review, including the tests and documentation it needed. With the tools studied, primarily Cursor with Claude 3.5 or 3.7 Sonnet, completion took a reported 19% longer when assistance was allowed. Participants subsequently believed assistance had saved them time. Timing the whole job captured work their impressions did not. The comparison applies to those developers, tasks and tools. METR’s study report [8]

Before comparing scores, specify the smallest gain worth changing systems for. Set separate limits for outcomes that must remain acceptable. For example, a team seeking shorter completion times might also cap serious defects and human review hours. These limits are guardrails. Reporting time, quality, review effort and cost separately lets the decision-maker see which constraint a faster or cheaper system would violate. AI Cost and Performance Engineering [13][14]

Cases, trials and outcome boundaries

An evaluation case describes one piece of work in enough detail to tell whether it succeeded. For a request to change a reservation, that includes the request itself, the starting reservation, the information available to the agent and the change the customer wants. One execution is a trial. To compare trials, specify what tools and help the system can use, how much time and how many retries it gets, when it must stop and how its work will be graded. These rules are the protocol. The software that runs it is the harness. [7][15]

In τ-bench, an agent serves a simulated customer using tools and domain policies. The benchmark compares the final database with an annotated target and checks information the agent must convey. Its authors identify a gap: an agent can reach the target state without obtaining required confirmation. To check whether the agent asked permission before requesting each change, match every write request in the transcript to an earlier approval for that target and value. Then read the reservation to check the final value and what the agent told the customer. If the order of committed changes matters, inspect an authoritative write history. An earlier value may already have been overwritten by the time of the final read. τ-bench’s task and reward definitions [16][17]

Record execution status separately from the grader’s verdict. A refusal can satisfy a prohibition. A malformed return can fail a format requirement. At a timeout, the deadline has been missed, but an external write may still complete. A crashed verifier leaves the property it was checking unresolved. Even a successful read has semantics: an eventually consistent DynamoDB read can omit a recent successful write. Record the key, read target, consistency mode and observation time before interpreting an absent value. DynamoDB read consistency [18][19][20]

Harbor makes a case inspectable as files. instruction.md gives the task, task.toml supplies configuration, and the environment and tests specify where work happens and how to inspect it. A separate verifier receives declared artifacts. If a needed artifact never reaches it, its verdict cannot establish the artifact’s contents. Similarly, a fresh trial container needs its mounted files and external state prepared for the case. Harbor task structure · Software Engineering Fundamentals [15][21]

Representative work and diagnostic cases

The target population is the work named in the claim. The sampling frame is the work available to draw from. A random sample of escalated conversations can describe those conversations while missing every request resolved automatically. Specify the period and exclusions, and keep related turns from one episode identifiable. A slice is a subgroup relevant to the decision. Its share of traffic, or prevalence, determines how much it contributes to an overall traffic estimate. Data Quality and Curation [11][22]

Pooling gives each case equal weight. Averaging the two rates gives each slice equal weight. For a traffic estimate, use the proportions of the intended traffic. For finding failures, it can be worth putting most of the effort into exceptions. Keep the resulting challenge set separate: a defect on one valid challenge case demonstrates a possibility, while its frequency in a deliberately difficult collection says little about how often customers encounter it. [24][23]

Diagnostic cases can also test a relationship without supplying a complete answer. CheckList changes incidental place names in sentiment examples where the judgment should stay the same. Adding a negative statement can support a directional check that sentiment should not become more positive. Verify that the transformation really has that meaning for the chosen input. Lyft similarly seeded support scenarios with sampled requests and varied their conditions. Every generated variation needs a plausible situation and a defensible expected action. CheckList · Synthetic Data [25][26]

Controlled collections and shared tasks

A benchmark fixes tasks, assessment procedures and reporting conditions so that different systems can be compared. An early practical problem was comparing ways of indexing documents for retrieval. With funding awarded to Aslib in July 1957, Cyril Cleverdon’s team at Cranfield indexed the same 18,000 documents using four descriptor systems. They controlled indexing time and balanced document categories. If one system retrieved different material, the choice of collection and time spent indexing were less plausible explanations. Cleverdon’s 1962 report [27]

The first Text Retrieval Conference (TREC), in November 1992, brought independent research groups into a coordinated comparison on large full-text collections. A shared topic described the information need and what would make a document relevant. Each system could formulate its own search query. Exhaustive human judgments were impractical at that scale. Organizers took the top 200 documents per topic from each participating run, pooled them, removed duplicates and had that subset judged. The judgments could be reused across systems, though relevant documents could remain outside the pool. TREC-1 overview [28]

Shared tasks also made it possible to ask whether learning transferred across problems. In 2018, Alex Wang and colleagues introduced the General Language Understanding Evaluation benchmark, GLUE, with nine English tasks spanning different domains, difficulties and amounts of training data. Some test sets stayed private. An average summarized performance across the suite, while an expert-built diagnostic collection probed phenomena such as negation. Inspecting how a model handled a negated statement could reveal a weakness too small to move the suite average appreciably. GLUE paper [29]

From fixed answers to executable outcomes

A benchmark saturates when leading systems have little headroom left for a score to distinguish further progress. By 2019 this was happening on GLUE, even though particular tasks and language phenomena remained difficult. SuperGLUE retained two difficult GLUE tasks, added harder ones and broadened the answer formats. It sustained a comparison across systems by changing the test. Its scores therefore answer a different question from scores on the original collection. SuperGLUE [30]

Sometimes the comparison needs a better assessment. One SWE-bench test required an exact deprecation warning absent from the issue given to the developer. A valid repair could fail on its wording. OpenAI and the SWE-bench authors had professional developers screen issues, tests and environments. In August 2024 they released a filtered subset of 500 tasks. A score on the subset has a different task mix and denominator from a score on the original. Introducing SWE-bench Verified [35]

A score can rise because more tasks are genuinely solved. Other explanations require different investigations. Prior exposure can make a task familiar. Trying many configurations and reporting the best can select favorable noise. Direct optimization can also exploit a mismatch between a measured property and the goal, a problem associated with Goodhart’s law. DeepMind describes an agent rewarded for raising the bottom face of a block: flipping the block earned the reward without stacking it. Here the behavior itself shows exactly where the proxy and the goal diverge. Specification gaming [36][37][38][39]

Requirements, graders and denominators

A grader applies a rule to an observation. A metric summarizes observations numerically. It is often easiest to build the first grader around a failure someone can already recognize. In their account of Rechat’s real-estate assistant Lucy, Hamel Husain and Emil Sedgh describe finding invalid placeholders in application data and encoding defects in inexpensive, repeatable assertions. A placeholder check has a specific purpose. Passing it leaves the usefulness of the rest of the message open. Rechat’s domain-specific evaluations [40][20]

In software testing, a procedure for deciding whether behavior is correct is called a test oracle. Exact matching works for a unique identifier. Executable tests can accept different implementations that satisfy the same property. Reference similarity measures resemblance to one or more examples. A rubric lets an assessor evaluate specified qualities, and a preference identifies which alternative the assessor favors. Each requires evidence suited to its claim. A schema check can establish that an amount is numeric. Establishing that the amount is owed requires the transaction. A faithfulness grader can check whether a report’s claims follow from its sources even when the sources are mistaken. Structured Outputs and Tool Calling · Ragas Faithfulness [20][41][42][43]

EvalPlus gives a concrete false acceptance. A generated function had to return the sorted, unique elements common to two lists. It sorted intermediate values, then converted them through a set, losing the ordering guarantee. It passed the original tests. Additional valid inputs exposed an unsorted result from the same program. The defect was present throughout. What changed was the grader’s ability to detect it. EvalPlus, Figure 1 [44]

Coverage of the assessment

Of 100 attempts, 90 returned artifacts and 80 received quality verdicts. The pass rate among assessed artifacts is 72/80, or 90%. Verified success across requests is 72/100. Even if every ungraded artifact passes on inspection, only 82 requests could have met both requirements. Record critical failures separately too. Letting good wording offset an unauthorized action in a weighted average would change the acceptance rule. [45][46]

Errors made by the grader

Precision is the fraction of acceptances that are valid, 36/42. Recall is the fraction of valid artifacts accepted, 36/40. The six false acceptances matter to two different readers. An auditor inspecting accepted work finds 6 errors among 42 acceptances. A developer investigating how invalid outputs escape the grader finds 6 escapes among 60 invalid artifacts. Give the denominator with the rate. When it is zero, the rate is undefined. [48]

Human standards and disputed judgments

A rubric states the property being judged, the observable difference between grades and examples on either side of the boundary. A factual judgment requires access to evidence. A policy judgment may require domain expertise. A reader preference calls for people whose preferences the application is meant to serve. Pilot the rubric on actual outputs. Collect independent decisions before discussion, hide irrelevant system identities, vary presentation order and allow both ties and insufficient evidence. A tie expresses equal preference. Insufficient evidence means the assessor cannot make the comparison. [42][33][49]

In the EvalGen study, two participants wanted different results when extracting names from text containing #Nike. One wanted to exclude names found only in hashtags. The other wanted to return Nike without the hash. Both received a checker that looked for a hash character in the output. It would accept Nike under either rule. It could detect a stray #, but deciding whether the name belonged in the output required looking at the input. Writing down the expected result for this one case exposed both the disagreement and the limit of the check. EvalGen study [50]

Adjudication asks reviewers to explain their decisions against the evidence and intended rule. Correct mistakes, clarify ambiguous instructions and revisit affected cases, including earlier agreements. Preserve unresolved judgments and their reasons. Agreement before discussion measures independent consistency. Consensus afterward records what reviewers settled together. Reviewers can agree on a mistaken reference. For multistep work, separate diagnostic credit from task completion: a correct lookup about the wrong person demonstrates the lookup skill while failing the original request. Data Quality and Curation [49][51]

Model judges as measurement instruments

A large language model (LLM) judge applies an assessment rule to work produced by a system. It may grade one answer against a rubric or compare two answers. A reference-supported judge receives evidence such as a policy, source document or expected answer. A reference-free judge can estimate quality before an authoritative answer exists. If a necessary observation is missing, give the judge a way to return an unjudgeable verdict. Asking it to infer that an email was sent cannot recover a missing delivery record. [52][53][17]

At YouTube Ads, Preetika Bhateja and Daniel Bump describe agreeing on examples of passing and failing outputs before scaling human ratings. The team samples expert and model judgments and examines disagreements. This makes a falling score investigable: did the output violate the requirement, did reviewers interpret it differently, or did the judge misapply it? How Evals and Prompts Shape Agent Behavior [9]

Use development examples to refine the judge, then freeze its rubric, prompt, model and threshold. Measure false acceptances and rejections on held-out, independently assessed work, including the costlier errors within each critical slice. Count ties and missing verdicts separately. MT-Bench reported 85% GPT-4–human preference agreement and 81% human–human agreement in one comparison excluding ties. Those numbers describe preferences on those pairs. A product team still needs to measure how often its judge accepts an actual policy violation. MT-Bench and Chatbot Arena study [54][55][47]

Testing irrelevant influences

Probe the judge by changing a feature that ought to be irrelevant. Reverse the same two answers and track the content selected. Two votes for the first position now select opposite answers. A stochastic judge can also reverse itself at fixed order, so repeat both presentations and compare the selection frequency of the same answer. Other probes vary redundant length or confident phrasing while preserving substance. On two news-summary datasets, model judges favored their own summaries more than human quality comparisons would predict. A judge can also follow a candidate’s reasoning into the same error. Independent references help test that possibility. [56][57][58][33]

Candidate text is untrusted input to the grader. In JudgeDeceiver experiments, adding an optimized sequence to an inferior answer induced tested judges to select it. Compare clean and modified candidates in both orders in the actual harness. Budget independent inspection of accepted work as well as flagged work. Otherwise the judge’s most consequential misses can be the cases nobody sees. Repeat validation when the judge or the work changes. JudgeDeceiver experiments · AI Security [59][60][61]

Contamination and assessment independence

A holdout reserves cases to assess a system after development choices are complete. Contamination occurs when exposure undermines that independence. A repository snapshot is legitimate input to an issue-solving task. If the retrieval index also supplies the accepted patch, a passing repair supports a different claim. Check what could reach the generator through training, prompt examples or retrieval, and what could reach the people tuning the grader. Keep task instructions accessible while restricting solutions and final assessment feedback. Validation without leakage [22][32][39]

Split by the source of dependence. Keep turns from one customer episode together, and place paraphrases derived from one problem on the same side of a split. Separate by time when the intended claim concerns future tasks. Training on translated or paraphrased benchmark material has increased scores in controlled experiments while escaping exact-overlap checks. Semantic matching can both miss exposure and flag innocent similarities. Fresh tasks reduce some exposure risks, though a public task and an undisclosed training history rarely permit a guarantee. LiveBench’s 2024 design proposed monthly refreshes and automatically checkable answers, a choice that also excluded open-ended work such as composing an email. LiveBench design [22][39][62]

Even a score can carry information from the test into development. If a team tries many prompts and keeps the highest scorer, it favors prompts that happen to do well on those particular cases. To assess the chosen prompt, freeze it and test it on fresh cases. To assess the method for choosing a prompt, repeat the whole choice: set aside a group of cases, select a prompt using only the remaining cases, and score that prompt on the group set aside. Repeat with other held-out groups. Each score then reflects how well the selection method worked on a new group, including occasions when it picked a lucky winner. Cawley and Talbot on selection bias [38]

Preserving executions and assessments

Give a run an identity when the system executes, and give a verdict its own identity when a grader assesses the evidence. Rescoring asks what a different assessment says about saved work. Rerunning asks what the application produces on another execution. LangSmith’s December 2024 SDK made this distinction explicit: supplying an experiment identifier reused stored outputs, while supplying an application function produced new ones. A trace is the linked execution record that lets a grader find the relevant observations. LangSmith release account · Identify the system that ran [63][17]

If both generator and grader change, score both sets of saved outputs with each grader. For each grader, compare the gain it gives the new generator on the same cases. If the graders disagree on a saved output, the output itself is fixed. Check the documented criteria to see whether the standard changed. Two implementations can apply the same rubric differently, and a model grader can vary across repeated judgments. Keep the four verdicts linked to their case. Rescoring requires the evidence each grader needs, which may be missing from truncated tool results or redacted messages. [63][50][56][66]

Name what a second investigator could repeat. Recomputing a score from saved verdicts can be exact even when regenerating the outputs is impossible. OpenAI’s archived seed guidance described consistency as best effort even with matching request parameters and backend fingerprints. A seed cannot restore a changed hosted model or a missing database snapshot. Independent replication takes a further step, collecting new data to address the same question. National Academies on reproducibility and replication [67][68]

Attempts and the deployed selection policy

When a system gets several attempts, the same results can answer different questions. For each task, single-attempt success is the chance of passing on one try. With k independent tries, pass@k is the chance that at least one passes, and pass^k is the chance that every try passes. The suite averages these probabilities across tasks. If a service generates several answers and delivers just one, grade the answer it selects as well. A checker used during evaluation may recognize a good answer that the service itself fails to pick. Sayash Kapoor on evaluating agents [31][34][69]

This task contributes a success to the any-pass count and a failure to the first-attempt, all-pass and delivered counts. A repair after feedback is different again. In Uber’s fries example, the quality check rejected the portion size and plating. Its feedback went into a second edit that passed. Those two edits constitute one run of a feedback policy. To compare it with another policy, record assistance, resets, retry limits, stopping, abstentions and total time and cost. Use a common budget for an equal-resource comparison. If comparing systems as offered, disclose what each spent to deliver its result. Reasoning and Test-Time Compute · Agent Runtimes and Harness Engineering [71][72][65]

Sampling units and uncertainty

Drawing new tasks changes the work being tested. Rerunning the same task shows how the system’s actions vary on that task. Asking another assessor to grade the same output can reveal disagreement about the verdict. To estimate a success rate across new tasks, start by counting independently sampled tasks. A confidence interval expresses uncertainty in the estimated rate. If we repeatedly drew samples in the same way and computed a 95% interval from each, about 95% of those intervals would contain the population success rate under the method’s assumptions. The percentage describes the interval procedure. Predicting whether an individual answer will pass is a separate question. [73][74][42]

Rare failures require particular care. Even if independent cases each have a 1% failure probability, the chance of seeing no failures in 100 trials is $0.99^{100}$, about 37%. And a thousand runs of one familiar task leave unseen tasks just as unseen. An interval can be narrow under the sampled conditions while missing a whole class of requests. [2][11]

A bootstrap estimates sampling variation by drawing units from the sample with replacement and recalculating the statistic many times. Draw at the levels that the design actually varies. DigiWorld keeps its curated apps fixed. Within each app it resamples task scenarios, environmental variations and stochastic runs. Repeating only the last step would miss the sensitivity to different starting screens or data configurations. The resulting suite interval still describes those apps. It supplies no sample of additional apps. DigiWorld paper · Pierluca D’Oro’s explanation [75][74]

Paired gains and meaningful differences

Run baseline and candidate on the same cases with the same assessment. A matched comparison retains which case improved or regressed. For binary outcomes, count a gain when only the candidate passes and a loss when only the baseline passes. Subtract losses from gains, then divide by the number of paired cases. Preserve the individual losses so a critical failure remains visible. [76][77]

For uncertainty in the gain, calculate the difference within each pair. If there are prespecified repetitions, first summarize each version’s trials within each task. For representative, independently sampled tasks, bootstrap the task differences and calculate their mean on each draw. This preserves the pairing and weights tasks equally. When tasks share a source episode, keep dependent cases together and resample at the independent group level with the intended weighting. Separate intervals for the two systems omit information about how often they fail on the same tasks. SciPy’s paired bootstrap [76][78]

Plan around the smallest worthwhile difference and the expected variability between pairs. Statistical power is the chance that the analysis will detect an effect of a specified size when it exists. More independent cases can improve precision, but even a precisely measured gain can be too small to justify a change. Prespecify the primary comparison and critical slices. Searching many candidates or testing many hypotheses creates opportunities for accidental wins. Confirm exploratory findings on fresh cases and account for multiple tests. Fix the stopping rule in advance or use a sequential method designed for repeated inspection. Stopping an ordinary fixed-sample test as soon as it looks favorable changes its error rate. [79][38][80][14]

An ablation changes one component under otherwise matched conditions to test its contribution. In a synthetic data-pipeline remediation benchmark, a learned action policy matched the observed success level of an equivalent hand-written policy on a compact set of incident states. That result gives a team a reason to consider the simpler implementation. Establishing equivalence within a meaningful margin requires enough precision to rule out substantial differences. Remediation policy comparison [81][14]

Assessing probability calibration

Suppose a system attaches a probability to the event that its answer will pass an independent correctness check. If it assigns 0.8 to many comparable answers, do about eight in ten pass? This agreement between forecast probabilities and observed frequencies is calibration. In a reliability diagram, group independent labeled cases by forecast. Plot each group’s mean forecast horizontally and its fraction correct vertically. Points below the diagonal indicate overprediction in that sample. Show counts and uncertainty because ten cases provide much less evidence than a thousand. The choice of groups matters too: combining overprediction and underprediction can hide both. Probability calibration [1][3]

In the figure, both panels use the same 40 labeled answers, of which 20 pass. A constant forecast of 0.5 agrees with the overall frequency but gives no guidance about which answer is more likely to pass. The second forecaster assigns a higher score to a group where 14 of 20 pass, compared with 6 of 20 in the lower group. It separates the groups in the right direction, a useful ranking, yet its forecast of 0.9 exceeds the observed 0.7. Accuracy counts correct decisions at a chosen decision rule. Ranking and probability calibration describe different properties of the scores behind those decisions. From probabilities to actions [3][82]

Scoring individual forecasts

Identify where the number came from. The probability of generating a token concerns the next unit of text. A percentage stated in an answer is itself generated text. Agreement among several models can reflect shared mistakes. Each signal needs independent outcome labels before it can stand for probability of correctness. In factual question answering, the calibration of verbal estimates depended on the model and how confidence was elicited. Calibration can also deteriorate when inputs change, as experiments with increasing image corruption demonstrated. Tokenization · Study of elicited confidence [84][33][85]

Abstention and the complete review workflow

Abstention withholds an answer. Deferral transfers responsibility for the case to a person or another process. Raising a score cutoff can restrict automation to easier cases. To assess that choice, measure automatic coverage, the fraction of eligible cases handled automatically, and selective risk, the mean loss among those cases. A risk–coverage curve plots those two quantities at different cutoffs. Ranking can be sufficient to select cases, even if the score itself is poorly calibrated. Choose the cutoff on development data and assess the resulting policy on independent work. [82][22]

What happens to everyone below the cutoff? Suppose a shift brings 80 cases and reviewers can finish 25. Deferring a quarter sends 20 to the queue. Deferring half sends 40 and leaves at least 15 unfinished by shift end without another route. A fall in automatic errors could coexist with worse service across all 80 cases. Count reviewer errors, time spent and requests still waiting at the deadline. Capacity must fit the period in which work arrives. Monthly spare capacity cannot finish this shift’s work on time. Research on capacity-constrained deferral demonstrates the distinction with simulated analysts. Workload-constrained deferral [86][82]

Reviewer performance also depends on what the interface shows. Duolingo inserted fabricated copy-typing alerts into legitimate historical English Test sessions and showed them to proctors in their normal workflow. Reviewers accepted half the planted alerts despite scoring above 90% on their usual accuracy checks. No test taker was affected. The denominator here is deliberately false alerts shown to reviewers. It measures a failure of review under that condition and says nothing about how often the live detector raises a false alert. Duolingo’s review study · Design Engineering and AI Interfaces [87]

Offline evaluation and replay limits

Offline evaluation assesses cases outside the live workflow. A recorded history provides the information available up to a decision. Replay can test which question a changed agent would ask at that point. Suppose the original question asked for a delivery address and the alternative offers to reissue a card. The recorded reply contains an address because the customer heard the first question. Feeding that reply to the alternative agent invents a response to the offer. At the fork in the figure, continuing the new path requires a new observation or a model of how the customer and environment would respond. [88][89]

Reconstruct what was accessible at decision time, including document versions, permissions, tool responses and state. A later correction may be useful for grading the old answer while leaking the solution if supplied as input. Tool behavior, latency and the mix of people seeking help can also change. Such changes in the conditions generating the observations are called distribution shift. Identify which condition matters to the proposed use. When deployment conditions shift [65][90][11]

A simulator can generate a reply to a new question, but its usefulness depends on whose behavior it reproduces. Lyft found that a model playing a support customer explained problems patiently in the style of a helpful assistant. After adapting it to actual customer utterances, the team reported lower offline scores. Changing the simulated user had changed the difficulty of the work. Deciding whether the new scores better predict service quality requires observing real interactions. Lyft’s support evaluation · RL Environments and Simulators [91][92]

A requirement can be missing in both offline and live assessment. OpenAI reported favorable offline evaluations and small-scale live preference signals for its April 2025 GPT-4o update, alongside expert concerns about behavior. Its launch checks lacked a specific assessment for sycophancy, the tendency to agree with or flatter a user inappropriately. Rollback began on April 28. More observations of the same preference signal would still leave the behavioral requirement inadequately tested. OpenAI’s postmortem [93]

Shadow runs, canaries and causal comparisons

Exposure determines which consequences can occur. When Zed evaluated Zeta2 in shadow, the model produced edit suggestions from real inputs but the editor hid them. Engineers could inspect the suggestions. Only after Zed displayed them in a small live rollout could it observe whether a developer accepted one. Acceptance is useful evidence about the suggestion. Determining whether it saved work requires following the edit through completion against a comparison workflow. How Zed developed Zeta2 [94][95]

For a canary, use metric windows long enough to observe the consequences that matter. Reverting traffic stops further exposure. Already issued refunds or sent messages require separate reconciliation. For an experiment, specify what receives the assignment and how its outcome will be counted. Requests from one person share a user. When people share work or a review queue, one person’s assignment can change another person’s outcome. This interference can make the observed difference unlike the effect of switching everyone. Assigning teams together can reduce spillovers at the cost of fewer independent units. Association, action, and causal effect [96][13][79]

Randomization protects the comparison among units that enter it. In a follow-up productivity study, METR found that some developers declined to participate or withheld issues they expected AI to accelerate. Random assignment of the submitted tasks could not recover those missing tasks. Outcomes can disappear after assignment too. Under selective review, a system’s own decisions determine which cases receive expert labels. A missing label, an unanswered survey and an outcome still pending are three different observations. Join executions to eventual outcomes and track eligibility, response rates and follow-up time alongside the result. METR’s follow-up · Connect operations to outcomes · Sample for the question [98][99][79]

From failed requirement to tested explanation

Start with the requirement and the evidence that appears to contradict it. Check whether the task was feasible, the reference was accurate, the environment supplied what the system needed, and the grader could inspect the result. Cranfield investigators put the missed document beside the question, indexing record and search record. They could then distinguish a poorly expressed question from an indexing or search problem and send doubtful cases for further review. In an agent trace, the last error may be several steps downstream of the first consequential mismatch. Cranfield’s analysis of failures · Build a defensible execution account [100][88][101]

At Rechat, unsent emails were among the application failures Husain and Sedgh used to develop repeatable checks. A case like this needs a precise boundary. Did the assistant compose the requested email, ask the tool to send it, or achieve the required delivery outcome? Each question points to a different observation. Preserve the smallest case that reproduces the discrepancy. A trace can suggest whether the tool was never called or its feedback was misinterpreted. The assistant’s own explanation cannot settle which change would repair the behavior. [40][17][102]

To test a feedback hypothesis, one possible experiment runs the same task in two conditions. In one, supply the suspected faulty tool feedback. In the other, replace only that feedback with a verified result. Fix the initial state, instructions, model, tool implementation and budget. Randomize run order, repeat the comparison and independently check downstream completion. Improvement supports a contribution from the feedback under these conditions. The agent’s interpretation of that feedback might still be part of the defect. Keep the confirmed case for regression testing, probe related cases for the same weakness, and evaluate the repair on fresh work outside the investigation. From traces to experiments [102][103][104]

Bounded decisions and revalidation

A replacement may be valuable if it costs less and preserves quality within an acceptable loss. For a higher-is-better score, let $d$ be candidate minus baseline. Choose a justified loss margin $m>0$ before seeing the result. Non-inferiority requires evidence that $d>-m$. With a two-percentage-point margin, a suitable lower confidence bound of −1 point clears the boundary. A bound of −4 points leaves an unacceptable loss plausible. Two-sided equivalence additionally requires ruling out meaningful differences in the other direction. Failing to detect a difference from zero can leave both questions unanswered. Non-inferiority and equivalence [105][14]

Record the tested versions, protocol, population, comparison, uncertainty, remaining risks, decision owner and conditions for reassessment. An evaluation recommendation supplies evidence to the person authorized to release the service. Preserve stable regression checks for known defects alongside refreshed samples of the work people now bring. A revised grader may require rescoring. A changed dependency, population or routing cutoff can require fresh trials and a new capacity check. Software Engineering Fundamentals · AI Platform Engineering [97][106][107][63]

Frequently asked questions

Open questions

1. 01

   How can a shared benchmark remain both fresh and comparable? Rephrased solutions can escape overlap checks, while replacing tasks changes the difficulty of the collection. Comparing systems on overlapping old and fresh task families, with documented exposure and changes in case mix, would help distinguish progress from a change of test. [39][62]

2. 02

   When does sending more work to reviewers improve the outcome for everyone served? Routing changes which cases receive expert labels, and review capacity varies across shifts. Following automatic, reviewed and unfinished cases through their deadlines, with independently sampled quality checks and actual review times, would reveal whether lower automatic error compensates for delays and reviewer mistakes. [99][86]

3. 03

   Which simulated interactions predict the effect of a change in live service? Making a simulated customer more demanding can lower all scores without improving the choice between candidates. Comparing predicted candidate differences with later workflow outcomes across user groups would identify the decisions for which the simulator is informative. [92][79]