Contents
  1. Purpose and development
    1. What research automation contributes
    2. Questions, hypotheses, and predictions
    3. Turning points in automated discovery
  2. Design an informative comparison
    1. Measure the property in the question
    2. Controls and competing causes
    3. Ablations and interactions
  3. Preserve the experiment
    1. Specify the protocol before observing
    2. Trace findings to observations
    3. Execute without moving the target
  4. Choose the next experiment
    1. Improvement versus information
    2. Search from accumulated observations
    3. Parallelism and cheaper trials
  5. Interpret and update
    1. Negative, inconclusive, or invalid
    2. Confirm beyond the search
    3. Choose the follow-up
  6. Establish scope and decide
    1. Reproduction and independent replication
    2. Stop with a bounded claim
    3. Evaluate the complete campaign
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

Autoresearch

Research automation helps turn ideas into experiments and observations into subsequent work. Its attraction is practical: software can carry out repetitive implementation, execution, and analysis while people concentrate on consequential decisions. But faster experimentation makes the design of those decisions more important. This chapter explains how to specify what an experiment should establish, preserve what actually happened, and distinguish a promising result from a finding that survives scrutiny.

Purpose and development

What research automation contributes

Autoresearch is a broad label for AI-assisted or AI-directed research, not a single architecture. The work may involve proposing changes, executing experiments, inspecting results, or deciding what to try next. A script can automate a planned comparison; an agent selects actions using observations, as explained in Agent Engineering. Neither arrangement requires a team of agents.

Three achievements deserve different descriptions. Optimization finds a better artifact against a defined objective. An empirical regularity is a repeatable relationship among observations. An explanation proposes why that relationship occurs. Finding a useful regularity can be a discovery without establishing its mechanism; improving an artifact need not establish either. Langley, Simon, and Bradshaw made the distinction between empirical laws and explanatory theories explicit in their 1984 BACON report.

For example, the inspected Karpathy autoresearch protocol establishes a baseline, runs fixed-duration training changes, and keeps or discards candidates. That is bounded optimization. It is also distinct from changing the researcher itself: in Richard Socher’s usage, improving a separate small model is autoresearch, not full recursive self-improvement. The useful starting point is therefore the research question, not the autonomy label.

Questions, hypotheses, and predictions

A research question identifies what remains unresolved. A proposed solution names something to try. An optimization objective tells a search which outcomes to prefer. A hypothesis instead makes a provisional, testable claim; a prediction states an observation expected under specified conditions. Falsifiability means that some observations could count against the claim together with its assumptions. This extends the programmer’s practice of testing explanations of a failure.

A better result does not by itself distinguish explanations. Consider a hypothetical data-filtering study: a filter removes selected records, and a model learns from the remaining examples. Suppose that model performs better on the assessed tasks. Filtering may have removed unsuitable records, changed the proportions of topics encountered during training, or done both. Repeating the original comparison could confirm the improvement without separating these explanations. A useful follow-up changes the conditions so their predictions differ.

These are proposed explanations and conditional predictions, not observed results.
ExplanationDistinguishing predictionRemaining ambiguity
Removing unsuitable records contributesA benefit remains when topic proportions are held comparable.The matching procedure may also change which records are available.
Changed topic proportions explain the benefitReproducing those proportions without the same record rejection reproduces the benefit.Record suitability and topic composition may interact.

Strong Inference, John Platt’s 1964 methodological argument, emphasizes alternative hypotheses and experiments that distinguish them. The original article is useful guidance, not a guarantee of decisive tests. Exploration remains legitimate: unexpected observations can generate new hypotheses. They become predictions only when stated before the observations used to test them. Recording that boundary prevents a convincing retrospective explanation from being presented as an advance prediction.

Turning points in automated discovery

Automated discovery predates language models. Several complementary traditions developed around different bottlenecks: finding patterns, choosing informative experiments, manipulating equipment, searching programs, and preparing research reports. Statistical experimental design addresses whether comparisons are interpretable; automating their execution does not replace that responsibility.

These developments delegated different responsibilities rather than forming a ladder of general scientific autonomy.
LandmarkContribution and boundary
BACON — 1977: empirical-law searchPatrick Langley’s rule-based system found regularities and derived attributes that guided further observations in a programmer-defined artificial universe. Observable attributes and simulated laws were supplied; the implementation did not handle noisy data.
Robot Scientist Adam — 2009: hypothesis-directed laboratory workRoss King and colleagues connected biological knowledge, hypotheses, experiment planning, laboratory execution, and interpretation. This was a specialized yeast-research system with substantial human-built infrastructure.
Mobile robotic chemist — 2020: adaptive physical searchBenjamin Burger and colleagues’ Liverpool system combined laboratory instruments, a mobile robot, and observation-guided batch selection. Researchers supplied hypotheses defining ten mixture variables.
FunSearch — December 14, 2023: executable program searchBernardino Romera-Paredes and colleagues paired a frozen language model with executable evaluation and evolutionary selection. A human-supplied program skeleton restricted the editable logic.
The AI Scientist — August 12, 2024: research-workflow automationChris Lu and colleagues’ first preprint connected ideas, code changes, experiments, plots, manuscripts, and automated review. Its demonstrations began with human-written experimental templates.

The recurring boundary is what the system receives from outside: a representation of the problem, permitted interventions, instruments, objectives, and standards of acceptance. Progress can expand those boundaries without removing them. To understand a contemporary system, identify which research decisions it makes and which decisions its designers have already made.

Design an informative comparison

Measure the property in the question

Operationalization translates a property in a claim into something observable and a procedure for measuring it. Specify the intended quantity, operating conditions, calculation, reference checks, and uncertainty. A precisely computed value can still measure the wrong property. The international vocabulary of metrology distinguishes the quantity intended to be measured, the procedure, and the resulting value with its uncertainty.

In the filtering study, a dataset is the collection of records used for a task. Model fitting adjusts learned behavior using examples; Machine Learning Fundamentals explains that process. Downstream assessment tests the resulting model. Checking whether a filter accepts suitable records and checking whether those records produce a better model are different measurements. Raw answers, calculated task accuracies, and an interpretation of why performance changed must remain distinguishable.

DataComp-LM, Jeffrey Li and colleagues’ June 2024 study, compared dataset interventions through trained-model performance, fixing model and training settings within each scale. In the reported comparison, 1.4-billion-parameter models consumed 28.8 billion tokens, the model’s units of text. The RefinedWeb reproduction scored 27.5 on Core; a fastText text-classifier filter trained with OpenHermes 2.5 and ELI5 positive examples scored 30.2. Core aggregates 22 task accuracies normalized between random guessing and perfect performance. The scores measure the trained models, not the accuracy of the filters’ record judgments.

An evaluation establishes behavior under its specified conditions, not usefulness everywhere. See What an evaluation establishes and Choose checks that match the requirement. The experimental step is to choose a comparison that makes the measurement bear on the question. The distinction between record quality and downstream utility is developed further in Controlled comparison.

Controls and competing causes

An intervention is a deliberate change. A control is the reference condition against which its effect is interpreted. Confounding occurs when the comparison mixes the proposed effect with another influence that it cannot separate. If a filtered dataset is tested with a different model and a larger training allowance, the difference cannot be attributed to filtering alone.

For a proposed total filtering-policy comparison, establish the following relationship between conditions.
PropertyComparison
Starting collectionUse the same eligible source collection.
Filtering policyDeliberately vary the selection rule.
Model and training allowanceKeep the recipe and declared allowance comparable.
AssessmentUse the same target population and scoring procedure.
Resulting topic proportionsAllow policy-induced changes when estimating the total policy effect.

The last row matters. A consequence of the intervention is not automatically a confounder. Holding topic proportions fixed instead tests a narrower question: whether filtering contributes beyond the composition change. Both questions are useful, but their answers are not interchangeable. This is the distinction between an association and a specified intervention developed in Association, action, and causal effect.

Different controls answer different questions

Common eligible source → reference selection and filtering policy → two separate collections

Total-policy comparison
Reference recordsFiltered records

Keep each collection’s resulting topic proportions; they may differ.

↓

Train a separate model on each

Compare assessed outcomes with composition changes included.

Composition-matched comparison
Reference recordsFiltered records

Additional operation: match topic proportions across the two collections.

↓

Train a separate model on each matched collection

Ask whether a contribution remains beyond composition change.

Both comparisons use the same model recipe, declared training allowance, target population and scoring procedure.

A proposed comparison, not a reported DataComp-LM experiment. Matching changes the comparison and may change other record properties; it does not uniquely identify a mechanism.

An experimental unit is independently assigned a condition. For a training comparison, it may be a fresh run; the intended claim determines whether independent datasets are also needed. Random assignment or randomized run order prevents the schedule from systematically favoring a condition. Blocking compares conditions within groups that share a relevant nuisance factor, such as machine or data sample. Interference arises when one unit’s treatment affects another’s outcome—for example, concurrent runs competing for shared resources. Randomization does not remove that interaction. What each design supports explains the broader design choices.

Controls can also check the measurement process itself. Burger’s mobile robotic chemist included two baseline samples per batch to monitor stability. Such reference measurements help detect a drifting process; they serve a different purpose from the treatment comparison.

Ablations and interactions

An ablation disables or removes a component to test its contribution within a specified system. Its answer is conditional: removing a component from one configuration does not establish that the component is useless in every configuration. A factor is a deliberately varied property. An interaction means that changing one factor has a different effect at different settings of another.

For a small filtering experiment, consider deduplication, which handles repeated records, and topic reweighting, which changes topics’ relative training exposure. Their detailed mechanisms belong in Data Quality and Curation and Mixture weight is not corpus size. Crossing two enabled/disabled factors produces four conditions, following a full factorial design.

Denote each condition’s mean response by Ydr, with deduplication indexed first and reweighting second.

Compare deduplication-only with the reference, then combined with reweighting-only. If those two differences differ beyond uncertainty, deduplication’s contribution depends on reweighting. Changing one factor while permanently fixing the other would miss this. The interaction establishes a conditional response, not a unique explanation for it.

Compare the two deduplication effects

Ydr is the condition’s mean response: d = deduplication, r = reweighting; 0 off, 1 on.

ReweightingDeduplication offDeduplication onOn minus off
OffReference
Y00
Deduplication only
Y10
Δoff = Y10 − Y00
OnReweighting only
Y01
Combined
Y11
Δon = Y11 − Y01

Interaction: Δon − Δoff = (Y11 − Y01) − (Y10 − Y00)

Compare deduplication’s contribution with reweighting off and on. Repeated independent runs supply uncertainty for these condition means and their contrast.

Every condition must remain a coherent pipeline with comparable training allowances. Four conditions are not four adequate samples: repeated, properly assigned runs are needed to estimate variation. Zhengyao Jiang’s Aiden account reports a useful combination of model-architecture changes, quantization—representing numbers with fewer bits—and tokenization changes that alter how text is divided into model inputs. The reported gain motivates testing combinations; without controlled contrasts, it does not establish an interaction’s magnitude or cause.

Preserve the experiment

Specify the protocol before observing

An experimental protocol specifies changes, measurements, and interpretation. Its primary outcome answers the main question. Prespecify the smallest meaningful difference; detectability is not importance. Required repetitions depend on variation and needed precision, not a universal run count.

A practical protocol makes these choices inspectable.
ChoiceWhat to specify
ComparisonIntervention, reference, eligible units, assignment, and preserved conditions.
MeasurementPrimary outcome, calculation, observation window, and reference checks.
InterpretationMeaningful effect, uncertainty analysis, and planned repetitions.
ExceptionsExclusions, missing observations, execution failures, and permitted repairs.
ContinuationAllowed adaptations, analysis times, resource limits, and stopping reasons.

A pilot experiment checks feasibility and measurement. Revise defective metrics or impractical conditions, recording the pilot’s influence; design-shaping observations are not an untouched test.

Preregistration records questions and analysis plans before outcomes are observed. It makes prediction distinguishable from retrospective explanation; it does not validate the instrument. Jacob Thomas’s proposed persona study illustrates the distinction: its conditions, rubric, and directional predictions were described as fixed in advance, while the larger experiment remained unrun. Exploratory deviations can be valuable when reported as such.

Trace findings to observations

Provenance is a result’s origin and production history. Building on dataset lineage, experimental provenance separates planned conditions from actual execution and retained outputs. The W3C PROV model distinguishes artifacts, activities that use or generate them, and responsible actors. Applied here, an attempt uses identified inputs and generates observations; analysis produces a separate interpretation.

An experiment manifest should connect the following records rather than replace them with a summary.

  • Intent — Question, hypothesis, protocol version, proposed intervention, and parent candidate.
  • Actual execution — Applied change, exact code and data membership, configuration, model identity, environment or instrument conditions, and randomness controls.
  • Observations and analysis — Raw outputs, measurement calculations, analysis versions, and resulting claims.
  • Disposition — Completion, failure, exclusion, negative or inconclusive finding, plus reasons and human interventions.

Keep proposed predictions distinct from observed measurements. A revised analysis should produce a new conclusion linked to the original observations, not overwrite their history. In the 2026 A-Lab correction, manual reanalysis supported 36 of 40 originally reported successes; four remained inconclusive from X-ray diffraction alone. X-ray diffraction measures patterns used to identify crystal structures. Reinterpreting those measurements was not a new synthesis campaign.

New analysis, unchanged observations

Rectangles: artifacts · rounded pills: activities

Identified inputsProtocol, code, data, configuration
↓ used by
One execution attempt
↓ generates
One retained observation artifactUnchanged original observations
↙ same artifact used by both ↘
Original analysis

↓ supports

Original conclusion
Revised analysis

↓ supports

Revised conclusion
A generic provenance structure, not a reconstruction of A-Lab’s records. Reanalysis preserves both conclusions and adds no experimental replicate. Provenance enables inspection; it does not certify correctness.

A commit identifies source, not the data or environment it encountered. A seed identifies part of randomness control, not the complete procedure. A generated summary records an interpretation, not necessarily its supporting observations. Preserving the dependency graph makes inspection possible; it does not make the experiment valid.

Execute without moving the target

An agent can feed execution results into its next choice, but experimental execution needs an additional invariant: the comparison must retain its meaning. Separate editable candidate code from protected measurement rules and confirmation data. The inspected Karpathy protocol permits changes to train.py while prohibiting changes to prepare.py and its evaluator. Those are instructions, not proof that access controls enforce the boundary.

Enforcement belongs outside the proposer: restrict which artifacts it can modify and which data it can read. Tool availability does not establish authorization. Nor does a successful command establish that the intended intervention ran; inspect the effective configuration and resulting artifacts. Interpret execution results covers the execution distinction, while Bounded execution, not safe code covers containment.

This proposed architecture separates candidate writes, controlled execution, retained observations and protected assessment. Proposer and candidate cannot write evaluator code or read evaluator-only reference data. Fixed assessment inputs and containment do not establish correctness or independent confirmation.

A faster candidate is useful only if it still performs the required computation. A GPU kernel is a function executed in parallel on a graphics processing unit; GPU Programming and Kernel Optimization explains that execution model. Recursive’s 2026 research report describes invalid kernel candidates that cached outputs, relied on persistent state, or exploited timing details. Stronger correctness checks were needed before speed counted as progress. Human inspection remained part of validation, and the team acknowledged that errors could still be missed.

A protocol deviation is a departure from the declared procedure. Repairing a typo may restore the intended intervention; changing the dataset, resource allowance, or measurement window creates a different condition unless the protocol already permits it. Record the repair and reassess comparability. A crash supplies no numerical outcome unless failure itself is a defined response. Through interruption, retain the protocol, applied candidate, attempt status, observations, and repair history; checkpointing mechanics belong to the runtime.

Choose the next experiment

Improvement versus information

One experiment may be best for improving an artifact and another for understanding it. Expected information gain means expected reduction in uncertainty, calculated under explicit hypotheses and possible observations. Because the outcome is unknown, the calculation averages over possibilities using starting assumptions, often called priors. An experiment expected to perform poorly can still be valuable if the competing explanations predict different failures.

Returning to the hypothetical filtering study, the purpose changes which comparison is useful.
ExperimentPossible valueWhat may remain unresolved
Try a stronger combined filtering policyFind a better-performing dataset.Both record suitability and topic composition may predict improvement.
Compare filtering with topic proportions matchedTest whether a contribution remains beyond composition.Matching may alter other properties; both explanations may contribute.
Recreate the topic proportions without the same filterTest whether composition alone reproduces the benefit.Failure to reproduce it does not identify the missing mechanism uniquely.

Model-discrimination design formalizes this distinction, but informativeness depends on the alternatives considered. If all candidate explanations are wrong, separating them can be misleading. Experiment selection must also account for cost, delay, and whether the result could change a decision. The same principle appears in Allocate the next computation: additional work earns its cost through a better eventual decision, not through activity alone.

Search from accumulated observations

An experimental campaign is a related sequence of experiments pursuing a question or objective. Its search space defines the permitted candidates. A search policy determines how previous observations affect the next choice. Restricting the space makes comparisons easier to manage, but also excludes solutions the system cannot express.

Choose a policy for the experimental purpose and candidate representation.
PolicyHow it choosesMain consideration
Fixed designExecutes prespecified contrasts.Useful for interpreting factor contributions.
Random searchSamples configurations from declared distributions.A simple baseline; ranges and sampling distributions matter.
Local variationModifies a retained candidate.Can remain confined to nearby ideas.
Evolutionary selectionVaries candidates retained according to evaluation.Requires informative evaluation and an effective variation mechanism.
Bayesian optimizationUses predicted outcomes and uncertainty to select evaluations.Useful when evaluations are expensive and the predictive model is appropriate.

Random Search for Hyper-Parameter Optimization, Bergstra and Bengio’s 2012 paper, explains why grids waste trials when only a few dimensions strongly affect performance: many grid points repeat values along those important dimensions. Random sampling spreads trials across them. This is a baseline for configuration search, not a universal result about generated programs.

In Bayesian optimization, a probabilistic surrogate model predicts outcomes and uncertainty across candidate settings. An acquisition function uses those predictions to choose the next trial. Its measured outcome then updates the surrogate, changing what is worth testing next. Snoek and colleagues’ 2012 account includes expected improvement per second: a promising but slow trial may be less attractive than a cheaper one.

A new observation changes the next trial

Maximize a noiseless score on nine permitted settings. The initial observations are at x = 0, 0.5 and 1; six values remain hidden from selection.

Settings x < 0.5 cost 1 s. This control changes predicted cost, never the score or observations. The denominator is a fixed cost estimate here, not a learned duration model.

EI chooses
x = 0.875 · EI 0.1173

EI / second chooses
x = 0.25 · 0.0120 score/s

Aligned surrogate and acquisition plots. Dots are observed scores; solid line is predicted mean; band is plus or minus one posterior standard deviation. Bars show acquisition only at unobserved permitted settings.Predicted score · mean ± 1 SD-20123Expected improvement / second (score/s)0.0120000.1250.1250.250.250.3750.3750.50.50.6250.6250.750.750.8750.87511Candidate setting x

Next: x = 0.25 · predicted 0.135 ± 0.698 · 1 s · Best observed score: 1.339

Observed values and acquisition calculations

x0: 0.2000 · x0.5: 0.1691 · x1: 1.3392

xMeanSDEISecondsEI/s
0.1250.17970.49680.001610.0016
0.25 · next0.13470.69830.012010.0120
0.3750.11490.49480.001110.0011
0.6250.36120.49480.0045120.0004
0.750.72860.69830.0735120.0061
0.8751.14870.49680.1173120.0098
Model assumptions and expected improvement

Zero-mean Gaussian-process prior; covariance k(x,z) = exp(−(x−z)² / (2 × 0.22²)), variance 1. Observation noise is zero; 10⁻¹⁰ diagonal jitter stabilizes the solve. Hyperparameters stay fixed. The band describes this model’s uncertainty, not measured calibration.

With best observed score b, prediction μ and standard deviation σ, EI = (μ−b) Φ(z) + σ φ(z), z = (μ−b)/σ. Φ and φ are the standard-normal CDF and density. Equal acquisition values choose smaller x. Trials reveal one value, update the surrogate and exclude that setting from further selection.

Synthetic score function, fixed Gaussian-process assumptions and deterministic reset. Expected improvement concerns score gains, not information gain or the probability that an explanation is true.

Language models can supply the variation mechanism without serving as calibrated uncertainty estimators. FunSearch generated editable program fragments and selected them through execution. Researchers then inspected generated programs, recognized useful symmetry, and restricted later searches accordingly. Candidate generation, evaluation, and interpretation remained distinct contributions. General algorithms for searching partial solutions apply, but the experimental question still determines what makes a candidate worth testing.

Parallelism and cheaper trials

A pending evaluation has started but has not returned an outcome. Parallel batches save elapsed time, but their candidates are chosen before all earlier results are available. A pending-aware policy accounts for what unfinished trials might reveal; repeatedly applying an unchanged acquisition rule can select work already running.

Completion order is not neutral when duration depends on the candidate. Updating only from fast or successful trials conditions the observed sample on completion. Keep pending, failed, and completed attempts visible rather than treating unavailable results as poor scores. Shared-resource contention is a separate problem: it changes the conditions under which outcomes are produced. Resource changes have altered failure rates and available solution strategies even with model, harness, and tasks held fixed in coding-agent experiments.

Fidelity describes an approximation’s relationship to the target setting. Multi-fidelity optimization guides expensive evaluations with smaller datasets or shorter training. Candidate-dependent approximation errors can eliminate eventual winners.

Compare unchanged candidates and assessments across training lengths, measured in epochs—passes through training data. Filtering thresholds control record retention. Check for ranking reversals; reserve resources for target-setting evaluation and independent confirmation.

A cheap ranking can discard the target winner

CandidateFixed threshold1 epoch10 epochs
A0.270% · winner73%
B0.562%81% · winner
C0.866%76%
Invented accuracies for unchanged candidates and assessment; only training length changes. Keeping only the 1-epoch winner A discards B, the 10-epoch winner. This shows a possible reversal, not its frequency.

Interpret and update

Negative, inconclusive, or invalid

Effect size is the magnitude of a measured difference, such as an absolute change in task accuracy. Interpret it alongside uncertainty and the smallest consequential difference. Identify the sources of uncertainty explains why repeated measurements can vary. Here the decision is whether the result supports the prediction, rules out a useful benefit, or remains unresolved.

A result’s disposition should preserve the distinction between the phenomenon and the ability to measure it.
DispositionInterpretation
Meaningful negative findingA valid, precise comparison rules out the proposed benefit under tested conditions.
InconclusiveValid observations remain compatible with materially different decisions.
Invalid measurementThe procedure or its assumptions do not support the intended interpretation.
Execution failureThe intended procedure did not complete; retain the failure without inventing its missing outcome.

Nondetection is not equality: wide intervals can include harm and benefit. Equivalence testing rejects effects outside justified practical bounds, supporting practical—not exact—equality.

Same estimate, different precision

Three independent studies with estimates and confidence intervals, compared against minus1 and plus1 percentage-point bounds.Precise negligible0.1 [-0.3, 0.5] ppInconclusive0.1 [-2.5, 2.7] ppMeaningful benefit2.5 [1.5, 3.5] pp-4-2-10+1+2+4+5Accuracy change (percentage points)

Shading: prespecified practical bounds [−1, +1] pp. Only the first interval is wholly inside. Each row assumes valid measurements; failure or invalid measurement is not a point on this axis.

Invented independent studies with fixed-analysis 90% confidence intervals. An interval wholly within hypothetical prespecified ±1 percentage-point bounds supports equivalence under the corresponding two one-sided tests at 5%. Practical equivalence is not exact equality.

Pseudoreplication treats dependent observations as if they were independent experimental units. A thousand readings from one prepared sample do not provide a thousand independently prepared samples. Likewise, many training seeds on one dataset investigate run variation without adding datasets. A control can weaken an explanation, but remaining alternatives and shared measurement errors still limit the conclusion.

Confirm beyond the search

The winner’s curse arises because selecting the largest noisy result also selects favorable noise. In their 2010 model-selection study, Cawley and Talbot demonstrated continued improvement in a selection criterion while independent test performance deteriorated. Even an approximately unbiased measurement can become optimistic after selection. This is a failure mechanism, not a claim that every selected gain disappears.

Multiple comparisons offer repeated chances to select favorable results. Adaptive search uses earlier scores—even with cases hidden—to shape proposals. Freeze the candidate, claim, and analysis before untouched confirmation; feedback-guided revisions make confirmation data development data. See Protect the independent assessment.

Separate selection from confirmation

Example

Selection and confirmation use separate information paths.

Initial candidate
Development observationsAggregate feedback included.
Revised candidate
Frozen candidate, claim and analysisAnalysis also frozen.
Untouched observationsIntended-population sample.
Confirmation analysis
Bounded conclusion
Development observations, including aggregate scores, guide revision and selection. Candidate, claim and analysis are frozen before confirmation. Confirmation does not select the candidate being confirmed; its findings may guide later revisions, which need a new confirmation basis.
Read the diagram as text
  • Initial candidate.
  • Development observations. Aggregate feedback included.
  • Revised candidate.
  • Frozen candidate, claim and analysis. Analysis also frozen.
  • Untouched observations. Intended-population sample.
  • Confirmation analysis.
  • Bounded conclusion.
  • Initial candidate → Development observations: Evaluated.
  • Development observations → Revised candidate: Informs revision.
  • Revised candidate → Frozen candidate, claim and analysis: Frozen.
  • Development observations → Frozen candidate, claim and analysis: Informs selection.
  • Frozen candidate, claim and analysis → Confirmation analysis: Defines assessment.
  • Untouched observations → Confirmation analysis: Independent input.
  • Confirmation analysis → Bounded conclusion: Supports.

Optional stopping makes the stopping time depend on observed results—for example, ending a comparison as soon as it looks favorable. Ordinary fixed-sample intervals are not designed for repeated opportunities to stop this way. A confidence sequence provides intervals whose coverage guarantee applies across observation times, allowing result-dependent stopping under its stated assumptions. Howard and colleagues develop these procedures, generally trading wider intervals for monitoring flexibility. This guarantee does not automatically correct arbitrary candidate selection, multiple hypotheses, invalid measurements, or dependence among observations.

The practical choice is a planned fixed assessment or an appropriate sequential design, not an improvised sequence of significance checks. Preserve rejected trials and analysis changes so a reader can distinguish evidence that selected the candidate from evidence that tested the frozen claim.

Choose the follow-up

A result should change the next research decision, not merely the leaderboard. Retain a hypothesis when its predictions survive useful tests; weaken or narrow it when support is limited to particular conditions. Refine it when an observation reveals a missing condition. Abandon it when the contradiction concerns its central claim and the measurement is trustworthy. An unexpected result can redirect the campaign without changing what earlier experiments predicted.

Match subsequent work to the uncertainty it can reduce.
Unresolved issueUseful follow-upRemaining limit
Run-to-run noiseRepeat the declared comparison with appropriate independent units.More repetitions do not repair a wrong measurement.
Questionable readingsTest reference cases with known expected behavior.Passing reference cases does not validate every unknown case.
Competing causesAdd a contrast on which their predictions differ.An observed difference may still admit other explanations.
Uncertain scopeTest new data or a new operating setting.Success extends support only to the settings represented.

In AlphaLab, workers inspect training behavior and evaluation results, then return postmortems to a strategist. That is a useful implementation pattern for observation-driven selection, not a guarantee that the next hypothesis improves. In A-Lab, observed reaction intermediates informed subsequent recipe choices and helped avoid routes producing the same unsuccessful products. Both examples make retained outcomes useful for deciding what not to repeat.

Establish scope and decide

Reproduction and independent replication

Following the National Academies’ terminology, computational reproduction recomputes results using the same data and methods; replication addresses the same scientific question with newly collected data. Usage varies across fields. Here, independent replication additionally requires identifying which relevant dependencies or possible errors the new test does not share. Different personnel alone do not supply that independence. The basic record requirements are covered in Preserve the run and the assessment.

Different procedures challenge different dependencies; this is not a universal ranking.
ProcedureWhat changesWhat remains shared
Inspect saved outputsInterpretation or reviewOriginal execution and observations
Rerun the computationExecution attemptData, implementation, and measurement design
Vary initialization seedsOne source of stochastic variationDataset and selection history
Collect new experimental unitsEmpirical observationsPossibly the same implementation and instrument
Implement an independent testImplementation-specific dependenciesPossibly the same data and conceptual assumptions
Test another settingOperating context or populationPossibly the same method and measurement

Variance research separates initialization, data ordering, augmentation, data sampling, and search. Repeating only initialization can miss substantial data-sampling variation; bootstrap resampling still reuses observed data. Seeds are also not a complete determinism contract: PyTorch’s documentation distinguishes random-generator control from deterministic algorithms and does not guarantee identical results across releases or platforms.

Another model agreeing with an interpretation is another judgment, not a new empirical test. If results disagree, investigate changed data, implementation, instruments, and conditions before choosing which conclusion to retain. When an unavailable dependency prevents exact reproduction, name that limit and specify the narrower claim a different test can address.

Stop with a bounded claim

Stopping one run, ending candidate search, completing confirmation, and resolving a research question are different decisions. The inspected Karpathy instructions impose a run timeout while directing the overall search to continue until human interruption. A-Lab instead stopped searching a target when it became the majority product or available recipes were exhausted. Neither a timeout nor exhausted recipes establishes that no better solution exists.

Record why the work stopped, because the reason limits the conclusion.
ReasonPermitted interpretation
Resources or safety limitFurther work was not permitted; the scientific question may remain open.
Measurement defectInterpretation must wait for a valid procedure.
Adequate precisionThe declared decision may be supported within the tested scope.
Contradicted predictionRevise or reject the affected claim, subject to measurement and assumptions.
Low expected valueAvailable follow-ups are unlikely to improve the decision enough to justify their costs.

Futility means that further work under the present design is unlikely to establish the intended effect. It does not mean the hypothesis is false. The term is formalized in adaptive-trial guidance; using it for computational campaigns is a methodological adaptation, not a clinical regulatory requirement.

Validity and novelty also need separate judgments. Adam’s researchers found that some apparently new findings were already known but absent from its database. A-Lab’s correction clarified that its novelty language meant new to its prediction platform, not necessarily new to science. Reviewers must examine methods, observations, and prior work rather than merely endorse a generated report.

End with a bounded claim: identify the intervention and reference, the setting and population, the measured difference and uncertainty, and the alternatives still unresolved. If confirmation was unfinished, say so. A useful candidate with an unresolved explanation is a legitimate outcome; calling it a completed discovery would erase the work still needed.

Evaluate the complete campaign

The best artifact and the process that produced it are different assessment objects. To test whether automation helped, compare credible human-directed, fixed-search, and AI-directed procedures with declared information access, candidate space, resources, and review conditions. Hold execution support fixed to study proposal quality; hold proposals fixed to study execution reliability. Changing experimental-design assistance asks another question. Measure the value of discretion explains the architectural comparison.

Keep campaign outcomes separate, and mark unavailable measurements as unavailable rather than zero.
OutcomeAccounting boundary
Useful conclusionsInclude informative negative findings and unresolved claims, not just winners.
Experiment expenseInclude failed trials, screening, and confirmation.
Elapsed timeMeasure through a defensible conclusion, including waiting and review.
Human effortInclude setup, intervention, verification, and integration.
Survival of findingsState which selected claims underwent confirmation or replication.

Parameter Golf’s May 2026 retrospective reports independent reproduction of every record-track leaderboard submission. It also describes agents propagating invalid approaches and submission volume exceeding manual inspection capacity, prompting automated triage for human review. That establishes verification work alongside optimization, not matched net researcher-hours saved. Similarly, The AI Scientist reported that successful execution could conceal incorrect implementation; automated manuscript review was not independent replication.

The mobile robotic chemist provides a different accounting boundary. Its 688 experiments over eight days found a light-driven hydrogen-producing formulation measured at 21.05 micromoles per hour, versus 3.36 for its baseline. The workflow required approximately two years of development; labor savings were estimates rather than a randomized human comparison. Physical experiment throughput and the total effort needed to establish the workflow are both relevant.

These cases do not form matched experimental arms, and they do not establish general productivity superiority. They show why evaluation must follow the whole campaign. Automation contributes when it helps reach useful, inspectable conclusions with acceptable resources—not simply when it generates more candidates, runs, or reports.

Open questions

  1. Reliable observation remains a bottleneck when scientifically important distinctions are small, specialized, or poorly represented in model training. Better proposal generation cannot compensate for systematically missed features. Progress would mean validating observation systems against domain references and showing that improved perception changes experimental decisions correctly.

  2. Research agents need better ways to move beyond familiar local changes without abandoning testable questions. The difficulty is distinguishing a genuinely informative new direction from an elaborate variation or a rediscovered public idea. Progress would combine broader proposals with prospective tests and traceable attribution.

  3. Net research productivity remains difficult to establish because generation, execution, and review costs move together. Matched campaign studies would need to follow all attempted work through confirmation and measure human effort as well as useful conclusions. Faster output alone cannot reveal whether verification has become the new bottleneck.

  4. Historical rediscovery tests remain vulnerable to knowledge already stored in model parameters. Restricting retrieved papers to an earlier period does not remove that knowledge. Stronger evidence would distinguish reconstruction using supplied observations from recall or recombination of later work.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

30 matching talks

TalkSpeakerEventYear
Richard SocherAI Engineer World's Fair 20262026
Stefania DrugaAI Engineer World's Fair 20252025
Ievgen VakulenkoAI Engineer World's Fair 20242024
Kobie CrawfordAI Engineer Europe 20262026
Jacob E. ThomasAI Engineer World's Fair 20262026
Anju KambadurAI Engineer Summit 20252025
Remy GuercioAI Engineer Europe 20262026
Anirban ChatterjeeAI Engineer World's Fair 20262026
Boris StarkovAI Engineer Europe 20262026
Hubert MisztelaAI Engineer World's Fair 20242024
Sachin KumarAI Engineer World's Fair 20262026
Ben KunkleAI Engineer Europe 20262026
Diane LinAI Engineer World's Fair 20262026
Jerry Wu, Wyatt MarshallAI Engineer World's Fair 20252025
Armanas PovilionisAI Engineer World's Fair 20262026
Jesse HanAI Engineer World's Fair 20252025
The Pipeline Is Dead

Transcript reviewed

Iris ten TeijeAI Engineer World's Fair 20262026
Robotics: why now?

Transcript reviewed

Quan Vuong, Jost Tobias SpringenbergAI Engineer World's Fair 20252025
Danny Gollapalli, Ben Hylak, Zubin KotichaAI Engineer Europe 20262026
Erina KaratiAI Engineer World's Fair 20262026
Tejas BhaktaAI Engineer World's Fair 20262026
Vayum Arora, Zhengyao Jiang, Dixing Xu, Dhruv SrikanthAI Engineer World's Fair 20262026
Zubin AysolaAI Engineer World's Fair 20262026
Hariprasad SudharshanAI Engineer NYC 2026 Online Talks2026
Elie BakouchAI Engineer World's Fair 20262026
Miguel González Fernández, Corby RossetEvent unknownUnknown
Shubhankar SrivastavaAI Engineer World's Fair 20262026
Lakshya AgrawalAI Engineer World's Fair 20262026
Tim SweeneyAI Engineer World's Fair 20262026
Hrishi OlickelAI Engineer NYC 2026 Online Talks2026

References

Coverage and source review
Processed transcripts
23 processed in full · 4 in the curated path
Automated source review
Passed
Metadata candidates
11 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Multi-fidelity Bayesian Optimisation with Continuous Approximations

    https://proceedings.mlr.press/v70/kandasamy17a/kandasamy17a.pdf

  2. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses

    https://pure.tue.nl/ws/portalfiles/portal/80918653/lakeequi2017.pdf

  3. Randomized policy comparison and experimental uncertainty

    https://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTheWebSurvey.pdf

  4. karpathy/autoresearch: program.md

    https://raw.githubusercontent.com/karpathy/autoresearch/master/program.md

  5. First Steps Toward Automated AI Research

    https://www.youtube.com/watch?v=pWXUkLP9uWM&t=824

  6. Heuristics for Empirical Discovery

    https://www.ri.cmu.edu/pub_files/pub3/langley_pat_1984_2/langley_pat_1984_2.pdf

  7. First Steps Toward Automated AI Research

    https://www.youtube.com/watch?v=pWXUkLP9uWM&t=885

  8. Strong Inference

    https://pages.cs.wisc.edu/~markhill/science64_strong_inference.pdf

  9. The preregistration revolution

    https://psychologicalsciences.unimelb.edu.au/__data/assets/pdf_file/0007/2888098/The-preregistration-revolution.pdf

  10. BACON: A Production System That Discovers Empirical Laws

    https://www.ijcai.org/Proceedings/77-1/Papers/057.pdf

  11. The Automation of Science

    https://www.researchgate.net/publication/24254152_The_Automation_of_Science

  12. NIST: Completely randomized designs

    https://www.itl.nist.gov/div898/handbook/pri/section3/pri331.htm

  13. A Mobile Robotic Chemist

    https://strathprints.strath.ac.uk/74759/1/Burger_etal_Nature_2020_A_mobile_robotic.pdf

  14. Mathematical discoveries from program search with large language models

    https://www.nature.com/articles/s41586-023-06924-6

  15. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    https://arxiv.org/html/2408.06292v3

  16. JCGM 200:2012 — International vocabulary of metrology

    https://www.bipm.org/documents/20126/2071204/JCGM_200_2012.pdf

  17. DataComp-LM: In search of the next generation of training sets for language models

    https://arxiv.org/html/2406.11794v3

  18. NIST AI RMF Playbook: Measure

    https://airc.nist.gov/airmf-resources/playbook/measure/

  19. NIST randomized blocks: applying controlled comparisons to RAG evidence

    https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm

  20. The Target Trial Framework for Causal Inference From Observational Data: Why and When Is It Helpful?

    https://pubmed.ncbi.nlm.nih.gov/39961105/

  21. What exactly is ‘N’ in cell culture and animal experiments?

    https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2005282

  22. Guardrails and interference in online experiments

    https://arxiv.org/pdf/2212.11366

  23. NIST Engineering Statistics Handbook: One variable at a time

    https://www.itl.nist.gov/div898/handbook/pri/section2/pri212.htm

  24. OpenThoughts: Data Recipes for Reasoning Models

    https://arxiv.org/html/2506.04178v1

  25. NIST Engineering Statistics Handbook: Full factorial designs

    https://www.itl.nist.gov/div898/handbook/pri/section3/pri333.htm

  26. NIST: replication and randomized experimental runs

    https://www.itl.nist.gov/div898/handbook/pmd/section3/pmd33.htm

  27. How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI

    https://www.youtube.com/watch?v=iCj_ATyThvc&t=399

  28. NeurIPS Paper Checklist Guidelines

    https://neurips.cc/public/guides/PaperChecklist

  29. Practical Guide to Controlled Experiments on the Web

    https://ai.stanford.edu/~ronnyk/GuideControlledExperiments.pdf

  30. Adaptive Designs for Clinical Trials of Drugs and Biologics

    https://www.fda.gov/media/78495/download

  31. Generalization in Adaptive Data Analysis and Holdout Reuse

    https://arxiv.org/html/1506.02629

  32. The Miranda Hypothesis: How Hamilton (the Musical) Poisoned Your Persona Evals

    https://www.youtube.com/watch?v=IJXjTLPzvAU&t=2081

  33. PROV-DM: The PROV Data Model

    https://www.w3.org/TR/prov-dm/

  34. Author Correction: An autonomous laboratory for the accelerated synthesis of inorganic materials

    https://www.nature.com/articles/s41586-025-09992-y

  35. An autonomous laboratory for the accelerated synthesis of inorganic materials

    https://www.nature.com/articles/s41586-023-06734-w

  36. Reproducibility and Replicability in Science: Understanding Reproducibility and Replicability

    https://www.nationalacademies.org/read/25303/chapter/6

  37. PyTorch documentation: Reproducibility

    https://docs.pytorch.org/docs/main/notes/randomness.html

  38. OWASP Access Control

    https://owasp.org/www-community/Access_Control

  39. Guide, Verify, Solve: The Engineering Discipline Agentic Development Demands

    https://www.youtube.com/watch?v=03l29gJXpCE&t=566

  40. First Steps Toward Automated AI Research

    https://www.recursive.com/articles/first-steps-toward-automated-ai-research

  41. Quantifying infrastructure noise in agentic coding evals

    https://www.anthropic.com/engineering/infrastructure-noise

  42. Effective harnesses for long-running agents

    https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

  43. Optimal Bayesian Design for Model Discrimination via Classification

    https://arxiv.org/html/1809.05301

  44. On Optimal Game-Tree Search Using Rational Meta-Reasoning

    https://www.ijcai.org/Proceedings/89-1/Papers/053.pdf

  45. Random Search for Hyper-Parameter Optimization

    https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

  46. How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI

    https://www.youtube.com/watch?v=iCj_ATyThvc&t=679

  47. Autonomous Agents for Scientific Tasks - Sina Shahandeh, RADiCAIT

    https://www.youtube.com/watch?v=XLEYtv3cMlw&t=37

  48. Practical Bayesian Optimization of Machine Learning Algorithms

    https://www.cs.princeton.edu/~rpa/pubs/snoek2012practical.pdf

  49. National Academies: inference with missing outcomes

    https://www.ncbi.nlm.nih.gov/books/NBK209900/

  50. Accounting for Variance in Machine Learning Benchmarks

    https://proceedings.mlsys.org/paper_files/paper/2021/file/0184b0cd3cfb185989f858a1d9f5c1eb-Paper.pdf

  51. NIST experimental-design terminology: replication, interactions and confounding

    https://www.itl.nist.gov/div898/handbook/pri/section7/pri7.htm

  52. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation

    https://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley10a.pdf

  53. Best practices for the human evaluation of automatically generated text

    https://aclanthology.org/W19-8643.pdf

  54. Time-uniform, nonparametric, nonasymptotic confidence sequences

    https://arxiv.org/html/1810.08240

  55. Generalization in Adaptive Data Analysis and Holdout Reuse

    https://proceedings.neurips.cc/paper_files/paper/2015/file/bad5f33780c42f2588878a9d07405083-Paper.pdf

  56. BioWatch PCR Assays: Key Components of a Performance Standard

    https://www.ncbi.nlm.nih.gov/books/NBK305483/?report=printable

  57. ALPHALAB: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs — Brendan Rappazzo

    https://www.youtube.com/watch?v=kiqubc5b5Yo&t=690

  58. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    https://arxiv.org/html/2306.05685v4

  59. OWASP LLM10:2025 Unbounded Consumption

    https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/

  60. What Parameter Golf taught us

    https://openai.com/index/what-parameter-golf-taught-us/

  61. First Steps Toward Automated AI Research

    https://www.youtube.com/watch?v=pWXUkLP9uWM&t=411

  62. The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

    https://arxiv.org/html/2504.08066v1

  63. Autonomous Agents for Scientific Tasks - Sina Shahandeh, RADiCAIT

    https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1030

  64. How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI

    https://www.youtube.com/watch?v=iCj_ATyThvc&t=347

  65. LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries

    https://www.youtube.com/watch?v=gbM7k3NauNI&t=777

  66. Autonomous Agents for Scientific Tasks - Sina Shahandeh, RADiCAIT

    https://www.youtube.com/watch?v=XLEYtv3cMlw&t=476

  67. ALPHALAB: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs — Brendan Rappazzo

    https://www.youtube.com/watch?v=kiqubc5b5Yo&t=643

  68. ALPHALAB: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs — Brendan Rappazzo

    https://www.youtube.com/watch?v=kiqubc5b5Yo&t=599

  69. How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI

    https://www.youtube.com/watch?v=iCj_ATyThvc&t=142

  70. Verifiable Environments for AI in Biology — Kenny Workman, LatchBio

    https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=170

  71. Verifiable Environments for AI in Biology — Kenny Workman, LatchBio

    https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=463

  72. Verifiable Environments for AI in Biology — Kenny Workman, LatchBio

    https://www.youtube.com/watch?v=3ZMUiFaQ3qg&t=731