Experimental investigation and useful progress
Experimental investigation connects a question to a proposed intervention, observed consequences, interpretation, and another decision. Automation can assume different responsibilities along this path; executing an experiment is only one of them.
The running investigation is constructed: find a database cache policy that reduces misses during bursts of one-time requests within fixed memory. A cache retains data for reuse. Here, a page is an equal-sized storage unit; a miss means the requested page is absent. Eviction chooses a resident page to discard.
One evidence-producing research pass
Observations require interpretation before they support a finding.
Read the diagram as text
- Research question.
- Proposed intervention.
- Experimental protocol.
- Execution.
- Recorded observations.
- Interpretation.
- Next experiment.
- Bounded finding.
- Research question → Proposed intervention: Motivates.
- Proposed intervention → Experimental protocol: Operationalized as.
- Experimental protocol → Execution: Specifies.
- Execution → Recorded observations: Produces.
- Recorded observations → Interpretation: Evidence for.
- Interpretation → Next experiment: Unresolved; further test justified.
- Interpretation → Bounded finding: Acceptance requirements met.
A useful optimization changes an operating outcome favorably. An empirical finding describes a supported relationship under tested conditions. Neither requires an entirely new idea: Aiden's reported research contributions largely implemented and combined ideas from human papers and community work. Valuable execution and original discovery are distinct accomplishments.
Novelty needs separate investigation. After the robot scientist Adam produced experimentally supported findings, literature checking revealed that some were already known but missing from its knowledge base.
Hypotheses, predictions, and executable protocols
A hypothesis is a testable proposed relationship or explanation. Falsifiability requires observations that could count against it. Competing hypotheses make an experiment informative when its possible outcomes distinguish their predictions.
| Explanation | Expected observation | Observation requiring revision |
|---|---|---|
| Reuse protection prevents displacement | Previously reused pages survive one-time bursts and miss less when requested again. | No returning-page benefit despite verified displacement in the control. |
| Extra memory explains the gain | The candidate retains more pages because its actual allowance increased. | The gain persists after memory accounting and controlled capacity comparisons. |
| Changed inputs explain the gain | The candidate receives a different request sequence or omits difficult requests. | Identical eligible inputs and complete request accounting preserve the difference. |
| The miss counter is faulty | An independent recount disagrees with reported misses. | Separately implemented checks agree on the recorded events. |
An experimental protocol specifies the intervention, comparison, eligible inputs, measurements, analysis, and decision rules. Operationalization translates a goal into an observable outcome. Measurement validity concerns whether that outcome answers the intended question; a well-implemented counter can faithfully measure the wrong thing.
| Part | Specified contract |
|---|---|
| Cache model and memory | Use read-only demand paging: fetch pages when requested, without prefetching. Total cache memory includes resident pages and policy metadata. Record resident capacity separately; equal page capacity need not mean equal total memory. |
| Eligible inputs and initial state | Define the intended workload population and eligible sessions. Preserve complete request sequences; specify separate initialization, warm-up, measurement windows, and state reset for each policy. |
| Primary outcome and analysis | Count misses over all eligible measured requests within each session. Declare whether the final estimate weights sessions equally or by request count, and fix the analysis timing before interpreting results. |
| Correctness and operating constraints | Require unchanged query results. Measure runtime separately and enforce memory limits. Fewer misses need not reduce complete workload runtime when bookkeeping or other costs dominate. |
An oracle is a procedure that checks correctness. Agreement with reference query results supports the checked behavior, not the reuse-protection explanation. Oracles, rubrics and human judgments develops that boundary; Controlled offline comparisons explains matched cases. The protocol table makes these measurement commitments inspectable without another diagram.
Controls, independent units, and mechanism tests
The control is the reference condition. Least recently used eviction, or LRU, discards the page unused for longest. A confounder is another factor that varies with the intervention and provides an alternative explanation. Prediction and causal effects explains why attribution requires more than observing a difference.
An experimental unit receives a treatment as a unit. Here, assign a policy to an entire independently sampled session, not individual requests: one request changes cache contents and therefore later outcomes. Thousands of requests within one session are not thousands of independent workload samples. Treating them that way is pseudoreplication and can understate uncertainty.
Cross both factors
ExampleProtection's effect is compared across both burst conditions.
Read the diagram as text
- Matched sessions.
- Bursts present.
- Bursts absent.
- Protection on.
- Protection off.
- Protection on.
- Protection off.
- Matched sessions → Bursts present: Insert bursts.
- Matched sessions → Bursts absent: Omit bursts.
- Bursts present → Protection on: Enable.
- Bursts present → Protection off: Disable.
- Bursts absent → Protection on: Enable.
- Bursts absent → Protection off: Disable.
Run baseline and candidate on matched sessions with separately initialized state. Blocking groups comparable conditions, such as workload and hardware, so comparisons occur within those groups. Randomize execution order to avoid consistently running one policy after the machine warms up. Shared contention still needs control; random order does not remove interference.
An ablation removes or disables a component to investigate its contribution. Disable reuse protection while preserving a valid policy and its allocated resources. Otherwise, changed capacity or broken behavior can explain the result. Comparisons of training-data ingredients illustrate the same issue: an unfiltered dataset with more examples also changes training effort unless that difference is controlled.
For the diagnostic comparison, cross protection enabled/disabled with disruptive bursts present/absent. Keep recurring requests, the returning-request window, resident capacity, and allocated metadata identical across protection toggles. Burst insertion is the intentional second intervention. Measure returning-page misses on a common denominator separately from whole-workload misses.
Four treatment combinations are not four adequate samples. Repeat them across independent sessions and estimate variation at that level. A conditional benefit supports the predicted response pattern, but other mechanisms can produce it; component removal does not automatically identify the underlying cause.
Exploratory search and protected confirmation
Exploration uses observations to select candidates and generate explanations. Confirmation tests a fixed prediction against new observations. Preregistration records the question and analysis decisions before outcomes are observed, making later changes visible. It does not validate measurements or enforce data separation.
Feedback can cross a boundary without exposing raw cases. A score summary that changes a cache detector's threshold makes that cohort development evidence for the revised candidate. Independent data boundaries applies to human steering as well as direct agent access.
Feedback changes evidence's role
ExampleHidden cases can influence a revision through their reported score.
C1 is fixed before E is assessed.
Read the diagram as text
- Cohort E.
- Candidate C1.
- E protected.
- Result R1.
- E used for development.
- Revised candidate C2.
- Fresh cohort F.
- F protected.
- Cohort E → E protected: Initial role.
- Cohort E → Result R1: Assessment inputs.
- Candidate C1 → Result R1: Evaluated candidate.
- Cohort E → E used for development: Changed role.
- Result R1 → Revised candidate C2: Guides revision.
- Candidate C1 → Revised candidate C2: Revised into.
- Fresh cohort F → F protected: Reserved role.
- Freeze. C1 is fixed before E is assessed. Active: Cohort E, Candidate C1, E protected. New: Cohort E, Candidate C1, E protected.
- Assess. Evaluation produces R1 from C1 and E. Active: Cohort E, Candidate C1, E protected, Result R1. New: Result R1.
- Revise. R1 shapes C2; E now supplied development feedback. Active: Cohort E, Candidate C1, Result R1, E used for development, Revised candidate C2. New: E used for development, Revised candidate C2.
- Prepare confirmation. Freeze C2; reserve fresh F for its assessment. Active: Cohort E, Candidate C1, Result R1, E used for development, Revised candidate C2, Fresh cohort F, F protected. New: Fresh cohort F, F protected.
A proposed access arrangement gives search workers development results while a separate assessment role controls confirmation workloads and results. Freeze the complete candidate—including detectors and routing thresholds—alongside the analysis and claim. Authorization must enforce those permissions outside the model.
Separate score channels can help organize this boundary. AlphaLab describes public scores returned to its research harness and a private held-out leaderboard for users. That arrangement alone leaves open whether private feedback subsequently influences selection. Failed confirmation can motivate further development, but a revision needs appropriately independent assessment.
Resource limits and evidence-based stopping
Stopping one execution, ending candidate search, and concluding an investigation are different decisions. Reserve confirmation effort before search consumes the available budget. No universal allocation fraction follows from the method; it depends on the cost of obtaining decision-relevant evidence.
| Level | Trigger | Permissible interpretation |
|---|---|---|
| Individual run | Timeout, resource limit, incorrect results, or completed measurement window. | Record termination and partial observations. The inspected autoresearch protocol separately bounds individual runs while leaving overall search open to human interruption. |
| Candidate search | Total compute, elapsed time, candidate count, pending commitments, or review allowance reaches its limit. | Stop new admissions and account for outstanding work. Exhaustion states what could be afforded, not whether the hypothesis is false. |
| Investigation | Planned confirmation supports a decision; uncertainty cannot be resolved; or further testing is futile under the remaining design. | Make a bounded conclusion, or end explicitly without one. Analysis timing and evidential stopping rules belong in the plan. |
The minimum meaningful effect is the smallest change that would alter the engineering decision. Specify it before interpreting outcomes. Statistical detectability and practical importance differ: enough observations can expose a change too small to justify maintaining a more complicated cache policy.
Futility means insufficient prospect of establishing the intended effect under the remaining design and resources. This computational use adapts established statistical terminology; clinical stopping rules do not prescribe software budgets. Safety violations require stopping, and invalid measurement machinery requires repair before further evidential interpretation.
A confidence sequence is a sequence of intervals designed to cover the target jointly across observation times under stated assumptions. Ordinary fixed-time intervals lack that automatic repeated-inspection guarantee. Sequential flexibility generally costs precision; choosing a valid construction is part of the experimental design.
Limits need enforcement across concurrent jobs and restarts, including already committed work. Budgets across concurrency and restart covers those mechanics. A resource counter that merely reports overspending cannot prevent another experiment from starting.
Experimental identity and provenance
Experimental provenance links proposals, implemented candidates, execution attempts, observations, and decisions. The following manifest is a practical design, not a universal schema. Its purpose is to identify what actually ran and preserve the evidence needed to inspect the conclusion.
| Record | Retain |
|---|---|
| Scientific intent | Hypothesis and protocol versions, prospective predictions, analysis timing, decision rules, and the reason for each revision. |
| Candidate identity | Parent candidate, actual executed files and uncommitted changes, configuration, and commands. HEAD alone can omit modifications; see Repository and execution baselines. |
| Inputs | Dataset membership, request order, split role, and initialization. A snapshot identifies a data state; a manifest identifies membership. See Dataset releases and reproducible lineage. |
| Execution conditions | Attempt identity, retry parent, randomness settings, dependencies, hardware, resource allocation, contention, termination reason, and actual consumption. Resource changes can alter feasibility and available strategies. |
| Model-mediated decisions | Model and prompt versions affecting proposals or judgments, sampling settings, and necessary input/output references. Restrict sensitive content; operational metadata does not require indiscriminate transcript retention. |
| Evidence and disposition | Measurement version, raw observations, uncertainty, exclusions and reasons, failed work, selection decisions, and total research cost—not just the winner's cost. |
Candidate identity survives retries
ExampleA decision uses retained outcomes, including unsuccessful attempts.
Read the diagram as text
- Hypothesis / protocol H1.
- Parent C0.
- Candidate C1.
- Inputs / environment S1.
- Attempt A1.
- Attempt A2.
- Outcome bundle O1.
- Decision D1.
- Hypothesis / protocol H1 → Candidate C1: Motivates.
- Parent C0 → Candidate C1: Derived candidate.
- Candidate C1 → Attempt A1: Executed.
- Candidate C1 → Attempt A2: Executed.
- Inputs / environment S1 → Attempt A1: Inputs.
- Inputs / environment S1 → Attempt A2: Inputs.
- Attempt A1 → Attempt A2: Retry lineage.
- Attempt A1 → Outcome bundle O1: Produces.
- Attempt A2 → Outcome bundle O1: Produces.
- Outcome bundle O1 → Decision D1: Evidence.
A file reference is not proof that its artifact remains accessible, and available artifacts are not proof of successful recomputation. Record unavailable dependencies and access restrictions explicitly.
Candidate selection and informative experiments
The search space is the set of permitted candidate changes; the search policy chooses what to try next. Existing code abstractions shape which changes are easy to express. They therefore influence search even when the agent could theoretically implement many alternatives.
| Policy | How it chooses | Comparison requirement |
|---|---|---|
| Random sampling | Draw configurations independently from declared distributions. | Specify ranges, distributions, evaluation access, and resources. Hyperparameter-search evidence does not establish a baseline for unrestricted program generation. |
| AI-proposed changes | Use prior work and observed behavior to propose implementations. | Track proposal sources and available feedback. Useful implementation can reuse existing ideas without establishing novelty. |
| Bayesian optimization | Fit a surrogate—a predictive model of outcomes—and select trials with an acquisition function combining predicted performance, uncertainty, and possibly cost. | Requires a suitable parameterized space and credible modeling assumptions; it is not automatically appropriate for arbitrary code changes. |
Exploitation favors candidates predicted to perform well; exploration probes uncertain candidates. An expensive trial may already be pending when another choice is made. Accounting for its possible results can avoid launching a duplicate. Cost-aware selection can favor expected improvement per unit time rather than the largest expected improvement alone.
In the cache search, a large redesign is pending, another parameter setting looks promising, and a cheap protection toggle could distinguish explanations. The toggle may have greater decision value even if it is unlikely to win the leaderboard. Value comes from how possible observations could change the eventual action, after accounting for their cost.
Preserve distinct proposal families rather than relaunching superficial variants of one idea. Record which results were available at selection time, including unfinished jobs. AlphaLab's worker postmortems illustrate feedback into subsequent proposals, although their usefulness depends on the interpretation of those observations.
Execution within protected experimental boundaries
An experiment harness runs specified candidates and records measurements under constraints. Its controls should protect evaluator code, confirmation access, resource allowances, and authoritative result admission from candidate edits. These are requirements for the proposed system; written instructions prohibiting evaluator changes do not themselves enforce isolation.
Interfaces can remove misleading paths. In Weco's reported preprocessing search, a shared training/testing interface allowed contamination; a stricter interface separated those paths. This illustrates why permitted operations matter, without establishing that one interface prevents every form of reward hacking.
Proposals cannot authorize themselves
ExampleExecution permission and result admission remain outside editable candidate code.
Read the diagram as text
- Proposer.
- Editable candidate.
- Authorization / resource gate. Outside candidate authority.
- Isolated runner.
- Protected evaluator. Candidate cannot edit scoring.
- Authoritative record writer.
- Rejected or stopped.
- Proposer → Editable candidate: Artifact: permitted edit.
- Editable candidate → Authorization / resource gate: Artifact: submit for checks.
- Authorization / resource gate → Isolated runner: Control: checks pass; budget available.
- Authorization / resource gate → Rejected or stopped: Control: checks fail or limit reached.
- Isolated runner → Protected evaluator: Data: execution outputs.
- Isolated runner → Rejected or stopped: Control: execution terminated early.
- Protected evaluator → Authoritative record writer: Data: checked measurements.
- Rejected or stopped → Authoritative record writer: Data: failure evidence.
The agent chooses a proposed next action; the runtime checks its preconditions and permissions. Plan, observe, and choose the next useful action explains this distinction. Changing the protocol requires explicit authorization and a new version rather than quietly redefining success during search.
Bounded roles can separate proposal, implementation, and checking, but different agent names do not establish independent verification. Combine repeatable computational checks with complementary review methods. Tests as evidence about a change explains why checked implementation properties remain narrower than a research explanation.
Initialize cache and process state separately, constrain shared-resource interference, and inspect actual side effects. Sandboxes and Execution Isolation covers containment mechanics. A retry remains linked to its original attempt; restarting work does not create an independent workload sample or erase earlier failure evidence.
Papers, candidate outputs, and experiment logs remain untrusted input when a model reads them. Embedded instructions can target an evaluator as readily as a research agent. Keep scoring separate from production credentials and write-capable tools; independently authorize any downstream action prompted by a score.
Trial validity and evidential outcomes
Execution validity and evidential status need separate fields. A completed process can produce incorrect results, while a valid experiment can reveal deterioration. The following constructed ledger preserves both dimensions; its qualitative outcomes are not observed cache results.
| Candidate / attempt | Execution validity | Observation and uncertainty | Interpretation / next action |
|---|---|---|---|
| C1 / A1 | Valid; retain all eligible measurements. | Development misses improve; repeatable effect unresolved. | Keep as a candidate for further testing, not an accepted finding. |
| C2 / A1 | Valid; no exclusions. | Deterioration under the tested condition. | Evidence against benefit there; retain the result and investigate applicability. |
| C3 / A1 | Valid; no exclusions. | Interval permits both meaningful benefit and harm. | Inconclusive; additional independent observations may be needed. |
| C4 / A1 | Invalid comparison: crash or incorrect query results. | Partial logs retained; intended miss outcome unavailable. | Repair before performance interpretation; link any retry as C4 / A2. |
A nonsignificant difference does not establish no effect. Practical equivalence instead tests whether effects outside prespecified meaningful bounds can be excluded. An imprecise comparison can establish neither improvement nor equivalence; equivalence does not mean exact equality.
Reporting only completed runs estimates performance conditional on completion. If failures concentrate on difficult workloads, that subset can be misleading. Preserve missing outcomes and exclusion reasons. A timeout may leave miss performance unknown while establishing that the attempt failed the operating-time requirement.
Selected results and adaptive uncertainty
The winner's curse is inflation from selecting a favorable noisy estimate. Even an approximately unbiased evaluation can become optimistic after choosing its best observed candidate.
Multiple comparisons provide more opportunities for chance wins. Optional stopping means deciding when to stop from accumulated results. These are distinct problems: fixing inspection time does not remove candidate-selection effects. An interval computed for one fixed comparison does not automatically account for the search that selected it.
Selection favors favorable noise
ExampleA lower reassessment can coexist with a real benefit.
Development
A=1, B=2, C=3, D=4.
Scroll sideways if the figure extends beyond the screen.
- 1. Assumed population effect
- 2. A
- 3. B, C, D
Read coordinates and regions as data
X: 0.5–4.5 dimensionless; Y: 0–8 misses per 1,000 requests, increasing up. Axes scaled independently; screen angles and distances are not comparable.
(0.5, 3); (4.5, 3)
(1, 7)
(2, 2); (3, 4); (4, 1)
Selected A: (1.2, 7.3)
Fresh reassessment
Only selected A is reassessed.
Scroll sideways if the figure extends beyond the screen.
- 1. Assumed population effect
- 2. A
Read coordinates and regions as data
X: 0.5–4.5 dimensionless; Y: 0–8 misses per 1,000 requests, increasing up. Axes scaled independently; screen angles and distances are not comparable.
(0.5, 3); (4.5, 3)
(1, 2)
| Design | Required commitment | Remaining limitation |
|---|---|---|
| Fixed confirmation | Freeze candidate, population, outcome, and analysis time before inspecting confirmation. | Inference still depends on valid measurement and independent units. |
| Planned comparison family | Specify the family and use multiplicity control suited to its claims. | Treating each comparison in isolation ignores the additional chances for false positives. |
| Sequential monitoring | Use a justified confidence-sequence construction and its observation assumptions. | Time-uniform coverage does not automatically repair adaptive candidate selection, dependence, or invalid measurements. |
Uncertainty in scores and differences explains interval interpretation. Here, the analysis must also respect the recorded selection history and session-level evidence. Adding requests from one session can improve its measurement without supplying the independent sessions needed for a broader workload claim.
Interpretation and discriminating follow-ups
Post-hoc explanations are constructed after seeing outcomes. They can generate valuable new hypotheses, but the observations that suggested them do not become prospective confirmation. Preserve the original prediction beside the revision.
| Observation | Unresolved issue | Next test | Decision consequence |
|---|---|---|---|
| Development improvement | Repeatable benefit or proposed mechanism. | Protected confirmation or a discriminating ablation. | Separate retaining the candidate from accepting its explanation. |
| Harm outside bursts | Whether usefulness is conditional. | Test the narrowed prediction prospectively. | Restrict scope; any new detector or threshold creates a revised candidate. |
| Wide uncertainty | Too few independent observations or unstable conditions. | Add independent sessions under a declared sampling plan; diagnose variation. | Resolve the decision only if the additional evidence is informative. |
| Counters disagree | Measurement correctness. | Repair and independently check event accounting. | Pause score optimization until the measurement is usable. |
One observation may remain compatible with several explanations. Retain contradictions rather than narrating every outcome as progress. Further investigation is worthwhile when its possible results could improve a decision enough to justify its cost; unresolved scientific questions can remain after a responsible engineering stop.
Fresh repetition and independent replication
Terminology varies. Computational reproducibility means consistent recomputation using the same data and procedures. Replication tests the scientific question with new data. Here, independent replication additionally requires meaningful separation of relevant investigators or implementations, reducing specific shared dependencies.
| Rerun | Execution / randomness | Workload / measurement / investigator | Evidence added |
|---|---|---|---|
| Recorded replay | Saved outcomes; no new execution. | All retained. | Reconstructs prior behavior; adds no observations. |
| Fresh repetition | New execution; specified randomness. | Same recorded inputs and procedures. | Tests recomputation, not new workload coverage. |
| New random settings | New execution and relevant randomness. | Same workloads and measurement. | Tests stochastic variation; a seed matters only where randomness enters. |
| New sessions | New execution. | New representative workloads; otherwise fixed. | Tests the finding against new workload evidence. |
| Independent replication | New execution and data. | Meaningfully separate investigator or implementation; dependencies disclosed. | Tests the finding with reduced shared error paths. |
Restoration, replay, and diagnostic evidence covers runtime mechanics. Executing code on saved inputs is fresh execution; reading saved outputs is not. Independently implementing a miss counter can challenge measurement errors, but another model endorsing the same report supplies neither new data nor an independent experiment.
Investigate disagreement through versions, workload conditions, measurement differences, and expected variation before combining results. Byte-identical output is not the universal criterion. Averaging incompatible conditions can hide precisely the dependence that the replication revealed.
Bounded findings and accountable decisions
A finding binds a claim to its tested conditions. External validity concerns applicability beyond those conditions. A cache policy supported on the sampled sessions and memory allowance is not thereby supported across other traffic patterns or database deployments.
Finding record
- Claim and scope — Identify candidate and protocol, target and sampled workloads, total-memory accounting, measured effect and uncertainty, and the conditions excluded from the claim.
- Evidence and remaining alternatives — Link confirmation, mechanism tests, negative results, and replication records. State which explanations remain compatible with the observations.
- Effort and disposition — Report total compute and human effort, retained artifacts, unresolved limits, and the accountable decision.
Legitimate outcomes include retaining an optimization within tested conditions, rejecting its proposed mechanism, or ending without sufficient evidence. Miss reduction alone does not establish database latency improvement. The analogous infrastructure distinction is between network microbenchmark gains and reduced time to train the actual model.
| Responsibility | Human decision |
|---|---|
| Question and domain fitness | Decide whether the problem matters and whether measurements and sampled conditions represent the intended use. |
| Research acceptance | Review remaining alternatives and novelty. Set mandatory acceptance checks outside the agent's discretion. |
| Consequential action | Authorize the specific deployment, publication, or external change. A favorable result does not authorize a changed target or scope. |
Research output must also be usable. Aiden describes a quality gate before submitting a pull request, separating local experimentation from community contribution. The existence of a gate does not specify its acceptance criteria; those criteria and the evidence satisfying them still need inspection.
Open questions
Measuring useful hypothesis generation remains difficult. Successful implementation can continue after promising ideas run out, while novelty is hard to score automatically. Progress would mean prospective tests where proposed experiments distinguish alternatives on unfamiliar problems, rather than merely producing additional executable variants.
Maintaining confirmation independence across a long research program requires controlling indirect feedback, including human summaries and stopping decisions. Fresh evidence can be expensive. Progress would be a documented access-and-query policy with guarantees matching the actual adaptive workflow, rather than an indefinitely reused private leaderboard.
Independent measurement remains an unresolved scaling constraint when multiple model reviewers share blind spots. Agreement can improve confidence in a judgment process without establishing factual correctness. Progress would pair reviewer validation with separately implemented measurements and targeted tests of shared failure modes.
The net research value of AI search needs matched comparisons. Proposal generation, failed trials, confirmation, and human review all consume resources, while unrestricted code search lacks an obvious random baseline. Progress would compare declared search spaces and total effort using independently confirmed, decision-relevant findings as outcomes.
