# Autonomous Agents for Scientific Tasks - Sina Shahandeh, RADiCAIT

Sina Shahandeh, RADiCAIT

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/XLEYtv3cMlw-autonomous-agents-scientific-tasks) · [Video](https://www.youtube.com/watch?v=XLEYtv3cMlw)

## Autonomous Agents for Scientific Tasks

A CT-to-PET research workflow shows how explicit problem hierarchies and specialist model review can help coding agents move beyond local optimization.

Familiarity with model training, convolutional networks, and coding agents will help; PET imaging and registration are explained as they arise.

### When the optimization loop runs out of ideas

[0:12](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=12s)

Give a coding agent a model, an error metric, and permission to change the code. It can run experiments and search for changes that reduce the error—the basic hill-climbing pattern Sina Shahandeh introduces through [Karpathy’s autoresearch](https://github.com/karpathy/autoresearch). The harder question is what happens when the next useful experiment requires a new idea.

In open-ended scientific work, implementation can continue after useful progress has stalled. An agent can edit code and run experiments over substantial data, yet exhaust the hypotheses that would make those experiments worthwhile. Shahandeh describes this as a shortage of research taste. His opening comparison shows agents plateauing while capable humans continue improving, with the strongest humans improving further; it is an illustration of the problem, not a benchmark result established here.

![A paper title above a Human vs Agent line chart; blue human curves rise while orange and green agent curves flatten.](https://ai.engineer/api/talk-essay-media/10fe501592b14596eeac1511655e1c7f4b4fc50e1f927d31ea695a2d400cab53)

Human and agent performance over time, with annotations showing agents plateauing while humans continue improving.

The scientific method separates these activities: observe a situation, ask a question, formulate a hypothesis, implement an experiment, and learn from its result. Memory and organized records of activity help prevent repeated mistakes. Shahandeh regards that learning machinery as comparatively tractable. **The central challenge is generating a hypothesis worth testing next.**

### A concrete research objective: CT to synthetic PET

[2:50](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=170s)

At [RADiCAIT](https://www.radicait.tech/), the motivating problem is **in silico PET**: generating a synthetic PET image from a CT scan. Consider a pulmonary nodule visible in CT. The clinical question is whether it is cancerous, and a PET scan can provide information about tissue activity. The demonstration places a chest CT beside a synthetic PET image, with the nodule marked in both.

![Grayscale chest CT beside a colored in silico PET image, connected by an AI arrow, with green lung-nodule boxes.](https://ai.engineer/api/talk-essay-media/e63193b6b1e09d8316a4b057d82639e680c7428f44f1d9c39653b99d0d93fb1d)

Diagnostic CT and in silico PET shown side by side, with a lung nodule marked in each.

PET reflects radioactive-tracer uptake: tissues that absorb more tracer appear brighter, and tumors often exhibit that behavior. The image-translation model attempts to learn the relationship between anatomical structure in CT and the activity represented in PET. This is the research objective of the example, not a demonstration that a generated image can replace a clinical PET scan or establish a diagnosis.

Such an objective spans years of research, so it must be broken into smaller goals, each with its own experimental loop. One is model training. The described GAN architecture encodes CT and decodes it into PET; making that experiment useful also requires assembling a suitable dataset. Shahandeh estimates that preparing good data accounts for 80% of the work in this workflow. With architecture and data in place, fidelity metrics compare synthetic PET with real PET. Those measurements define a starting point for further optimization.

### From local adjustments to a different architecture

[5:16](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=316s)

The Codex run Shahandeh presents follows several branches: some changes improve the system, some do nothing, and others lead to dead ends. Eventually progress saturates. The existing model uses a **2.5D** arrangement: it stacks CT slices as channels but processes them with 2D convolutions. That representation becomes the concrete opportunity for a broader hypothesis.

| Approach | Treatment of CT slices | Search implication |
| --- | --- | --- |
| Existing 2.5D model | Slices stacked as channels; 2D convolutions | Tune within the current architecture |
| Proposed 3D alternative | Use 3D convolutions | Test a different architectural choice |

In this run, familiar adjustments such as hyperparameter changes did not naturally lead the agent to the larger architectural move. Shahandeh intervened with suggestions, including considering 3D convolutions and reading papers for alternatives. The limitation concerns the search behavior he observed, not a rule preventing autoresearch from changing architectures. The engineering problem is how to make broader hypothesis generation part of the loop instead of relying on those human interruptions.

### Make decomposition an explicit action

[6:49](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=409s)

The intervention begins before another training experiment: ask the agent to decompose the problem. The top-level objective remains specific—translate pulmonary-nodule CT patches into equivalent PET—but the search now has named components to inspect. Shahandeh compares this explicit decomposition to prompting a model to reason step by step. The key is to request the decomposition as an action, rather than assume it will happen implicitly while the agent edits code.

The component inventory reaches beyond the neural network itself:

- **Data preparation:** the inputs and preparation work on which training depends.
- **Learning architecture:** the model that performs the translation.
- **Training loss:** the objective used to train it.
- **Operations:** the machinery that runs the modeling workflow.
- **Metrics and evidence:** how changes are evaluated.
- **Peripheral scripts:** supporting code around the model.

This gives hypothesis generation several places to look, including components that a narrow request to improve the model might overlook.

The coding agent inspects the codebase and produces a hierarchy of linked documents describing those components. The documents make the problem’s structure available for subsequent reasoning: what matters, what can change, and where a change belongs. The purpose is not documentation for its own sake. **The hierarchy defines the areas the agent should search for improvements.**

### Follow the hierarchy down to code

[8:38](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=518s)

The Obsidian walkthrough shows how the generated documents connect a research goal to implementation:

1. Start with a level-one `README` containing the problem statement and the documentation entry point.
2. Follow links to level-two components, which lead to more specific level-three components.
3. Enter the model-architecture branch and inspect the generator document, which describes the existing 2.5D generator.
4. Continue to the actual implementation code at the leaves of the hierarchy.

Each document is simple; the value comes from the linked path between levels.

Build this structure before starting the optimization loop. The agent can then reason through the hierarchy step by step, moving from the objective to a component and finally to an implementation. That gives the next hypothesis a location in the system, rather than leaving it as an isolated suggestion.

### Generate candidates, then challenge them

[9:55](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=595s)

Once the hierarchy exists, ask for ideas that improve the objective using its components. A bare codebase-plus-goal instruction, like the role played by `program.md` in the autoresearch example, leaves the agent to choose its own search coverage. The hierarchy supplies an explicit scaffold. Shahandeh suggests generating 100 candidate solutions as an illustrative target, not a reported count of successful experiments. The displayed proposal table organizes hypotheses by component and pairs proposed changes with evaluation hooks.

![A slide about generating hypotheses through an adversarial or collaborative loop, above a table of configuration, runtime, and data-labeling proposals.](https://ai.engineer/api/talk-essay-media/2987dee8481e5a9a181c41d343266f94f50e15e61c46df25cf849a1a33128f93)

Component-by-component hypotheses pair proposed changes with evaluation hooks.

The resulting search can consider modifications across the system. In Shahandeh’s example, this process surfaces a three-dimensional alternative to the initial architecture—a more substantial change than another local adjustment. It remains a hypothesis to test, not evidence that 3D convolutions have already improved the model. Another agent or model can review the plan collaboratively, developing the ideas, or adversarially, challenging their assumptions.

The experiment loop still starts from a codebase, metrics, and a goal. It now includes a more deliberate hypothesis stage before implementation: formulate the required change, implement it, evaluate the outcome, and repeat. Shahandeh reports that better hypotheses make this process more effective; the talk does not quantify the gain.

### Registration needs more than a numerical score

[12:06](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=726s)

A second example returns to data preparation: **image registration**, the process of aligning scans. The demonstration overlays grayscale CT with colored PET activity. The liver shows high uptake, while the lung regions appear dark. Because the scans were acquired at different times, their initial positions do not match. Registration must bring them into alignment before the paired images can be used reliably together.

This is not merely a matter of shifting one image. Respiration, patient movement, changes in the body, different scanners, and different positioning can all complicate the correspondence. Registration is therefore another optimization problem with multiple components and multiple ways to fail.

Some of its quality judgments are qualitative, and a poor result does not necessarily identify the bug that caused it. A scientist may inspect images one by one and ask whether the result makes sense. Two concrete checks in the demonstration are whether a lung mask actually covers the lungs and whether scans have been cropped or truncated correctly. Each processing stage can require its own review-and-improvement loop.

### Bring specialist models into the loop

[14:47](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=887s)

The coding agent need not perform every kind of review itself. A model with stronger multimodal capabilities can inspect an image and assess alignment. A stronger reasoning model can help generate hypotheses or critique changes. Shahandeh names [**GPT-5.5 Pro**](https://developers.openai.com/api/docs/models/gpt-5.5-pro) for the latter role, attributing its usefulness to greater inference-time compute. These are distinct jobs: judging what an image shows and deciding what experiment should follow.

Skills make those consultations available inside the agent workflow. Shahandeh describes an image-review skill using **Gemini**, then introduces Peter Steinberger’s [Oracle CLI](https://github.com/steipete/oracle) for packaging code and data and sending that context to an external model API. The relevant integration is the consultation step; current Oracle defaults or API examples should not be assumed to be the configuration used in the demonstration.

Close the loop by returning both the implementation and its outcome to the reviewer. The review must connect what changed with what happened, assess whether the result makes sense, and propose the next action. Shahandeh reports better hypothesis generation and more useful critique through this process, based on his experience rather than a quantified comparison. A second model is useful here because its judgment feeds the next experiment, not simply because it supplies another opinion.

### The remaining problem of scientific observation

[16:41](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1001s)

This workflow gives the scientific loop a more rigorous hypothesis stage, but it does not make every part autonomous. Shahandeh considers implementation comparatively solved when the work can run inside a basic simulated environment. That qualification matters: a task executable in a computational box gives the agent a much more accessible experimental setting.

**Better hypotheses still depend on reliable observations.** Returning to the scientific images, Shahandeh argues that contemporary general multimodal models do not reliably recognize tiny details such as pulmonary nodules. He attributes that weakness to insufficient training on scientific images and data. The grid of axial scans and colored overlays illustrates the kind of material a reviewer must interpret; the presence of an image-review model does not itself establish that it can notice the scientifically decisive feature.

![A four-by-four grid of grayscale axial scans with colored overlays and a vertical color scale.](https://ai.engineer/api/talk-essay-media/4a332e21ab63f880ba3be6e33e1226188a7cd339fddd6c460259d4d511894545)

PET/CT registration quality-control views across multiple axial slices.

Fine-tuning or other adaptation could help models observe those details more like a trained scientist. Until then, observation remains a major obstacle to closing the entire loop and building a fully autonomous scientist in a data center.

### Give additional computation a structure

[18:03](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1083s)

Hypothesis quality remains the other limiting capability. Shahandeh compares hierarchy scaffolding to the way chain-of-thought prompting can elicit stronger reasoning from an existing model, using **GPT-4** as an analogy rather than a claim about where that technique originated. His forecast is that models pretrained to decompose and compartmentalize problems more effectively may eventually need less explicit scaffolding.

For the present workflow, decomposition is a deliberate sequence: expose the hierarchy, inspect its components, and look for improvements within each one. That structure gives additional test-time computation somewhere useful to go. More generated tokens can support broader hypothesis search around a defined research objective, while implementation and observed outcomes keep the search connected to the experiment loop.

## Resources

- [Karpathy's autoresearch](https://github.com/karpathy/autoresearch) — A compact autonomous training loop that edits model code, runs fixed-budget experiments, and retains improvements in validation bits per byte.
- [Oracle](https://github.com/steipete/oracle) — A tool for bundling prompts and project files for another model's review, with saved sessions and follow-up support.
- [GPT-5.5 Pro API documentation](https://developers.openai.com/api/docs/models/gpt-5.5-pro) — Current model documentation covering reasoning settings, supported inputs, and API capabilities.
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) — Foundational research on prompting language models with intermediate reasoning examples.

## Slides

### Humans and agents over longer horizons

The chart shows agents improving early and leveling off by 24 hours, while both human groups continue climbing through 14 days. Shahandeh attributes the gap to humans continuing to generate ideas.

[0:50](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=50s) [A chart compares human and coding-agent Elo ratings over time, showing human scores continuing to rise beyond 24 hours.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0df6ed18e7de31949e80bbaad48175354bf6ab40bb874045eb6cdecd7869d44.webp)

Humans Still Beat AI in the Long Horizon:
Revisiting Test-Time Scaling in the Agent Era

June 16, 2026 · Qiuyang Mang¹, Kaiyuan Liu², Bo Peng³, Shreyas Pimpalgaonkar⁴, Luke Zettlemoyer², Alex Dimakis¹,⁴, Alvin Cheung¹
¹ UC Berkeley · ² University of Washington · ³ Princeton University · ⁴ Bespoke Labs
2026 · # test-time-scaling · # LLM-agents · # scaling-laws · research

Human vs Agent
Legend: top10-humans; top50-humans; Claude Code Opus-4.6; Codex GPT-5.5
Y-axis: Elo Rating
Y-axis ticks: 1000, 1200, 1400, 1600, 1800
X-axis: time
Timeline: 1h, 2h, 4h, 8h, 16h, 24h, 2d, 4d, 8d, 14d
agents sprint early
then plateau by 24h
humans keep climbing for days
1368
1092
1853
1587

Partial adjacent chart legend: Discarded; Kept; Running best

### Hypothesis generation

Shahandeh describes science as a loop that learns from each experiment. He identifies generating good hypotheses as a difficult bottleneck for agents trying to improve.

[1:54](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=114s) [The slide identifies hypothesis generation as a bottleneck for autonomous scientific agents and illustrates the scientific method as a cycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/154654b114c31235eedf8ccd740dee586031c2acc8555f032a3994db77d4b08b.webp)

The bottleneck for Autonomous Scientific Agents is
Hypothesis Generation.

Scientific method cycle:
Observation
Question
Hypothesis
Experiment
Analysis
Conclusion

Partial chart labels:
1853
1587

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### CT to in silico PET

RADiCAIT is building a way to generate PET images from CT scans. The example marks a lung nodule on both the diagnostic CT and the in silico PET image.

[2:52](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=172s) [A diagnostic chest CT and an in silico PET image appear side by side, with a lung nodule outlined in each.](https://assets.ai.engineer/sites/aiecode2025/sha256/00172273ce292eb38dee627aaaafd64b232b81bd5e84858c1833f30188492c75.webp)

CT → In Silico PET
Coronal View

Diagnostic CT
Lung Nodule

AI →

In Silico PET
Lung Nodule
Relative Uptake
Scale ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0

Decompose the problem into /g[illegible]

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### Decompose the problem into goals and loops

Shahandeh uses CT-to-PET translation to illustrate breaking a long research process into goals and loops. Training a machine learning model is one of those steps.

[4:06](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=246s) [The CT-to-PET example appears above text proposing that the problem be decomposed into a goal and a loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f2816ef0de882a9fded6d3a6ebd9f9bd551cd329bf528ffaf0d231430eae20f.webp)

Decompose the problem into /goal
/loop

CT → In Silico PET
Coronal View
Diagnostic CT
In Silico PET
Lung Nodule
Lung Nodule
AI
Color scale: [illegible]

Prepare [cropped]
Train [cropped]

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### CT to PET Generator

The generator encodes CT slices and decodes them into synthetic PET, which is compared with real PET to assess image fidelity.

[4:25](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=265s) [A diagram shows CT slices entering a generator that produces synthetic PET for comparison with real PET.](https://assets.ai.engineer/sites/aiecode2025/sha256/b83c50f56f10805fadb5731dbf02a3b477c9ba059c34e11f01445cc796acfc41.webp)

Prepare data
Train ML

CT to PET Generator
CT
Generator
Synthetic PET
Real PET

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

[4:42](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=282s) [A generator transforms stacked CT slices into a synthetic PET image for comparison with a real PET image.](https://assets.ai.engineer/sites/aiecode2025/sha256/12bdecdc5e44147d664574203cee7a264d643e62b753aa40780c9218cf7e66db.webp)

Autoresearcher climbs

CT to PET Generator
CT
Generator
Synthetic PET
Real PET

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### CT to PET Generator

Stacked CT slices enter a generator that produces synthetic PET for comparison with real PET. The initial 2.5D model treated slices as channels for 2D convolutions.

[5:44](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=344s) [Stacked CT slices enter a generator, producing synthetic PET for comparison with real PET.](https://assets.ai.engineer/sites/aiecode2025/sha256/4bd2ae009e5ff0396c3e98498d6d60f22dce757ef2015fec43986c70690f0890.webp)

CT to PET Generator

CT
Generator
Synthetic PET
Real PET

### NoduleGAN Pipeline: Component Map

The map breaks CT-to-PET prediction into four domains, then traces 11 components to their source files. Data preparation, model training, evaluation and export, and operations each have a place in the hierarchy.

[7:09](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=429s) [A hierarchy maps the CT-to-PET pipeline into contracts and data, learning, evidence and export, and operations.](https://assets.ai.engineer/sites/aiecode2025/sha256/4366aafc55e8d2cfdf7886dbd42fcd23a9427ea412f7012c3d601b49f7ddaafd.webp)

DOCUMENTATION HIERARCHY
NoduleGAN Pipeline · Component Map
From the problem statement down to the source files — the whole GAN pipeline as one four-level abstraction tree.

1 Problem — what the system does
2 Domain — four cooperating areas
3 Component — the 11 subsystems
4 Code — concrete source files

LEVEL 1 · PROBLEM
Translate pulmonary-nodule CT patch stacks into equivalent PET / SUV patch predictions
Paired CT / PET patches → validate & select → train CT→PET GAN → export IPN-compatible checkpoint

LEVEL 2 · DOMAIN A
Contracts & Data
Settings, pair discovery, clinical labels, pruning, and tensors.

COMPONENT 01
Configuration & Runtime Contracts
Strict typed YAML — the contract every stage reads from.
config.py  configs/base.yaml

COMPONENT 02
Data Indexing, Validation & Labels
PairRecords, array checks, cuboid-level PET-positive labels.
indexer.py  validation.py  clinical_labels.py

COMPONENT 03
Selection Audit & Pruning
Fingerprinted HU/SUV metrics prune the training pool.
selection_audit.py  selection.py

COMPONENT 04
Dataset Loading & Augmentation
Crop, augment, normalize and collate CT/PET tensors.
dataset.py

LEVEL 2 · DOMAIN B
Learning Core
The networks, the objective, and the reusable training loop.

COMPONENT 06
Model Architecture
2.5D ResUNet generator + PatchGAN discriminator.
generator.py  discriminator.py  factory.py  utils.py

COMPONENT 07
Training Objective & Losses
LSGAN + L1 + lesion-weighted L1 + VGG + feature matching.
objective.py  regression.py  perceptual.py

COMPONENT 08
Trainer & Experiment Orchestration
One training loop; run layout, provenance and export handoff.
trainer.py  experiment.py

LEVEL 2 · DOMAIN C
Evidence & Export
Cross-validation evidence, clinical metrics, checkpoint handoff.

COMPONENT 05
Grouped Splits & Research CV
Leakage-safe folds recommend the final-fit epoch.
splits.py  run_kfold.py

COMPONENT 09
Evaluation Metrics
Image metrics + SUV-threshold clinical metrics by cuboid.
metrics.py

COMPONENT 10
Checkpoint Export & Contract
Generator-only weights as an IPN-Inference patch checkpoint.
inference/export.py  train_full.py

LEVEL 2 · DOMAIN D
Operations & QA
Operator entry points, experiment tracking, and behavior tests.

COMPONENT 11
Operational Scripts, Tracking & Tests
The operator surface that drives, observes and guards every run.
RUN
validate_data.py  train_full.py  run_kfold.py
OBSERVE
tracking.py  summarize_results.py  show_tracking_dashboard.py
PROTECT
tests/  audit_hu_suv_dataset.py

L1 · PROBLEM
L2 · DOMAIN
L3 · COMPONENT

A · Contracts & Data  B · Learning Core  C · Evidence & Export  D · Operations & QA
Generated from docs/abstractions/ · click any component to open its Level-3 page

### Prompt for Hierarchical Abstraction Documents

The prompt asks a coding agent to document a codebase in layers, from the problem statement through components to source-file links. The hierarchy gives the agent a way to reason about individual components.

[7:53](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=473s) [A Markdown prompt asks for layered codebase documentation ending in links to source files.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ca6bd36748d695ebd05fe05e603f7d5c86d5fb2b02ec05f49146e5b001ac41a.webp)

# Function: Creating Hirearchal Abstraction Documents

We are working on a pipeline for the following problem <problem_title>. Here is a description of the problem:
<problem_description>

Read the code given below to understand the current implementation of the model described in the problem description.
I would like you to create a series of markdown document explaining the codebase in several layers of abstraction.
1- First level is the general statement from problem description stated above
2- Second level concerned with the overall components of the problem, List all components and write a short paraphraph explaining what the component does in a single md file.
3- Third level is a detailed description of each component above. For each component write a separate md file. You can add sudo code or short snippts of the code from the files in these documents. If we are close enough to the code, you can add a link to the file.
4- You can continue to lower levels depending on the complexity of the componenents. A well organized abstraction is very important. continue as level three until the end is a link to the actual code files.

### Code:
<code_dump>

Interface text:
Program.md > # Function: Creating Hirearchal Abstraction Documents
Preview
Markdown

### Documentation Link Graph

In Obsidian, generated documentation appears as a graph connecting the README and component pages to code and tests, giving the coding agent a component-level view of the problem.

[8:34](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=514s) [A dark graph connects component documents, source files, tests, and a README.](https://assets.ai.engineer/sites/aiecode2025/sha256/69a192443dbf901c2374c6b3bd031551e09f4770540b2198cdf9b701613f8dbf.webp)

run_improvement_experiments.py
tracking.py
summarize_results.py
train_full.py
run_kfold.py
trainer.py
experiment.py
selection_audit.py
show_tracking_dashboard.py
operational-scripts-tracking-and-tests
README
config.py
audit_hu_suv_dataset.py
dataset.py
trainer.py
trainer-and-experiment-orchestration
validate_model.py
validate_data.py
selection.py
factory.py
run_kfold.py
test_full_training.py
selection-audit-and-training-pool-pruning
train_full.py
model-architecture
generator.py
evaluation-and-metrics
level-2-components
validation.py
experiment.py
test_model_shapes.py
grouped-splits-and-research-cv
splits.py
data-indexing-validation-and-labeling
base.yaml
clinical_labels.py
utils.py
configuration-and-runtime-contracts
discriminator.py
test_evaluation_metrics.py
metrics.py
checkpoint-export-and-downstream-contract
test_kfold_selection_scope.py
dataset-loading-and-augmentation
indexer.py
test_splits.py
config.py
test_dataset_augmentation.py
training-objective-and-losses
dataset.py
[illegible]ort.py
README_TRAINING_PIPELINE
README_NODULE_CHECKPOINT_STUDY
test_loss_options.py
objective.py
regression.py

### Primary Runtime Flow and Code Landmarks

The documentation traces the runtime from configuration and data indexing through training to an exported checkpoint, then links to the main code locations.

[8:50](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=530s) [Documentation lists the pipeline's runtime sequence and links to key code locations.](https://assets.ai.engineer/sites/aiecode2025/sha256/fbdeab32856af9e9289d0e77f05e4c7cfc616efcd577e77085c679061b115177.webp)

Start with Level 2: Components. It names every major subsystem and explains what each one owns.

Then use the component documents under components/ for Level 3 and lower. Each component page works from a conceptual responsibility down to concrete files and functions. The lowest-level sections end in links to the actual code files.

Primary Runtime Flow

configs/base.yaml
  -> nodulegan.config.load_config
  -> nodulegan.data.indexer.build_pair_index
  -> selection audit resolution and optional record selection
  -> validation and cuboid-level PET-positive labels
  -> train_full.py or run_kfold.py
  -> FoldTrainer
  -> model, losses, metrics, checkpoints
  -> artifacts/exports/ipn_patch_model_best_<run_id>.pth

Fast Code Landmarks

• Default full-training CLI: ../../scripts/train_full.py
• Research K-fold CLI: ../../scripts/run_kfold.py
• Central config schema: ../../src/nodulegan/config.py
• Dataset loader: ../../src/nodulegan/data/dataset.py
• Generator and discriminator: ../../src/nodulegan/models/
• Training loop: ../../src/nodulegan/training/trainer.py
• Run orchestration and export: ../../src/nodulegan/training/experiment.py

Interface text:
text

### Model Architecture

The model-generated hierarchy documents the 2.5D CT-to-PET generator, discriminators, and factory, then links to the underlying code.

[9:13](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=553s) [Documentation describes the CT-to-PET model responsibility, generator structure, and discriminator inputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/502f9bebf655fbd609c5cca95f7746cee69b1c00b1440e9b484976f5d78b7030.webp)

model-architecture

Model Architecture

Level 3: Responsibility

The model component builds the CT-to-PET GAN networks. It exposes a generator for PET prediction and a discriminator for adversarial supervision. The factory functions hide architecture options from the trainer so training code only asks for build_generator(cfg.model) and build_discriminator(cfg.model).

Level 4: Generator

`FixedResUnetPlusPlus` is a legacy-compatible 2.5D generator:

(B, 7, 50, 50) -> (B, 3, 50, 50)

Its structure is:

enc1 -> enc2 -> enc3 -> bottleneck
       <- dec3 <- dec2 <- dec1
       skip connections at matching spatial scales
       final 1x1 convolution + sigmoid

The generator uses 2D convolutions. The seven CT slices are input channels, and the three PET slices are output channels. Spatial sizes that do not match exactly after transposed convolution are repaired with bilinear interpolation before concatenating skip connections.

Level 4: Discriminators

Both discriminator variants are PatchGAN-style networks over concatenated CT and PET channels:

input channels = data.ct_channels + data.pet_channels

For the base contract, that is 7 + 3 = 10 channels.

PatchDiscriminator is the v1 variant with spectral normalization and BatchNorm.
PatchDiscriminatorV2 is a smaller variant tuned for smaller crops, with optional instance [illegible]

Interface text: text

[9:28](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=568s) [Model documentation showing factory configuration, weight initialization, and code links.](https://assets.ai.engineer/sites/aiecode2025/sha256/313bf51f10234ff541ad50fd6400f28a087ea99dcc23f7a1898a75a2d0ce868c.webp)

For the base contract, that is 7 + 3 = 10 channels.

PatchDiscriminator is the v1 variant with spectral normalization and BatchNorm. PatchDiscriminatorV2 is a smaller variant tuned for smaller crops, with optional instance normalization and a denser patch map. Both return:

logits, intermediate_features

Intermediate features are used by the feature-matching generator loss.

Level 4: Factory Boundary

The factory chooses implementation classes from config:

generator = FixedResUnetPlusPlus(
    input_nc=cfg.input_channels,
    output_nc=cfg.output_channels,
    base_channels=cfg.base_channels,
)

discriminator_input_nc = cfg.input_channels + cfg.output_channels

This keeps model construction centralized and makes it easier for tests and scripts to smoke-test shape compatibility.

Level 5: Weight Initialization

Convolution and linear weights use a normal initialization with std=0.02; BatchNorm weights are initialized near 1.0. Warm-start checkpoints can bypass generator initialization when training.init_generator_checkpoint is configured.

Code Links

• Generator: ../../../src/nodulegan/models/generator.py
• Discriminators: ../../../src/nodulegan/models/discriminator.py
• Model factory: ../../../src/nodulegan/models/factory.py
• Weight initialization: ../../../src/nodulegan/models/utils.py
• Model smoke-test CLI: ../../../scripts/validate_model.py
• Shape tests: ../../../tests/test_model_shapes.py

Interface text: Python

### Generating Hypotheses from the Hierarchy

The model uses the code hierarchy to generate component-level improvements, each paired with an evaluation hook. The proposed collaborative or adversarial loop can then refine those ideas.

[9:56](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=596s) [A slide introduces a collaborative hypothesis loop and shows a table of configuration hypotheses and evaluation hooks.](https://assets.ai.engineer/sites/aiecode2025/sha256/984a9454c87e554b25fc6455831feca751135c8138f9944e7c916a22ea10c2a6.webp)

Generate hypothesis against the heirarachy in an advesserial/collaborative loop

Component-By-Component Hypotheses

Configuration And Runtime Contracts

ID | Hypothesis or change | Evaluation hook
H011 | Add explicit generator_version and loss_version fields instead of overloading loose architecture toggles. | Config diff and shape smoke tests prevent accidental mixed-contract runs.
H012 | Version the dataset contract in export sidecars: spacing, crop, channel count, PET scale, label policy, selection fingerprint. | Downstream evaluator refuses incompatible checkpoints.
H013 | Add a promotion config that names all benchmark roots, label definitions, and thresholds. | One command produces internal, FDG, Emory, and SegMed comparison tables.
H014 | Separate pet_train_max from pet_export_max only if deliberately testing calibration. | Detects silent SUV inflation/deflation.
H015 | Add a config-level endpoint_target enum: PET fidelity, FDG threshold, strict clinical, downstream blend. | Makes checkpoint scoring and report language match experiment intent.
H016 | Add config provenance for upstream exporter version and Diagnostic-CT-Pipeline commit. | Explains performance changes caused by data export drift.

Data Indexing, Validation, And Clinical Labels

ID | Hypothesis or change | Evaluation hook
H017 | Compare positive_slice_mode of center, any, and window under cuboid-level labels. | Label distribution and FDG-threshold metrics.

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### Model architecture hypotheses

The plan proposes wider CT context, a 3D or hybrid generator, and multi-scale crops. The highlighted generator aims to model axial texture more naturally, at the cost of more memory and compute.

[11:43](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=703s) [A highlighted document section describes larger axial context, a 3D or hybrid generator, and multi-scale crop inputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/5816b4c6bb32a1dcf8323947d7f8cb5712c5617cc34fee8608cf1ffb2e130b03.webp)

H004 - Larger Axial Context
• Change: test 11, 15, 21, or full cuboid CT input slices, with 3, 5, or 7 PET output slices.
• Why it may help: uptake and registration context may be outside the current 7-slice CT stack, especially with thick lesions or off-center nodules.
• Implementation: regenerate or reuse patch exports with more channels, update config/model/export sidecars, and add IPN compatibility handling.
• Evaluation: compare against current 7-to-3 contract in grouped CV and downstream inference.
• Risk: breaks current checkpoint shape contract and may require downstream model-loader changes.

H005 - True 3D Or 2D/3D Hybrid Generator
• Change: replace channel-stacked 2D convolutions with 3D convolutions, factorized 2D+1D convolutions, or a 2D encoder plus axial attention.
• Why it may help: current 2.5D network cannot model axial texture as naturally as a volumetric model.
• Implementation: add a model variant behind model.generator_version and keep output export explicit.
• Evaluation: PET peak localization and axial consistency, not just center-slice SSIM.
• Risk: more memory and compute; may underperform on small data without regularization.

H006 - Multi-Scale Crop Inputs
• Change: feed both tight lesion-centered crop and wider context crop, either as separate encoder streams or a pyramid.
• Why it may help: nodule appearance needs fine detail, while lung location, vessel context, and surrounding anatomy may influence expected uptake.
• Implementation: update dataset/export to load two crops, add dual-stream generator, and preserve downstream crop metadata.
• Evaluation: subgroup analysis by small nodules, peripheral/central nodules, and false-hot background cases.
• Risk: context can become a shortcut if not domain-regularized.

Other visible text:
[illegible]y in
Evaluation hook
[illegible]ion fields instead of — Config diff and shape smoke tests prevent accidental mixed-contract runs.
[illegible]cing, crop, channel — Downstream evaluator refuses incompatible checkpoints.
[illegible]ots, label definitions, — One command produces internal, FDG, Emory, and SegMed comparison tables.
[illegible]only if deliberately — Detects silent SUV inflation/deflation.
[illegible]idelity, FDG threshold, — Makes checkpoint scoring and report language match experiment intent.
[illegible]on and Diagnostic-CT- — Explains performance changes caused by data export drift.
Evaluation hook
[illegible]nd window under — Label distribution and FDG-threshold metrics.

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### Anatomy of a loop

The codebase produces output metrics. If the goal is not achieved, the loop moves from a hypothesized change to implementation, then back to the codebase.

[11:50](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=710s) [A cycle connects codebase, output metrics, hypothesizing a required change, and implementing the change.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6edf9f468149c0cf085aa620bb29e611e84f4945ecd7c3f31ace10006fa0b75.webp)

Anatomy of a loop

Codebase
Output Metrics
/goal achieved? No
Hypothesize a required change
Implement the change

Anatomy of a loop
Codebase
Implement the change

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### 3D voxel-level alignment between CT and PET scans

Axial slices overlay colored PET activity on grayscale CT images, providing a visual check of scan registration.

[12:22](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=742s) [A grid of axial CT slices displays colored PET overlays for registration quality review.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8d1062dbcaec3d96730baa5929f7ce1d64dc83254993d56b4cd4152ffdbcd25.webp)

3D voxel-level alignment between CT and PET scans

PET/CT Registration QC - Axial Views
Slice 14, Slice 16, Slice 19, Slice 22
Slice 25, Slice 28, Slice 30, Slice 33
Slice 36, Slice 39, Slice 42, Slice 44
Slice 47, Slice 50, Slice 53, Slice 56
Color scale: SUV
Color scale ticks: 0.5, 1.0, 1.5, 2.0, 2.5

Adjacent cropped content:
3D voxel-level alignm[illegible]
Slice 12, Slice 21, Slice 31, Slice 41

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### Anatomy of a loop: later return

For CT/PET alignment, the loop evaluates output metrics, hypothesizes a change, and updates the codebase. Some metrics are qualitative, and the underlying issue can be hard to identify.

[13:56](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=836s) [Two loop diagrams connect a codebase, output metrics, hypothesizing a change, and implementing it.](https://assets.ai.engineer/sites/aiecode2025/sha256/1032f170592e9d84829a26fce615acf9fbf596170f77f4993c381f7937f50094.webp)

Anatomy of a loop

Codebase → Output Metrics
/goal achieved? No
Hypothesize a required change → Implement the change → Codebase

Anatomy of a loop

Codebase → Output Metrics
/goal ach[illegible]
Hypothesize a required change → Implement the change → Codebase

[illegible]

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### Anatomy of a loop

The loop evaluates a codebase’s output, hypothesizes a change if the goal is unmet, and implements it. A multimodal model can review images, while another model generates hypotheses.

[14:53](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=893s) [A feedback loop connects a codebase, output metrics, a proposed change, and implementation; model labels identify the review and hypothesis steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/de5489d96e0f0353fdc1b7450f03cad179b76488ddf8700a0811ec0d8d5e1285.webp)

Anatomy of a loop

Codebase → Output Metrics
/goal achieved? No
Hypothesize a required change → Implement the change → Codebase

gemeni-3.5-pro
gpt-5.5-pro

step9_pet_final_pre_crop.png
Axial

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

### Skills for image review and auxiliary models

Shahandeh uses an image-query skill to get Gemini’s second opinion on images, and Oracle CLI to send code and data to GPT-5.5 Pro for hypotheses and critique.

[15:41](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=941s) [A slide shows an image-review skill definition and names a command-line tool for consulting an auxiliary model.](https://assets.ai.engineer/sites/aiecode2025/sha256/7804f80494d8acd592fddfe7c1aa016beab2c7847cd6788c9d72f7c4daeafa78.webp)

> Skills for image review

---
name: image-query
description: Upload a local image to Gemini and ask a prompt about it. Use when Codex needs Gemini multimodal image understanding for a second opinion on screenshots, scans, plots, or other local images, then feed the returned answer back into the main agent loop.
---

# Image Query

Use the bundled CLI instead of writing one-off Gemini code. The helper uploads a local image with the Gemini Files API, optionally assembles a markdown bundle from selected repo files and freeform notes, uploads that bundle as extra context, reads `GEMINI_API_KEY` from the environment or `/home/sina/.env`, runs `gemini-3.1-pro-preview` with `HIGH` thinking and high media resolution, streams the answer to stdout, and deletes the temporary Gemini uploads unless told not to.

## Quick Start

> Skills for auxillary out-of-the-loop models

@steipete/oracle CLI

Interface text:
AIE_conference > presentation2.tldr
.codex > skills > image-query > SKILL.md > # Image Query > ## Workflow
MADE WITH TLDRAW

### Anatomy of a loop

When the goal is not met, the loop uses output metrics to hypothesize a change, implement it, and update the codebase before measuring again.

[16:20](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=980s) [The annotated feedback loop returns, showing codebase, metrics, hypothesis, and implementation steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/f6721919ee6bf2115a1671a72eb52e2c4629b7d01b24e5ddd21e22338ca6110e.webp)

Anatomy of a loop

Codebase
Output Metrics
gemeni-3.5-pro
/goal achieved? No
Hypothesize a required change
gpt-5.5-pro
Implement the change

Output Metrics
/goal achieved? No
Hypothesize a required change

[illegible]

Interface text:
AIE_conference > presentation2.tldr
MADE WITH TLDRAW

## Full transcript

[0:00](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=0s) Hello, everyone. My name is Sina Shahandeh. A qui-- my pleasure to present you this talk about, uh, running autonomous agents for scientific tasks.

[0:12](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=12s) Um, let's, let's dive in. Um, so here, um, I, I think everyone is quite familiar with the concept of, uh, autoresearcher or autoresearch. Uh, uh, this is the original Andrej Karpathy's, um, uh, GitHub repo, where, uh, we have a ML model, and we ask a coding agent to, uh, uh, find, s-find a particular metrics, and then

[0:37](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=37s) optimize the code in order to minimize the error. Basically, do a hill climb over a model optimization.

[0:45](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=45s) Now, for many of these coding tasks, um, this works very well. Uh, but when the problems become very much open-ended and, um, y-

[0:56](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=56s) sometimes long horizon, m-- like most of the scientific task, y-you have this case where, uh, AI agents, uh, usually kind of saturate, uh, to a certain level. Um, simply, they, they are very good at implementation of the, of the code or changing the, m-- or running the experiments over lots of data and so on.

[1:14](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=74s) But the problem is they run out of ideas, uh, or, you know, what people call them research taste. Now, you can see that in these situations, uh, you know, good humans keep going, uh, higher, and the top one percent humans, you know, they keep even, uh, improving better and better over time.

[1:32](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=92s) Um, now, um, uh, the, the, the really-- the difference, um, from here is the way that, uh, good ideas, uh, can... Or, or good hypothesis on how the model could be improved or how the, the problem could be solved, um, keep, um, keep coming, coming up, humans keep coming up.

[1:50](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=110s) So in the scientific task, uh, you, you know, you're, uh, the, you have this, um, scientific method, um, where, you know, we observe a situation, we make questions, uh, we come up with a good hypothesis and come up with a hypothesis of how to solve this problem, and then we do experiment and implement and do the experiment,

[2:08](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=128s) um, do the loop. Um, and the, as each of the iterations, we learn something and we improve. Now,

[2:15](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=135s) uh, the components of a learning component, uh, I think those are all questions of memory and implementation from learning the mistakes, but it-- which is one of the bottlenecks of, uh, using coding agents.

[2:25](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=145s) But I think this quite solved by just simply organizing patterns of a-activity. I think what is much more difficult is coming up with a hypothesis. So how can we come up with a good hypothesis, good ideas for our, uh, coding agents to keep improving better, uh, the process?

[2:42](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=162s) And this is something core things that I would like to kind of, um, um, focus on this talk, uh, and plus a little bonus at the end.

[2:50](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=170s) Um, so, um, let's look at a problem here w-we're trying to achieve as an example, so we have a good, good idea of what we're trying to do. So here is what we're doing at RADiCAIT.

[2:58](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=178s) We're building, uh, in silico PET, meaning you have a CT, and we want to generate, um,

[3:06](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=186s) a PET, PET image, um, a PET scan, uh, from the CT scan. Example of this you can see here, um, we have CT scans and of slices of bo-- um, scans of the, of, of the, of a patient.

[3:18](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=198s) Uh, they might have a nodule in the, in the lung. And the question is: Is this cancerous or not? Um, is it a lung cancer? So they do a PET scan, which is a difficult process and very time-consuming and, uh, to do.

[3:29](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=209s) Um, but here we do an ML-- through an ML model image translation, one can kind of change the modality, learn the structure of the body, and infer what would happen in a PET scan if, um, the hyper-- the activity of the tissue.

[3:43](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=223s) So, you know, certain tissues absorb more radioactive tracer, and they shine up in this PET scan. Um, and, um, the, the tumors, usually that's the case. Now, to generate this relationship, uh, we need, um, we need a model, uh, to, um, to do the translation, but the problem itself has many components.

[4:05](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=245s) So really, uh, this, like any other scientific task, the problem is decomposing the problem, uh, entire long-term horizon, two years, ten years research process into steps. And each of those steps is really fundamentally are a goal, are, are, are a, are a loop.

[4:21](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=261s) So I'm gonna focus on one of these particular ones right now, and that is on training of a machine learning model. Uh, uh, so we have here, um, you know, decoder, encoder type of situations.

[4:33](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=273s) Um, so, uh, en-encoding the CT and then decoding it into PET. Um, so that's the typical GAN model which kind of generates the, the image. Now, for this, we can kind of define these kind of, you know, the architecture, and we try it, we capture data, uh, and do all the 80% of the work to basically bringing

[4:51](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=291s) the good data set. And here, uh, we create the metrics and so on, and, um, around the image, m-- you know, fidelity of synthetic PET, uh, to, to real PET and so, so on.

[5:03](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=303s) But the challenge is: How can we improve this situation, uh, given a certain, um, in-initial points? And we'll go back to our idea of, uh, hill climb, uh, around this, uh, optimization.

[5:16](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=316s) So you can see an example of iterations coming from a real run in Codex, where, you know, we improve the, the data and so on, and then we, the model goes around and tries to do the optimization.

[5:28](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=328s) And you can see there's a range of possibilities, and some of them become dead end, uh, some of them not, don't improve anything. But, um, we kind of saturate, uh, at a certain point.

[5:38](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=338s) And then you need a, really need a good idea. A good idea has to come up. So in this case, we have a slices of CT, and we feed these as a channel into the model.

[5:47](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=347s) So initial problem, initial, uh, model, uh, that was trained was two and a half D. So treating each CT slice as a, uh, be the 2D convolutions, but stacked with a channel.

[5:58](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=358s) Now, if you give this to a, um, a typical ML model, it would not think about it as, you know, go through, uh, hyperparameters or, or you know, some sort of You know, playing around with problems that it knows, but it wouldn't do a very radical change.

[6:12](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=372s) Uh, for example, to come up with a 3D, uh, uh, idea, uh, convolutions, or change the whole problem upside down. So to create those ideas, um, for, uh, for, for the, for the model to try, um, I had to kind of in the midst of the Codex loop, say:

[6:31](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=391s) Oh, what about, what about this idea? What about that idea? Or go read papers out there and see what the papers are, are, are saying, what other peoples are trying.

[6:38](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=398s) Um, so to induce, uh, that hypothesis generation, we need to do something about, about, uh, about our, our ML model, uh, uh, LLM models. So

[6:49](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=409s) this is a trick, um, that I found that working very ef-efficiently, and it's very similar to that chain of thoughts step-by-step problem. First is to decompose the problem into its subcomponents, but it's an explicit ac- um, um, action.

[7:05](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=425s) So, you know, y-you could ask an LLM, actually go through your problem here. In this case, you know, create a-- translate pulmonary nodules CT patches into equivalent PET. Um, that's our top problem that I just explained to you.

[7:18](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=438s) And then it has components into it. So different domain in this case, you can see, um, you know, the, the, the data component, the

[7:28](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=448s) actual core, the, the learning, the architecture, the training loss, the, you know, the operational part of the, the ML modeling, the metrics and evidence, uh, uh, and, um, you know, the peripheral scripts that kind of run, run the model.

[7:43](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=463s) And data preparation itself is very important pieces. Now, what we have here is this hierarchy of, of components of this model that's induced. So this itself can be eas-- induced very easily using a prompt.

[7:56](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=476s) Uh, so basically, you know, y-you... a coding agent can itself go in with this prompt of go in through this code base and create this series of, uh, hyperp-- um, documents that are linked to each other.

[8:10](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=490s) Now, what we're trying to do is, is to give our, um, uh, our coding agent a-ability to look at this problem as component, uh, where it might not, uh, do so, and then induce a, a, a change within those plethora.

[8:28](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=508s) That's exactly what, uh, human researchers would do. Uh, the question is, look at what's important, what's not important. So let me show, let me show y- this graph here in Obsidian.

[8:38](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=518s) So the documents that's generated out of this hierarchy generation would-- looks like something like this, right? So there's a, there's an initial README, says: Here's a problem statement, the level one, and here is our, you know, uh, the documentation.

[8:52](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=532s) And then when we go to a next level,

[8:55](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=535s) uh, of the, of the problem, you see here, um, um, the, the level two. Level two itself goes in and have other level three, uh, components, uh, going on.

[9:06](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=546s) So one of these pieces, for example, is the model architecture. There's all those kind of components that I showed you in the graph that you can see here. And each of these documents are very simple.

[9:14](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=554s) We just ask them, the model to, to document different levels of, um, this hierarchy as it goes in, um, through the process. So here you can see the generator.

[9:24](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=564s) Now it says it's a 2.5D generator and so on, and we go, um, shows the details. And at the end of these hierarchy components is the actual code, um, that is, that is in our system.

[9:36](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=576s) Now, uh, what we do during-- Before we actually start the loop, the question becomes, okay, now that you have this hierarchy, uh, now the-- our model can st-start thinking, uh, step-by-step through the hierarchy and going forward, uh, for the actual, uh, implementation.

[9:55](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=595s) So we ask, um, then, uh, the task is: Given this hierarchy that you have, come up with ideas that can improve, improve this problem. And you can see here, before, uh, y-y if I just s-say, say: Here's our code base and here's my objective goal, optimize this process, similar to what originally Karpathy's README file in this program.md

[10:18](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=618s) had. It would not, it would not generate, it would saturate after a while. But now that we have this hypothesis generation process that can work in an adversarial and collaborative way, um, now you have a process where, uh, you can create, say, you know, um, you, you basically create, um, one hundred, um,

[10:39](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=639s) you could create, um, uh, one hundred, uh, uh, 100 different, uh, solutions. So this goes in, and because we have all the components, we actually have a pr- uh, have a process in which going and, uh, modifying every component of this.

[10:55](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=655s) So it becomes a very much more comprehensive search because you have a scaffold that our reasoning LLMs can go in and make a decision around improving each of these.

[11:07](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=667s) And, uh, very much you can see here that it can come up with the idea around, uh, you know, creating a three-dimensional, um, uh, structure rather than a 2D.

[11:16](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=676s) So basically creates a much more radical change that it otherwise would have not come up or, or would have missed. Now, you can also review this, of course, um, uh, this plan, uh, w-with a, with a, um, adversarially and, um, or collaboratively with another agent or another model to basically loop in and, uh, and improve this.

[11:40](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=700s) For-- Ultimately, the idea would be, um, to go through this i-in-in-in-in-in-in a loop like, um, um, similar to like this. So we have basically, um, you know, a code base, a metrics, um, a goal achieved.

[11:55](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=715s) You know, then you hypothesize the required change and then kind of go in and implement. Uh, and this loop can go m-much faster and much better because here we create a better hypothesis.

[12:06](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=726s) Um, now, let me give you another example, uh, where, um, we also in the parts, the data preparation

[12:15](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=735s) Uh, another way to kind of improve this scientific loop, um, is around the concepts of, um, um... So you can see an example of it as a registration. I wanna show you how you can use these additional skills and additional kind of collaborative and adversarial, uh, um,

[12:32](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=752s) process within your loop. Problem is that we have this, uh, case where we have to do, um, uh, image registration. It's another type of optimization loop in the sense that you, you s-- you-- the idea here is you can see two process, two, two, two scans are overlapped here.

[12:49](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=769s) So the CT scan, the gray, and then the PET scan is the color one that shows the metabolic activity of the, of the tissue. So here you see liver is high and the lung is, is black.

[13:00](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=780s) It, it is not absorbing that, uh, radiotracer. Um, now at the beginning, these two scans are not aligned because they, they scan at different times. Uh, at the beginning, they'll look like this, um, it's off.

[13:14](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=794s) And through that process of optimization, which we call image registration, we need to align these images together. This itself is-- has a goal to make a good registration, but there are many different components to it, many metrics, and sometimes the-- there's a very long process.

[13:30](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=810s) It, it can fail in different aspects of this. As this, uh, you know, the bodies may not be the same. W- well, the, the body's not the same because the, the, you know, because respiration, because changes in the system, uh, y- they, they're not always aligned and the patient's moving and coming in the next day, next day

[13:46](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=826s) or next week do the same scan, uh, with a different scanner would be very different, um, positioning. So to do so,

[13:56](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=836s) um, we have to kind of align these, and we go through the same loop here, but, uh, we have many different, um, type of metrics

[14:05](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=845s) here. Some of them are not necessarily, um,

[14:10](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=850s) uh, quantitative, but qualitative, and, uh, also hard to know which bug or which issue has, has come across. So as part of that loop process, um, w- one of the idea is that as a, as a scientist, I would, would, uh, come here and look at these images one by one myself and could have tried to say,

[14:28](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=868s) "Oh, is this kind of quality control metric?" kind of story where you look at this image, say, "Does this make sense?" So here you can see a lung mask.

[14:35](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=875s) Is the mask applied correctly on the lungs, uh, or, or not? If this process, uh, next steps is good, is the scans properly, uh, uh, truncated, uh, or cropped?

[14:47](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=887s) Um, but each of these processes, there's a loop in itself, and it could be this concept adversarial could also be collaborative. So as part of the metrics, uh, you can kind of call in another model that has a better multimodal capability, uh, to review the image.

[15:06](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=906s) Uh, and what's the opinion about this image? Does the scans correctly aligned? Um, uh, and then as part of the hypothesis generations or critique of the changes that has happened, you can invoke the-- another model such as, uh, uh, that is much more better at reasoning, uh, 5.5 Pro, uh, to generate the hypothesis because they have much

[15:28](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=928s) bigger, um, test time compute and, and inference time compute. To do so, y- you need to equip your loop with additional skills to do these, these processes by themselves.

[15:40](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=940s) And it's very easy now in the coding agents, of course. So here I, I build a, a skill for review of that image generated with, say, with a Gemini model and so on.

[15:51](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=951s) So as part of that loop, you can kind of implement these, these loops here. Uh, and of course, to use GPT 5 and Pro, that's something what we really like is Peter Steinberger's Oracle CLI, which packages the code and packages the data and send them to API, uh, of the, um, the, the GPT 5.5 Pro, and then

[16:11](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=971s) we'll be able to kind of collect-- uh, close the loop, um, uh, on the, uh, on the outcome. And you-- I've seen a much better improvement during the... especially the hypothesis improve generation, uh, but also the critique of is this implementation was really effective because every loop that goes in, the loop basically asks, say, "This will be

[16:30](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=990s) implemented, this is the outcome of it. Does this make sense? What would you do next?" So the process of the loop is very important, uh, to kind of keep going, uh, and improving the model.

[16:41](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1001s) Out of this story, I think what you see as this, um, back to the concept of the, uh, uh, autonomous researcher, is that you have the ability to go through this loop of scientific discovery with much more rigor into the hypothesis.

[16:56](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1016s) And, uh, and I think implementation is already quite a bit solved, especially if your problem is, um, you know, ba- you know, in a, in a, in a basic-- can be done in a basic simulated environment, uh, in, in, in a box.

[17:10](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1030s) Of course, uh, s- the big thing in science is, um, better observations. Uh, the multimodal, uh, model, currently they lack very much understanding of these components. So for example, here, most, um, uh, understanding of these tiny, tiny changes, for example, the existence of this nodule, no LLM today is able to identify these very well because it's just

[17:33](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1053s) not simply trained on scientific, uh, images and scientific data. So there's a big space for future work for people who build these models to be able to fine-tune them or be able to adjust them so they can-- the, the model themselves can close the loop by observing the same way as a trained scientist would be able to

[17:51](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1071s) observe that particular, uh, issue. And then, um, so that's I think one of the biggest bottlenecks for, uh, for not having a full, you know, scientist, uh, in a, in a, in a data center.

[18:03](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1083s) Uh, and then the next thing would be this, the strength of this hypothesis. But I feel the same way as the chain of thoughts came on top of the, um, a basic, um, uh, base model like, you know, 4.0, GPT, GPT 4.0 and, a-and allowed it to do much more better reasoning.

[18:25](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1105s) This type of trick with this type of, um, hierarchical structures is the same trick basically applied on the current model. And I think going forward with the newer models that are h-- much better pre-trained to compartmentalize the problems and break down the problems, uh, you-- we probably get, um, need less and less of these tricks down the

[18:45](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1125s) road. But in the meantime, in the absence of very good models, I think you, you can do something like this where, uh, manually, well, in a different explicit ste-steps, decompose a problem into hierarchies, and then going through those hierarchies for, uh, for, um, improvements in each component of it.

[19:03](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1143s) And basically that allows a very, uh, structured way to scale the test time compute to, to generate more and more tokens on this problem and, uh, close the, uh, and continue the loop.

[19:16](https://www.youtube.com/watch?v=XLEYtv3cMlw&t=1156s) Well, thank you very much for this talk. Uh, uh, hope, uh, to see you around.
