Real-time Experiments with an AI Co-Scientist

A heat pad warms a micro:bit connected to Jacdac sensors. Stefania Druga sends its readings to a science assistant, which interprets the surroundings as stable, dark, and quiet and reports a temperature of 26 degrees. The interaction resembles pair programming: a person works with physical equipment while an AI collaborator helps interpret the results. Here, the shared working material is sensor data rather than source code.

A reading alone does not explain what an experiment is trying to accomplish. Druga adds that missing context through an experiment protocol: the intended experiment, its conditions, and its constraints. Subsequent feedback about sensor readings or microscope images can then address that experiment instead of merely describing the room. The LabMind dashboard brings those observations together with the assistant conversation.

For a repeated experiment, a custom page preserves the workflow and provides real-time monitoring and plotting. During the demonstration, Druga reconnects Jacdac before continuing the live monitoring. The progression is from a single observation, to an observation interpreted against a protocol, to a reusable place to follow an experiment over time.

Leaving an experiment running introduces another problem: keeping the relevant object in view. Druga demonstrates reCamera, a camera that runs its tracking model on the device itself. She describes the displayed hardware and components as open source, selects person tracking, and starts the camera moving to follow its target. This is object tracking, not full autonomy.

A custom model could instead track crystal growth or another experimental object. Wi-Fi connectivity allows flexible placement; Druga also proposes combining observation with changes to experimental conditions, such as raising or lowering temperature. The live demonstration establishes the person-detection behavior: the camera turns toward the room and detects audience members before she stops it. It does not demonstrate crystal tracking or automatic temperature control.

Back on the sensor dashboard, the temperature has been increasing, although Druga suspects the heat pad has stopped. There is no specific scientific experiment governing this opening demonstration. It establishes the available interactions—measurement, interpretation, and tracking—before the longer experiments put them to use.

Scientific work produces more data and literature than a researcher can readily inspect. A co-scientist can help analyze that material, generate hypotheses, and expose blind spots in prior work or in the researcher's own reasoning. Druga illustrates the potential with the possibility of testing 100 hypotheses instead of one; this is a motivation for broader exploration, not a measured throughput result. She reports that the prototype's parts cost under $300 and that it took two weeks to build. She presents it as an open-source project others could adapt.

The inspiration was Google's Towards an AI co-scientist, which coordinates agents built on Gemini 2.0. Its work begins with existing papers and data. Agents perform complementary scientific tasks: analyzing and summarizing papers, inspecting data, searching literature and the web, and ranking candidate hypotheses. Their combined output gives a researcher a plan to investigate. The useful unit is the orchestration of these activities, rather than a single answer to a scientific question.

Druga describes the gene-transfer example as 12 years of scientific work versus two days for the AI co-scientist. The comparison needs its task boundary: the paper reports two days to generate a hypothesis using prior literature, while Imperial's account describes more than ten years of investigation, including laboratory work. Researchers already knew the answer from unpublished experiments. The particular result concerned cf-PICIs interacting with diverse phage tails to expand host range; it was not a demonstration that the model knew nothing about gene transfer. An unpublished answer and a general audit of training-data exclusion are different claims.

The system also proposed new liver-fibrosis targets that experts tested in the laboratory. Google's report specifies anti-fibrotic activity in human hepatic organoids, a preclinical model rather than evidence of treatment efficacy in patients. Together, hypothesis rediscovery and new laboratory-tested candidates suggest applications in drug discovery, healthcare, bacterial resistance, and materials research.

The next step is to bring the assistant into an experiment while it is happening. Instead of asynchronously producing a plan from existing data, it could formulate hypotheses from current laboratory observations—or help interpret a robot breaking down as the failure unfolds. Real-time context changes which evidence is available when a decision is made.

Druga connects this direction to David Silver and Richard Sutton's Welcome to the Era of Experience. Their proposed research agenda moves beyond predictions based only on human-created datasets toward learning through continuous environmental experience. Images, sensor measurements, and audio streams provide channels for that experience. LabMind demonstrates live multimodal observation and response; continuously learning from those observations is a further ambition, not a capability established by the demo.

The application is a React frontend accepting Jacdac sensors over USB, multiple webcams, text, and voice. Voice works in both directions: the user can speak to the assistant and hear its response. Druga describes the inputs as becoming webhooks that feed a backend connected to a real-time Gemini API.

The more detailed information flow starts with physical sensors. Jacdac reaches the browser through WebUSB, and frontend hooks expose observations to a context assembly layer. Whenever a message is sent, that layer checks which modalities are present—text, voice, images, and chat history—and constructs the context for the API request. The response returns to the application. This describes the demonstrated architecture without assuming a particular current Gemini SDK or transport.

The important implementation boundary is between collecting an observation and deciding what that observation means for the current experiment. Context injection must adapt both to connected sensors and to protocols created during the session. A TypeScript implementation of that assembly boundary can keep the pieces explicit:

type Observation =
  | { kind: "sensor"; name: string; value: number; unit: string }
  | { kind: "image"; source: string; data: Blob }
  | { kind: "audio"; data: Blob };

type Protocol = {
  goal: string;
  conditions: string[];
  constraints: string[];
};

type Turn = {
  role: "user" | "assistant";
  text: string;
};

function assembleContext(input: {
  text?: string;
  protocol?: Protocol;
  observations: Observation[];
  history: Turn[];
}) {
  return {
    protocol: input.protocol ?? null,
    history: [...input.history],
    text: input.text?.trim() || null,
    sensors: input.observations.filter(o => o.kind === "sensor"),
    images: input.observations.filter(o => o.kind === "image"),
    audio: input.observations.filter(o => o.kind === "audio"),
  };
}

This produces an application-level context object for a backend adapter to encode. Empty modality arrays preserve the distinction between an absent input and an actual observation; a protocol accompanies the data rather than replacing it.

With the software pipeline in place, the next question is what the available hardware can measure. Druga approaches this like cooking: lay out the ingredients before choosing a recipe. Her inventory includes sensor inputs, outputs, cameras, cables, and boards. Each experiment has to fit that inventory.

Three further constraints narrow the choice: the experiment must be measurable in real time, safe to conduct at home, and practical to travel with. Crystallization and fermentation fit. Druga builds a portable laboratory in her Tokyo apartment and records the longer crystal-growth experiments in full, allowing changes that take longer than a stage demonstration to be inspected afterward.

The crystallization setup follows a simple sequence:

1. Add salt to hot water to prepare a solution that becomes supersaturated as it cools.
2. Cool the solution gradually, allowing nucleation and crystal growth.
3. Minimize disturbance while monitoring conditions, including temperature and humidity.

Druga identifies gradual cooling and humidity as important to the resulting crystal formation. The measurements of interest include dissolution curves, the locations where crystals first form—nucleation sites—and subsequent growth rates. Temperature and concentration affect those processes.

The microscope recording visibly moves because a fan blows air cooled by ice across the setup. That is the cooling mechanism, but it also disturbs the view. Alongside the images, sensor values change throughout the recording. Exporting those values as CSV makes it possible to inspect the cooling trajectory afterward and ask whether cooling was too fast or too slow, and whether crystals grew under those conditions. Druga also places samples in the refrigerator, in the room, and at other temperatures to create comparison groups.

For the analysis, she uses a separate script to read the CSV and create plots comparing temperatures and crystal-growth rates across samples. Those analysis visualizations are not yet integrated live into the platform. This is distinct from the opening dashboard's live sensor plotting: the longer experiment still moves through recording, export, and separate analysis.

Druga's surprising observation is that crystal formation occurs in bursts rather than as steady, gradual growth. She interprets the bursts as sudden formation once critical saturation is reached. A sped-up recording covers 40 minutes of crystal growth while temperature and humidity change. The longer record makes these transitions visible in a way that a single microscope image cannot.

Fermentation provides another experiment with controllable inputs and observable changes. Druga varies the salt and sugar in dough and changes the temperature, then measures growth rate and CO₂. She shows a sped-up recording of data collection but moves through the example quickly, without presenting detailed comparative results.

For readers wanting to try the interaction, the talk distinguishes two versions:

The education version requires users to supply their own API access and is intended for teaching children. The laboratory application's deployment and release remain future work at this point in the recording.

Moving beyond an observational demo requires equipment that can manipulate experiments as well as record them. Druga points to an open-source ecosystem spanning recreated laboratory equipment, pipetting, analysis, and sample handling. The Jubilee Motion platform and UW laboratory-automation ecosystem connect this direction to the University of Washington, where she completed her PhD. She also mentions an open bioreactor.

The examples include droplet manipulation, robotic liquid handling, and mixing vials. Druga recalls an April workshop at UW as a week of building scientific-automation tools. If this was the university's Pathways to Open-Source Hardware for Laboratory Automation event, its organizer lists April 13–15, 2025. The broader engineering opportunity is to connect the assistant's observations and proposed experiments to existing machines that can carry out physical operations.

The proposed next step is a loop between physical experiments and simulations. Camera images, sensor readings, and voice observations would inform simulations grounded in actual laboratory conditions. Those simulations could explore beyond the small number of experiments practical to run physically.

For crystal growth—or a more consequential application such as bacterial-colony growth—the sequence would be to collect observations, simulate growth under the observed laboratory conditions, identify favorable settings, and then reproduce those settings in the physical experiment. Simulation integration is future work, extending the current measurement-and-analysis workflow toward choosing what to try next. The closing diagram shows this two-way relationship between the experimental dashboard and collections of sample images.

Druga closes by directing readers to her website for projects, papers, and open-source work, including her education projects. She also points to an upcoming AI education summit, connecting the laboratory tools back to making experimentation accessible to more learners.