Real-time Experiments with an AI Co-Scientist

Talk transcript

1. 0:00

   [on-hold electronic music] My name is Stefania.

2. 0:16

   I'm so glad you made it until the, yeah, uh, last day of the conference and came to the robotics track. So we're gonna start with a live demo, uh, and then we'll switch to the presentation, just like to s- to kind of like swap things around.

3. 0:31

   So I'm gonna try to connect the microscope over here, uh, and let's see the other camera,

4. 0:39

   and some sensors. So my talk is about real-time, uh, science ex-- co- co-scientists. So think about pair programmers. How many of you use any form of copilot for coding?

5. 0:52

   Right. So it's just like that, but for doing things in the real world, like, uh, science experiments. So you can see here now I have this board, which is a micro:bit board with Jacdac connected, and it's measuring the temperature.

6. 1:05

   And I actually have a heat pad so I can increase the temperature and make it very, very hot, and hopefully it's not gonna melt the board. Uh, and then I can send that to my, uh, science assistant.

7. 1:16

   It's gonna analyze the data and kind of give me an answer in real time. So it's, it's telling me like, "Okay, the ambient conditions indicate like stable, dark, quiet environments," um, because it has like...

8. 1:27

   It knows it can measure all these different sensors, no buttons, um, but it sees that the temperature is twenty-six degrees. And then I can a- actually give it more context.

9. 1:36

   I can go and create a protocol and say like, "This is the type of experiment that I wanna do," um, "so whenever you give me feedback about the data or the images that I'm sending from the microscope," which I can also do, um, it's gonna be in the context of that experiment with those conditions and the, those

10. 1:53

    constraints. And of course, like, if you are actually doing that experiment several times, you can go ahead and create a custom page for your experiment, um, and it will monitor the data from the...

11. 2:07

    We're, we're gonna connect again the Jacdac. Um, it's gonna monitor and plot the data in real time, and the cool thing is also that if you want to leave your experiment running and go away, right?

12. 2:20

    Like, you could actually have cameras that are autonomous, to go back to the previous talk. They're not fully autonomous, but they can track objects. So this camera is called reCamera.

13. 2:31

    Also, by the way, all the hardware I'm showing you and all the components are open source. So right now, uh, I'm gonna set it to track me as a person

14. 2:42

    and start the tracking, and it's running a model on the camera itself, so it's gonna start like moving around to see me and see where I'm going. Um, but you could train a custom model like to track like crystal growth or specific objects that you want to monitor and in real time.

15. 3:01

    And this runs on Wi-Fi, so it can be... You can place it anywhere, um, and then control like the, the conditions in the experiments, like increase the temperature or decrease the temperature.

16. 3:13

    So, oh, it sees all of you. That's awesome. [laughs]

17. 3:17

    Um, so I'm gonna stop this for now.

18. 3:21

    So that's like a short... And here we're seeing like our temperature kind of increasing. I think my heat pad stopped. Um, but that's just like a, a simple like demo of like without having a specific experiment in mind.

19. 3:37

    And now I'm gonna switch to, oops, uh, my slides and actually show you what happens when you record much longer experiments and what it can do. Um, why do we care about co-scientists?

20. 3:51

    We're not talking yet about real-time, uh, co-scientists, just co- co-scientists in general. Like, why should you care about it? We know that there is a, a data overload in science, and there's a lot of complexity, a lot of things to parse through and analyze, and AI can really help with that.

21. 4:07

    Um, we also know that beyond the analysis of data fast at scale, they can help with generating new hypothesis and maybe identifying blind spots in prior work and in your own thinking.

22. 4:19

    And we also know that it can-- they can speed things up, right? So instead of testing one hypothesis at a time, you could test 100 hypothesis. So I got inspired to build this demo.

23. 4:31

    Uh, and by the way, the cost of all the parts, uh, is under $300, and it took me two weeks to build it, and it's open source, so you can play with it and hack it.

24. 4:42

    Um, so I got inspired to do this because of this demo from DeepMind. So this paper got published, uh, two months ago on AI co-scientists, and it was actually showing, um, not in the real life, but it was actually showing what happens when you're analyzing papers and data.

25. 5:01

    If you have a multitude of agents that can perform the different roles that we do as scientists, what are the results? So what are the roles that we do as scientists?

26. 5:11

    Like, we analyze papers, we summarize them, we look at data, we rank the options, we rank the different hypothesis. So they actually created, uh, an orchestration of different agents.

27. 5:24

    Each of them was actually working on Gemini two-- Gemini 2.0. And then, uh, with this orchestration of like ranking the different results, the different hypothesis, doing search online, doing search on papers, come up with a plan for the researcher to use.

28. 5:42

    And not only that they did that, but they actually tested it against prior discoveries, and this was super interesting. So they tested it against prior discoveries on gene transfer mechanisms.

29. 5:54

    So a discovery that took scientists 12 years, uh, it took the AI scientist two days to come up with without having seen the data. So it, it was not aware of how the gene transfer works.

30. 6:06

    This was not part of its training. Um, so, uh, we-- And

31. 6:11

    That, that was one verification by trying to replicate past results. But another one was, like, to come up with completely new hypothesis. So they used it for liver fibrof- fibrosis target treatment discovery, and the AI co-scientist came up with target, uh, uh, drugs that were actually efficient in the lab, in wet lab.

32. 6:32

    They were tested by experts. So it's not a science fiction. It's not like something in the future. It's actually happening now. Um, and for me, what was inspiring about this is, like, when we can make real discoveries in drug, drug discovery like, um, healthcare treatment, bacterial resistance, new type of materials, I think that can

33. 6:57

    be accelerated doing it, uh, in real time with a scientist. This clicker doesn't work very well. So the vision was instead of just doing it like o- on the data, like not sync, like async, and giving the researchers a plan based on the data that exists, what if we do it in real time, right?

34. 7:20

    And we formulate this hypothesis based on the empirical data we're observing in the lab in real time. Or it could be like when you're observing the robot, like, breaking down in real time, right?

35. 7:31

    So what motivated me was the results from the AI co-scientist, but also this vision from Silver and Sutton, which is they, they just published this paper, "Welcome to the Era of Experience," which is fantastic.

36. 7:44

    It's very short. If you haven't read it, I highly recommend it. And they really talk about how we're going past the era of human data, where we're only indexing and making predictions b- uh, based on the datasets that we created, and going to an era where the AI learns from the continuous environment in which we operate, right?

37. 8:06

    And especially with multimodal, when we have real-time data from images, from sensors, from audio streams, like, that is possible.

38. 8:17

    So I hope I convinced you by now that real time matters, and I wanted to show you, like, how I use my system with longer experiments. So, um, you already saw, like, the, the chat image, and I had to find experiments that I could actually do at home.

39. 8:34

    So this is the overview of the, of the system. So it's, it's a very simple React app, and it's has, like, all the input sources that you've seen live in action, like the Jacdac sensors via USB, different webcams.

40. 8:49

    You can add as many webcams as you want. Text input. It actually works with voice. I forgot to show that. You can talk to it, and it talks back.

41. 8:57

    Um, and then, uh, all of these, like, inputs become webhooks, and I can... I, I'm gonna go into more detail how I optimize each one of them.

42. 9:07

    And that gets sent to a backend which communicates with the real-time AI API. I, I use the Gemini in this case. And

43. 9:16

    I had to, uh, this is the information flow. So, um, you have the physical sensors, and for the Jacdac, I'm actually using the WebUSB API. Um, that goes to the front-end hooks, which gets pulled by the context assembly.

44. 9:30

    And the context assembly actually is always checking what sort of modalities are present. Do we have text? Do we have voice? Do we have image? Do we have a chat history?

45. 9:39

    And depending on what modality, uh, modalities are present, every single time, uh, I'm sending a message, it builds this context that gets sent to the API that gives me back a response.

46. 9:52

    And, uh, this is how the unified context assembly works. So

47. 9:58

    I guess, like, the, the very important piece here is how you're doing this con- uh, context injection depending on what sensors you have connected and the type of experiment you're defining, and making this dynamic on the fly, like when you create the protocols.

48. 10:17

    And that was the, the, tha- those were the ingredients for the code. Now let's, now let's talk about the ingredients for the hardware. So when I start doing the, the experiments, I had to list and put out all the parts that I had at my, my disposal.

49. 10:32

    Kind of like cooking, right? So it's like, "What can I make with all these sensors?" Like, these are my inputs. These are my outputs. Uh, these are my, all my cameras [chuckles] and all my cables, like, and how many boards I have.

50. 10:44

    What sort of experiments could I create with that? And

51. 10:48

    I also had the constraint of doing experiments that I can measure in real time for this talk, and also that are safe for me to do at home and I can travel with, right?

52. 10:58

    So that really narrowed down the list of things that I could play with. So one thing that I did was exploring crystallization. So you can see there, like, the crystals that I was growing, and that took longer, and I mon- I recorded the whole thing, and fermentation.

53. 11:14

    And let's dive into each one of those. So this was my lab in a box in our apartment in Tokyo. Um, the people coming to the house were very confused every day. [laughs]

54. 11:25

    Um, there was something... No explosions, though, so that was good. Uh, so for, for crystal growth, how many of you reco- remember this from chemistry? Okay. Okay. That's awesome.

55. 11:37

    So I don't need to go into great detail, but basically the, the principle is quite simple. Like, you oversaturate the solution by adding salt in hot water, and then you're, like, slowly cooling it off, and that creates this process, this, uh, this nucleation and the growth of crystals.

56. 11:55

    Now, the trick in having beautiful crystals is how you do the cooling off, right? It needs to be gradual. You need to not move, like, the, um, the liquid or the object.

57. 12:07

    So the gradual cooling and the level of humidity is what gives you beautiful crystal formation. So the main things we wanna measure in this example is what are, are, are curves for how fast the salt d- dis- uh, dissolves.

58. 12:24

    Like, where are the places where the crystals are being formed? Those are called nucleation sites. What's the growth rate of the crystals? And, um, that is affected by temperature concentration.

59. 12:35

    And oh yeah, I forgot to play this. This was, like, a sped up. So the, this was a recording from the microscope. It's moving because it actually has a fan, uh, uh, uh, blowing cool air from ice.

60. 12:50

    Uh, so that's why, like, the camera is moving. That was, uh, how I was, like, cooling it off. But you can see the data in the sensors changing on the side, and this is, like, being recorded for an extended period of time.

61. 13:01

    And then I get a CSV from the, the sensor values, and I can go and analyze that, um, and kind of get a sense for, like, what happened. Was I cooling it off too fast or too low, too, uh, slow?

62. 13:15

    Um, were my crystal, like, growing or not? And then I also played with different samples. So I had samples that I would put in the fridge and in the room and, like, at different temperatures so I could also measure and have, like, control groups for that.

63. 13:29

    And then once I have that CSV data, I can plot it, right? So I can actually see what was my crystal growth rate, um, and if my temperature was, like, the best, and I can plot that for the different samples.

64. 13:43

    And right now, I don't have that integrated live on the platform, but that's coming. Um, so I had to write a separate script to get the CSV data and create these visualizations.

65. 13:55

    Um, and the insight I got from this is that the crystal formation is actually, uh, not gradual, that it happens in bursts. Um, so that, that was something that surprised me when I looked at the data.

66. 14:07

    So once the critical saturation is reached, there's, like, sudden crystal growth formation. So it's not, like, something that it's kind of, like, slowly growing, but, uh, it grows in bursts.

67. 14:16

    Uh, and here is, like, 40 minutes recording of the crystal growth, like, super sped up, so you can kind of see how the humidity and the temperature was changing.

68. 14:28

    And the, the next one was... How am I doing on time? Because I have a lot of things. Okay. Uh, I might skip over fermentation just in interest of time, but, um, I'm sure all of you have dealt with fermentation in sh- some shape or form, even if it was liquid having to benefit the, the outf- uh,

69. 14:48

    the end result. Um, but the, the insight here is to actually control how much salt and sugar we put into the different, uh, dough and then measure and, and then also change the temperature and measure the growth rate, uh, and the CO2.

70. 15:05

    And, uh, that was the fermentation, and I also have, like, a sped up recording of the data collection.

71. 15:14

    And, uh, I already showed you the reCamera and the fact that it's mobile and it can track objects. Um, this is the education version. Uh, if you wanna play with some of these things and just kind of do your own experiments, uh, please, like...

72. 15:28

    The lab mine that I showed you is not deployed yet, but it will be, and the code will be open source. But this you can play with. You need to put your own API. [laughs]

73. 15:37

    But it runs on the phone, and you can test the camera for now. Uh, and if you have a micro:bit or you wanna connect mine, um, you're welcome to, to try it.

74. 15:47

    And let me know if the QR doesn't work. And yeah, this, this is a version kind of, like, for, to teach it to kids. Um, and I wanted to end by talking a little bit about the open source ecosystem that can support this type of initiative to go beyond the demo and kind of like a cool experimentation

75. 16:07

    to a real s- uh, real solution in, uh, for scientists and builders. So there's an entire open source ecosystem for recreating all the lab equipment, but then also for creating machines that can support the automation of pipetting and, like, doing all the, um, analysis that you need to do in a lab and all the manipulation that you

76. 16:30

    need to do in a lab. The Jubilee Motion platform, it's actually coming from University of Washington where I did my PhD. It's open source. Um, and there's an open bioreactor, and there's actually a workshop that just took place in April at UW where they had people for a week hacking on creating different solutions for automating la- uh,

77. 16:53

    s- um, scientific experiments in the lab. So anything from, like, droplet manipulation to robot handling liquids and mixing, like, vials and things like that, and other application with Jubilee.

78. 17:06

    So definitely check them out. And for the future, uh, this is my last slide, you can imagine that based on the samples that we're collecting from these cameras and sensors and voice, it's much easier to create simulations.

79. 17:26

    So we need to, we n- uh, don't need to be limited by the experiments we're doing in real life, but those experiments that we're doing in real life are gonna inform realistic simulations, right?

80. 17:37

    So I could actually create a simulation of my crystal growth, but more useful stuff, right, like bacteria colony growth, and run those simulations with the lab conditions and then identify what are the best conditions for the experiment and create those conditions in the real life.

81. 17:55

    So very excited about integrating simulation, and I think that's where the, this is going. Uh, that was me. Uh, if you wanna read more about, uh, my projects and the paper and all the open source projects, I do a lot of work in education as well.

82. 18:11

    It's all on my website. And as Ben mentioned this morning, we are going to do an AI education summit, uh, and very excited about that. Uh, some of you who saw my talk, like, in New York or last year know how much, how passionate I am about, about education.

83. 18:28

    Um, so I hope you can join us for the summit. I really appreciate you coming to the talk today, and, uh, I'll be around. Thank you so much. [upbeat music]