Autonomous Agents for Scientific Tasks - Sina Shahandeh, RADiCAIT

Talk transcript

1. 0:00

   Hello, everyone. My name is Sina Shahandeh. A qui-- my pleasure to present you this talk about, uh, running autonomous agents for scientific tasks.

2. 0:12

   Um, let's, let's dive in. Um, so here, um, I, I think everyone is quite familiar with the concept of, uh, autoresearcher or autoresearch. Uh, uh, this is the original Andrej Karpathy's, um, uh, GitHub repo, where, uh, we have a ML model, and we ask a coding agent to, uh, uh, find, s-find a particular metrics, and then

3. 0:37

   optimize the code in order to minimize the error. Basically, do a hill climb over a model optimization.

4. 0:45

   Now, for many of these coding tasks, um, this works very well. Uh, but when the problems become very much open-ended and, um, y-

5. 0:56

   sometimes long horizon, m-- like most of the scientific task, y-you have this case where, uh, AI agents, uh, usually kind of saturate, uh, to a certain level. Um, simply, they, they are very good at implementation of the, of the code or changing the, m-- or running the experiments over lots of data and so on.

6. 1:14

   But the problem is they run out of ideas, uh, or, you know, what people call them research taste. Now, you can see that in these situations, uh, you know, good humans keep going, uh, higher, and the top one percent humans, you know, they keep even, uh, improving better and better over time.

7. 1:32

   Um, now, um, uh, the, the, the really-- the difference, um, from here is the way that, uh, good ideas, uh, can... Or, or good hypothesis on how the model could be improved or how the, the problem could be solved, um, keep, um, keep coming, coming up, humans keep coming up.

8. 1:50

   So in the scientific task, uh, you, you know, you're, uh, the, you have this, um, scientific method, um, where, you know, we observe a situation, we make questions, uh, we come up with a good hypothesis and come up with a hypothesis of how to solve this problem, and then we do experiment and implement and do the experiment,

9. 2:08

   um, do the loop. Um, and the, as each of the iterations, we learn something and we improve. Now,

10. 2:15

    uh, the components of a learning component, uh, I think those are all questions of memory and implementation from learning the mistakes, but it-- which is one of the bottlenecks of, uh, using coding agents.

11. 2:25

    But I think this quite solved by just simply organizing patterns of a-activity. I think what is much more difficult is coming up with a hypothesis. So how can we come up with a good hypothesis, good ideas for our, uh, coding agents to keep improving better, uh, the process?

12. 2:42

    And this is something core things that I would like to kind of, um, um, focus on this talk, uh, and plus a little bonus at the end.

13. 2:50

    Um, so, um, let's look at a problem here w-we're trying to achieve as an example, so we have a good, good idea of what we're trying to do. So here is what we're doing at RADiCAIT.

14. 2:58

    We're building, uh, in silico PET, meaning you have a CT, and we want to generate, um,

15. 3:06

    a PET, PET image, um, a PET scan, uh, from the CT scan. Example of this you can see here, um, we have CT scans and of slices of bo-- um, scans of the, of, of the, of a patient.

16. 3:18

    Uh, they might have a nodule in the, in the lung. And the question is: Is this cancerous or not? Um, is it a lung cancer? So they do a PET scan, which is a difficult process and very time-consuming and, uh, to do.

17. 3:29

    Um, but here we do an ML-- through an ML model image translation, one can kind of change the modality, learn the structure of the body, and infer what would happen in a PET scan if, um, the hyper-- the activity of the tissue.

18. 3:43

    So, you know, certain tissues absorb more radioactive tracer, and they shine up in this PET scan. Um, and, um, the, the tumors, usually that's the case. Now, to generate this relationship, uh, we need, um, we need a model, uh, to, um, to do the translation, but the problem itself has many components.

19. 4:05

    So really, uh, this, like any other scientific task, the problem is decomposing the problem, uh, entire long-term horizon, two years, ten years research process into steps. And each of those steps is really fundamentally are a goal, are, are, are a, are a loop.

20. 4:21

    So I'm gonna focus on one of these particular ones right now, and that is on training of a machine learning model. Uh, uh, so we have here, um, you know, decoder, encoder type of situations.

21. 4:33

    Um, so, uh, en-encoding the CT and then decoding it into PET. Um, so that's the typical GAN model which kind of generates the, the image. Now, for this, we can kind of define these kind of, you know, the architecture, and we try it, we capture data, uh, and do all the 80% of the work to basically bringing

22. 4:51

    the good data set. And here, uh, we create the metrics and so on, and, um, around the image, m-- you know, fidelity of synthetic PET, uh, to, to real PET and so, so on.

23. 5:03

    But the challenge is: How can we improve this situation, uh, given a certain, um, in-initial points? And we'll go back to our idea of, uh, hill climb, uh, around this, uh, optimization.

24. 5:16

    So you can see an example of iterations coming from a real run in Codex, where, you know, we improve the, the data and so on, and then we, the model goes around and tries to do the optimization.

25. 5:28

    And you can see there's a range of possibilities, and some of them become dead end, uh, some of them not, don't improve anything. But, um, we kind of desaturate, uh, at a certain point.

26. 5:38

    And then you need a, really need a good idea. A good idea has to come up. So in this case, we have a slices of CT, and we feed these as a channel into the model.

27. 5:47

    So initial problem, initial, uh, model, uh, that was trained was two and a half D. So treating each CT slice as a, uh, be the 2D convolutions, but stacked with a channel.

28. 5:58

    Now, if you give this to a, um, a typical ML model, it would not think about it as, you know, go through, uh, hyperparameters or, or you know, some sort of You know, playing around with problems that it knows, but it wouldn't do a very radical change.

29. 6:12

    Uh, for example, to come up with a 3D, uh, uh, idea, uh, convolutions, or change the whole problem upside down. So to create those ideas, um, for, uh, for, for the, for the model to try, um, I had to kind of in the midst of the Codex loop, say:

30. 6:31

    Oh, what about, what about this idea? What about that idea? Or go read papers out there and see what the papers are, are, are saying, what other peoples are trying.

31. 6:38

    Um, so to induce, uh, that hypothesis generation, we need to do something about, about, uh, about our, our ML model, uh, uh, LLM models. So

32. 6:49

    this is a trick, um, that I found that working very ef-efficiently, and it's very similar to that chain of thoughts step-by-step problem. First is to decompose the problem into its subcomponents, but it's an explicit ac- um, um, action.

33. 7:05

    So, you know, y-you could ask an LLM, actually go through your problem here. In this case, you know, create a-- translate pulmonary nodules CT patches into equivalent PET. Um, that's our top problem that I just explained to you.

34. 7:18

    And then it has components into it. So different domain in this case, you can see, um, you know, the, the, the data component, the

35. 7:28

    actual core, the, the learning, the architecture, the training loss, the, you know, the operational part of the, the ML modeling, the metrics and evidence, uh, uh, and, um, you know, the peripheral scripts that kind of run, run the model.

36. 7:43

    And data preparation itself is very important pieces. Now, what we have here is this hierarchy of, of components of this model that's induced. So this itself can be eas-- induced very easily using a prompt.

37. 7:56

    Uh, so basically, you know, y-you... a coding agent can itself go in with this prompt of go in through this code base and create this series of, uh, hyperp-- um, documents that are linked to each other.

38. 8:10

    Now, what we're trying to do is, is to give our, um, uh, our coding agent a-ability to look at this problem as component, uh, where it might not, uh, do so, and then induce a, a, a change within those plethora.

39. 8:28

    That's exactly what, uh, human researchers would do. Uh, the question is, look at what's important, what's not important. So let me show, let me show y- this graph here in Obsidian.

40. 8:38

    So the documents that's generated out of this hierarchy generation would-- looks like something like this, right? So there's a, there's an initial README, says: Here's a problem statement, the level one, and here is our, you know, uh, the documentation.

41. 8:52

    And then when we go to a next level,

42. 8:55

    uh, of the, of the problem, you see here, um, um, the, the level two. Level two itself goes in and have other level three, uh, components, uh, going on.

43. 9:06

    So one of these pieces, for example, is the model architecture. There's all those kind of components that I showed you in the graph that you can see here. And each of these documents are very simple.

44. 9:14

    We just ask them, the model to, to document different levels of, um, this hierarchy as it goes in, um, through the process. So here you can see the generator.

45. 9:24

    Now it says it's a 2.5D generator and so on, and we go, um, shows the details. And at the end of these hierarchy components is the actual code, um, that is, that is in our system.

46. 9:36

    Now, uh, what we do during-- Before we actually start the loop, the question becomes, okay, now that you have this hierarchy, uh, now the-- our model can st-start thinking, uh, step-by-step through the hierarchy and going forward, uh, for the actual, uh, implementation.

47. 9:55

    So we ask, um, then, uh, the task is: Given this hierarchy that you have, come up with ideas that can improve, improve this problem. And you can see here, before, uh, y-y if I just s-say, say: Here's our code base and here's my objective goal, optimize this process, similar to what originally Karpathy's README file in this program.md

48. 10:18

    had. It would not, it would not generate, it would saturate after a while. But now that we have this hypothesis generation process that can work in an adversarial and collaborative way, um, now you have a process where, uh, you can create, say, you know, um, you, you basically create, um, one hundred, um,

49. 10:39

    you could create, um, uh, one hundred, uh, uh, 100 different, uh, solutions. So this goes in, and because we have all the components, we actually have a pr- uh, have a process in which going and, uh, modifying every component of this.

50. 10:55

    So it becomes a very much more comprehensive search because you have a scaffold that our reasoning LLMs can go in and make a decision around improving each of these.

51. 11:07

    And, uh, very much you can see here that it can come up with the idea around, uh, you know, creating a three-dimensional, um, uh, structure rather than a 2D.

52. 11:16

    So basically creates a much more radical change that it otherwise would have not come up or, or would have missed. Now, you can also review this, of course, um, uh, this plan, uh, w-with a, with a, um, adversarially and, um, or collaboratively with another agent or another model to basically loop in and, uh, and improve this.

53. 11:40

    For-- Ultimately, the idea would be, um, to go through this i-in-in-in-in-in-in a loop like, um, um, similar to like this. So we have basically, um, you know, a code base, a metrics, um, a goal achieved.

54. 11:55

    You know, then you hypothesize the required change and then kind of go in and implement. Uh, and this loop can go m-much faster and much better because here we create a better hypothesis.

55. 12:06

    Um, now, let me give you another example, uh, where, um, we also in the parts, the data preparation

56. 12:15

    Uh, another way to kind of improve this scientific loop, um, is around the concepts of, um, um... So you can see an example of it as a registration. I wanna show you how you can use these additional skills and additional kind of collaborative and adversarial, uh, um,

57. 12:32

    process within your loop. Problem is that we have this, uh, case where we have to do, um, uh, image registration. It's another type of optimization loop in the sense that you, you s-- you-- the idea here is you can see two process, two, two, two scans are overlapped here.

58. 12:49

    So the CT scan, the gray, and then the PET scan is the color one that shows the metabolic activity of the, of the tissue. So here you see liver is high and the lung is, is black.

59. 13:00

    It, it is not absorbing that, uh, radiotracer. Um, now at the beginning, these two scans are not aligned because they, they scan at different times. Uh, at the beginning, they'll look like this, um, it's off.

60. 13:14

    And through that process of optimization, which we call image registration, we need to align these images together. This itself is-- has a goal to make a good registration, but there are many different components to it, many metrics, and sometimes the-- there's a very long process.

61. 13:30

    It, it can fail in different aspects of this. As this, uh, you know, the bodies may not be the same. W- well, the, the body's not the same because the, the, you know, because respiration, because changes in the system, uh, y- they, they're not always aligned and the patient's moving and coming in the next day, next day

62. 13:46

    or next week do the same scan, uh, with a different scanner would be very different, um, positioning. So to do so,

63. 13:56

    um, we have to kind of align these, and we go through the same loop here, but, uh, we have many different, um, type of metrics

64. 14:05

    here. Some of them are not necessarily, um,

65. 14:10

    uh, quantitative, but qualitative, and, uh, also hard to know which bug or which issue has, has come across. So as part of that loop process, um, w- one of the idea is that as a, as a scientist, I would, would, uh, come here and look at these images one by one myself and could have tried to say,

66. 14:28

    "Oh, is this kind of quality control metric?" kind of story where you look at this image, say, "Does this make sense?" So here you can see a lung mask.

67. 14:35

    Is the mask applied correctly on the lungs, uh, or, or not? If this process, uh, next steps is good, is the scans properly, uh, uh, truncated, uh, or cropped?

68. 14:47

    Um, but each of these processes, there's a loop in itself, and it could be this concept adversarial could also be collaborative. So as part of the metrics, uh, you can kind of call in another model that has a better multimodal capability, uh, to review the image.

69. 15:06

    Uh, and what's the opinion about this image? Does the scans correctly aligned? Um, uh, and then as part of the hypothesis generations or critique of the changes that has happened, you can invoke the-- another model such as, uh, uh, that is much more better at reasoning, uh, 555 Pro, uh, to generate the hypothesis because they have much

70. 15:28

    bigger, um, test time compute and, and inference time compute. To do so, y- you need to equip your loop with additional skills to do these, these processes by themselves.

71. 15:40

    And it's very easy now in the coding ages, of course. So here I, I build a, a skill for review of that image generated with, say, with a Gemini model and so on.

72. 15:51

    So as part of that loop, you can kind of implement these, these loops here. Uh, and of course, to use GPT 5 and Pro, that's something what we really like is Peter Spanberger's Oracle CLI, which packages the code and packages the data and send them to API, uh, of the, um, the, the GPT 5.5 Pro, and then

73. 16:11

    we'll be able to kind of collect-- uh, close the loop, um, uh, on the, uh, on the outcome. And you-- I've seen a much better improvement during the... especially the hypothesis improve generation, uh, but also the critique of is this implementation was really effective because every loop that goes in, the loop basically asks, say, "This will be

74. 16:30

    implemented, this is the outcome of it. Does this make sense? What would you do next?" So the process of the loop is very important, uh, to kind of keep going, uh, and improving the model.

75. 16:41

    Out of this story, I think what you see as this, um, back to the concept of the, uh, uh, autonomous researcher, is that you have the ability to go through this loop of scientific discovery with much more rigor into the hypothesis.

76. 16:56

    And, uh, and I think implementation is already quite a bit solved, especially if your problem is, um, you know, ba- you know, in a, in a, in a basic-- can be done in a basic simulated environment, uh, in, in, in a box.

77. 17:10

    Of course, uh, s- the big thing in science is, um, better observations. Uh, the multimodal, uh, model, currently they lack very much understanding of these components. So for example, here, most, um, uh, understanding of these tiny, tiny changes, for example, the existence of this nodule, no LLM today is able to identify these very well because it's just

78. 17:33

    not simply trained on scientific, uh, images and scientific data. So there's a big space for future work for people who build these models to be able to fine-tune them or be able to adjust them so they can-- the, the model themselves can close the loop by observing the same way as a trained scientist would be able to

79. 17:51

    observe that particular, uh, issue. And then, um, so that's I think one of the biggest bottlenecks for, uh, for not having a full, you know, scientist, uh, in a, in a, in a data center.

80. 18:03

    Uh, and then the next thing would be this, the strength of this hypothesis. But I feel the same way as the chain of thoughts came on top of the, um, a basic, um, uh, base model like, you know, 4.0, GPT, GPT 4.0 and, a-and allowed it to do much more better reasoning.

81. 18:25

    This type of trick with this type of, um, hierarchical structures is the same trick basically applied on the current model. And I think going forward with the newer models that are h-- much better pre-trained to compartmentalize the problems and break down the problems, uh, you-- we probably get, um, need less and less of these tricks down the

82. 18:45

    road. But in the meantime, in the absence of very good models, I think you, you can do something like this where, uh, manually, well, in a different explicit ste-steps, decompose a problem into hierarchies, and then going through those hierarchies for, uh, for, um, improvements in each component of it.

83. 19:03

    And basically that allows a very, uh, structured way to scale the test time compute to, to generate more and more tokens on this problem and, uh, close the, uh, and continue the loop.

84. 19:16

    Well, thank you very much for this talk. Uh, uh, hope, uh, to see you around.