AI Engineer World's Fair 2024
LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries
Read the talk
Scientific Discovery Needs Reasoning Before Retrieval
A petunia experiment that produced the opposite of its intended result exposes a harder task for RAG: connecting scattered observations into a grounded scientific hypothesis.
From a talk by Hubert Misztela
Before you start: Basic familiarity with embeddings and retrieval-augmented generation is helpful; no biology background is required.
Why did adding a gene remove the color?
Could an LLM connect unexpected biological results before scientists recognize their common cause? Hubert Misztela approaches that question from generative molecular design at Novartis, where he works with small molecular graphs. His scientific-discovery project with medicinal chemist Derek Lowe starts with a particularly useful puzzle: researchers trying to intensify petunia flower color obtained the opposite result.
In the early 1990s, scientists overexpressed a gene expecting stronger pigmentation. Instead, the flowers lost purple pigment. Similar surprises appeared in other biological subdomains, but the phenomena had different names. Researchers could see that something was interfering with gene activity without knowing the shared explanation.
In Misztela’s account, connecting and resolving the three phenomena took roughly eight years and led to a Nobel Prize in medicine. That delay creates a concrete discovery task: find a plausible common cause from the literature available before the answer was known. Perhaps the necessary clues were already present, scattered across disciplines. Asking a modern model for the answer is easy; determining whether it can assemble those clues is harder.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieval inherits the difficulty of the question
Ordinary retrieval-augmented generation starts by processing documents and representing them as embeddings. At query time, it embeds the question in the same latent space, retrieves nearby chunks, and supplies them to a response generator. This works naturally when the question points toward text that already contains an answer.
Different RAG improvements repair different parts of that pipeline. HyDE, ranking, and document indexing are not interchangeable: improving the match between a question and answer-bearing text is a different problem from selecting the best documents or chunks. But those improvements still depend on understanding what evidence the question requires. A convoluted question can defeat that understanding before retrieval begins. Misztela describes this as “trying to run before we crawl.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reason over the question and the representation
Reasoning can happen before retrieval on either side of the search. Routing interprets a question before deciding where to send it; the query cannot simply be thrown at an embedding index and expected to produce the right evidence. On the document side, GraphRAG and GraphReader represent knowledge using graphs. The shared architectural move is to process meaning before selecting material, whether by understanding the request or by changing how document knowledge is represented.
After retrieval, an LLM usually performs the final reasoning, but that is a design choice rather than a requirement. The more semantically complex the question, the more work may be needed before retrieval as well. A useful pipeline therefore becomes reasoning → embedding → retrieval → reasoning, with each reasoning stage chosen for a specific job.
This resembles the molecular-design workflow Misztela already uses. An encoder–decoder model, such as a variational autoencoder, represents molecular graphs in a latent space. The system then navigates that space to find points useful for a desired objective. RAG also selects points from an embedding space, but usually through similarity to a question.
| Workflow | Representation | Selection objective |
|---|---|---|
| Standard RAG | Document embeddings | Similarity to the question |
| Molecular design | Molecular latent space | Optimize a desired property |
The analogy broadens what counts as reasoning: goal-directed optimization over a latent space can perform part of the job. The system need not express every operation as an LLM-generated chain of thought.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How many concepts must the evidence connect?
A needle-in-a-haystack test asks a system to locate a particular piece of information inside a large context. Misztela uses the example of finding the ingredient for a perfect pizza. A multi-needle test extends the task to several pieces of information. But counting answer fragments captures only part of the difficulty: the relationship between the question and those fragments matters too.
The database analogy makes that relationship explicit:
| Question concepts | Evidence locations | Relationship |
|---|---|---|
| One | One | One-to-one |
| One | Several | One-to-many |
| Several | One | Many-to-one |
| Several | Several | Many-to-many |
A question can combine several concepts that must jointly identify one relevant passage, or it can require concepts scattered across multiple chunks. In either case, searching for the whole question as one undifferentiated vector may hide the structure that determines what to retrieve.
Multiple concepts are a signal to consider reasoning before retrieval. First determine what the question asks the system to connect; then decide which embeddings to retrieve and how to assemble their evidence. The petunia problem is difficult precisely because the desired answer must connect observations that originated in separate biological subdomains.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the operation, then choose its executor
Reasoning here means processing information in a logical way, not merely asking for chain of thought. That includes several distinct operations:
- Aggregation and arithmetic: combine observations or calculate over data.
- Logical and causal reasoning: deduce relationships, or extract entities and causal links from which further hypotheses can be proposed.
- Algorithmic reasoning: carry out a defined computational procedure.
- Probabilistic reasoning: update beliefs under uncertainty; Misztela describes attempts to make LLMs reason in a Bayesian fashion.
- Structured reasoning: use compositional relationships within a structure, which need not be causal.
- Geometric reasoning: infer transformations and relationships in visual arrangements, as in ARC.
The historical ARC Prize launch offered a $1 million-plus prize pool for open solutions to ARC-AGI. Its geometric tasks illustrate how far reasoning can extend beyond familiar text questions.
Expecting a single LLM to perform all these operations well is an unnecessary architectural constraint. A causal-inference library can handle a causal computation. An agent can generate Python and send it to a REPL for algorithmic work. The LLM may interpret the task and select a tool while the specialist tool executes the operation. That division applies to reasoning after retrieval just as much as to reasoning before it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build for one question—and a historical evidence boundary
Return to the biological problem: find the common cause of three phenomena. This requires causal reasoning and a retrieval strategy that can connect multiple observations. It also changes the economics of the system. The goal is not to answer many different questions over a fixed dataset. It is to answer one persistent scientific question, processing the dataset repeatedly to extract every relevant theme.
The answer is already part of modern scientific knowledge: RNA interference, whose discovery Misztela dates to 1998. A model trained on sources such as Wikipedia may know it before receiving any retrieved papers. Groundedness therefore matters as much as retrieval quality. A relevance classifier that examines the entire dataset can be attractive when exhaustive processing is affordable, even if it costs more than embedding search. The architecture depends on the required reasoning, the question–answer relationship, and the tradeoff between computation and grounding.
The proposed historical experiment follows a short sequence:
- State the scientific question and classify the reasoning and evidence relationships it requires.
- Restrict the retrieval corpus to papers published before the discovery.
- Reconstruct the pre-discovery setting and test whether the system can assemble a useful explanation from that evidence.
Misztela calls the restriction a knowledge cutoff. It limits the retrieved evidence, but it does not remove the answer from the model’s pretrained knowledge. This distinction becomes consequential in the experiments, where the first output does draw on later knowledge.
Working repeatedly on this question resembles training an agent on a particular scientific problem, in the sense of developing and testing its approach against that setting; no weight-training procedure is specified. The larger ambition is to reuse the same discovery scheme across multiple scientific problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover facts, connect them, then explain them
Success has several levels. The first is to recover what the corpus already knows: relevant facts, existing hypotheses, and relationships among them. This is broader than extracting a short answer to a narrow question. The useful output is a connected account of the evidence surrounding the problem.
Next come less obvious links, especially between the three biological subdomains. Ideally, the system would recover all relevant facts rather than only the most conspicuous ones. Beyond that lies new hypothesis generation: taking the observed relationships one step further to propose a connection that the corpus does not explicitly state. The highest ambition is to explain how the phenomenon happens, not merely identify related observations. Misztela’s joke about meeting in Sweden makes clear that this final level remains an aspiration.
The distinction between discovering a phenomenon and explaining its mechanism needs care. The human discovery identified an effective trigger before the complete biochemical machinery was understood. A system can therefore make meaningful progress by identifying what causes an effect without yet providing a complete account of how that effect is produced.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Select a score cluster instead of an arbitrary chunk count
The experiments start with a practical retrieval question: how many chunks should be returned? Rather than choosing a fixed top k, Misztela proposes Jenks natural breaks. Group the scalar query-to-chunk similarity or distance scores into classes, considering variation within and between groups, then retrieve the best-scoring class. These are groups of scores, not clusters fitted directly to the full embedding vectors.
The highest-similarity group—or lowest-distance group—provides an adaptive chunk count. Misztela’s intuition is that a coherent top group may contain useful evidence. Low score variance alone, however, does not establish scientific informativeness, and the number of score classes still has to be chosen.
The talk mentions a Python snippet. A compact implementation of the score-selection idea can use Jenkspy. Here, higher values mean greater similarity; the chunk IDs and scores are illustrative inputs:
python
import jenkspy
chunks = [
{"id": "paper-a:1", "similarity": 0.91},
{"id": "paper-b:2", "similarity": 0.89},
{"id": "paper-c:1", "similarity": 0.52},
{"id": "paper-d:3", "similarity": 0.49},
{"id": "paper-e:2", "similarity": 0.15},
{"id": "paper-f:1", "similarity": 0.12},
]
scores = [chunk["similarity"] for chunk in chunks]
breaks = jenkspy.jenks_breaks(scores, n_classes=3)
# The final class is open on the left and closed on the right.
selected = [
chunk for chunk in chunks
if breaks[-2] < chunk["similarity"] <= breaks[-1]
]
print([chunk["id"] for chunk in selected])
This selects the members of the strongest score class rather than specifying their count in advance. For a distance measure, selection must instead target the lowest-valued class.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask whether evidence advances a hypothesis
The initial RAG output used knowledge from after the discovery. Stronger instructions—strict prompting—reportedly improved grounding. The next experiment passed every dataset chunk through an LLM relevance classifier, asking whether it was relevant to the scientific problem. That basic classifier added little: its judgments were largely redundant with embedding distance.
The more useful change was to judge each paper in the context of hypothesis advancement. Instead of asking only whether a paper concerned the topic, the prompt asked the model to approach it more like a scientist evaluating evidence. Misztela reports that this produced the literature’s DNA-related hypotheses together with an RNA-related hypothesis, which he identifies as the correct direction.
| Experiment | Selection or grounding change | Reported observation |
|---|---|---|
| Initial RAG | Retrieve supporting chunks | Used post-discovery knowledge |
| Strict prompting | Strengthen evidence restrictions | Grounding improved |
| Basic relevance classifier | Judge every chunk for relevance | Mostly duplicated embedding distance |
| Hypothesis-aware assessment | Judge papers for hypothesis advancement | Surfaced an RNA-related hypothesis |
The important shift is the criterion for relevance. A passage can resemble the question without helping distinguish explanations; evidence that advances a hypothesis has a more specific role. Misztela reports movement toward the known result after reasoning over both the question and the dataset before retrieval. These are qualitative observations, without a numerical performance evaluation or a demonstration that pretrained answer leakage was eliminated.
Scientific discovery therefore asks more of the architecture than ordinary Q&A. Classifying the problem’s reasoning demands and question–evidence relationships helps determine where additional processing belongs; the generalized needle-in-a-haystack view is one proposed design aid, not a claim to the best architecture.
There is also a practical alternative to making embedding search ever more elaborate: when the workload permits it, test exhaustive LLM assessment of the corpus. For one persistent scientific question, spending computation on every chunk may be reasonable. Compare that approach with distance-based selection and check whether the LLM’s judgments actually improve the evidence available for discovery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Generates a hypothetical document from a query, embeds it, and retrieves similar real documents.
Uses entity graphs and community summaries to answer questions spanning an entire document collection.
An agent plans and explores a graph representation of long documents to gather evidence for answers.
The original announcement of a $1 million-plus prize pool for open solutions to ARC-AGI.
Further reading
Python implementation of Fisher-Jenks natural breaks, with examples for calculating numeric class boundaries and groups.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hello, everyone. Um, there's a lot of interesting talks, so, so thank you for being here.
- 0:18
Um, my name is Hubert. Uh, I work for pharma, uh, Novartis. On a daily basis, I work on, uh, designing of small molecules with generative AI, so designing, designing small graphs which would try to fit our d- uh, fix our diseases, cure.
- 0:34
Um, but today I would like to talk about, um, scientific reasoning, scientific discoveries, um, specifically how to use LLMs for that. And this is a joint work with my colleague Derek Lowe, who's a medicinal chemist.
- 0:47
You might know it-- know him if you are, uh, interested in drug design.
- 0:52
So today, I'm gonna talk about a few things, right? Probably I'm gonna leave you with more questions than responses and more problems than solutions, but I think it's worthwhile.
- 1:01
Um, so first I'm, I'm gonna show you an interesting paradox in biology which led to a Nobel Prize discovery. Then I'm gonna be talking about, uh, reasoning with RAGs.
- 1:12
And you've seen many different themes of RAGs and agentic, uh, RAGs during this conference, so I think it's, it's fitting really nicely to all of the, um, the other dov- topics.
- 1:23
Um, scientific discovery framework, which is how we can really build something, how we can build a, the system, or RAG specifically, which would help us in those scientific discoveries.
- 1:33
And in the end, I will show you a few, a few, um, uh, examples of the experiments. Okay. I think this is doesn't working. Yeah. So back in 1990s, early '90s, um, scientists were trying to improve, uh, color of the flowers, petunia flowers.
- 1:51
You know, this is what you do when you're a scientist. So they were trying to, um, overexpress the specific gene. So they wanted to get a color stronger. But what they get, they get the color flipped, which was really surprising.
- 2:05
Nobody really knew what was happening and why, why would th- was this happening, right? And this kind of, um, results they were getting, biologists were getting from different subdomains of biology.
- 2:14
I'm not gonna get into the details. I'm not a biologist. I'm a computer scientist, so I'm gonna save you all of that. But the interesting thing here is that there were three different, uh, set of results, set of experiments from different subdomains, even with different names.
- 2:30
Everybody knew it was related to some-something was messing with, with genes, but nobody knew why, right? So these three things, only after eight years, more or less, um, has been resolved, these three phenomena.
- 2:44
And this, this is what led to, to a Nobel Prize, uh, in biology, right? Uh, in medicine. And, um, so the question is:
- 2:55
Can we really use LLMs to speed this up, right? It took eight years. There might be something in the literature which could maybe give a hint of how we can go faster, right?
- 3:05
And the question is: What are the underlying causes, or what might be the underlying causes of these three phenomena, right? And now you're all thinking, "Okay, let's run the RAG.
- 3:14
Let's ask, ask the question. Probably it's gonna, it's gonna get it quite quickly," right?
- 3:19
Let's pause for a second and try to think how we can set this up and try to ex-experiment. But before it, um... Okay. We all know what, what RAG is, naive RAG is.
- 3:30
We have a question. We process the question. We process previously before, uh, the data. We represent both of them in a given latent space. We, we do some kind of retrieval over the embeddings, and you get the response.
- 3:43
And there is a lot of, a lot of different tricks. I, I wanted to show you just a b-bunch of them because these tricks, like height, for example, or, or ranking, um, or document indexing, are things relating to different pieces of RAG pipeline, and they are all for different purposes, slightly different purposes, right?
- 4:03
They might m- try to, try to fix the problem of matching between the question and answer. They might try to, uh, fix the problem of the, the right retrieval of the top documents or top chunks, and so on.
- 4:16
But the problem is, if LLMs, and this is really interesting paper, if LLMs, and we know that, cannot respond a little bit, little bit more, more catchy questions, right?
- 4:29
A little bit more, more convoluted questions, how we can expect that LLMs would s- understand the question and then on top of that, uh, give us the right, uh, right response with the RAG, right?
- 4:41
So we are, we are trying to run before we crawl.
- 4:46
So I was thinking about it and, and, um, when you look at the literature, the papers, and at the solutions during this conference, you're gonna realize there is, there are themes kind of repeatedly referring to something which is happening before the retrieval.
- 5:04
And one example of, of reasoning before the retrieval
- 5:10
is, uh, is, is routing. Routing, which we have in agentic flows, is really the idea of doing something with the question because we cannot throw it at, at, at, at the embeddings and just get the response, right?
- 5:22
So you... I-I'm re- I'm, I'm, I'm naming it r-reasoning for, for a purpose, and you're gonna, you're gonna understand it in, in a few slides, right? So you need to understand a little bit more, we need to understand a little bit more what's the question about, right?
- 5:37
And on the other hand, not only with the question because we also try to, um, do something with the data, right? So GraphRAG or a graph reader, very interesting papers.
- 5:52
If you look at them, you're gonna, you're gonna see that they are trying to represent the knowledge from the, from these documents in a little bit different way, specifically about the, uh, with graphs.
- 6:01
You might use different representations in many different ways, but the point is
- 6:06
We are doing something before the retrieval, which really is a reasoning to deal with the more semantically complex question.
- 6:15
The reasoning after the retrieval usually hap- usually is done by, by, uh, delegated to LLM, right? Because this is all about LLMs, but it, that doesn't have to be the case.
- 6:27
Um, so the point here really is that, uh, between more convoluted the semantics of the question, um,
- 6:37
the mo- the harder it's gonna be for your standard RAG without those reasoning steps. So you probably wanna, want, you wanna try to add these reasoning steps before it, and I'm gonna show you some experiments with that.
- 6:47
Um, and small by the way is that, um, as I was saying, I work on a little bit different, um, flavor of, of, of AI, right? Which is generative, but for graphs.
- 7:00
So kind of old style, encoder-decoder, variational autoencoder, you have a latent space, and you navigate that.
- 7:06
So when I saw this thing which I draw, which is reasoning, embedding,
- 7:12
uh, retrieval, and then again reasoning, I realized that this is very similar to what we do in... It, it's very well known, uh, pipeline in chemistry, where you take a dataset, which in that case are molecules.
- 7:27
You represent them in the latent space, which are embeddings effectively, and then you navigate the embeddings to optimize for that. So for gi- for a given purpose. So you have the latent space, and you wanna extract the, the points which are the most interesting for your purpose, right?
- 7:42
So in our case, we are trying to extract the embeddings which are the most similar to the question, right? So we do simple similarity. But in other domains, like chemistry, we run optimization over that latent space, and this is type of a, type o- type of reasoning really, right?
- 7:57
So I was like, "Aha, okay." So kind of common theme is appearing here.
- 8:05
Now the question is, do you really know, do you really need it? How do I know if my question is complex or not? So, um, we all know needle-in-a-haystack, right?
- 8:17
Um, test or experiment or benchmark. So I'm not, I'm, I'm not gonna be focusing on that too much, so let me go quite quickly. We know, like, find the perfect ingredient, uh, uh, the, the ingredient to build the perfect pizza, right?
- 8:33
And a little bit extended version of that is multi, uh, multi-needle-in-haystack, uh, discussed yesterday on one of the tracks as well, right? So
- 8:44
when you think about it, this is not only about the response which you're gonna get, but this is also about the question, and really relationship between the question and the response.
- 8:54
And as I was saying, between more convoluted semantically the question is, the harder it's gonna be. So how we can generalize it? And I was thinking about databases, right?
- 9:03
I'm an engineer, so one needle, one response, or one, one, one, one, uh, concept, one piece of information is like one-to-one relationship, like in databases. Then you have one to n, and then appeared to my head that, okay, if I have a few concepts put together in one question, this is the multi,
- 9:27
uh, th- this is a multi, multi thing versus finding one, one needle. And then you can extend it still to a few concepts hidden in a few chunks of the document.
- 9:35
So really, you, you need to do a little bit more. You already see that, okay, you need to pass somehow the, the, the complexity of the question so that you can get the right embeddings extracted.
- 9:48
And this is what, what kind of, um, triggers your reasoning. If you have a few concepts in the question, you probably need the reasoning before the retrieval so that you know what to retrieve and how to do it. [clears throat]
- 10:02
Sorry. So I was, uh, I was mentioning reasoning, right? So what, what reasoning really is, and there are many, uh, views on that because, for example, we use chain of thought as a, as a type of reasoning, right?
- 10:15
But I'm thinking about reasoning as processing information in some logical way, right? So we all know how we can do the, uh, do the aggregation, how we can do simple arithmetic over, over the data.
- 10:27
You, you... We al- we also know what's logical reasoning, right? Um, so there's also causal reasoning. If you extract specific entities from the data and causal relationships between them, you might ask LLM, "Okay, what, what more you can hypothesize about this data, or more, what more y- maybe you can deduce?"
- 10:45
Right? And there are papers about it, and this, this kind of starts working. Then you have algorithmic reasoning, which I mentioned before. Um, you have also probabilistic reasoning, so we ex- we were trying to, to, to make LLMs reason in a probabilistic fashion, right, like a Bayesian inference.
- 11:02
There's also structured way of reasoning. This is a little bit different from causal because you might have a structure, and you can expect some kind of compositionality over, over that, that structure, right?
- 11:12
So, um, and there's also, of course, uh, ARC
- 11:17
Kaggle competition, uh, recently released with one million dollar prize for closer getting to, to AGI, right? This is reasoning over, over, over geometry. This is... Th- that's why it's so challenging because this is not typical type of reasoning, right?
- 11:32
So why am I saying this? Because usually what we do, we try to ex- we, we expect LLMs to perform all of these reasoning well, all of these reasonings well.
- 11:46
And do we really need LLM for that? Probably yes or maybe not, but you can also delegate these, these reasoning types to specific tools which are specialist in that.
- 11:57
So you can do causal inference with, with libraries for, for causal inference. You can do, uh, algorithmic reasoning with specific Python, right? That's why we have a REPL as a, as a tool, right, attached to, to your, to your agent so that you can generate the code, and this is, this is kind of algorithmic reasoning, right?
- 12:16
So let's, let's try to, um, come back to the problem and think how we, how we can solve it really, right? So we wanna find the cause of these three phenomenas, and we define the type of reasoning, the type of re-retrieval, which is the relation between question and, and, and an answer.
- 12:33
And the particularities in our case are also that
- 12:36
we are not pre... We don't wanna build a RAG to respond to many different questions over a specific data set, but we want to respond to only one question, right?
- 12:46
This... and this is the question. We want to respond this really one question, and we wanna process our data set as many times as we possibly can so that we extract all of the relevant themes in that.
- 12:57
So but the trick here is that LLMs during training have seen Wiki-Wikipedia, so they know about this problem in RNA interference because it happened in 1998. So we need the groundedness, right?
- 13:10
And this is another aspect of it. Um, so when you think about different methods and where you are in your specific use case in the computational efficiency versus ground-groundedness, uh, you'll...
- 13:22
you immediately see, okay, which approaches might be more suitable for, for my case. And as, as I'm gonna show you later on, um, uh, relevance classifier is one of the main things which we use because we can process all of, all of the data set.
- 13:36
Um, so overall, um, designing choices, uh, you can see here, and th-this depends usually upon your specific question you have, the reasoning you're, you, you, you need, the relationship between question and answering, and another aspect like, like groundedness versus efficiency.
- 13:56
Um, so when we have, when we have defined what type of question it is, so how we really wanna test whether our solution is capable of doing this kind of discovery, right?
- 14:08
So you state the question, you define the, the type of question, but then you need to do the knowledge cutoff, which is basically you need to remove from the LLM the knowledge about the discovery.
- 14:17
So what we do, we use RAG, and in that RAG we have only the, the, the, the scientific papers from before the discovery, right? So that it doesn't cheat.
- 14:27
So we, we present it in, in the state of the situation which was before the discovery, so that it can... we can simulate that situation, right? And this is kind of a training.
- 14:38
You can think about it as a agent training in that specific... in, on that specific scientific problem, on that specific data set, and the over-overarching goal is to, uh, have a system which would be able to make those discoveries on many different scientific problems and applying this, the same, the same scheme.
- 14:59
Um, now the question is, how, how do we define the, the, the success of, of, of, uh, this experiment? So the first level of success would be, okay, find some hypothesis or what we, what we really know from this data set, data set, right?
- 15:14
So it's not that we are asking specific question. We want RAG to extract, okay, what do you know about this specific... a-anything related really to that problem.
- 15:24
And, uh, don't read the text. Forgive me the amount of text on, on these slides. Um, but look at the graph, right? So we want RAG to find those relationships between different facts in our data set.
- 15:37
Then the next level would be to find, like, less obvious links because you remember there were three different subdomains of biology, right? So the, the, the, those, those linkages are not obvious.
- 15:48
So we wanted to get little bit, little bit further. Ideally, if it's exhaustive, so finding all of the facts from the data set, right? And the next level would be doing new hy- making new hypothesis, right?
- 16:02
Which would be, okay, if I see this, I can go maybe one step further and hypothe-hypothesize about a, a new, new relationship. And the i-ideal situation or, like, the, the, the high level of success, I'm not expecting to, to have this anytime soon, otherwise we're gonna meet in Sweden.
- 16:19
But, uh, this is really about finding not only what can be, uh, what, what is related between, between the, those different facts in the literature, but also explaining how that happens.
- 16:29
And the interesting thing is that when the discovery happened made by humans, we didn't know how it was happening. We know what was happening, but we didn't know how.
- 16:39
Okay, so the last part, I'm gonna go quite quickly because I have just, uh, less than two minutes. So the naive RAG we all know. Uh, I'm gonna show a very, very, very nice trick, which is not very prevalent actually, because when you have, uh, the, the, the, the, the, the distances, similarities between the question embedding and
- 16:58
the, the embedding in your database, um, you might ask yourself, of course, how many embeddings are... how many chunks are, are enough, right, really? But you can do it...
- 17:06
think about it mathematically, because if you, if you see those distances, you can calculate the variances within the different clusters of those, uh, embeddings and between them. So this is called, uh, Jenkins natural breaks.
- 17:21
Very simple thing to do, and you can extract the top, top cluster, which would be kind of representing, uh, the lowest variance between, between the chunks extracted. So they are kind of, uh, telling you the, the probably, the, the...
- 17:34
giving you the most information, right? Um, uh, there is a snippet how you can do it. Um, I'm gonna go quite quickly. In this case, w- we got, uh, we didn't get the ground as well because it, it was, uh, using the knowledge from, from after the discovery, so we need to, uh, make the, the prompt stronger,
- 17:53
right? So we call it strict prompting, and then it, and it, and it got m- uh, better, right? And then what we do, we did the relevance classifier, which is really passing all of the chunks in, in our data set through the LLM and asking whether this is relevant to our problem.
- 18:08
Unfortunately, that wasn't very much, uh, informative because it was kind of redundant to distance of the, from the embeddings, right? There was a discussion yesterday about it. But when we, when we, uh, did the, um, the analysis of how relevant, um, each paper is in the context of, uh, hypothesis advancing, so we, we built a more sophisticated
- 18:30
prompt and thinking about, little bit more about the scientist. This is when we got, uh, the hypothesis a little bit further, right? So it, it found us all of the hypothesis in the literature related to DNA, but one to the RNA, and this is the right one, right?
- 18:44
So we see already that it's going in the right direction. I'm gonna skip it in the interest of time. You're gonna have access to it. So my point here is when we started doing this reasoning over the question and the database before the retrieval, we started getting, uh, closer to the results, which are ground true results without
- 19:00
cheating. Um, so the conclusion is scientific discovery requires, uh, solving harder, uh, than simple Q&A problems. Knowing your problem can help define more efficient RAG architecture. Needle-in-a-haystack might be generalized.
- 19:13
I'm not saying that this is, like, the best way of doing it, but this is already you see something new and interesting. And harder problems might need, uh, reasoning, right?
- 19:21
And of course, you cannot forget about the brut-brute force because if you have the use case where you can do it,
- 19:29
check your, check your LLM. Maybe it's gonna, it's gonna do better than the distance embeddings.
- 19:35
Um, so thank you very much. Uh, forgive me rushing. [audience applauding]
- 19:40
I hope this was helpful. [upbeat music]