# LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries

Hubert Misztela, Novartis

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/gbM7k3NauNI-llm-scientific-reasoning-make-ai-capable-nobel) · [Video](https://www.youtube.com/watch?v=gbM7k3NauNI)

## LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries

A petunia experiment that produced the opposite of its intended result exposes a harder task for RAG: connecting scattered observations into a grounded scientific hypothesis.

Basic familiarity with embeddings and retrieval-augmented generation is helpful; no biology background is required.

### Why did adding a gene remove the color?

[0:18](https://www.youtube.com/watch?v=gbM7k3NauNI&t=18s)

Could an LLM connect unexpected biological results before scientists recognize their common cause? Hubert Misztela approaches that question from generative molecular design at Novartis, where he works with small molecular graphs. His scientific-discovery project with medicinal chemist Derek Lowe starts with a particularly useful puzzle: researchers trying to intensify petunia flower color obtained the opposite result.

In the early 1990s, scientists overexpressed a gene expecting stronger pigmentation. Instead, the flowers lost purple pigment. Similar surprises appeared in other biological subdomains, but the phenomena had different names. Researchers could see that something was interfering with gene activity without knowing the shared explanation.

![Unintuitive Phenomena slide with purple, variegated, and mostly white flowers and text describing the unexpected loss of pigment.](https://ai.engineer/api/talk-essay-media/9eecbead24d1eaacf80858d87090a4e0d3f28fefc3735bcf4c9fc2c7aca28674)

Gene overexpression unexpectedly produced flowers with less purple pigment.

In Misztela’s account, connecting and resolving the three phenomena took roughly eight years and led to a Nobel Prize in medicine. That delay creates a concrete discovery task: **find a plausible common cause from the literature available before the answer was known**. Perhaps the necessary clues were already present, scattered across disciplines. Asking a modern model for the answer is easy; determining whether it can assemble those clues is harder.

### Retrieval inherits the difficulty of the question

[3:19](https://www.youtube.com/watch?v=gbM7k3NauNI&t=199s)

Ordinary retrieval-augmented generation starts by processing documents and representing them as embeddings. At query time, it embeds the question in the same latent space, retrieves nearby chunks, and supplies them to a response generator. This works naturally when the question points toward text that already contains an answer.

Different RAG improvements repair different parts of that pipeline. [HyDE](https://arxiv.org/abs/2212.10496), ranking, and document indexing are not interchangeable: improving the match between a question and answer-bearing text is a different problem from selecting the best documents or chunks. But those improvements still depend on understanding what evidence the question requires. A convoluted question can defeat that understanding before retrieval begins. Misztela describes this as “trying to run before we crawl.”

![Question and data processing paths converge on retrieval and response, with augmentation references and a paper title at upper right.](https://ai.engineer/api/talk-essay-media/2307ab5016a0893d6aa6d4cc47b43012c21c7db7ac18582f045f38bbc3bb35ac)

The naive RAG pipeline alongside “Easy Problems That LLMs Get Wrong.”

### Reason over the question and the representation

[4:46](https://www.youtube.com/watch?v=gbM7k3NauNI&t=286s)

Reasoning can happen before retrieval on either side of the search. **Routing** interprets a question before deciding where to send it; the query cannot simply be thrown at an embedding index and expected to produce the right evidence. On the document side, [GraphRAG](https://arxiv.org/abs/2404.16130) and [GraphReader](https://arxiv.org/abs/2406.14550) represent knowledge using graphs. The shared architectural move is to process meaning before selecting material, whether by understanding the request or by changing how document knowledge is represented.

After retrieval, an LLM usually performs the final reasoning, but that is a design choice rather than a requirement. The more semantically complex the question, the more work may be needed before retrieval as well. A useful pipeline therefore becomes **reasoning → embedding → retrieval → reasoning**, with each reasoning stage chosen for a specific job.

This resembles the molecular-design workflow Misztela already uses. An encoder–decoder model, such as a variational autoencoder, represents molecular graphs in a latent space. The system then navigates that space to find points useful for a desired objective. RAG also selects points from an embedding space, but usually through similarity to a question.

| Workflow | Representation | Selection objective |
| --- | --- | --- |
| Standard RAG | Document embeddings | Similarity to the question |
| Molecular design | Molecular latent space | Optimize a desired property |

The analogy broadens what counts as reasoning: goal-directed optimization over a latent space can perform part of the job. The system need not express every operation as an LLM-generated chain of thought.

### How many concepts must the evidence connect?

[8:05](https://www.youtube.com/watch?v=gbM7k3NauNI&t=485s)

A needle-in-a-haystack test asks a system to locate a particular piece of information inside a large context. Misztela uses the example of finding the ingredient for a perfect pizza. A multi-needle test extends the task to several pieces of information. But counting answer fragments captures only part of the difficulty: the relationship between the question and those fragments matters too.

The database analogy makes that relationship explicit:

| Question concepts | Evidence locations | Relationship |
| --- | --- | --- |
| One | One | One-to-one |
| One | Several | One-to-many |
| Several | One | Many-to-one |
| Several | Several | Many-to-many |

A question can combine several concepts that must jointly identify one relevant passage, or it can require concepts scattered across multiple chunks. In either case, searching for the whole question as one undifferentiated vector may hide the structure that determines what to retrieve.

**Multiple concepts are a signal to consider reasoning before retrieval.** First determine what the question asks the system to connect; then decide which embeddings to retrieve and how to assemble their evidence. The petunia problem is difficult precisely because the desired answer must connect observations that originated in separate biological subdomains.

### Choose the operation, then choose its executor

[10:02](https://www.youtube.com/watch?v=gbM7k3NauNI&t=602s)

Reasoning here means processing information in a logical way, not merely asking for chain of thought. That includes several distinct operations:

- **Aggregation and arithmetic:** combine observations or calculate over data.
- **Logical and causal reasoning:** deduce relationships, or extract entities and causal links from which further hypotheses can be proposed.
- **Algorithmic reasoning:** carry out a defined computational procedure.
- **Probabilistic reasoning:** update beliefs under uncertainty; Misztela describes attempts to make LLMs reason in a Bayesian fashion.
- **Structured reasoning:** use compositional relationships within a structure, which need not be causal.
- **Geometric reasoning:** infer transformations and relationships in visual arrangements, as in ARC.

The historical [ARC Prize launch](https://arcprize.org/blog/launch) offered a $1 million-plus prize pool for open solutions to ARC-AGI. Its geometric tasks illustrate how far reasoning can extend beyond familiar text questions.

![Nine teal hexagons labeled Simple Aggregation, Algebraic, Logical, Causal, Algorithmic, Probabilistic, Structured, Geometric, and Others, beside an arcprize.org link.](https://ai.engineer/api/talk-essay-media/1321f9d9d8192a1331f4f7e09e44560aa86faff1d3ff4a3b77d970e01ac9a44d)

Reasoning types extend from aggregation and algebra to causal, probabilistic, and geometric reasoning.

Expecting a single LLM to perform all these operations well is an unnecessary architectural constraint. A causal-inference library can handle a causal computation. An agent can generate Python and send it to a REPL for algorithmic work. The LLM may interpret the task and select a tool while the specialist tool executes the operation. That division applies to reasoning after retrieval just as much as to reasoning before it.

### Build for one question—and a historical evidence boundary

[12:16](https://www.youtube.com/watch?v=gbM7k3NauNI&t=736s)

Return to the biological problem: find the common cause of three phenomena. This requires causal reasoning and a retrieval strategy that can connect multiple observations. It also changes the economics of the system. The goal is not to answer many different questions over a fixed dataset. It is to answer **one persistent scientific question**, processing the dataset repeatedly to extract every relevant theme.

The answer is already part of modern scientific knowledge: RNA interference, whose discovery Misztela dates to 1998. A model trained on sources such as Wikipedia may know it before receiving any retrieved papers. Groundedness therefore matters as much as retrieval quality. A relevance classifier that examines the entire dataset can be attractive when exhaustive processing is affordable, even if it costs more than embedding search. The architecture depends on the required reasoning, the question–answer relationship, and the tradeoff between computation and grounding.

The proposed historical experiment follows a short sequence:

1. **State the scientific question** and classify the reasoning and evidence relationships it requires.
2. **Restrict the retrieval corpus** to papers published before the discovery.
3. **Reconstruct the pre-discovery setting** and test whether the system can assemble a useful explanation from that evidence.

Misztela calls the restriction a knowledge cutoff. It limits the retrieved evidence, but it does not remove the answer from the model’s pretrained knowledge. This distinction becomes consequential in the experiments, where the first output does draw on later knowledge.

Working repeatedly on this question resembles training an agent on a particular scientific problem, in the sense of developing and testing its approach against that setting; no weight-training procedure is specified. The larger ambition is to reuse the same discovery scheme across multiple scientific problems.

### Recover facts, connect them, then explain them

[14:59](https://www.youtube.com/watch?v=gbM7k3NauNI&t=899s)

Success has several levels. The first is to recover what the corpus already knows: relevant facts, existing hypotheses, and relationships among them. This is broader than extracting a short answer to a narrow question. The useful output is a connected account of the evidence surrounding the problem.

Next come less obvious links, especially between the three biological subdomains. Ideally, the system would recover all relevant facts rather than only the most conspicuous ones. Beyond that lies **new hypothesis generation**: taking the observed relationships one step further to propose a connection that the corpus does not explicitly state. The highest ambition is to explain how the phenomenon happens, not merely identify related observations. Misztela’s joke about meeting in Sweden makes clear that this final level remains an aspiration.

![Levels of Success slide showing Level 3 Advanced Hypothesis Generation and Level 4 Predictive Hypothesis Generation, with text and node-link diagrams.](https://ai.engineer/api/talk-essay-media/62da051c6cec04d90d1a3d4a3b8ac69aa3e130d05a5b5c160fa70b7d7c7e9add)

Advanced and predictive hypothesis generation define the higher levels of discovery success.

The distinction between discovering a phenomenon and explaining its mechanism needs care. The human discovery identified an effective trigger before the complete biochemical machinery was understood. A system can therefore make meaningful progress by identifying what causes an effect without yet providing a complete account of how that effect is produced.

### Select a score cluster instead of an arbitrary chunk count

[16:39](https://www.youtube.com/watch?v=gbM7k3NauNI&t=999s)

The experiments start with a practical retrieval question: how many chunks should be returned? Rather than choosing a fixed top `k`, Misztela proposes **Jenks natural breaks**. Group the scalar query-to-chunk similarity or distance scores into classes, considering variation within and between groups, then retrieve the best-scoring class. These are groups of scores, not clusters fitted directly to the full embedding vectors.

The highest-similarity group—or lowest-distance group—provides an adaptive chunk count. Misztela’s intuition is that a coherent top group may contain useful evidence. Low score variance alone, however, does not establish scientific informativeness, and the number of score classes still has to be chosen.

The talk mentions a Python snippet. A compact implementation of the score-selection idea can use [Jenkspy](https://github.com/mthh/jenkspy). Here, higher values mean greater similarity; the chunk IDs and scores are illustrative inputs:

```python
import jenkspy

chunks = [
    {"id": "paper-a:1", "similarity": 0.91},
    {"id": "paper-b:2", "similarity": 0.89},
    {"id": "paper-c:1", "similarity": 0.52},
    {"id": "paper-d:3", "similarity": 0.49},
    {"id": "paper-e:2", "similarity": 0.15},
    {"id": "paper-f:1", "similarity": 0.12},
]

scores = [chunk["similarity"] for chunk in chunks]
breaks = jenkspy.jenks_breaks(scores, n_classes=3)

# The final class is open on the left and closed on the right.
selected = [
    chunk for chunk in chunks
    if breaks[-2] < chunk["similarity"] <= breaks[-1]
]

print([chunk["id"] for chunk in selected])
```

This selects the members of the strongest score class rather than specifying their count in advance. For a distance measure, selection must instead target the lowest-valued class.

### Ask whether evidence advances a hypothesis

[17:34](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1054s)

The initial RAG output used knowledge from after the discovery. Stronger instructions—strict prompting—reportedly improved grounding. The next experiment passed every dataset chunk through an LLM relevance classifier, asking whether it was relevant to the scientific problem. That basic classifier added little: its judgments were largely redundant with embedding distance.

The more useful change was to judge each paper in the context of **hypothesis advancement**. Instead of asking only whether a paper concerned the topic, the prompt asked the model to approach it more like a scientist evaluating evidence. Misztela reports that this produced the literature’s DNA-related hypotheses together with an RNA-related hypothesis, which he identifies as the correct direction.

| Experiment | Selection or grounding change | Reported observation |
| --- | --- | --- |
| Initial RAG | Retrieve supporting chunks | Used post-discovery knowledge |
| Strict prompting | Strengthen evidence restrictions | Grounding improved |
| Basic relevance classifier | Judge every chunk for relevance | Mostly duplicated embedding distance |
| Hypothesis-aware assessment | Judge papers for hypothesis advancement | Surfaced an RNA-related hypothesis |

The important shift is the criterion for relevance. A passage can resemble the question without helping distinguish explanations; evidence that advances a hypothesis has a more specific role. Misztela reports movement toward the known result after reasoning over both the question and the dataset before retrieval. These are qualitative observations, without a numerical performance evaluation or a demonstration that pretrained answer leakage was eliminated.

Scientific discovery therefore asks more of the architecture than ordinary Q&A. Classifying the problem’s reasoning demands and question–evidence relationships helps determine where additional processing belongs; the generalized needle-in-a-haystack view is one proposed design aid, not a claim to the best architecture.

There is also a practical alternative to making embedding search ever more elaborate: **when the workload permits it, test exhaustive LLM assessment of the corpus**. For one persistent scientific question, spending computation on every chunk may be reasonable. Compare that approach with distance-based selection and check whether the LLM’s judgments actually improve the evidence available for discovery.

## Resources

- [HyDE: Precise Zero-Shot Dense Retrieval without Relevance Labels](https://arxiv.org/abs/2212.10496) — Generates a hypothetical document from a query, embeds it, and retrieves similar real documents.
- [From Local to Global: A Graph RAG Approach](https://arxiv.org/abs/2404.16130) — Uses entity graphs and community summaries to answer questions spanning an entire document collection.
- [GraphReader](https://arxiv.org/abs/2406.14550) — An agent plans and explores a graph representation of long documents to gather evidence for answers.
- [ARC Prize's 2024 Launch](https://arcprize.org/blog/launch) — The original announcement of a $1 million-plus prize pool for open solutions to ARC-AGI.
- [Jenkspy](https://github.com/mthh/jenkspy) — Python implementation of Fisher-Jenks natural breaks, with examples for calculating numeric class boundaries and groups.

## Slides

### Presenting Sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=gbM7k3NauNI&t=5s) [Microsoft presenting sponsor card.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f227e03d5413a3f145480f3998b2dc3d19988d231e026e1f23d34ccc7102bb5.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and Platinum Sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=gbM7k3NauNI&t=10s) [AWS innovation sponsor and MongoDB, Google Cloud, and Neo4j platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/97ca19d6f33fddbda91019b40e3ae319a4bdca0457055c7967ee5848a232dc8c.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Content

The talk moves from a Nobel Prize–linked biology paradox to RAG with reasoning, a scientific rediscovery framework, and an example using pre-retrieval reasoning.

[1:03](https://www.youtube.com/watch?v=gbM7k3NauNI&t=63s) [Agenda covering a biology paradox, RAG with reasoning, a scientific rediscovery framework, and a pre-retrieval reasoning example.](https://assets.ai.engineer/sites/aiecode2025/sha256/321d7d520ce6943b5ca88f05797e8abadaa8bbb79e08b5ecb4755b229f2db812.webp)

Content

• Paradox in biology which led to Nobel Prize
• RAG with Reasoning
• Scientific re-discovery framework
• Example of RAG with simple pre-retrieval reasoning

Hubert Misztela - LLM Scientific Reasoning
1

AIE
Microsoft
smol ai

### Unintuitive Phenomena

Scientists tried to deepen petunia color by overexpressing a gene. Instead, some flowers showed less purple pigment, including variegated patterns.

[1:45](https://www.youtube.com/watch?v=gbM7k3NauNI&t=105s) [Three petunia flowers illustrate unexpectedly reduced purple pigment after gene overexpression.](https://assets.ai.engineer/sites/aiecode2025/sha256/d87db6725f0b0825050e1e4dfe497bb089585ede95a8d6e8b478ee1501750941.webp)

Unintuitive Phenomena

Back in 1990...

The overexpressed gene was expected to result in darker flowers, but instead caused some flowers to have less visible purple pigment, sometimes in variegated patterns.

Hubert Misztela - LLM Scientific Reasoning
2

AIE

### Discovery of RNAi

Co-suppression in plants, gene silencing in animals, and quelling in fungi appeared as separate, unexplained phenomena. Their connections were unclear, and foundational knowledge was incomplete.

[2:10](https://www.youtube.com/watch?v=gbM7k3NauNI&t=130s) [Slide describes three initially unexplained phenomena across biological subdomains, illustrated by plants, animals and fungi.](https://assets.ai.engineer/sites/aiecode2025/sha256/35b3f1fb72dab1c063151490f1c855702195f68f05e13a540eec914e32df9423.webp)

Discovery of RNAi

• Three separate, unexplained phenomena
• Loosely related, if at all
• Happening in different subdomains
• Foundational knowledge incomplete

Sparse, Unrelated Facts...
Co-Suppression
Gene Silencing
Quelling
plants
animals
fungi
RNA Biology

Hubert Misztela - LLM Scientific Reasoning
3

Interface text:
AIE
Microsoft
smol ai

### Discovery of RNAi

The slide connects co-suppression, gene silencing and quelling to RNAi and asks whether LLMs could find their underlying cause faster than the eight years from the first recorded experiment to discovery.

[3:06](https://www.youtube.com/watch?v=gbM7k3NauNI&t=186s) [Co-suppression, gene silencing and quelling converge on RNAi in 1998; the slide asks whether an LLM could rediscover their underlying cause.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b3d7570bfef17bd5c3c191b126d09879949aca45eaa88a0e22c768efb9429f2.webp)

Discovery of RNAi

• From the first recorded experiment in 1990 to the discovery in 1998 it took 8 years, could we get there faster with LLMs?

Key Question:
Can we rediscover it with LLM?
Or at least could LLM point towards the right direction?

RNAi 1998
Co-Suppression
Gene Silencing
Quelling
RNA Biology

“What could be the underlying cause of C, S, Q?”

Hubert Misztela - LLM Scientific Reasoning
4

Interface text:
AIE
aws

### Generalize the problem

Before setting up the RAG experiment, Misztela pauses to generalize the problem. His prompt also offers a $200 tip.

[3:17](https://www.youtube.com/watch?v=gbM7k3NauNI&t=197s) [A prompt asks the audience to take a deep breath and generalize the problem, alongside overlapping colorful shapes.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9b3e9010f57d389307751839b923e23120e50c95a8a35fd35342d514824b021.webp)

Before trying to solve it, take a deep breath and think how to generalize it.

I’ll tip you $200.

Hubert Misztela - LLM Scientific Reasoning
6

Interface text:
AIE
Microsoft
smol ai

### Naive RAG

The question and prepared data are embedded separately, then feed retrieval to produce a response. HyDE, document indexing and reranking target different parts of the pipeline.

[3:39](https://www.youtube.com/watch?v=gbM7k3NauNI&t=219s) [Question and data processing pipelines feed retrieval and response, with examples of augmentation, indexing and reranking.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c7ed915cd1f770ab1dd58830cff6ec3a23eadac15a097e5f58cf05e7b4cd9f8.webp)

Naive RAG

Question processing:
Question → Augmentation → Embedding → Retrieval → Response
HyDE (2212.10496)
Multi-query by LangChain
MMR (by Carbonell et al.)
Cohere ReRank by Cohere
RankGPT (2311.01555)

Data processing:
Chunking → Augmentation → Embeddings → Retrieval
Document Summary Index by LlamaIndex

Hubert Misztela - LLM Scientific Reasoning
7

AIE
Microsoft
smol ai

### RAG with Reasoning

Reasoning can shape both the question and the data before retrieval. The diagram places another reasoning step after retrieval; routing and graph-based representations are examples of the earlier work.

[4:46](https://www.youtube.com/watch?v=gbM7k3NauNI&t=286s) [Question and data pipelines include reasoning before retrieval, followed by reasoning before the response.](https://assets.ai.engineer/sites/aiecode2025/sha256/71fc22c8494b410d4fbd0c2bc556679201851244e979a7fe4560c52bbb8d8b8d.webp)

RAG with Reasoning

Question processing:
Routing in agentic flows
Question → Augmentation → Reasoning → Embedding

Data processing:
Chunking → Augmentation → Reasoning → Embeddings

Both pipelines feed: Retrieval → Reasoning → Response

GraphRAG (2404.16130)
GraphReader (2406.14550v1)

Hubert Misztela - LLM Scientific Reasoning
8

AIE
Microsoft
smol ai

[4:57](https://www.youtube.com/watch?v=gbM7k3NauNI&t=297s) [Question and data processing pipelines feed retrieval, reasoning, and a response.](https://assets.ai.engineer/sites/aiecode2025/sha256/b210ef360e8ad7206852d37175b27d4b9fdc3a8bb69d5dab87e67ede7f453171.webp)

RAG with Reasoning

Question processing:
Routing in agentic flows
Question → Augmentation → Reasoning → Embedding

Data processing:
Chunking → Augmentation → Reasoning → Embeddings

Both pipelines feed: Retrieval → Reasoning → Response

GraphRAG (2404.16130)
GraphReader (2406.14550v1)

Hubert Misztela - LLM Scientific Reasoning
8

AIE

### RAG with Reasoning

Complex questions may need reasoning before retrieval. The diagram adds it to both question and data processing, then asks how to tell when a question is hard enough to need it.

[6:38](https://www.youtube.com/watch?v=gbM7k3NauNI&t=398s) [The RAG diagram adds that complex questions may require reasoning before retrieval and notes a connection to generative models over latent spaces.](https://assets.ai.engineer/sites/aiecode2025/sha256/3041eb12e4ecbb4dadc103aa8d3de878298c66bd57c6a6b3fc77a8bc20e9a005.webp)

RAG with Reasoning

Question processing:
Routing in agentic flows
Question → Augmentation → Reasoning → Embedding

The point: questions with complex semantics might require pre-retrieval reasoning

Data processing:
Chunking → Augmentation → Reasoning → Embeddings

Retrieval → Reasoning → Response

GraphRAG (2404.16130)
GraphReader (2406.14550v1)

BTW: This is a common theme in generative models over latent spaces

Hubert Misztela - LLM Scientific Reasoning
9

Interface text:
AIE
Microsoft
smol ai

[8:07](https://www.youtube.com/watch?v=gbM7k3NauNI&t=487s) [Diagram showing reasoning in question processing, data processing, and after retrieval.](https://assets.ai.engineer/sites/aiecode2025/sha256/afcae1cb06a311c92741ccbabeeff87763867b0696d1b9a4b76d67ebc690e30f.webp)

RAG with Reasoning

Question processing:
Routing in agentic flows
Question → Augmentation → Reasoning → Embedding

The point: questions with complex semantics might require pre-retrieval reasoning

Data processing:
Chunking → Augmentation → Reasoning → Embeddings

Retrieval → Reasoning → Response

GraphRAG (2404.16130)
GraphReader (2406.14550v1)

BTW: This is a common the[illegible]
models over latent [illegible]

But how do I even know if I need any of that, i.e. if my question is hard?

Hubert Misztela - LLM Scientific Reasoning
9

Interface text:
AIE
Microsoft
smol ai

### Generalized Needle in a Haystack (Question-Response Relation Type)

Misztela extends needle-in-a-haystack into four relationships: questions can involve one or many concepts and seek one or many pieces of information. His examples move from pizza alone to pizza, pierogies, and sushi.

[8:30](https://www.youtube.com/watch?v=gbM7k3NauNI&t=510s) [Slide compares one-to-one and one-to-many question-response relations using pizza ingredient examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/25f2ee0bc6d4e082d97de0503feea06123848630d9837c6e92c22288335b4d19.webp)

Generalized Needle in a Haystack
(Question-Response Relation Type)

• 1:1 Relation – one concept : one piece of information (NiH)
“What is the secret ingredient needed to build the perfect pizza?”
• 1:n Relation – one concept : many pieces of information (Multi NiH)
“What are the secret ingredients needed to build the perfect pizza?”

https://x.com/GregKamradt/status/1729573848893579488
Multi Needle in a Haystack (langchain.dev)
Hubert Misztela - LLM Scientific Reasoning
10

AIE
Microsoft
smol ai

[9:31](https://www.youtube.com/watch?v=gbM7k3NauNI&t=571s) [Four question-response relationship types are illustrated with food examples; the last example is partly obscured by the presenter.](https://assets.ai.engineer/sites/aiecode2025/sha256/622f24b555bf5adbf786e19ff32f3c138e42cd4c80ff9bec509a31c409afb673.webp)

Generalized Needle in a Haystack
(Question-Response Relation Type)

• 1:1 Relation – one concept : one piece of information (NiH)
“What is the secret ingredient needed to build the perfect pizza?”
• 1:n Relation – one concept : many pieces of information (Multi NiH)
“What are the secret ingredients needed to build the perfect pizza?”
• n:1 Relation – many concepts : one piece of information
“What secret ingredient is needed to build the perfect pizza and also to build the perfect pierogies?”
• n:n Relation – many concepts : many pieces of information
“What are the secret ingredients needed to build the perfect pizza, the perfect pierogies and [illegible] perfect sushi?”

https://x.com/GregKamradt/status/1729573848893579488
Multi Needle in a Haystack (langchain.dev)
Hubert Misztela - LLM Scientific Reasoning
10

AIE

### Reasoning Types

Misztela frames reasoning as processing information logically: aggregating data, performing arithmetic, or applying logical reasoning.

[10:01](https://www.youtube.com/watch?v=gbM7k3NauNI&t=601s) [Three hexagons label simple aggregation, algebraic, and logical reasoning.](https://assets.ai.engineer/sites/aiecode2025/sha256/617d2a37b4c346e91d6922f8e4c84d52874099708e676eae58fee0edf6caf35c.webp)

Reasoning Types

Simple Aggregation
Logical
Algebraic

IS
Hubert Misztela - LLM Scientific Reasoning
11

AIE
Microsoft
smol ai

[10:09](https://www.youtube.com/watch?v=gbM7k3NauNI&t=609s) [Three hexagons label simple aggregation, logical, and algebraic reasoning.](https://assets.ai.engineer/sites/aiecode2025/sha256/90ca8501381c688490a2a0f078cf26f2e63c83e60fdc45ea1d3e40b6ff97f3b3.webp)

Reasoning Types

Simple Aggregation
Logical
Algebraic

IS
Hubert Misztela - LLM Scientific Reasoning
11

AIE
Microsoft

### Causal Reasoning

Causal reasoning adds relationships between entities to the reasoning diagram. Misztela suggests asking an LLM to hypothesize or deduce more from those extracted relationships.

[10:29](https://www.youtube.com/watch?v=gbM7k3NauNI&t=629s) [The reasoning types diagram adds a causal hexagon alongside excerpts from two papers.](https://assets.ai.engineer/sites/aiecode2025/sha256/99a544382663ff30482458fcb7bf4585e1f782bd7540549ec3e8de9d6afe8cd4.webp)

Reasoning Types

Simple Aggregation
Logical
Algebraic
Causal

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Emre Kiciman* — Microsoft Research — [redacted]
Robert Ness — Microsoft Research — [redacted]
Amit Sharma — Microsoft Research — [redacted]
Chenhao Tan — University of Chicago — [redacted]

Causal Inference Using LLM-Guided Discovery

Aniket Vashishtha — Microsoft Research, India — [redacted]
Abbavaram Gowtham Reddy — IIT Hyderabad, India — [redacted]
Abhinav Kumar — Massachusetts Institute of Technology, USA — [redacted]
Saketh Bachu — IIT Hyderabad, India — [redacted]
Vineeth N Balasubramanian — IIT Hyderabad, India — [redacted]
Amit Sharma — Microsoft Research, India — [redacted]

Hubert Misztela - LLM Scientific Reasoning
11

AIE
Microsoft
smol ai

### Algorithmic Reasoning

The diagram adds algorithmic reasoning, represented by chess pieces, alongside four other reasoning types. A paper excerpt pairs the category with large language models as optimizers.

[10:47](https://www.youtube.com/watch?v=gbM7k3NauNI&t=647s) [The diagram adds algorithmic reasoning with a chess icon and an optimizer paper excerpt.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d1ff77a087690f00ae206409ca864d2c0a1240371c7a882020b523ae3679f72.webp)

Reasoning Types

Simple Aggregation
Algebraic
Logical
Causal
Algorithmic

Google DeepMind
Large Language Models as Optimizers
Chengrun Yang*, Xuezhi Wang, Yifeng Lu, Hanxiao Liu
Quoc V. Le, Denny Zhou, Xinyun Chen*
[redacted]
Google DeepMind    * Equal contribution

[illegible] IS
Hubert Misztela - LLM Scientific Reasoning
11

AIE
Microsoft
smol ai

### Probabilistic Reasoning

Probabilistic reasoning joins the other reasoning types in the diagram. Misztela describes efforts to make LLMs reason probabilistically, like Bayesian inference.

[10:54](https://www.youtube.com/watch?v=gbM7k3NauNI&t=654s) [The diagram adds probabilistic reasoning with a bar chart icon and a paper excerpt.](https://assets.ai.engineer/sites/aiecode2025/sha256/90861708bde8dbbde462040840a40753a686dc72e5bfef4dd29d825ebef8e069.webp)

Reasoning Types

Simple Aggregation
Algebraic
Logical
Causal
Algorithmic
Probabilistic

Probabilistic Reasoning in Generative Large Language Models
Aliakbar Nafar¹, Kristen Brent Venable² and Parisa Kordjamshidi³
¹,³ Michigan State University
² Florida Institute for Human and Machine Cognition
[redacted]

Hubert Misztela - LLM Scientific Reasoning
11

AIE
aws

### Reasoning Types

Structured reasoning can involve compositionality within a structure; geometric reasoning includes ARC. Misztela suggests using specialist tools, such as causal inference libraries or a Python REPL.

[11:00](https://www.youtube.com/watch?v=gbM7k3NauNI&t=660s) [The diagram adds structured reasoning with a network icon and a paper excerpt about compositionality.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b43a93141e7ab2cd77bf466d5b0acfc4f3d6ac6be2a881dc85e7c76fe52fcdf.webp)

Reasoning Types

Simple Aggregation
Algebraic
Logical
Causal
Algorithmic
Probabilistic
Structured

Faith and Fate:
Limits of Transformers on Compositionality

Nouha Dziri, Ximing Lu, Melanie Sclar,
Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin,
Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal,
Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi
[illegible affiliation markers]

Allen Institute for Artificial Intelligence
University of Washington
University of Southern California
University of Chicago
[redacted]

Hubert Misztela - LLM Scientific Reasoning
11

Interface text:
AIE
IS

[11:11](https://www.youtube.com/watch?v=gbM7k3NauNI&t=671s) [A diagram lists reasoning types, including causal, structured, probabilistic and geometric reasoning.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae76fa0008c45a9007f91789618ad34d6b0b5c29b1886fdeb66f6d01778bff6b.webp)

Reasoning Types

Simple Aggregation
Algebraic
Logical
Causal
Algorithmic
Probabilistic
Structured
Geometric
Others...

http://arcprize.org

Hubert Misztela - LLM Scientific Reasoning
11

Interface text:
AIE
IS
Microsoft
smol ai

### RNAi Rediscovery Problem

The task is to find one cause for three phenomena: narrow many papers to relevant material on the three topics, then use causal or structured reasoning to answer a single question.

[12:24](https://www.youtube.com/watch?v=gbM7k3NauNI&t=744s) [The slide describes retrieval and causal or structured reasoning for finding a common cause of three phenomena.](https://assets.ai.engineer/sites/aiecode2025/sha256/9bc107e2c2e1921be473cdc6340b18587d06e6b91bbcde84206d90a13b5a071c.webp)

RNAi Rediscovery Problem

INPUT: “What could be the underlying cause of C, S, Q?”

1) Retrieval Type [n:n]
What is really relevant in this haystack? (narrow down from many papers to a small number of those 3 topics)

2) Reasoning type: causal or structured
Causal
Structured

3) Particularities:
• Usually we prepare RAG for responding many questions. Here instead we really have just one question.

Diagram: X → C, S, Q

Hubert Misztela - LLM Scientific Reasoning
12

Interface text:
AIE

## Full transcript

[0:00](https://www.youtube.com/watch?v=gbM7k3NauNI&t=0s) [upbeat music] Hello, everyone. Um, there's a lot of interesting talks, so, so thank you for being here.

[0:18](https://www.youtube.com/watch?v=gbM7k3NauNI&t=18s) Um, my name is Hubert. Uh, I work for pharma, uh, Novartis. On a daily basis, I work on, uh, designing of small molecules with generative AI, so designing, designing small graphs which would try to fit our d- uh, fix our diseases, cure.

[0:34](https://www.youtube.com/watch?v=gbM7k3NauNI&t=34s) Um, but today I would like to talk about, um, scientific reasoning, scientific discoveries, um, specifically how to use LLMs for that. And this is a joint work with my colleague Derek Lowe, who's a medicinal chemist.

[0:47](https://www.youtube.com/watch?v=gbM7k3NauNI&t=47s) You might know it-- know him if you are, uh, interested in drug design.

[0:52](https://www.youtube.com/watch?v=gbM7k3NauNI&t=52s) So today, I'm gonna talk about a few things, right? Probably I'm gonna leave you with more questions than responses and more problems than solutions, but I think it's worthwhile.

[1:01](https://www.youtube.com/watch?v=gbM7k3NauNI&t=61s) Um, so first I'm, I'm gonna show you an interesting paradox in biology which led to a Nobel Prize discovery. Then I'm gonna be talking about, uh, reasoning with RAGs.

[1:12](https://www.youtube.com/watch?v=gbM7k3NauNI&t=72s) And you've seen many different themes of RAGs and agentic, uh, RAGs during this conference, so I think it's, it's fitting really nicely to all of the, um, the other dov- topics.

[1:23](https://www.youtube.com/watch?v=gbM7k3NauNI&t=83s) Um, scientific discovery framework, which is how we can really build something, how we can build a, the system, or RAG specifically, which would help us in those scientific discoveries.

[1:33](https://www.youtube.com/watch?v=gbM7k3NauNI&t=93s) And in the end, I will show you a few, a few, um, uh, examples of the experiments. Okay. I think this is doesn't working. Yeah. So back in 1990s, early '90s, um, scientists were trying to improve, uh, color of the flowers, petunia flowers.

[1:51](https://www.youtube.com/watch?v=gbM7k3NauNI&t=111s) You know, this is what you do when you're a scientist. So they were trying to, um, overexpress the specific gene. So they wanted to get a color stronger. But what they get, they get the color flipped, which was really surprising.

[2:05](https://www.youtube.com/watch?v=gbM7k3NauNI&t=125s) Nobody really knew what was happening and why, why would th- was this happening, right? And this kind of, um, results they were getting, biologists were getting from different subdomains of biology.

[2:14](https://www.youtube.com/watch?v=gbM7k3NauNI&t=134s) I'm not gonna get into the details. I'm not a biologist. I'm a computer scientist, so I'm gonna save you all of that. But the interesting thing here is that there were three different, uh, set of results, set of experiments from different subdomains, even with different names.

[2:30](https://www.youtube.com/watch?v=gbM7k3NauNI&t=150s) Everybody knew it was related to some-something was messing with, with genes, but nobody knew why, right? So these three things, only after eight years, more or less, um, has been resolved, these three phenomena.

[2:44](https://www.youtube.com/watch?v=gbM7k3NauNI&t=164s) And this, this is what led to, to a Nobel Prize, uh, in biology, right? Uh, in medicine. And, um, so the question is:

[2:55](https://www.youtube.com/watch?v=gbM7k3NauNI&t=175s) Can we really use LLMs to speed this up, right? It took eight years. There might be something in the literature which could maybe give a hint of how we can go faster, right?

[3:05](https://www.youtube.com/watch?v=gbM7k3NauNI&t=185s) And the question is: What are the underlying causes, or what might be the underlying causes of these three phenomena, right? And now you're all thinking, "Okay, let's run the RAG.

[3:14](https://www.youtube.com/watch?v=gbM7k3NauNI&t=194s) Let's ask, ask the question. Probably it's gonna, it's gonna get it quite quickly," right?

[3:19](https://www.youtube.com/watch?v=gbM7k3NauNI&t=199s) Let's pause for a second and try to think how we can set this up and try to ex-experiment. But before it, um... Okay. We all know what, what RAG is, naive RAG is.

[3:30](https://www.youtube.com/watch?v=gbM7k3NauNI&t=210s) We have a question. We process the question. We process previously before, uh, the data. We represent both of them in a given latent space. We, we do some kind of retrieval over the embeddings, and you get the response.

[3:43](https://www.youtube.com/watch?v=gbM7k3NauNI&t=223s) And there is a lot of, a lot of different tricks. I, I wanted to show you just a b-bunch of them because these tricks, like HyDE, for example, or, or ranking, um, or document indexing, are things relating to different pieces of RAG pipeline, and they are all for different purposes, slightly different purposes, right?

[4:03](https://www.youtube.com/watch?v=gbM7k3NauNI&t=243s) They might m- try to, try to fix the problem of matching between the question and answer. They might try to, uh, fix the problem of the, the right retrieval of the top documents or top chunks, and so on.

[4:16](https://www.youtube.com/watch?v=gbM7k3NauNI&t=256s) But the problem is, if LLMs, and this is really interesting paper, if LLMs, and we know that, cannot respond a little bit, little bit more, more catchy questions, right?

[4:29](https://www.youtube.com/watch?v=gbM7k3NauNI&t=269s) A little bit more, more convoluted questions, how we can expect that LLMs would s- understand the question and then on top of that, uh, give us the right, uh, right response with the RAG, right?

[4:41](https://www.youtube.com/watch?v=gbM7k3NauNI&t=281s) So we are, we are trying to run before we crawl.

[4:46](https://www.youtube.com/watch?v=gbM7k3NauNI&t=286s) So I was thinking about it and, and, um, when you look at the literature, the papers, and at the solutions during this conference, you're gonna realize there is, there are themes kind of repeatedly referring to something which is happening before the retrieval.

[5:04](https://www.youtube.com/watch?v=gbM7k3NauNI&t=304s) And one example of, of reasoning before the retrieval

[5:10](https://www.youtube.com/watch?v=gbM7k3NauNI&t=310s) is, uh, is, is routing. Routing, which we have in agentic flows, is really the idea of doing something with the question because we cannot throw it at, at, at, at the embeddings and just get the response, right?

[5:22](https://www.youtube.com/watch?v=gbM7k3NauNI&t=322s) So you... I-I'm re- I'm, I'm, I'm naming it r-reasoning for, for a purpose, and you're gonna, you're gonna understand it in, in a few slides, right? So you need to understand a little bit more, we need to understand a little bit more what's the question about, right?

[5:37](https://www.youtube.com/watch?v=gbM7k3NauNI&t=337s) And on the other hand, not only with the question because we also try to, um, do something with the data, right? So GraphRAG or a GraphReader, very interesting papers.

[5:52](https://www.youtube.com/watch?v=gbM7k3NauNI&t=352s) If you look at them, you're gonna, you're gonna see that they are trying to represent the knowledge from the, from these documents in a little bit different way, specifically about the, uh, with graphs.

[6:01](https://www.youtube.com/watch?v=gbM7k3NauNI&t=361s) You might use different representations in many different ways, but the point is

[6:06](https://www.youtube.com/watch?v=gbM7k3NauNI&t=366s) We are doing something before the retrieval, which really is a reasoning to deal with the more semantically complex question.

[6:15](https://www.youtube.com/watch?v=gbM7k3NauNI&t=375s) The reasoning after the retrieval usually hap- usually is done by, by, uh, delegated to LLM, right? Because this is all about LLMs, but it, that doesn't have to be the case.

[6:27](https://www.youtube.com/watch?v=gbM7k3NauNI&t=387s) Um, so the point here really is that, uh, between more convoluted the semantics of the question, um,

[6:37](https://www.youtube.com/watch?v=gbM7k3NauNI&t=397s) the mo- the harder it's gonna be for your standard RAG without those reasoning steps. So you probably wanna, want, you wanna try to add these reasoning steps before it, and I'm gonna show you some experiments with that.

[6:47](https://www.youtube.com/watch?v=gbM7k3NauNI&t=407s) Um, and small by the way is that, um, as I was saying, I work on a little bit different, um, flavor of, of, of AI, right? Which is generative, but for graphs.

[7:00](https://www.youtube.com/watch?v=gbM7k3NauNI&t=420s) So kind of old style, encoder-decoder, variational autoencoder, you have a latent space, and you navigate that.

[7:06](https://www.youtube.com/watch?v=gbM7k3NauNI&t=426s) So when I saw this thing which I draw, which is reasoning, embedding,

[7:12](https://www.youtube.com/watch?v=gbM7k3NauNI&t=432s) uh, retrieval, and then again reasoning, I realized that this is very similar to what we do in... It, it's very well known, uh, pipeline in chemistry, where you take a dataset, which in that case are molecules.

[7:27](https://www.youtube.com/watch?v=gbM7k3NauNI&t=447s) You represent them in the latent space, which are embeddings effectively, and then you navigate the embeddings to optimize for that. So for gi- for a given purpose. So you have the latent space, and you wanna extract the, the points which are the most interesting for your purpose, right?

[7:42](https://www.youtube.com/watch?v=gbM7k3NauNI&t=462s) So in our case, we are trying to extract the embeddings which are the most similar to the question, right? So we do simple similarity. But in other domains, like chemistry, we run optimization over that latent space, and this is type of a, type o- type of reasoning really, right?

[7:57](https://www.youtube.com/watch?v=gbM7k3NauNI&t=477s) So I was like, "Aha, okay." So kind of common theme is appearing here.

[8:05](https://www.youtube.com/watch?v=gbM7k3NauNI&t=485s) Now the question is, do you really know, do you really need it? How do I know if my question is complex or not? So, um, we all know needle-in-a-haystack, right?

[8:17](https://www.youtube.com/watch?v=gbM7k3NauNI&t=497s) Um, test or experiment or benchmark. So I'm not, I'm, I'm not gonna be focusing on that too much, so let me go quite quickly. We know, like, find the perfect ingredient, uh, uh, the, the ingredient to build the perfect pizza, right?

[8:33](https://www.youtube.com/watch?v=gbM7k3NauNI&t=513s) And a little bit extended version of that is multi, uh, multi-needle-in-haystack, uh, discussed yesterday on one of the tracks as well, right? So

[8:44](https://www.youtube.com/watch?v=gbM7k3NauNI&t=524s) when you think about it, this is not only about the response which you're gonna get, but this is also about the question, and really relationship between the question and the response.

[8:54](https://www.youtube.com/watch?v=gbM7k3NauNI&t=534s) And as I was saying, between more convoluted semantically the question is, the harder it's gonna be. So how we can generalize it? And I was thinking about databases, right?

[9:03](https://www.youtube.com/watch?v=gbM7k3NauNI&t=543s) I'm an engineer, so one needle, one response, or one, one, one, one, uh, concept, one piece of information is like one-to-one relationship, like in databases. Then you have one to n, and then appeared to my head that, okay, if I have a few concepts put together in one question, this is the multi,

[9:27](https://www.youtube.com/watch?v=gbM7k3NauNI&t=567s) uh, th- this is a multi, multi thing versus finding one, one needle. And then you can extend it still to a few concepts hidden in a few chunks of the document.

[9:35](https://www.youtube.com/watch?v=gbM7k3NauNI&t=575s) So really, you, you need to do a little bit more. You already see that, okay, you need to pass somehow the, the, the complexity of the question so that you can get the right embeddings extracted.

[9:48](https://www.youtube.com/watch?v=gbM7k3NauNI&t=588s) And this is what, what kind of, um, triggers your reasoning. If you have a few concepts in the question, you probably need the reasoning before the retrieval so that you know what to retrieve and how to do it. [clears throat]

[10:02](https://www.youtube.com/watch?v=gbM7k3NauNI&t=602s) Sorry. So I was, uh, I was mentioning reasoning, right? So what, what reasoning really is, and there are many, uh, views on that because, for example, we use chain of thought as a, as a type of reasoning, right?

[10:15](https://www.youtube.com/watch?v=gbM7k3NauNI&t=615s) But I'm thinking about reasoning as processing information in some logical way, right? So we all know how we can do the, uh, do the aggregation, how we can do simple arithmetic over, over the data.

[10:27](https://www.youtube.com/watch?v=gbM7k3NauNI&t=627s) You, you... We al- we also know what's logical reasoning, right? Um, so there's also causal reasoning. If you extract specific entities from the data and causal relationships between them, you might ask LLM, "Okay, what, what more you can hypothesize about this data, or more, what more y- maybe you can deduce?"

[10:45](https://www.youtube.com/watch?v=gbM7k3NauNI&t=645s) Right? And there are papers about it, and this, this kind of starts working. Then you have algorithmic reasoning, which I mentioned before. Um, you have also probabilistic reasoning, so we ex- we were trying to, to, to make LLMs reason in a probabilistic fashion, right, like a Bayesian inference.

[11:02](https://www.youtube.com/watch?v=gbM7k3NauNI&t=662s) There's also structured way of reasoning. This is a little bit different from causal because you might have a structure, and you can expect some kind of compositionality over, over that, that structure, right?

[11:12](https://www.youtube.com/watch?v=gbM7k3NauNI&t=672s) So, um, and there's also, of course, uh, ARC

[11:17](https://www.youtube.com/watch?v=gbM7k3NauNI&t=677s) Kaggle competition, uh, recently released with one million dollar prize for closer getting to, to AGI, right? This is reasoning over, over, over geometry. This is... Th- that's why it's so challenging because this is not typical type of reasoning, right?

[11:32](https://www.youtube.com/watch?v=gbM7k3NauNI&t=692s) So why am I saying this? Because usually what we do, we try to ex- we, we expect LLMs to perform all of these reasoning well, all of these reasonings well.

[11:46](https://www.youtube.com/watch?v=gbM7k3NauNI&t=706s) And do we really need LLM for that? Probably yes or maybe not, but you can also delegate these, these reasoning types to specific tools which are specialist in that.

[11:57](https://www.youtube.com/watch?v=gbM7k3NauNI&t=717s) So you can do causal inference with, with libraries for, for causal inference. You can do, uh, algorithmic reasoning with specific Python, right? That's why we have a REPL as a, as a tool, right, attached to, to your, to your agent so that you can generate the code, and this is, this is kind of algorithmic reasoning, right?

[12:16](https://www.youtube.com/watch?v=gbM7k3NauNI&t=736s) So let's, let's try to, um, come back to the problem and think how we, how we can solve it really, right? So we wanna find the cause of these three phenomenas, and we define the type of reasoning, the type of re-retrieval, which is the relation between question and, and, and an answer.

[12:33](https://www.youtube.com/watch?v=gbM7k3NauNI&t=753s) And the particularities in our case are also that

[12:36](https://www.youtube.com/watch?v=gbM7k3NauNI&t=756s) we are not pre... We don't wanna build a RAG to respond to many different questions over a specific data set, but we want to respond to only one question, right?

[12:46](https://www.youtube.com/watch?v=gbM7k3NauNI&t=766s) This... and this is the question. We want to respond this really one question, and we wanna process our data set as many times as we possibly can so that we extract all of the relevant themes in that.

[12:57](https://www.youtube.com/watch?v=gbM7k3NauNI&t=777s) So but the trick here is that LLMs during training have seen Wiki-Wikipedia, so they know about this problem in RNA interference because it happened in 1998. So we need the groundedness, right?

[13:10](https://www.youtube.com/watch?v=gbM7k3NauNI&t=790s) And this is another aspect of it. Um, so when you think about different methods and where you are in your specific use case in the computational efficiency versus ground-groundedness, uh, you'll...

[13:22](https://www.youtube.com/watch?v=gbM7k3NauNI&t=802s) you immediately see, okay, which approaches might be more suitable for, for my case. And as, as I'm gonna show you later on, um, uh, relevance classifier is one of the main things which we use because we can process all of, all of the data set.

[13:36](https://www.youtube.com/watch?v=gbM7k3NauNI&t=816s) Um, so overall, um, designing choices, uh, you can see here, and th-this depends usually upon your specific question you have, the reasoning you're, you, you, you need, the relationship between question and answering, and another aspect like, like groundedness versus efficiency.

[13:56](https://www.youtube.com/watch?v=gbM7k3NauNI&t=836s) Um, so when we have, when we have defined what type of question it is, so how we really wanna test whether our solution is capable of doing this kind of discovery, right?

[14:08](https://www.youtube.com/watch?v=gbM7k3NauNI&t=848s) So you state the question, you define the, the type of question, but then you need to do the knowledge cutoff, which is basically you need to remove from the LLM the knowledge about the discovery.

[14:17](https://www.youtube.com/watch?v=gbM7k3NauNI&t=857s) So what we do, we use RAG, and in that RAG we have only the, the, the, the scientific papers from before the discovery, right? So that it doesn't cheat.

[14:27](https://www.youtube.com/watch?v=gbM7k3NauNI&t=867s) So we, we present it in, in the state of the situation which was before the discovery, so that it can... we can simulate that situation, right? And this is kind of a training.

[14:38](https://www.youtube.com/watch?v=gbM7k3NauNI&t=878s) You can think about it as a agent training in that specific... in, on that specific scientific problem, on that specific data set, and the over-overarching goal is to, uh, have a system which would be able to make those discoveries on many different scientific problems and applying this, the same, the same scheme.

[14:59](https://www.youtube.com/watch?v=gbM7k3NauNI&t=899s) Um, now the question is, how, how do we define the, the, the success of, of, of, uh, this experiment? So the first level of success would be, okay, find some hypothesis or what we, what we really know from this data set, data set, right?

[15:14](https://www.youtube.com/watch?v=gbM7k3NauNI&t=914s) So it's not that we are asking specific question. We want RAG to extract, okay, what do you know about this specific... a-anything related really to that problem.

[15:24](https://www.youtube.com/watch?v=gbM7k3NauNI&t=924s) And, uh, don't read the text. Forgive me the amount of text on, on these slides. Um, but look at the graph, right? So we want RAG to find those relationships between different facts in our data set.

[15:37](https://www.youtube.com/watch?v=gbM7k3NauNI&t=937s) Then the next level would be to find, like, less obvious links because you remember there were three different subdomains of biology, right? So the, the, the, those, those linkages are not obvious.

[15:48](https://www.youtube.com/watch?v=gbM7k3NauNI&t=948s) So we wanted to get little bit, little bit further. Ideally, if it's exhaustive, so finding all of the facts from the data set, right? And the next level would be doing new hy- making new hypothesis, right?

[16:02](https://www.youtube.com/watch?v=gbM7k3NauNI&t=962s) Which would be, okay, if I see this, I can go maybe one step further and hypothe-hypothesize about a, a new, new relationship. And the i-ideal situation or, like, the, the, the high level of success, I'm not expecting to, to have this anytime soon, otherwise we're gonna meet in Sweden.

[16:19](https://www.youtube.com/watch?v=gbM7k3NauNI&t=979s) But, uh, this is really about finding not only what can be, uh, what, what is related between, between the, those different facts in the literature, but also explaining how that happens.

[16:29](https://www.youtube.com/watch?v=gbM7k3NauNI&t=989s) And the interesting thing is that when the discovery happened made by humans, we didn't know how it was happening. We know what was happening, but we didn't know how.

[16:39](https://www.youtube.com/watch?v=gbM7k3NauNI&t=999s) Okay, so the last part, I'm gonna go quite quickly because I have just, uh, less than two minutes. So the naive RAG we all know. Uh, I'm gonna show a very, very, very nice trick, which is not very prevalent actually, because when you have, uh, the, the, the, the, the, the distances, similarities between the question embedding and

[16:58](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1018s) the, the embedding in your database, um, you might ask yourself, of course, how many embeddings are... how many chunks are, are enough, right, really? But you can do it...

[17:06](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1026s) think about it mathematically, because if you, if you see those distances, you can calculate the variances within the different clusters of those, uh, embeddings and between them. So this is called, uh, Jenks natural breaks.

[17:21](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1041s) Very simple thing to do, and you can extract the top, top cluster, which would be kind of representing, uh, the lowest variance between, between the chunks extracted. So they are kind of, uh, telling you the, the probably, the, the...

[17:34](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1054s) giving you the most information, right? Um, uh, there is a snippet how you can do it. Um, I'm gonna go quite quickly. In this case, w- we got, uh, we didn't get the ground as well because it, it was, uh, using the knowledge from, from after the discovery, so we need to, uh, make the, the prompt stronger,

[17:53](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1073s) right? So we call it strict prompting, and then it, and it, and it got m- uh, better, right? And then what we do, we did the relevance classifier, which is really passing all of the chunks in, in our data set through the LLM and asking whether this is relevant to our problem.

[18:08](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1088s) Unfortunately, that wasn't very much, uh, informative because it was kind of redundant to distance of the, from the embeddings, right? There was a discussion yesterday about it. But when we, when we, uh, did the, um, the analysis of how relevant, um, each paper is in the context of, uh, hypothesis advancing, so we, we built a more sophisticated

[18:30](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1110s) prompt and thinking about, little bit more about the scientist. This is when we got, uh, the hypothesis a little bit further, right? So it, it found us all of the hypothesis in the literature related to DNA, but one to the RNA, and this is the right one, right?

[18:44](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1124s) So we see already that it's going in the right direction. I'm gonna skip it in the interest of time. You're gonna have access to it. So my point here is when we started doing this reasoning over the question and the database before the retrieval, we started getting, uh, closer to the results, which are ground truth results without

[19:00](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1140s) cheating. Um, so the conclusion is scientific discovery requires, uh, solving harder, uh, than simple Q&A problems. Knowing your problem can help define more efficient RAG architecture. Needle-in-a-haystack might be generalized.

[19:13](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1153s) I'm not saying that this is, like, the best way of doing it, but this is already you see something new and interesting. And harder problems might need, uh, reasoning, right?

[19:21](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1161s) And of course, you cannot forget about the brut-brute force because if you have the use case where you can do it,

[19:29](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1169s) check your, check your LLM. Maybe it's gonna, it's gonna do better than the distance embeddings.

[19:35](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1175s) Um, so thank you very much. Uh, forgive me rushing. [audience applauding]

[19:40](https://www.youtube.com/watch?v=gbM7k3NauNI&t=1180s) I hope this was helpful. [upbeat music]
