# Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs

Yuval Belfer, AI21 Labs

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/r9OwPx_HoV0-stop-chunking-like-its-2022) · [Video](https://www.youtube.com/watch?v=r9OwPx_HoV0)

## Stop Chunking Like It's 2022

A small chunk can find a precise fact and miss a relationship spread across a scene. Yuval Belfer explains how searching several chunk sizes and combining whole-document rankings improves recall—and what that flexibility costs.

![Selected presentation frame from Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs at 801 seconds](https://ai.engineer/api/talk-essay-media/9442cce671989aeb14631929da9404aaa9f914a1403a699d7410d3ee1b69f931)

Stop Choosing One Chunk Size: Multiscale Retrieval with Yuval Belfer

### Agentic search still depends on how data is organized

[0:13](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=13s)

[Yuval Belfer](https://x.com/yuvalinthedeep) of AI21 Labs opens with the recurring declarations that RAG—retrieval-augmented generation—is dead, followed by similar claims about chunking. Agents can explore files with tools such as `grep`, `ls`, and `find`. His objection concerns scale and variety: these tools alone do not make searching a large corpus efficient for many different kinds of questions. The work of dividing and indexing data still matters.

### Indexing sets the units; retrieval searches them

- **Indexing:** Before questions arrive, preprocess the corpus, choose how much text each chunk contains, and store the resulting representations in a vector database. This determines the units the system can search.
- **Retrieval:** For each question, search those units. Developers can adjust top K, the number of results returned, or experiment with hybrid search. Because they already have the questions, they can inspect how these changes affect the results.

Belfer finds retrieval tuning easier to engage with: the questions are concrete and the effects are visible. The earlier decision about what each searchable unit contains receives less attention.

![Selected presentation frame from Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs at 135 seconds](https://ai.engineer/api/talk-essay-media/cc8885018b31b0cceb5977be6bebe486701d57faf45ee078699318cea47af358)

Agentic search still depends on how data is organized

Agentic search does not remove the cost of a poor organization. Belfer imagines World Cup records arranged into a separate folder for each tournament, including 1998 and 2002. To answer which team won the most tournaments, an agent must visit the folders, find each winner, and aggregate the results. No individual tournament folder answers that question. An organization that suits a local lookup can make a cross-cutting question expensive, even when the agent navigates it correctly.

### A fixed chunk size trades detail against context

[4:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=255s)

Retrieval has become plumbing, Belfer says, and chunking often becomes a decision made once and forgotten. The familiar setup is to choose a size such as 512, add 10% or 20% overlap, index everything, and move on. Overlap lets neighboring chunks share some text, but the system still commits to a particular window. That window determines which information receives a searchable representation together.

- **Large chunks preserve surrounding context.** They keep more of a passage together, but their embeddings can blur a specific detail among other material. The whole picture survives while a small fact becomes harder to retrieve.
- **Small chunks concentrate local detail.** A focused passage can represent a contained fact more clearly, but it can separate that fact from the broader context needed to understand a relationship.

Belfer calls chunking lossy compression. The loss concerns the searchable representation: choosing a window favors some information and weakens other information. His claim is that the useful size depends on the query, even within one corpus. A fixed size that performs well on familiar questions can hide failures on questions that need a different amount of context. The experiments that follow test how much this difference matters.

![Selected presentation frame from Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs at 364 seconds](https://ai.engineer/api/talk-essay-media/fb7f554735accfeea3d83c1dbc99384fea18b8a129b142c8e1cf9e78a8d2a2e6)

A fixed chunk size decides which information survives

### Two Seinfeld questions need different windows

[6:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=375s)

The team created six differently chunked instances of each dataset. Belfer names windows of 2,000 and 1,000 alongside smaller sizes. The initial datasets were QMSum, containing meeting transcripts; NarrativeQA, involving questions about novels; and an in-house Seinfeld trivia dataset built around episode transcripts. Holding the underlying material constant let the team compare what each scale could retrieve.

### A contained fact and a contextual relationship

- **Focused lookup:** A question asking for the name of something associated with Jerry returns the relevant result at rank one with 100-token chunks, while larger windows place it beyond the first 50 results. Belfer's explanation is that a contained answer benefits from a narrow representation.
- **Contextual lookup:** A question asks whom Jerry describes as his nemesis and pure evil. Belfer identifies the answer as Newman, but reports that small chunks fail to retrieve it. Finding the relationship needs more surrounding transcript context than the focused lookup.

![Selected presentation frame from Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs at 450 seconds](https://ai.engineer/api/talk-essay-media/a85bc6909f9a70013248c19c408f9558f4220175e0b0b705aaecbe3befca5f7b)

Two Seinfeld questions need different windows

These questions make corpus-wide tuning an awkward compromise. A narrow window works for one question and fails for the other. The team therefore moves from asking which size wins on a dataset to asking how much performance is available when each query gets its favorable size.

### The oracle measures the cost of committing too early

[8:14](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=494s)

The oracle experiment uses the known answers to choose the best-performing chunk size for each query. It measures headroom: how well could the tested collection of sizes perform if the system always made the favorable choice? A real retrieval system cannot use this selector because it does not know the answer before searching.

Belfer describes plots with recall on the vertical axis and the number of retrieved chunks, K, on the horizontal axis. Each fixed-size curve shows recall at K for one window size. The oracle curve uses the best per-query outcomes across the tested sizes. Some fixed-size curves intersect, showing that a size's advantage changes with the retrieval cutoff. The oracle also captures differences between individual questions that disappear into dataset-level averages.

He reports a roughly 20%–40% recall gap between the oracle and fixed-size retrieval in the tested settings. That range describes an opportunity within those experiments. It does not establish that every system using 512 loses that amount, and the recording does not distinguish relative improvements from percentage-point differences.

### The question arrives after the window is frozen

The information problem has two ends. At indexing time, the system controls chunk size but does not know the incoming questions. At retrieval time, it has the question but the indexed windows are already fixed. Changing them would repeat the indexing work. The system can estimate future questions, but it cannot adjust a single preselected size to every query as it arrives.

Belfer contrasts the team's direction with Anthropic's contextual retrieval, which he describes as enriching chunks to improve their representations. His team instead keeps several window sizes available. Rather than making one chunk representation carry every kind of question, multiscale indexing supplies several searchable views of the same material.

### Search every scale, then combine rankings of whole documents

[11:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=675s)

Multiscale indexing turns the experiment's duplicated corpus into a retrieval strategy. Index the material at N window sizes, then send each incoming query to all N indexes. The six instances used in the experiment are one choice of N. The system searches every available scale instead of predicting a winning size and searching only that index.

![Selected presentation frame from Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs at 690 seconds](https://ai.engineer/api/talk-essay-media/269a93ebe971acc4694d33b89b1d0f80073cca8eb9aef37d9a580c5bd7bef37f)

Search every scale, then let documents receive the votes

Searching several indexes produces several rankings, but their entries initially refer to different objects. A short passage and a larger passage containing it are distinct chunks. Their lists do not rank a shared set of candidates. The team changes the unit of comparison: a chunk match retrieves its entire source document. Each scale can now rank the same document identities, even though it found them through differently sized passages.

### Document rankings become votes

Belfer describes the remaining step as voting. Each scale supplies a ranking of relevant documents; **Reciprocal Rank Fusion**, or RRF, combines those rankings into one list. This replaces the oracle's answer-aware choice with an aggregation of the available retrieval results. RRF worked best among the merging methods the team tried. Belfer describes its implementation as a simple script, without an additional learned reranking model.

Whole-document retrieval also separates the passage used to find information from the text supplied afterward. A narrow chunk can locate a document while the full document supplies surrounding context. Belfer connects this choice to growing model context windows. The recording does not quantify the downstream token cost or explain how documents that exceed a model's context window would be handled.

Different search windows, shared document candidates
Two example scales show the flow; the system searches all N scales. Chunk matches lead to whole documents, giving RRF shared candidates to combine.
- Source corpus — The same material is indexed at several window sizes.
- 100-token index — A narrow searchable representation.
- 1,000-token index — A larger searchable representation; additional scales follow the same flow.
- One incoming query — Search every available index.
- Document ranking from narrow matches — Retrieve the whole source document for a chunk match.
- Document ranking from larger matches — Rank the same document identities through a different scale.
- Reciprocal Rank Fusion — Aggregate document rankings into one result list.
- Source corpus → 100-token index (chunk at a narrow scale)
- Source corpus → 1,000-token index (chunk at a larger scale)
- One incoming query → 100-token index (search in parallel)
- One incoming query → 1,000-token index (search in parallel)
- 100-token index → Document ranking from narrow matches (chunk match → whole document)
- 1,000-token index → Document ranking from larger matches (chunk match → whole document)
- Document ranking from narrow matches → Reciprocal Rank Fusion (contribute document ranks)
- Document ranking from larger matches → Reciprocal Rank Fusion (contribute document ranks)

### Better recall costs memory—and leaves two choices open

[13:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=824s)

The final evaluation covers QMSum, NarrativeQA, Seinfeld, and FinanceBench. Belfer reports that multiscale fusion matches or beats the best fixed size across those datasets. He walks through a heat map whose rows represent fixed chunk sizes and the combined method, while columns represent recall cutoffs from one through ten. Combining scales has to place useful documents high in the final ranking; merely finding them somewhere among N searches would not be enough.

Belfer reports improvements reaching roughly 20%, 30%, or 40% in many tested settings. These are retrieval results, so they do not establish equivalent gains in generated-answer accuracy. He also mentions MTEB improvements between 10% and 40%, depending on the dataset, but does not walk through those results in the recording. The practical fusion results remain separate from the oracle: they combine rankings without knowing which scale will find the answer.

![Selected presentation frame from Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs at 841 seconds](https://ai.engineer/api/talk-essay-media/9a7edbc4e6b5a6a02c05eebd702488a8315e0db00d7af555b4c456edd3d1cc20)

Recall improves, while memory and scale selection remain the tradeoffs

### Memory increases; searches can overlap in time

- **Memory:** Belfer gives a roughly two-to-five-times memory cost for keeping the additional indexed copies. With a fixed number of scales, this is a multiplier on the corpus's storage requirements; memory still grows with the corpus.
- **Retrieval latency:** He reports little added latency because the searches can run in parallel and RRF adds little computation. Parallelism lets searches overlap rather than run one after another. The system still performs N retrieval calls for each question.

The recording provides no timing measurements or accounting of indexing cost and concurrent serving capacity. Overlapping the searches explains how extra work can add little elapsed time, but does not eliminate that work.

### Which scales, and which merger?

- **Scale selection remains unfinished.** Belfer says the tested choices, including 50, 100, and 200, were fairly arbitrary. The team still wants to determine which window sizes to use and how many indexed copies are necessary. Each additional scale spends memory and retrieval work, so their number matters alongside recall.
- **RRF remains a tested choice.** It performed best among the merging methods the team tried. Belfer explicitly leaves room for a better aggregation method; the experiments do not establish that RRF is the best possible merger.

Belfer's closing judgment returns to the infrastructure that agents inherit. Retrieval still matters, and a chunk size selected early should remain open to scrutiny. Multiscale indexing offers a simple experiment: keep several searchable views of the material, retrieve whole documents through each view, and combine their rankings. The reported recall gains give a reason to try it; the memory multiplier and unfinished scale selection determine what a team must weigh for its workload. He closes by pointing readers to the research blog, example code, and published Seinfeld dataset.

## Key takeaways

- The useful chunk size depends on the question: narrow windows can find contained facts while losing relationships that need surrounding context.
- The oracle's reported 20%–40% recall gap measures headroom using known answers. It cannot choose a scale for an unanswered query.
- Multiscale indexing searches N window sizes, maps chunk matches to whole documents, and combines comparable document rankings with Reciprocal Rank Fusion.
- Belfer reports matching or beating the best fixed size across four datasets at roughly two to five times the memory. Parallel searches limit added latency; scale selection and better fusion remain open questions.

## Resources

- [Yuval Belfer on X](https://x.com/yuvalinthedeep) — Speaker profile supplied with the recording, including discussion of query-dependent chunk sizes.

## Related talks

- [RAG Evaluation Is Broken! Here's Why (And How to Fix It)](https://ai.engineer/talks/Ywl4LsvHKzU-rag-evaluation-is-broken-heres-why-fix) — Belfer and Niv Granot examine aggregation questions that individual chunks cannot answer, extending the World Cup example into structured retrieval and evaluation.

## Slides

### Stop Chunking Like It’s 2022

Yuval Belfer introduces his case that chunking still has a role in retrieval.

[0:31](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=31s) [Title slide for Yuval Belfer’s talk about retrieval and chunking.](https://assets.ai.engineer/sites/aiecode2025/sha256/d218e032cd02df0e800f6af1e8bd0c76b4fd0f0fec74d3efe324b7c4a65c2d59.webp)

Stop Chunking like it's 2022

A talk about nothing (struck through) retrieval

Yuval Belfer
Sr. Developer Advocate
AI21 Labs

AI21

### Claims That RAG Is Dead

Two posts declare RAG dead: one proposes 20 Codex spark subagents to search a filesystem; LlamaIndex champions agentic retrieval. Belfer asks how many times RAG can die.

[0:56](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=56s) [Two social posts claim that agentic search or retrieval supersedes RAG.](https://assets.ai.engineer/sites/aiecode2025/sha256/347c6dc805940b0518d4f2ef026bcf6a0054e8785867583b7313179fe428b808.webp)

jason @jxnlco · Mar 5
Rag is dead. Spinning up 20 Codex spark subagents to search your filesystem is the solution.

LlamaIndex 🦙
@llama_index
RAG is dead, long live agentic retrieval!

Stop chunking like it’s 2022
AI21

Interface text:
56 replies · 15 reposts · 408 likes · 44K views

### Is Chunking Dead?

Jerry Liu argues that agents can expand context around a file, making chunk size less important. Belfer questions whether agentic search tools suffice when there is a lot of data.

[1:02](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=62s) [A social post argues that agents can dynamically expand context around a file.](https://assets.ai.engineer/sites/aiecode2025/sha256/034f6c166222117cb72aa659875f19dac6bdd3ee73be10b1eeed6d216ed66710.webp)

Jerry Liu
@jerryjliu0

RAG/retrieval might not be dead.

But chunking is dead.

There is no point overthinking your chunk size if the agent can dynamically expand context around a file.

11:18 PM · Jan 16, 2026 · 77.8K Views

Stop chunking like it’s 2022
AI21

### Agentic Search: How It Reads Your Repo

A revenue threshold query spans reports filed by company and year, so the slide says every file for every year must be opened. Belfer argues grep, ls, and find are insufficient for large datasets and varied queries.

[1:24](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=84s) [A financial-report directory example shows that answering a revenue threshold query may require opening every file for every year.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef0e8f9a1b7b2d3425e32899b36c568da3ed297806457757d19e9ecc11b1596a.webp)

Agentic search
How it reads your repo

~/financial-reports
reports/
  acme-corp/   2021.pdf  2022.pdf
  globex-inc/  2021.pdf  2022.pdf
  initech/     2021.pdf  2022.pdf
# filed by company, then year
$ grep -rl “revenue” reports/
? companies with revenue > $100M
→ must open every file, every year

Stop chunking like it’s 2022

### Stages of RAG

Indexing turns documents into chunks and embeddings to build an index offline. Retrieval searches that index and ranks results for each query.

[1:46](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=106s) [Indexing transforms documents into an index offline; retrieval searches it for each query.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0207ef11a4a23eb488b7c397152dc1a85294495c2b29637e45bf8342aba329d.webp)

The Bet
Stages of RAG

Offline · Once
Indexing
Documents → Chunks → Embeddings →
Index

→

Online · Every Query
Retrieval
Query → Search → Ranked results

Stop chunking like it’s 2022
AI21

### Agentic search killed retrieval tuning

Belfer argues that agentic search has likely displaced retrieval tuning, but still faces limits at large data volumes and high cost.

[2:39](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=159s) [Slide states that agentic search killed retrieval tuning.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c6fbf8bc6ce0d060c504d9a9c3d42b12ba7be045f86aaba778d84d564716c25.webp)

Agentic search
killed retrieval tuning.

Stop chunking like it’s 2022
AI21

### Where agentic search runs out

Belfer says agentic search gets costly at scale, while poorly organized data makes cross-folder queries inefficient. His World Cup example requires checking and aggregating results from every tournament folder.

[3:11](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=191s) [The slide lists scale, cost, and the thing underneath as limits of agentic search.](https://assets.ai.engineer/sites/aiecode2025/sha256/58abb69c61b221682cdf9b2b7f4aaf1ba8805e9cdfcf00bb6c5e654f5e6e3e4f.webp)

Where agentic search runs out

01 Scale
02 Cost
03 The thing underneath

Stop chunking like it’s 2022
AI21

### Retrieval got demoted to plumbing

Belfer argues retrieval has become background infrastructure: teams pick a chunk size and overlap, index their data, and often move on.

[4:09](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=249s) [Slide states that retrieval did not die; it was demoted to plumbing.](https://assets.ai.engineer/sites/aiecode2025/sha256/45b5a1df75f87017f8cda01f4aa63d0426ea781f912f6a18015c6477105d0d40.webp)

Retrieval didn't die.
It got demoted to plumbing.

Stop chunking like it’s 2022
AI21

### Day one of every RAG project

Belfer describes picking a chunk size such as 512, adding overlap, indexing everything, and then forgetting about the choice.

[4:16](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=256s) [The slide says someone picked 512 on day one of a RAG project.](https://assets.ai.engineer/sites/aiecode2025/sha256/331a4b8feefc90a9033ccbd4b754203fbe7082d89f30ae2049bb33a849b77bcd.webp)

Day one of every RAG project:

Someone picked 512

Stop chunking like it’s 2022
AI21

### Fixed chunking strategies

A large chunk keeps the contract’s surrounding clauses but can lose nuance in its embedding. A small chunk isolates the liability cap, leaving out the context around it.

[4:40](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=280s) [Examples compare large chunks, which retain context, with small chunks, which isolate detail.](https://assets.ai.engineer/sites/aiecode2025/sha256/93947a0eba8be471b31777ab3d947373c4ffb8e61487ef8f963faa27f2171b95.webp)

The Bet
Fixed chunking strategies

Big chunks → context, no detail.
The vendor shall maintain 99.95% uptime, measured monthly.
Service credits apply when availability falls below target.
Either party may terminate for material breach on 30 days notice.
Confidential information includes model weights and benchmarks.
Total liability is capped at fees paid in the prior twelve months.
Disputes are governed by the laws of the State of Delaware.

Small chunks → detail, no context.
The vendor shall maintain 99.95% uptime, measured monthly.
Service credits apply when availability falls below target.
Either party may terminate for material breach on 30 days notice.
Confidential information includes model weights and benchmarks.
Total liability is capped at fees paid in the prior twelve months.
Disputes are governed by the laws of the State of Delaware.

Stop chunking like it’s 2022
AI21

### Fixed chunking as lossy compression

Large chunks retain context but lose nuance; small chunks preserve detail but lose the bigger picture. Belfer calls chunking lossy compression because either choice loses information.

[5:04](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=304s) [Diagrams compare large chunks that retain context with small chunks that preserve detail.](https://assets.ai.engineer/sites/aiecode2025/sha256/9040a3f89f1116873e7b1d3d45473764cf8b632ee5457ce8e1fa4fe1af015fc5.webp)

The Bet

A fixed chunking is
lossy compression.

Big chunks → context, no detail.
Small chunks → detail, no context.

Stop chunking like it’s 2022
AI21

### There is no right chunk size

Belfer argues that chunk size depends on the query, so tuning a system to a dataset cannot establish one right size.

[5:12](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=312s) [A dark slide states that there is no right chunk size.](https://assets.ai.engineer/sites/aiecode2025/sha256/aea5261c758af62bf776b4413d913f1e7a7428da218be0c8c14115df21667b9d.webp)

There is no right chunk size.

Stop chunking like it’s 2022

ai21

### Chunk size and dataset optimization

Belfer argues that chunk size depends on the query, so it cannot be optimized once for an entire dataset.

[5:50](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=350s) [The slide states that there is no right chunk size and that it cannot be optimized per dataset.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc8ace02112d78448738acbf2f85a8ae2ae94f8611faf55f82dcc73875a95e9b.webp)

There is no
right chunk size.

You can’t optimize it per dataset.

Stop looking for one.

Stop chunking like it’s 2022
AI21

### The experiment: chunk sizes

The experiment indexes six copies of the same dataset at different chunk sizes, from 50 to 2,000 tokens. Larger chunks divide the corpus into fewer pieces.

[5:56](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=356s) [The same corpus is illustrated at chunk sizes of 50, 100, 200, 500, 1,000 and 2,000 tokens.](https://assets.ai.engineer/sites/aiecode2025/sha256/a172d2bcfed6f9b041d8802ab054a72ecd933dc0d21d18576f6df9860c063469.webp)

Evidence / The Experiment

Same corpus, indexed at

50  100  200  500  1000  2000 Tokens

Stop chunking like it’s 2022
AI21

### Across multiple corpora

The experiment compares chunk sizes across QMSum meeting transcripts and NarrativeQA questions about novels.

[6:33](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=393s) [The experiment slide identifies QMSum as a corpus of meeting transcripts.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2f2e5ed461d369c0fa1a98701d7f2d562a7c53e30a981c72f455bde1e11ef88.webp)

Evidence / The Experiment

Across multiple corpora

QMSum
meeting transcripts

Stop chunking like it’s 2022
AI21

[6:37](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=397s) [Slide listing QMSum meeting transcripts and NarrativeQA novels as experiment corpora.](https://assets.ai.engineer/sites/aiecode2025/sha256/366b63109719f83302695453c3ea04d47a9cf98720cfae27b680c1d7a26fee7b.webp)

Evidence / The Experiment

Across multiple corpora

QMSum
meeting transcripts

NarrativeQA
novels

Stop chunking like it’s 2022
AI21

### Experiment corpora

The experiment tests different chunk sizes across meeting transcripts, novels and Seinfeld trivia. The Seinfeld dataset contains questions about the show's transcripts and was built in-house.

[6:42](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=402s) [Slide listing QMSum meeting transcripts, NarrativeQA novels, and Seinfeld trivia.](https://assets.ai.engineer/sites/aiecode2025/sha256/5dcfb33664782a9610f59440315b518934a587be1aa3dc5009f8dc2089a795ed.webp)

Evidence / The Experiment

Across multiple corpora

QMSum
meeting transcripts

NarrativeQA
novels

Seinfeld
trivia about nothing

Stop chunking like it’s 2022
AI21

### Experiment corpora with Seinfeld meme

The experiment tested chunk sizes across meeting transcripts, novels, and an in-house Seinfeld trivia dataset built from episode transcripts.

[6:46](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=406s) [QMSum and NarrativeQA cards beside a Seinfeld meme about appearing in a RAG benchmark.](https://assets.ai.engineer/sites/aiecode2025/sha256/bdcbdafdeb8a5978b5625b63d773bdd1596b7d4b22f5889966473c10d32d8afb.webp)

Evidence / The Experiment

Across multiple corpora

QMSum
meeting transcripts

NarrativeQA
novels

IF ONLY I KNEW THAT I'D END UP
IN A RAG BENCHMARK

Stop chunking like it’s 2022
AI21

### Different questions favor different chunk sizes

For Jerry’s shirt, 100-token chunks rank first and 500-token chunks rank beyond 50. For Newman, whose name appears further down the scene, 500-token chunks rank first and 100-token chunks rank 38.

[7:01](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=421s) [Two Seinfeld questions show that 100-token and 500-token chunks each rank first for a different question.](https://assets.ai.engineer/sites/aiecode2025/sha256/65ba78419f1f423361627a2544649561a2be9754ef18b7dd3e67234dfcf771e4.webp)

Seinfeld / Trivia about nothing

Different questions want different chunk sizes.

“What’s the name for Jerry’s favorite shirt?”
JERRY: “It’s my best one, I call him Golden Boy.”
The whole answer sits in one short line.

100 TOKENS: rank 1
500 TOKENS: rank >50

“Who does Jerry describe as his nemesis and ‘pure evil’?”
JERRY: “I’ve looked into his eyes. He’s pure evil.” …
“Hellooo Jerry...”
The name - Newman - is further down the scene; only the larger window reaches it.

100 TOKENS: rank 38
500 TOKENS: rank 1

No single size serves both.

Stop chunking like it’s 2022
AI21

### The oracle experiment

The oracle uses the known answer to pick the best chunk size for each query, estimating the potential of query-specific retrieval. It cannot be used in production because the answer is already known.

[8:05](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=485s) [An oracle uses the known answer to choose the winning chunk size for each query; the slide notes this is impossible in production.](https://assets.ai.engineer/sites/aiecode2025/sha256/380b66db1d681b869b62be7089146f2720ab647fe69215f43f5c707389490bbc.webp)

Evidence / The Oracle

For each query, peek at the answer. Pick the chunk size that wins.

(impossible in production - that’s the point)

ANSWER
vs. gold answer
CANDIDATE CHUNK SIZES

Stop chunking like it’s 2022

### Chunk size performance vs. Oracle upper bound

Across three datasets, fixed chunk-size curves intersect and stay below the oracle, which retrospectively picks the best size for each query. The gap shows potential recall gains from choosing chunk sizes per query.

[8:35](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=515s) [Recall at K versus retrieved chunks for several fixed chunk sizes and an orange oracle curve that stays above them.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b4fd819d48df12f5f3b5da45e73b9e3fb6c1b229f575a32cad3ef3661af989a.webp)

Chunk size performance vs. Oracle upper bound

QMSum
Legend: W=50, W=100, W=200, W=500, W=1000, ORACLE
Y-axis: Recall@K
Y-axis ticks: 40, 50, 60, 70, 80, 90, 100
X-axis ticks: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Recall (↑)
# of Retrieved Chunks

Stop chunking like it’s 2022
AI21

[9:08](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=548s) [Recall at K curves compare fixed chunk sizes with an oracle across QMSum, NarrativeQA, and Seinfeld.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca384babbfd847cda1513ca8eb01b4c7b091abfd3b9aac0eafa09256427943a9.webp)

Chunk size performance vs. Oracle upper bound

QMSum
Legend: W=50, W=100, W=200, W=500, W=1000, ORACLE
Y-axis: Recall@K
Y-axis ticks: 40, 50, 60, 70, 80, 90, 100
X-axis: K
X-axis ticks: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

NarrativeQA
Legend: W=100, W=200, W=500, W=1000, ORACLE
Y-axis: Recall@K
Y-axis ticks: 40, 50, 60, 70, 80, 90, 100
X-axis: K
X-axis ticks: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Seinfeld
Legend: W=50, W=100, W=200, W=500, W=1000, ORACLE
Y-axis: Recall@K
Y-axis ticks: 40, 50, 60, 70, 80, 90, 100
X-axis: K
X-axis ticks: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Stop chunking like it’s 2022
AI21

### The cost of choosing 512

Belfer calls 512 an arbitrary chunk size: the gap to the oracle represents the cost of choosing a fixed size.

[9:42](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=582s) [The slide states that the gap is what choosing 512 costs on every query.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1ef4883474bd8e8570edc1beb4d3d78fc21c2721fdf40df50a8b4c9af39544c.webp)

Evidence

That gap is what
512 costs you.

Every query. Every day.
Silently.

Stop chunking like it’s 2022
AI21

### Stop choosing one size

Belfer argues for moving beyond a single chunk size: at indexing time, future queries are unknown; at retrieval time, the chunk size is already fixed.

[9:53](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=593s) [The slide says the right size cannot be known at query time and urges viewers to stop choosing one.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1e7410c830e03ccbfa39300eeaaf4a90b452bcab8d3815f93124d1baa4cc6e8.webp)

The Fix

We can’t know the right size at query time.
So stop choosing.

Stop chunking like it’s 2022
AI21

### Index time and query time

Indexing fixes the chunk size before the queries are known. At retrieval time, the query is available, but the chunk size is already locked.

[10:08](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=608s) [Indexing converts documents into chunks, embeddings, and an index with a locked chunk size; retrieval searches that index for each query.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dba29295de579534ada08e0fd3817652117e5e5335401fd15cf3ecd4c8b8872.webp)

The Bet

Index time and query time

Offline · Once
Indexing
Documents → Chunks → Embeddings → Index
Chunk size – Locked

Online · Every Query
Retrieval
Query → Search → Ranked results

Stop chunking like it’s 2022
AI21

### Prior work and chunk size

Contextual Retrieval enriches chunks, Late Chunking works in latent space, and RAPTOR uses hierarchical summaries. The slide argues that all three still choose a base chunk size.

[10:40](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=640s) [Contextual Retrieval, Late Chunking, and RAPTOR are compared, with the claim that each picks a base chunk size.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fd30d82bd7c337349d0c053abb4eef58ec71158f758013578529809c59b0a82.webp)

The Fix

Prior work is clever.
And still commits to one size.

Contextual Retrieval (Anthropic) — enriches chunks
Late Chunking (Jina) — chunks in latent space
RAPTOR — hierarchical summaries

They all pick a base chunk size. We don't.

Stop chunking like it’s 2022
AI21

### Multi-scale indexing

Multi-scale indexing chunks copies of the database at different window sizes into separate indices, then queries each index at retrieval time. The slide says it requires no learned chunking, LLM enrichment, or retraining.

[11:02](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=662s) [Multi-scale indexing uses sliding-window chunking at multiple sizes with separate indices, without learned chunking, LLM enrichment, or retraining.](https://assets.ai.engineer/sites/aiecode2025/sha256/78cbf5caf6675886ea4ea6d234c15f1c22915ac4a0a64c1d8ca8b7d2c871e7eb.webp)

The Fix

Multi–scale indexing

Same sliding–window chunking.
Multiple sizes. Separate indices.

No learned chunking
No LLM enrichment
No retraining

Stop chunking like it’s 2022
AI21

## Full transcript

[0:13](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=13s) Hi, everybody. Uh, thank you for, uh, coming today. Uh, welcome to a talk about nothing... Sorry, a talk about retrieval. Uh, my name is Yuval. Uh, I work at AI21, which is essentially an AI research lab. And today, I want to talk to you about something that most people don't wanna talk about, which is chunking. And I hope to convince you by the end that chunking isn't dead, and there

[0:43](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=43s) is something to do with that. And really, if you are at X, LinkedIn, wherever, you've probably have seen that RAG is dead, right? I think people also killed MCP lately. And RAG is dead again. Long live agentic, uh, retrieval, agentic search. And there is co- there come a time where you have to ask yourself, "How many times can RAG die?" Right? And even when someone says, "Well, RAG isn't dead," like,

[1:13](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=73s) uh, Jerry, the CEO of LlamaIndex, they still have to kill something, and apparently this something is chunking. Like, don't invest in it, don't do it. And this is the reason that people said that chunking is dead because everybody's using agentic search now, right? You have greps, you have ls, you have finds. All of these are great, but these are still not enough if you have a lot of data and you have

[1:42](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=102s) various amount of queries.

[1:46](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=106s) Just a second. Okay. And I think that the main reason that a lot of people don't like to talk about chunking, it's because it's not the fun part, right? In every RAG or, uh, files, uh, system, we have two stages. The first stage is the, like, the boring one, as you may. The one you do in the beginning. You have a lot of data. You have to pre-process it, you have to decide on the chunk size, and then you have to store everything in a vector DB. The

[2:16](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=136s) other part is the retrieval part, essentially the, the one that happens per query. This is something which is much easier to do, right? It's much easier to optimize. You can use all your queries, and then you can play with the max K, uh, top K, sorry. You can, uh, play with a hybrid search maybe, those kind of things. Much more fun to do retrieval tuning, right? Uh, so I will claim that if we have to kill something, if something has to be dead,

[2:46](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=166s) then it's probably retrieval tuning. And yes, agentic search probably killed that. And but still agentic search, even if we can accept the fact that it killed agen- uh, retrieval tuning, it's still not good enough when you have a lot of, right, scale, a lot of data. It costs a lot of money. I, I don't think I have to mention that anymore. Token maxing is, like, something that everybody's talking about. And the thing underneath, which is

[3:16](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=196s) if the data itself is not ordered in a right way, in your folders, in your directories, you still get something which is inefficient. So let's try to think of a, like, a timely example, right? The FIFA World Cup, uh, is now, and let's imagine that we have a dataset that contains of all the FIFA World Cup. So every directory is the, let's say, the '98 one, the 2002 one, and so on and so on. But if your query

[3:46](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=226s) asks whi- how, which team won the most World Cups, you can't just go to a folder and ask that. You have to go to every folder, see who won, and then aggregate this together, which is very inefficient. The answer, by the way, is Brazil, I hope, at least according to-  Whoo. Yeah, uh,  according to the time that this, uh, conversation's happening. So retrieval didn't actually die, okay? We're not killing anything in this, uh, lecture. It

[4:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=255s) is... got demoted into plumbing. And I think that everybody who worked on any RAG system know the feeling. Day one or week one or maybe even month one if you're very thorough, you're picking some sort of a chunk size. Let's say 512. Uh, maybe you're probably putting some overlap, right? 10, 20%, so on, indexing everything and forget all about it. And you can... Right? We talk a lot about the fixed, uh, chunking

[4:46](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=286s) strategies where if you chunk, uh, something which is too big, right? So you get the whole picture, which is nice, but you're losing a lot of the nuance and all the chunks will not get meaningful embeddings. Where if you will choose your chunks to be too small, you're getting the big picture lost and really it won't be as efficient. So what this tells us is that chunking is essentially a lossy compression. No matter what we're doing, we're losing something.

[5:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=315s) And I will, I will claim that there is no right chunk size. And a lot of you who worked on data will say, "No, but we have this, uh, corpus, we have this dataset, and we really used and we optimized our system to work very, very well on this data." And we thought so too. We had a lot of experience with it, with a lot of different types of agents and systems and workflows that you can really... and, hmm, right? You think about benchmarks, how easy it is to overfit

[5:45](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=345s) your model to a benchmark. But not with, uh, RAG. It doesn't happen there. And you cannot really optimize it per dataset, and I will claim that it is query-dependent. And how can I be so sure? How can I claim such a thing? Because we ran experiments and we tested, and now I'm gonna present it to you. So what we did, instead of saying what is the best chunk size per data, let's find out. Let's, let's actually take a dataset

[6:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=375s) and dupl-duplicate this dataset several times, in this case, six times. In every duplication, in every instance, the chunk size is different. So we have a database with a chunk size of two thousand, a database with a chunk size of one thousand, and so on and so on. And we did it with several datasets, so QMSum, which is a meeting transcript dataset, NarrativeQA, which is question answering on, uh, novels, and Seinfeld dataset, which is trivia

[6:45](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=405s) about, uh, nothing. Uh, not really. It's, uh, trivia, qu-trivia questions about the transcripts of, uh, Seinfeld. It's, uh, kind of a trolling dataset that we built, uh, in-house. We also published it if anybody wants the, the link at the end. And we tested on all of them to see what happens. And first of all, we just wanted to see, for every dataset, which chunk size is the best. And what we're seeing here is an example from the Seinfeld dataset, where essentially two

[7:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=435s) queries which are different by nature get different results, uh, based on the chunk size. So the first question, what is the name for Jerry's favorite shirt? You can see this is a very focused question, very specific question. The answer to it is probably very contained, and this is something that a smaller chunk size will do best in. And you can see, uh, rank one versus rank below fifty, uh, between one hundred tokens, uh, fixed at chunk size to one hundred.

[7:45](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=465s) Whereas a question like wh-who does Jerry describe as his nemesis and pure evil, which I'm not even that big of a Seinfeld fan, and I know it's Newman, uh, but if you look at the transcript, it's not something you can find that easily. And you can see that it really changes, right? If you use small chunk size, you will not get the answer. And what we did to really... After we ran all of these things and we've noticed that, we said, "What if we had an

[8:14](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=494s) oracle or a genie, if you want, that can tell us for every query what is the best chunk size to do retrieval for?" This essentially is the oracle experiment. This is what we wanted to know to see the potential. This is not... Right? We already have the answers, so we're not actually building a system here. We just want to see what is the potential that we have here. And what you can see here, 'kay, uh, in this, uh, graph, all the blue... First of all, the Y-axis is the recall. Higher is

[8:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=524s) better. Uh, the X-axis is the number of retrieved chunks, so it's recall at K versus K. You can see all the blue lines, probably indistinguishable, but each of them is a performance for a fixed chunk size, whereas the orange one is the oracle line. This is for every query, we took the best one out of these. And you can see it happens across several datasets. In a lot of them, you can actually see that the blue lines

[9:14](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=554s) inter, uh, intersect with each other, meaning that indeed for a lot of the datasets, no chunk size actually dominates. And what's more interesting is that there's a lot of potential. The gap which you can see between the orange line and all the blue lines is big. And when I say big, it's something like twenty to forty percent just from doing strategy on chunking, and very simple strategy, may I add. And

[9:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=584s) this is, like, the, this gap, this is what the choice of five hundred and twelve or a thousand or whatever, right? This number is just arbitrary. This is what it costs you. And I think that the, the problem here is... Like, it's a bit tricky because it's kind of like a p- uh, an information problem, that we don't have the information that we need at every stage. And what do I mean by that? If I'm looking at the indexing part where I do have control over the chunk

[10:14](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=614s) size, I don't know what the queries will be. I can guess. I can maybe estimate. I, I can try, but I don't know what the queries will be, so I cannot adjust my, uh, chunk size accordingly. And the retrieval part where I do have my queries, I cannot control the chunk size, right? It's already fixed, and I obviously will not do the entire process per query from the beginning. So, uh, we looked at, uh, prior works

[10:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=644s) such as notably Anthropic, uh, contextual retrieval, where they intru- enrich every chunk, and others that essentially try to improve the latent space, uh, of every chunk, uh, but this is not the direction that we went. All of them just stayed in the model of let's work with a fixed, uh, chunk size, whereas we took a different approach and we said, "Why commit to one when  we can commit to several?" And we call it the multi-scale indexing.

[11:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=675s) Es-essentially, we're just doing what we've seen before. So we're checking the database. We duplicate it and chunk it, uh, with several chu-chunk sizes or window sizes. And then... Sorry, um... And then this is what happens at the indexing, and then at retrieval time, we are querying all of them. So if we had N, uh, duplicates of the database and window sizes, we now have to run six different

[11:45](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=705s) retrieval calls per query. Uh, sorry, six is N. And how do we combine them? We, we obviously cannot use the oracle, right? The oracle is something that we have just for potential. In real life, we don't know the answer. Uh, but what we can do is to find some sort of merging algorithm. Now you would say when we look at it like this- What can be the issue? The fact that we have s- N ranking, but the rankings are for

[12:14](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=734s) chunks, and chunks with different sizes are not really comparable, right? So instead, we opted to do something which is pretty popular these days, and a lot of the RAG systems actually work like this, that instead of just retrieving the chunk, when we're getting a chunk, we're retrieving the entire document. Right? When context window grows, we want to give more and more context. And now, in this case, we have N, right, N, uh, rankings of the same documents because they're not

[12:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=764s) chunks anymore, and this we can compare. And in this case, you can think of retrieval as essentially just voting. Right? So it's not purely ranking. We don't have round ranking and then we're doing a re-rank. We're having N different ranks of the relevant documents, and we want to aggregate them all into one. That's why we are using something called RRF, Reciprocal Rank Fusion, okay, which is pretty much a simple, uh, formula. We tried several things.

[13:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=795s) This worked the best, and as you can see, it's not a model. It's not something, uh, that you have to do specifically, like, especially, this is just a simple script that takes really no time. And this is how the full, uh, how the full system looks like. So we have the indexing N times, then, uh, we query each query from every database, and we're using RRF to combine them all. And the results, you can guess that

[13:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=824s) they're good, otherwise I would not, uh, standing here and, uh-  ... being way too much, uh, uh, confident. Right? But you can see we tested across several datasets, uh, QMSum, NarrativeQA, Seinfeld, and also, uh, FinanceBench. Uh, we took all of them, and it matches the be- or beats the best fixed, uh, size. Let's see it in a graph. Uh, it's a bit hard to see here, so I'll walk it slowly. Every row here is a chunk

[14:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=855s) size, so you can see fifty, one hundred, and so on. The bottom row is our method, this one, the one that you t- do from all of them and then combine. And the... every column is a recall at something, so recall at one, two, three, up until ten. What you can see here is that two things, right? First of all, that across, m- like, recall at whatever, uh, our method still wins, which you can think

[14:45](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=885s) is very easy, but the fact that you have to combine all of them is not very... it's not something which is very trivial. And also, you can see that the quality actually increases. The heat map where you can see it's become much greener. And again, this was just something that I wanted to show in large. Here you can see, uh, all four of the datasets where we do achieve better results, uh, really quite, like, twenty, thirty, forty percent even in a lot of the things. Uh, also there are results that I

[15:15](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=915s) did not show you here, uh, which, uh, are on MTEB. Um, you can see in our blog, I will put the link later. We're getting there also a lot of improvements, somewhere between ten to forty percent depending on the dataset. Now, I'm, I'm, I'm not naive. I'm not gonna claim here that this costs nothing. Obviously, there is a cost, right? No free lunches. Everything has to come with something. And yes, this costs with extra memory. It costs something

[15:44](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=944s) between two to five to... O of one, right? A constant of additional memory where you have to keep all of those, uh, all those, uh, copies of the database. However, if you think about it latency-wise, it doesn't really affect that because you can do all the retrieval part parallelly, and also the RRF part doesn't really take a lot of time.

[16:10](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=970s) Uh, I will say that this was a very nice research project that we did, and we've got really, really cool results. There are things to do, right? There are places to improve. There are future work, uh, to do. More precisely, we want to understand how many chunk sizes do we want and, and which, right? The, the fact that we worked with fifty, one hundred, two hundred and so on was pretty arbitrary, uh, to be honest. So we do need to figure out how to compute this

[16:40](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=1000s) and how to know how many copies exactly do you need. Uh, also go beyond RRF, right? The fact that we're using RRF is because it worked the best from the methods that we used, but it doesn't mean that there is no better method. And if I need to leave you with something, I would say that agents didn't kill retrieval. N- nothing died. Come on. Uh, just infrastructure. And the part, the ba- bad part is that it's infrastructure from 2022,

[17:10](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=1030s) and with really simple, simple methods, you can take your RAG system or anything that has to do with storing data and then retrieve it with twenty to forty percent, again, without any, you know, something too sophisticated. So if you, uh, want to hear more about or read more about it, uh, you can read the, the blog. There is also an example code there and the Seinfeld dataset. Um, that's it. I'm Yuval. Thank you so much, uh, being

[17:40](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=1060s) here.
