Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs

Yuval Belfer of AI21 Labs opens with the recurring declarations that RAG—retrieval-augmented generation—is dead, followed by similar claims about chunking. Agents can explore files with tools such as grep, ls, and find. His objection concerns scale and variety: these tools alone do not make searching a large corpus efficient for many different kinds of questions. The work of dividing and indexing data still matters.

Indexing sets the units; retrieval searches them

• Indexing: Before questions arrive, preprocess the corpus, choose how much text each chunk contains, and store the resulting representations in a vector database. This determines the units the system can search.
• Retrieval: For each question, search those units. Developers can adjust top K, the number of results returned, or experiment with hybrid search. Because they already have the questions, they can inspect how these changes affect the results.

Belfer finds retrieval tuning easier to engage with: the questions are concrete and the effects are visible. The earlier decision about what each searchable unit contains receives less attention.

Agentic search does not remove the cost of a poor organization. Belfer imagines World Cup records arranged into a separate folder for each tournament, including 1998 and 2002. To answer which team won the most tournaments, an agent must visit the folders, find each winner, and aggregate the results. No individual tournament folder answers that question. An organization that suits a local lookup can make a cross-cutting question expensive, even when the agent navigates it correctly.

Retrieval has become plumbing, Belfer says, and chunking often becomes a decision made once and forgotten. The familiar setup is to choose a size such as 512, add 10% or 20% overlap, index everything, and move on. Overlap lets neighboring chunks share some text, but the system still commits to a particular window. That window determines which information receives a searchable representation together.

• Large chunks preserve surrounding context. They keep more of a passage together, but their embeddings can blur a specific detail among other material. The whole picture survives while a small fact becomes harder to retrieve.
• Small chunks concentrate local detail. A focused passage can represent a contained fact more clearly, but it can separate that fact from the broader context needed to understand a relationship.

Belfer calls chunking lossy compression. The loss concerns the searchable representation: choosing a window favors some information and weakens other information. His claim is that the useful size depends on the query, even within one corpus. A fixed size that performs well on familiar questions can hide failures on questions that need a different amount of context. The experiments that follow test how much this difference matters.

The team created six differently chunked instances of each dataset. Belfer names windows of 2,000 and 1,000 alongside smaller sizes. The initial datasets were QMSum, containing meeting transcripts; NarrativeQA, involving questions about novels; and an in-house Seinfeld trivia dataset built around episode transcripts. Holding the underlying material constant let the team compare what each scale could retrieve.

A contained fact and a contextual relationship

• Focused lookup: A question asking for the name of something associated with Jerry returns the relevant result at rank one with 100-token chunks, while larger windows place it beyond the first 50 results. Belfer's explanation is that a contained answer benefits from a narrow representation.
• Contextual lookup: A question asks whom Jerry describes as his nemesis and pure evil. Belfer identifies the answer as Newman, but reports that small chunks fail to retrieve it. Finding the relationship needs more surrounding transcript context than the focused lookup.

These questions make corpus-wide tuning an awkward compromise. A narrow window works for one question and fails for the other. The team therefore moves from asking which size wins on a dataset to asking how much performance is available when each query gets its favorable size.

The oracle experiment uses the known answers to choose the best-performing chunk size for each query. It measures headroom: how well could the tested collection of sizes perform if the system always made the favorable choice? A real retrieval system cannot use this selector because it does not know the answer before searching.

Belfer describes plots with recall on the vertical axis and the number of retrieved chunks, K, on the horizontal axis. Each fixed-size curve shows recall at K for one window size. The oracle curve uses the best per-query outcomes across the tested sizes. Some fixed-size curves intersect, showing that a size's advantage changes with the retrieval cutoff. The oracle also captures differences between individual questions that disappear into dataset-level averages.

He reports a roughly 20%–40% recall gap between the oracle and fixed-size retrieval in the tested settings. That range describes an opportunity within those experiments. It does not establish that every system using 512 loses that amount, and the recording does not distinguish relative improvements from percentage-point differences.

The question arrives after the window is frozen

The information problem has two ends. At indexing time, the system controls chunk size but does not know the incoming questions. At retrieval time, it has the question but the indexed windows are already fixed. Changing them would repeat the indexing work. The system can estimate future questions, but it cannot adjust a single preselected size to every query as it arrives.

Belfer contrasts the team's direction with Anthropic's contextual retrieval, which he describes as enriching chunks to improve their representations. His team instead keeps several window sizes available. Rather than making one chunk representation carry every kind of question, multiscale indexing supplies several searchable views of the same material.

Multiscale indexing turns the experiment's duplicated corpus into a retrieval strategy. Index the material at N window sizes, then send each incoming query to all N indexes. The six instances used in the experiment are one choice of N. The system searches every available scale instead of predicting a winning size and searching only that index.

Searching several indexes produces several rankings, but their entries initially refer to different objects. A short passage and a larger passage containing it are distinct chunks. Their lists do not rank a shared set of candidates. The team changes the unit of comparison: a chunk match retrieves its entire source document. Each scale can now rank the same document identities, even though it found them through differently sized passages.

Document rankings become votes

Belfer describes the remaining step as voting. Each scale supplies a ranking of relevant documents; Reciprocal Rank Fusion, or RRF, combines those rankings into one list. This replaces the oracle's answer-aware choice with an aggregation of the available retrieval results. RRF worked best among the merging methods the team tried. Belfer describes its implementation as a simple script, without an additional learned reranking model.

Whole-document retrieval also separates the passage used to find information from the text supplied afterward. A narrow chunk can locate a document while the full document supplies surrounding context. Belfer connects this choice to growing model context windows. The recording does not quantify the downstream token cost or explain how documents that exceed a model's context window would be handled.

The final evaluation covers QMSum, NarrativeQA, Seinfeld, and FinanceBench. Belfer reports that multiscale fusion matches or beats the best fixed size across those datasets. He walks through a heat map whose rows represent fixed chunk sizes and the combined method, while columns represent recall cutoffs from one through ten. Combining scales has to place useful documents high in the final ranking; merely finding them somewhere among N searches would not be enough.

Belfer reports improvements reaching roughly 20%, 30%, or 40% in many tested settings. These are retrieval results, so they do not establish equivalent gains in generated-answer accuracy. He also mentions MTEB improvements between 10% and 40%, depending on the dataset, but does not walk through those results in the recording. The practical fusion results remain separate from the oracle: they combine rankings without knowing which scale will find the answer.

Memory increases; searches can overlap in time

• Memory: Belfer gives a roughly two-to-five-times memory cost for keeping the additional indexed copies. With a fixed number of scales, this is a multiplier on the corpus's storage requirements; memory still grows with the corpus.
• Retrieval latency: He reports little added latency because the searches can run in parallel and RRF adds little computation. Parallelism lets searches overlap rather than run one after another. The system still performs N retrieval calls for each question.

The recording provides no timing measurements or accounting of indexing cost and concurrent serving capacity. Overlapping the searches explains how extra work can add little elapsed time, but does not eliminate that work.

Which scales, and which merger?

• Scale selection remains unfinished. Belfer says the tested choices, including 50, 100, and 200, were fairly arbitrary. The team still wants to determine which window sizes to use and how many indexed copies are necessary. Each additional scale spends memory and retrieval work, so their number matters alongside recall.
• RRF remains a tested choice. It performed best among the merging methods the team tried. Belfer explicitly leaves room for a better aggregation method; the experiments do not establish that RRF is the best possible merger.

Belfer's closing judgment returns to the infrastructure that agents inherit. Retrieval still matters, and a chunk size selected early should remain open to scrutiny. Multiscale indexing offers a simple experiment: keep several searchable views of the material, retrieve whole documents through each view, and combine their rankings. The reported recall gains give a reason to try it; the memory multiplier and unfinished scale selection determine what a team must weigh for its workload. He closes by pointing readers to the research blog, example code, and published Seinfeld dataset.