Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
Read the talk
Stop Choosing One Chunk Size: Multiscale Retrieval with Yuval Belfer
Chunk size changes what a retrieval system can find. Yuval Belfer explains why the useful scale depends on the query, how several indexes can vote on whole documents, and what the resulting recall gains cost.
From a talk by Yuval Belfer
At a glance
Ideas worth remembering
Chunk size trades focused detail against surrounding context. Different questions about the same corpus can need opposite choices.
The oracle's reported 20%–40% recall gap measures potential using known answers; it is not an operational method for selecting a scale.
Multiscale indexing searches N differently chunked indexes, maps matches to whole documents, and combines the document rankings with Reciprocal Rank Fusion.
Belfer reports matching or beating the best fixed size across four datasets, at roughly two to five times the memory. Parallel searches limit added latency, while the number of scales and the best fusion method remain open questions.
Agentic search still depends on how data is organized
Yuval Belfer of AI21 Labs opens by questioning the recurring claim that retrieval-augmented generation, or RAG, has become obsolete. Agents can search files with tools such as grep, ls, and find, but those tools do not remove the problem of finding relevant information across a large corpus with varied questions. His subject is the less glamorous work underneath that search: how a system divides and indexes its data.
He separates the system into two stages. Indexing happens before a query arrives: preprocess the corpus, choose a chunk size, and store the resulting representations in a vector database. Retrieval happens for each question. At that point, developers can experiment with top K—the number of results returned—or hybrid search. Retrieval tuning is attractive because the questions are available and the effects of a change are easier to inspect. The earlier decision about what each searchable unit contains tends to receive less attention.
A directory of World Cup records illustrates why an agent's search tools cannot compensate automatically for every organizational choice. Suppose each tournament has its own folder, including 1998 and 2002. A question about which team has won the most tournaments requires visiting the folders, extracting each winner, and aggregating the results. No single tournament folder answers the question. Belfer's point is about the work required: an organization that suits one kind of lookup can make a cross-cutting question expensive, even when an agent can navigate it correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A fixed chunk size decides which information survives
Belfer describes a familiar setup: choose a size such as 512, add perhaps 10% or 20% overlap, index the corpus, and move on. That choice remains inside the system while attention shifts toward retrieval behavior. The danger is that it quietly determines which relationships are represented together and which details have to compete inside a larger passage.
Large chunks preserve more surrounding context, but their embeddings can lose the distinctions that make a small fact searchable. Small chunks give individual details a clearer representation, but can separate those details from the context needed to understand them. Belfer calls chunking lossy compression: either choice sacrifices something. The loss concerns how the text is represented for retrieval, rather than a claim that the original documents have disappeared.
His central claim is that the useful chunk size depends on the query. A corpus-level optimum can conceal different requirements within the same dataset. Optimizing a fixed size against familiar questions may produce good benchmark results while leaving other kinds of questions poorly served. The experiment that follows tests whether choosing a different scale for each question would offer meaningful headroom.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two Seinfeld questions need different windows
The team created six differently chunked instances of each dataset. Belfer mentions sizes including 2,000 and 1,000, along with smaller windows. They tested QMSum, a meeting-transcript dataset; NarrativeQA, involving questions about novels; and an in-house Seinfeld dataset containing trivia questions about episode transcripts. The purpose was to compare retrieval behavior across scales on the same underlying material.
The Seinfeld examples make the tradeoff concrete. A focused question asking for the name of something associated with Jerry performs best with a small window: Belfer reports rank one with 100-token chunks, compared with a position beyond the first 50 results at larger scales. His explanation is that a contained answer benefits from a representation that keeps attention on the relevant detail.
A second question asks whom Jerry describes as his nemesis and pure evil. The answer is Newman, but Belfer reports that small chunks fail to retrieve the answer. Recognizing the relationship needs more surrounding transcript context than the narrow factual lookup does. These examples expose the difficulty of selecting one size for the entire corpus: the same transcripts support questions with opposite retrieval needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The oracle measures the cost of committing too early
To measure the opportunity, the team constructed an oracle experiment. For each query, they used the known answer to select whichever chunk size produced the best retrieval result. This is a retrospective comparison, not a deployable selector: a real system does not know the answer before searching. The oracle asks how well the tested collection of scales could perform if the choice were always favorable.
Belfer describes plots of recall at K against K, where K is the number of retrieved chunks. Each fixed chunk size has its own curve, and the oracle combines the best per-query outcomes. Several fixed-size curves intersect, so one size does not dominate across all the retrieval cutoffs shown. The oracle also benefits from differences between individual questions that a dataset-wide average can hide.
He reports a gap of roughly 20% to 40% between the oracle and fixed-size retrieval. That is evidence of potential within the tested settings, rather than a promise that every corpus loses that amount by choosing 512. The talk does not specify whether these percentages represent relative gains or percentage-point differences, so they should remain reported ranges rather than a more precise numerical conclusion.
The timing of the decision creates the problem. During indexing, the system controls chunk size but does not yet know the incoming questions. During retrieval, it knows the question but the chunks are already fixed. Rebuilding the corpus for each query would repeat the indexing work. Belfer contrasts the team's approach with methods such as Anthropic's contextual retrieval, which he describes as enriching chunks to improve their representations. His team instead changes how many scales the system makes available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search every scale, then let documents receive the votes
Multiscale indexing keeps several differently chunked versions of the corpus. With N scales, indexing produces N searchable instances and each incoming query triggers N retrieval calls. The six copies in the experiment are one choice of N. This preserves several ways to find a document without requiring the system to predict the correct chunk size before it searches.
The next decision is how to merge the results. A short chunk and a long chunk are different retrieval units, so their ranked lists cannot simply be treated as rankings of identical objects. The team resolves this by retrieving the whole document associated with a chunk match. Each scale then produces a ranking of the same document identities. The chunks serve as different routes to those documents, while the documents become the common units for fusion.
Belfer frames the combination as voting. Each scale supplies its ranking of relevant documents, and Reciprocal Rank Fusion, or RRF, aggregates those rankings into one list. The system therefore avoids needing an oracle to choose a winning scale. It also avoids adding a learned reranking model for this step: Belfer describes the fusion as a simple script and says RRF worked best among the merging methods the team tried.
Returning whole documents also gives the downstream system more context, which Belfer connects to growing model context windows. This separates the size of the passage used to find a document from the amount of text supplied afterward. It does not establish that every document will fit a downstream context window, and the talk does not quantify the token cost of passing whole documents onward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recall improves, while memory and scale selection remain the tradeoffs
The final evaluation includes QMSum, NarrativeQA, Seinfeld, and FinanceBench. Belfer reports that the combined method matches or beats the best fixed chunk size across these datasets. He describes comparisons at recall cutoffs from one through ten and gains reaching roughly 20%, 30%, or 40% in many of the tested settings. These are retrieval results; the talk does not establish an equivalent improvement in the accuracy of generated answers.
He also mentions MTEB results with improvements between 10% and 40%, depending on the dataset, but does not present those results in the recording. They extend his account of the research without supplying the same walkthrough as the four named evaluations. The practical result is also distinct from the oracle experiment: fusion must combine imperfect rankings without knowing which scale contains the answer.
The explicit cost is memory. Keeping the additional indexes requires roughly two to five times the memory, as Belfer describes it. For a fixed collection of scales, this is a multiplier on storage requirements; it still grows as the corpus grows. The method spends memory to preserve several retrieval representations of the same material.
Belfer reports little added retrieval latency because the searches can run in parallel and RRF adds little computation. Parallel execution explains how N searches can finish without taking N times as long, but the system still performs N retrieval calls. The recording does not provide timing measurements or quantify indexing cost and concurrent serving capacity, so the latency claim should not be read as a claim that the additional work is free.
Two choices remain open. The tested window sizes—including 50, 100, and 200—were fairly arbitrary, and the team still wants a way to determine which sizes to use and how many copies are necessary. RRF is also a successful choice among the methods they tested, rather than a demonstrated best possible merger. Both questions matter because extra scales consume memory and retrieval work, while their value depends on what new information they contribute.
Belfer ends by returning retrieval to its role as infrastructure for agents. Better search behavior can come from revisiting a simple indexing decision that was made early and then forgotten. Multiscale indexing supplies a concrete alternative: retain several views of the corpus, search them together, and combine their document rankings. His reported gains make that experiment worth considering, while the memory cost and unfinished scale-selection problem define what a team still needs to evaluate for its own workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:13
Hi, everybody. Uh, thank you for, uh, coming today. Uh, welcome to a talk about nothing... Sorry, a talk about retrieval. Uh, my name is Yuval. Uh, I work at AI21, which is essentially an AI research lab. And today, I want to talk to you about something that most people don't wanna talk about, which is chunking. And I hope to convince you by the end that chunking isn't dead, and there
- 0:43
is something to do with that. And really, if you are at X, LinkedIn, wherever, you've probably have seen that RAG is dead, right? I think people also killed MCP lately. And RAG is dead again. Long live agentic, uh, retrieval, agentic search. And there is co- there come a time where you have to ask yourself, "How many times can RAG die?" Right? And even when someone says, "Well, RAG isn't dead," like,
- 1:13
uh, Jerry, the CEO of LlamaIndex, they still have to kill something, and apparently this something is chunking. Like, don't invest in it, don't do it. And this is the reason that people said that chunking is dead because everybody's using agentic search now, right? You have greps, you have ls, you have finds. All of these are great, but these are still not enough if you have a lot of data and you have
- 1:42
various amount of queries.
- 1:46
Just a second. Okay. And I think that the main reason that a lot of people don't like to talk about chunking, it's because it's not the fun part, right? In every RAG or, uh, files, uh, system, we have two stages. The first stage is the, like, the boring one, as you may. The one you do in the beginning. You have a lot of data. You have to pre-process it, you have to decide on the chunk size, and then you have to store everything in a vector DB. The
- 2:16
other part is the retrieval part, essentially the, the one that happens per query. This is something which is much easier to do, right? It's much easier to optimize. You can use all your queries, and then you can play with the max K, uh, top K, sorry. You can, uh, play with a hybrid search maybe, those kind of things. Much more fun to do retrieval tuning, right? Uh, so I will claim that if we have to kill something, if something has to be dead,
- 2:46
then it's probably retrieval tuning. And yes, agentic search probably killed that. And but still agentic search, even if we can accept the fact that it killed agen- uh, retrieval tuning, it's still not good enough when you have a lot of, right, scale, a lot of data. It costs a lot of money. I, I don't think I have to mention that anymore. Token maxing is, like, something that everybody's talking about. And the thing underneath, which is
- 3:16
if the data itself is not ordered in a right way, in your folders, in your directories, you still get something which is inefficient. So let's try to think of a, like, a timely example, right? The FIFA World Cup, uh, is now, and let's imagine that we have a dataset that contains of all the FIFA World Cup. So every directory is the, let's say, the '98 one, the 2002 one, and so on and so on. But if your query
- 3:46
asks whi- how, which team won the most World Cups, you can't just go to a folder and ask that. You have to go to every folder, see who won, and then aggregate this together, which is very inefficient. The answer, by the way, is Brazil, I hope, at least according to- Whoo. Yeah, uh, according to the time that this, uh, conversation's happening. So retrieval didn't actually die, okay? We're not killing anything in this, uh, lecture. It
- 4:15
is... got demoted into plumbing. And I think that everybody who worked on any RAG system know the feeling. Day one or week one or maybe even month one if you're very thorough, you're picking some sort of a chunk size. Let's say 512. Uh, maybe you're probably putting some overlap, right? 10, 20%, so on, indexing everything and forget all about it. And you can... Right? We talk a lot about the fixed, uh, chunking
- 4:46
strategies where if you chunk, uh, something which is too big, right? So you get the whole picture, which is nice, but you're losing a lot of the nuance and all the chunks will not get meaningful embeddings. Where if you will choose your chunks to be too small, you're getting the big picture lost and really it won't be as efficient. So what this tells us is that chunking is essentially a lossy compression. No matter what we're doing, we're losing something.
- 5:15
And I will, I will claim that there is no right chunk size. And a lot of you who worked on data will say, "No, but we have this, uh, corpus, we have this dataset, and we really used and we optimized our system to work very, very well on this data." And we thought so too. We had a lot of experience with it, with a lot of different types of agents and systems and workflows that you can really... and, hmm, right? You think about benchmarks, how easy it is to overfit
- 5:45
your model to a benchmark. But not with, uh, RAG. It doesn't happen there. And you cannot really optimize it per dataset, and I will claim that it is query-dependent. And how can I be so sure? How can I claim such a thing? Because we ran experiments and we tested, and now I'm gonna present it to you. So what we did, instead of saying what is the best chunk size per data, let's find out. Let's, let's actually take a dataset
- 6:15
and dupl-duplicate this dataset several times, in this case, six times. In every duplication, in every instance, the chunk size is different. So we have a database with a chunk size of two thousand, a database with a chunk size of one thousand, and so on and so on. And we did it with several datasets, so QMSum, which is a meeting transcript dataset, NarrativeQA, which is question answering on, uh, novels, and Seinfeld dataset, which is trivia
- 6:45
about, uh, nothing. Uh, not really. It's, uh, trivia, qu-trivia questions about the transcripts of, uh, Seinfeld. It's, uh, kind of a trolling dataset that we built, uh, in-house. We also published it if anybody wants the, the link at the end. And we tested on all of them to see what happens. And first of all, we just wanted to see, for every dataset, which chunk size is the best. And what we're seeing here is an example from the Seinfeld dataset, where essentially two
- 7:15
queries which are different by nature get different results, uh, based on the chunk size. So the first question, what is the name for Jerry's favorite shirt? You can see this is a very focused question, very specific question. The answer to it is probably very contained, and this is something that a smaller chunk size will do best in. And you can see, uh, rank one versus rank below fifty, uh, between one hundred tokens, uh, fixed at chunk size to one hundred.
- 7:45
Whereas a question like wh-who does Jerry describe as his nemesis and pure evil, which I'm not even that big of a Seinfeld fan, and I know it's Newman, uh, but if you look at the transcript, it's not something you can find that easily. And you can see that it really changes, right? If you use small chunk size, you will not get the answer. And what we did to really... After we ran all of these things and we've noticed that, we said, "What if we had an
- 8:14
oracle or a genie, if you want, that can tell us for every query what is the best chunk size to do retrieval for?" This essentially is the oracle experiment. This is what we wanted to know to see the potential. This is not... Right? We already have the answers, so we're not actually building a system here. We just want to see what is the potential that we have here. And what you can see here, 'kay, uh, in this, uh, graph, all the blue... First of all, the Y-axis is the recall. Higher is
- 8:44
better. Uh, the X-axis is the number of retrieved chunks, so it's recall at K versus K. You can see all the blue lines, probably indistinguishable, but each of them is a performance for a fixed chunk size, whereas the orange one is the oracle line. This is for every query, we took the best one out of these. And you can see it happens across several datasets. In a lot of them, you can actually see that the blue lines
- 9:14
inter, uh, intersect with each other, meaning that indeed for a lot of the datasets, no chunk size actually dominates. And what's more interesting is that there's a lot of potential. The gap which you can see between the orange line and all the blue lines is big. And when I say big, it's something like twenty to forty percent just from doing strategy on chunking, and very simple strategy, may I add. And
- 9:44
this is, like, the, this gap, this is what the choice of five hundred and twelve or a thousand or whatever, right? This number is just arbitrary. This is what it costs you. And I think that the, the problem here is... Like, it's a bit tricky because it's kind of like a p- uh, an information problem, that we don't have the information that we need at every stage. And what do I mean by that? If I'm looking at the indexing part where I do have control over the chunk
- 10:14
size, I don't know what the queries will be. I can guess. I can maybe estimate. I, I can try, but I don't know what the queries will be, so I cannot adjust my, uh, chunk size accordingly. And the retrieval part where I do have my queries, I cannot control the chunk size, right? It's already fixed, and I obviously will not do the entire process per query from the beginning. So, uh, we looked at, uh, prior works
- 10:44
such as notably Anthropic, uh, contextual retrieval, where they intru- enrich every chunk, and others that essentially try to improve the latent space, uh, of every chunk, uh, but this is not the direction that we went. All of them just stayed in the model of let's work with a fixed, uh, chunk size, whereas we took a different approach and we said, "Why commit to one when we can commit to several?" And we call it the multi-scale indexing.
- 11:15
Es-essentially, we're just doing what we've seen before. So we're checking the database. We duplicate it and chunk it, uh, with several chu-chunk sizes or window sizes. And then... Sorry, um... And then this is what happens at the indexing, and then at retrieval time, we are querying all of them. So if we had N, uh, duplicates of the database and window sizes, we now have to run six different
- 11:45
retrieval calls per query. Uh, sorry, six is N. And how do we combine them? We, we obviously cannot use the oracle, right? The oracle is something that we have just for potential. In real life, we don't know the answer. Uh, but what we can do is to find some sort of merging algorithm. Now you would say when we look at it like this- What can be the issue? The fact that we have s- N ranking, but the rankings are for
- 12:14
chunks, and chunks with different sizes are not really comparable, right? So instead, we opted to do something which is pretty popular these days, and a lot of the RAG systems actually work like this, that instead of just retrieving the chunk, when we're getting a chunk, we're retrieving the entire document. Right? When context window grows, we want to give more and more context. And now, in this case, we have N, right, N, uh, rankings of the same documents because they're not
- 12:44
chunks anymore, and this we can compare. And in this case, you can think of retrieval as essentially just voting. Right? So it's not purely ranking. We don't have round ranking and then we're doing a re-rank. We're having N different ranks of the relevant documents, and we want to aggregate them all into one. That's why we are using something called RRF, Reciprocal Rank Fusion, okay, which is pretty much a simple, uh, formula. We tried several things.
- 13:15
This worked the best, and as you can see, it's not a model. It's not something, uh, that you have to do specifically, like, especially, this is just a simple script that takes really no time. And this is how the full, uh, how the full system looks like. So we have the indexing N times, then, uh, we query each query from every database, and we're using RRF to combine them all. And the results, you can guess that
- 13:44
they're good, otherwise I would not, uh, standing here and, uh- ... being way too much, uh, uh, confident. Right? But you can see we tested across several datasets, uh, QMSum, NarrativeQA, Seinfeld, and also, uh, FinanceBench. Uh, we took all of them, and it matches the be- or beats the best fixed, uh, size. Let's see it in a graph. Uh, it's a bit hard to see here, so I'll walk it slowly. Every row here is a chunk
- 14:15
size, so you can see fifty, one hundred, and so on. The bottom row is our method, this one, the one that you t- do from all of them and then combine. And the... every column is a recall at something, so recall at one, two, three, up until ten. What you can see here is that two things, right? First of all, that across, m- like, recall at whatever, uh, our method still wins, which you can think
- 14:45
is very easy, but the fact that you have to combine all of them is not very... it's not something which is very trivial. And also, you can see that the quality actually increases. The heat map where you can see it's become much greener. And again, this was just something that I wanted to show in large. Here you can see, uh, all four of the datasets where we do achieve better results, uh, really quite, like, twenty, thirty, forty percent even in a lot of the things. Uh, also there are results that I
- 15:15
did not show you here, uh, which, uh, are on MTEB. Um, you can see in our blog, I will put the link later. We're getting there also a lot of improvements, somewhere between ten to forty percent depending on the dataset. Now, I'm, I'm, I'm not naive. I'm not gonna claim here that this costs nothing. Obviously, there is a cost, right? No free lunches. Everything has to come with something. And yes, this costs with extra memory. It costs something
- 15:44
between two to five to... O of one, right? A constant of additional memory where you have to keep all of those, uh, all those, uh, copies of the database. However, if you think about it latency-wise, it doesn't really affect that because you can do all the retrieval part parallelly, and also the RRF part doesn't really take a lot of time.
- 16:10
Uh, I will say that this was a very nice research project that we did, and we've got really, really cool results. There are things to do, right? There are places to improve. There are future work, uh, to do. More precisely, we want to understand how many chunk sizes do we want and, and which, right? The, the fact that we worked with fifty, one hundred, two hundred and so on was pretty arbitrary, uh, to be honest. So we do need to figure out how to compute this
- 16:40
and how to know how many copies exactly do you need. Uh, also go beyond RRF, right? The fact that we're using RRF is because it worked the best from the methods that we used, but it doesn't mean that there is no better method. And if I need to leave you with something, I would say that agents didn't kill retrieval. N- nothing died. Come on. Uh, just infrastructure. And the part, the ba- bad part is that it's infrastructure from 2022,
- 17:10
and with really simple, simple methods, you can take your RAG system or anything that has to do with storing data and then retrieve it with twenty to forty percent, again, without any, you know, something too sophisticated. So if you, uh, want to hear more about or read more about it, uh, you can read the, the blog. There is also an example code there and the Seinfeld dataset. Um, that's it. I'm Yuval. Thank you so much, uh, being
- 17:40
here.