AI Engineer World's Fair 2025
Vector Search Benchmark[eting]
Read the talk
Vector Search Benchmark[eting]
Contradictory vector-search rankings often begin with different workloads, configurations and quality targets. A useful benchmark makes those conditions explicit.
From a talk by Philipp Krenn
Before you start: Familiarity with vector search and basic latency measurements will help; the article explains the approximate-search quality tradeoff.
How can every vendor be faster?
Why are vector-search benchmarks so hard to compare? When Philipp Krenn asks the audience about their experience, that is the problem they raise. One benchmark puts a product ahead; another reverses the ranking. In Krenn’s telling, almost any vendor can occupy either side of X is faster than Y.
Those contradictory winners pose a more useful question than whether everyone is lying: what changed between the comparisons? Workloads, settings and reporting choices can change the outcome. When a selected result becomes a broad sales claim, benchmarking becomes benchmarketing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Similar conditions can still favor one system
A glossy chart does not establish a fair comparison. Start with the use case: does the test represent what your system must do? Krenn illustrates the problem with a cartoon in which a squid and a housecat face puzzles in matching water tanks. The surroundings are similar, but their suitability for the participants is not. A benchmark can apply the same conditions to two systems while making one system’s strengths decisive.
The next step is selective publication. Run several scenarios, reject the ones where your system looks ordinary or weak, and keep the one where it wins. Then generalize that narrow result into a claim about everything. Krenn’s example of being forty percent faster is an invented marketing headline, not a measured finding presented in the talk. The missing information is the set of scenarios discarded along the way.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A read-only dataset leaves out operational work
Read-only tests are attractive because they are easier to repeat and compare. They can also use an optimized static representation of the data. Krenn observes that these tests dominate the vector-search benchmarks he encounters, even though many real workloads continue to change while serving queries.
Once writes enter the workload, the benchmark needs more than a dataset and a query set. It needs a representative read/write ratio. That adds parameters and makes the experiment harder to design, but omitting them changes the question being answered: performance against an optimized, stationary dataset may not represent performance while that dataset is being updated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Filtering can increase the search work
Filtering introduces a counterintuitive case. In a conventional datastore, a restrictive filter often reduces the work needed for subsequent ranking. Krenn asks us to imagine a filter that admits twenty percent of the records. For HNSW, an approximate nearest-neighbor graph index, fewer eligible records can instead mean more candidate exploration.
The search still needs to find enough acceptable neighbors. If candidates encountered during exploration fail the filter, it may have to examine more candidates before collecting the requested results.
| Search situation | Effect of a restrictive filter |
|---|---|
| Filter records, then rank survivors | Fewer records to rank |
| Explore an approximate-neighbor graph | More exploration may be needed to find eligible neighbors |
The distinction is between the size of the eligible result set and the work required to discover it. A smaller set does not guarantee a cheaper search.
This slowdown is conditional: implementations have filtering optimizations, and different filter scenarios exercise them differently. A benchmark publisher can select a filter ratio that suits its implementation or choose a workload that hits a recently added optimization. Read-only state, read/write mix and filter behavior therefore belong in the explanation of the result, not outside it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check which releases actually competed
A benchmark can change its winner simply by updating one side. Krenn reports seeing his company’s software represented by a roughly eighteen-month-old version while the publisher uses its own release from the preceding one or two months. This is an attributed example of unequal software freshness, not a comparison between identified releases.
Keeping competitors current requires work: someone must follow their releases, understand changed behavior and learn the appropriate configuration. It is natural for a team to concentrate on its own improvements. But leaving the competing systems untouched also conveniently improves the appearance of the publisher’s progress. A refreshed benchmark needs scrutiny of every product version, not just its publication date.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Familiar defaults are not neutral defaults
Bias does not require deliberate manipulation. Engineers know how their own architecture works, which workloads suit it and which settings make it perform well. That familiarity shapes scenario selection before anyone sees the competing results. Defaults can carry the same bias.
Several choices determine whether a benchmark fits a system:
- Shards: How the data is split, how shards are allocated and how large each shard becomes.
- Hardware and memory: Instance size, instance configuration and memory allocation.
- Data representation: Total dataset size, whether the working data fits in memory and how it is compressed.
- Access patterns: Which parts of the data the workload repeatedly touches.
A configuration chosen because it works well for one product may be a poor fit for another, even if nobody intended that outcome. Equal settings are not sufficient evidence of equally suitable settings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast results must still be comparable results
There is also outright cheating. Krenn invokes Volkswagen’s emissions scandal as an analogy for looking good under test conditions, then recalls a joke CI project that makes tests pass. The satirical Volkswagen package captures that idea: improving the reported outcome without improving what the test is meant to establish.
Vector search has a subtler way to produce an impressive speed result. With approximate nearest-neighbor search, search breadth and implementation parameters affect which neighbors are recovered. Changing those parameters can change both execution time and result quality. A speed comparison needs a quality condition. Otherwise, two fast-looking numbers may describe different search outcomes.
Report precision and recall alongside performance, and make their definitions explicit. For nearest-neighbor approximation, recall measures recovery of the exact nearest neighbors; it does not automatically measure relevance to a human reader. A simple per-query recall_at_k check compares returned IDs with the exact top k IDs:
python
def recall_at_k(exact_ids, returned_ids, k):
if k <= 0 or len(exact_ids) < k:
raise ValueError("Provide at least k exact neighbors, with k > 0")
expected = set(exact_ids[:k])
if len(expected) != k:
raise ValueError("Exact neighbor IDs must be unique")
recovered = expected.intersection(returned_ids[:k])
return len(recovered) / k
This makes the missing dimension concrete: returning quickly is only part of the result; recovering the intended neighbors is another.
Omitting quality is not necessarily intentional. Krenn recounts an internal benchmark that nearly reached publication without precision and recall checks. People accustomed to Boolean filters—where a record either passes or fails—had overlooked the additional quality tradeoff introduced by approximation. Adding those checks materially changed the comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One exceptional query can dominate the headline
Even when individual measurements are defensible, their presentation can exaggerate the scope of an advantage. Krenn recalls roughly twenty measurements: eighteen or nineteen showed similar performance, while one optimized use case was about ten times faster. Aggregating the results made the systems appear much farther apart overall.
He tentatively remembers the exceptional case as a change in sort direction on a particular data type. The exact query and aggregation formula are not specified, so the anecdote cannot establish a general speedup. Its useful lesson is to inspect the distribution: does an advantage appear across the workload, or does one special case carry the claim?
The marketing destination is a simple headline, such as being five times faster than a competitor. That is another illustrative slogan, not a verified overall result here. Once the headline replaces the per-query evidence, readers can no longer see whether the improvement applies to the work they actually do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeated measurements expose gradual regressions
A benchmark is difficult to challenge or use if its publisher leaves out the pieces required to reproduce it. The constructive response is to make tests automated and repeatable. Krenn describes Elastic’s nightly benchmarks: a broad set of tests run against the system’s own evolving performance.
The displayed chart shows a sharp latency drop after an optimization. The repeatable practice matters more than assigning that example a speedup multiplier: each night’s run measures the combined changes made during the day, allowing the team to see when performance changes.
Krenn uses the slow-boiling-frog story as a metaphor for accumulated regressions. An illustrative one-percent slowdown today may escape notice; another two-percent slowdown the following week may also seem unremarkable. Over time, those small losses can become substantial. A continuous history makes that drift visible instead of leaving the team to discover it after performance has already deteriorated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn your workload into a repeatable evaluation
To make a purchasing or architecture decision, the benchmark must represent your own workload. Specify the data size and structure, read/write ratio, actual queries, acceptable latency and available hardware. An external benchmark may match some of those conditions without matching the combination that determines your result. Running the local experiment takes work, but skipping it leaves you hoping that someone else’s conditions transfer to yours.
Krenn’s concrete tool is Rally, an Elasticsearch benchmarking framework. A track describes the data and operations to run. That gives the evaluation a repeatable structure:
- Define the data and queries that represent the workload.
- Run that workload against the system.
- Change settings or hardware and repeat the same workload to assess the effect.
Krenn describes using this approach to compare their own system with itself, including nightly testing.
For a cross-product evaluation, use the corresponding tools for the other systems and reproduce the same workload conditions. Rally is not being presented as a universal harness for every vector database. The question is what each system can deliver for your data, queries and hardware—not which product owns the most impressive unrelated chart.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A flawed benchmark can still tell you where to look
Finding a flaw makes it tempting to discard the entire benchmark. Krenn recommends asking a second question before doing that: what can still be learned from the scenario the publisher chose? Even an unfair comparison can reveal where a vendor believes its system performs especially well.
The chosen workload can point toward a system’s strengths, preferred access patterns or recently improved behavior. Treat those as leads to investigate rather than accepting the headline—or dismissing every observation because one part of the comparison is weak.
That leaves a practical ending: use published benchmarks to discover what to test, then use your own workload to establish what to trust. Without that work, the decision still rests on benchmarketing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Elastic's framework for running reproducible Elasticsearch benchmarks.
Public performance histories and methodology for daily Elasticsearch regression testing.
A satirical JavaScript package that detects CI environments and makes tests pass.
Further reading
Define benchmark data, queries, indexing operations, concurrency, warmup and request schedules.
Explains why metadata filters can increase HNSW traversal work and how Lucene adapts ACORN-1 to reduce it.
Documents search breadth and index parameters that affect nearest-neighbor accuracy, speed and memory.
Research on extending HNSW to support vector similarity searches constrained by structured predicates.
Read the complete timestamped transcript
- 0:00
[upbeat music] Um, I'm Philipp.
- 0:17
Uh, let's talk a bit about benchmarking, benchmarketing. Who has heard the term benchmarketing before? [laughing]
- 0:24
Not, not many. Okay. Um, so benchmarketing is that what you get in most of the benchmarks out there, um, even though nobody calls their own benchmarks like that. So let's take a quick look.
- 0:35
Um, who is using vector search today? I assume many. Who likes the performance of their vector search? [laughing]
- 0:44
Okay. Um, who has looked at various performance benchmarks around, uh, vector search? Okay, quite a few. How did you like those benchmarks?
- 0:53
Very hard to compare against each other.
- 0:54
Yes. Um, that is, um... I have this X is faster than Y. The problem is that you can have pretty much any product in X or any product in Y.
- 1:04
I think at this point, we have benchmarks for every single vendor being both faster and slower than all of their competitors, um, which is a bit of a funny sign for benchmarks.
- 1:14
Like, why is that? Like, are benchmarks just bad? Is everybody lying? Like, how does it happen, uh, that benchmarks are so different? Um, don't be shy. Come up here.
- 1:24
We have some more chairs up here. Um,
- 1:28
also, don't trust the glossy charts. Like, the better-looking the benchmarks are sometimes, uh, the worse the, the underlying material is. Um, so let's dive into it, like what to do and what not to do with benchmarks.
- 1:42
Um, come up here. We have two more chairs up here. Um, the first thing, and that's one of the biggest ones, is finding the right use case. Um, and the right use case, of course, depends on you-- for your system.
- 1:56
Um, this is one of my favorite benchmarking comic. It's under similar conditions, we are comparing two systems, and one is doing much better than the other system. And that is pretty much exactly what many system or many companies are doing with their own vendor benchmarks.
- 2:12
They find-
- 2:13
We got, we got the-
- 2:14
... a scenario that they like and probably their competit-competitors don't like so much, and then run the benchmark, and then afterwards they will say how much better they are.
- 2:23
This is like a very common pattern for benchmarks. Or to take it in a slightly different directions, um, uh, different direction, if you try to build a benchmarking dataset, and maybe you've seen that with your own, uh, marketing team, it's like you run various scenarios and then you see, like, this is no good, this is no good,
- 2:42
this is definitely no good. But here you have struck gold. Um, because here, um, them is down here and us is up here. So we will take this one, um, and then we'll generalize, and generalize, and generalize and say, like, "For everything, we're forty percent faster than the competition."
- 2:59
And you forget about all the others. Um, that is one of the very common things that you do with benchmarks. Um, and I--
- 3:07
I would just call out a couple of things, especially for vector search. Um, one of the things, depending on the underlying data structure that you have and, like, the s- how the system is built, um, most of the benchmarks, because it's much easier and also more reproducible, um, will be read-only, oftentimes with an optimized format.
- 3:24
The problem is most of our workloads are not read-only. But pretty much all the benchmarks that you see out there are for read-only datasets, um, because it's much easier to compare.
- 3:35
Versus if you have a specific dataset, um, and then you would need to find the right read-write ratio, and, like, there's so many parameters that you can do, um, that most people just don't, and then you have a benchmark that is not really representative of what you're trying to do.
- 3:50
Um, another thing is, um, one very counterintuitive thing about vector search is that at least for HNSW, filtering makes things slower. So if you come from a regular data store, uh, and you normally you have a restrictive filter, and you reduce the dataset, let's say, like, only twenty percent of the things make it through the, the filter,
- 4:11
and you then try to rank those, it will be faster. The way vector search works, or at least for a approximate, uh, nearest neighbor search, is that that will actually make it slower because you need to look at a lot more candidates and then find actually what remains after the filtering.
- 4:26
There are various optimizations to do that, um, but that's another trick that you can do, depending if you-- how good or how bad you are with filters, that you just tune the right scenario to find your sweet, sweet spot.
- 4:37
Um, or that you have just built in some optimizations, and then you're making sure that you have a scenario that hits exactly these optimizations that your competition doesn't have.
- 4:48
Um, another thing that I see quite frequently, especially as people, uh, update their benchmarks, they normally always update their own software, but they don't update their competitors'. Um, we see that in various benchmarks, um, where we are also present as a company, that our version is probably eighteen months old, and the competitor has something that came out
- 5:08
in the last one or two month. Um, and then obviously they will do quite a bit better there. Um,
- 5:16
so yeah, you only updated your own version and the other ones. I can understand, you don't want to keep up with the changes of all your competitors. Uh, you won't-- don't wanna figure out, like, what changed and how are things, uh, configured the right way in the newer version.
- 5:28
Um, you're mostly focused on the interesting things that you're doing yourself.
- 5:33
It is also convenient for your own performance. Um,
- 5:38
there's also, like, a lot of implicit biases. It's like you, you know how your system is built and what it is built for and how it works well. And then you might often pick scenarios, uh, that work well for those scen-- uh, for that use case.
- 5:50
And then it might not even be intentional, but then you might pick something that is just not a very good fit, um, for, uh, your competitors. The same goes for defaults.
- 6:00
It's like how you split up shard, how you allocate memory, how-- what instance size or instance configuration you pick. Um-
- 6:08
You might not even mean it in a bad way or you haven't even benchmarked the competitor against it on that specific machine. Uh, but you might just pick something that you know that works well for you, and then conveniently it's not so great for your competitors.
- 6:21
Um, it could be like the shard size, the total data size, what fits into memory, how you compress something, uh, the overall memory settings, the, the general data access patterns.
- 6:30
There are a ton of different ways how you could tweak that, uh, one way or another. Um, and then there is
- 6:39
obviously cheating. Uh, my, my favorite example for that is, um, [audience chuckles]
- 6:45
if you got the, the Volkswagen a while ago, like they, they had some creative ways, uh, to look good in the benchmarks for like how, how much exhaust, uh, they were producing.
- 6:56
Um, I think there was even a, a fun project, uh, that was, uh, like a Volkswagen plu- plug-in for UCI server that would just always make your tests pass.
- 7:06
Um, which is kind of like the same thing here. And cheating can be many things. For vector search specifically, um, approx-- especially when you do approximate nearest neighbor search, um, where it's depending on how you pick the parameters and like how, how widely you search, um, the, the quality of the results will be different or potentially be
- 7:29
different. So precision recall is normally what we do for information retrieval, and if you have different parameters or different implementations and you don't include those in the benchmark, um, you're comparing totally different things.
- 7:41
And then it's not totally uncommon to see, uh, that precision and benchmarked are, I'm not sure if intentionally or unintentionally skipped, um, for, for performance. Um, but that you then see wildly different numbers, but the results are actually totally different or the quality is totally different.
- 7:59
It's like obviously you can produce crap results very quickly, um, but that is probably not the point of what you do in your benchmarks. We actually had some of those ourselves with people who were, who were less experienced in vector search benchmarks.
- 8:13
They forgot about that quality attribute because for other types of searches, for example, if you have a, a Boolean filter where something passes and something doesn't pass, this doesn't exist.
- 8:22
Um, that they almost published something without looking at precision and recall, and it did make quite a bit of a difference, um, even in our own benchmarks. Um, I sometimes see creative, um, statistics.
- 8:35
Um, so one thing that I've recently seen that was very funny is like there were like I think twenty data points or so, or twenty different things measured, and in eighteen or nineteen, two systems were quite similar.
- 8:46
And then one system basically introduced some optimization for one of the use cases and made it like ten times faster than the competitor. Um, and then they found a way to kind of like even out the statistics across all of them, and then they said like, "Overall we're that faster."
- 9:01
But they found this one weird edge case. I think it was like instead of descending sorting, it was ascending sorting or something like that, um, on one specific data type, and that one was much faster.
- 9:13
And then they evened it out enough that it looked like overall the systems were wildly different. But it was basically like one specific query on one of the many things, uh, you looked at.
- 9:23
And then you have like the headline, because that's mostly what you're going for in the benchmarks, right? You want to have this benchmark five times faster than X. Um, again, coming back to the glossy charts, those are the things that you, you do, um, and it's kind of like the, the money shot in, in benchmarketing.
- 9:41
Um, and of course, then there are the problems when da- data doesn't reproduce. You don't publish all the pieces that anybody can actually run this to try it out.
- 9:49
Um, those don't make it easy either. Um, so how do we make benchmarks better or me- more meaningful? Um, they should definitely be automated and reproducible. Um, so what we internally do, for example, we have a very wide set of-- we call them the nightly benchmarks.
- 10:06
That's why up here it says nightly. Um, I just pulled this one out because it was nice. We, we optimized one specific thing, I think like, I don't know, a hundred x or ten x or whatever the ratio was.
- 10:18
Uh, but we, we optimized this, which was nice. But the important thing here is we run this benchmark every night, and then we put all the, the changes that we had during the day together to see if the performant change-- performance changes.
- 10:30
Why are we doing that? To avoid the slow-boiling frog problem. Who knows the slow-boiling frog?
- 10:38
It's a very French, uh, thing. It's like, you know, you throw the frog into the boiling water, and the frog will jump out again. And you put the frog into the cold water and slowly heat it up, and the frog will stay in the water.
- 10:49
And the same thing kind of like applies to benchmarks. If you make a change today and it makes something one percent slower, you're probably not going to see it.
- 10:57
And next week you make another change where it gets two percent slower, and over time, your performance just decreases a lot. Um, so you're the, the frog sitting in the warming up water and being boiled.
- 11:08
So you want to avoid that with your own system, that you don't want to boil your own performance. Um, so you want to have nightly benchmarks where you see how things are changing and evolving over time.
- 11:18
Um, second, this is a bit of an unfortunate thing, but you will need to do your own benchmarks. Um, don't trust anybody's benchmarks, and they-- probably nobody has done exactly the scenario that you want to have.
- 11:33
It's like you need to know this is my data size, and this is my data structure, and this is the read/write ratio, and this is how exactly my queries look like, and this is the latency that I am willing to accept, and this is the type of hardware that I have.
- 11:46
There are so many ways that make the existing benchmarks probably not a hundred percent meaningful for you, that if you want to really be sure, there is no way around doing that yourself.
- 11:57
Otherwise, you will buy somebody's, uh, glossy benchmarks, and you will have to trust them and hope for the best that it actually behaves the same way. Um,
- 12:07
it's unfortunate because it always means work, um, and you need to put in that work, but there's no easy way around that. Um, we have like A tool we call Rally.
- 12:15
It's basically you create a so-called track, and then you run the track where you say, like, "This is the data, this is the query," and then you can optimize that.
- 12:23
And then you can tune, like, the settings or the hardware or whatever else. It's only against us and ourselves. This is how we would do nightly benchmarks with ourselves.
- 12:31
Uh, but that is how you can then benchmark and figure out, for my hardware, what can I actually get out in terms of performance? And then you will need to take whatever other tools you want to evaluate and do something similar to see, like, for exactly your workload, um, how does that compare?
- 12:46
Um, one final thing, and that is also what I need to keep repeating internally for ourselves. If you find a flaw in a benchmark, it's easy to call the entire benchmark crap and then ignore everything that comes out of it.
- 13:00
What is much smarter is if y- if you look at it and still figure out, can I learn something from that? Even if it's like somebody pro- produces a, a bad benchmark against a competitor, it can tell you, like, where they think their sweet spot is, or what is the scenario that they pick for themselves, or what
- 13:16
they try to highlight. Because even then you can learn, like, what are the potential strengths of a system, uh, where do they shine, uh, what works well and what doesn't work well.
- 13:24
So all of that makes it easier to find something useful if you want to look. But it's of course much easier to say like, "All of this is crap, and we'll ignore it, and we'll call it BS."
- 13:35
And then you move on. Um, I hope this was useful. Um, I'm afraid you will have to do your own benchmarks to get proper results. Um, otherwise you will have to, uh, believe the benchmarketing.
- 13:48
Um, let me take a quick picture with you so I can prove to my colleagues that I've been working today. [laughs]
- 13:54
Um, can you all wave and say benchmarking?
- 13:57
Benchmarking. [laughs]
- 14:00
Thanks a lot. Um, enjoy the heat. And I'll hand over to the next speaker. [upbeat music]