Vector Search Benchmark[eting]

Talk transcript

1. 0:00

   [upbeat music] Um, I'm Philipp.

2. 0:17

   Uh, let's talk a bit about benchmarking, benchmarketing. Who has heard the term benchmarketing before? [laughing]

3. 0:24

   Not, not many. Okay. Um, so benchmarketing is that what you get in most of the benchmarks out there, um, even though nobody calls their own benchmarks like that. So let's take a quick look.

4. 0:35

   Um, who is using vector search today? I assume many. Who likes the performance of their vector search? [laughing]

5. 0:44

   Okay. Um, who has looked at various performance benchmarks around, uh, vector search? Okay, quite a few. How did you like those benchmarks?

6. 0:53

   Very hard to compare against each other.

7. 0:54

   Yes. Um, that is, um... I have this X is faster than Y. The problem is that you can have pretty much any product in X or any product in Y.

8. 1:04

   I think at this point, we have benchmarks for every single vendor being both faster and slower than all of their competitors, um, which is a bit of a funny sign for benchmarks.

9. 1:14

   Like, why is that? Like, are benchmarks just bad? Is everybody lying? Like, how does it happen, uh, that benchmarks are so different? Um, don't be shy. Come up here.

10. 1:24

    We have some more chairs up here. Um,

11. 1:28

    also, don't trust the glossy charts. Like, the better-looking the benchmarks are sometimes, uh, the worse the, the underlying material is. Um, so let's dive into it, like what to do and what not to do with benchmarks.

12. 1:42

    Um, come up here. We have two more chairs up here. Um, the first thing, and that's one of the biggest ones, is finding the right use case. Um, and the right use case, of course, depends on you-- for your system.

13. 1:56

    Um, this is one of my favorite benchmarking comic. It's under similar conditions, we are comparing two systems, and one is doing much better than the other system. And that is pretty much exactly what many system or many companies are doing with their own vendor benchmarks.

14. 2:12

    They find-

15. 2:13

    We got, we got the-

16. 2:14

    ... a scenario that they like and probably their competit-competitors don't like so much, and then run the benchmark, and then afterwards they will say how much better they are.

17. 2:23

    This is like a very common pattern for benchmarks. Or to take it in a slightly different directions, um, uh, different direction, if you try to build a benchmarking dataset, and maybe you've seen that with your own, uh, marketing team, it's like you run various scenarios and then you see, like, this is no good, this is no good,

18. 2:42

    this is definitely no good. But here you have struck gold. Um, because here, um, them is down here and us is up here. So we will take this one, um, and then we'll generalize, and generalize, and generalize and say, like, "For everything, we're forty percent faster than the competition."

19. 2:59

    And you forget about all the others. Um, that is one of the very common things that you do with benchmarks. Um, and I--

20. 3:07

    I would just call out a couple of things, especially for vector search. Um, one of the things, depending on the underlying data structure that you have and, like, the s- how the system is built, um, most of the benchmarks, because it's much easier and also more reproducible, um, will be read-only, oftentimes with an optimized format.

21. 3:24

    The problem is most of our workloads are not read-only. But pretty much all the benchmarks that you see out there are for read-only datasets, um, because it's much easier to compare.

22. 3:35

    Versus if you have a specific dataset, um, and then you would need to find the right read-write ratio, and, like, there's so many parameters that you can do, um, that most people just don't, and then you have a benchmark that is not really representative of what you're trying to do.

23. 3:50

    Um, another thing is, um, one very counterintuitive thing about vector search is that at least for HNSW, filtering makes things slower. So if you come from a regular data store, uh, and you normally you have a restrictive filter, and you reduce the dataset, let's say, like, only twenty percent of the things make it through the, the filter,

24. 4:11

    and you then try to rank those, it will be faster. The way vector search works, or at least for a approximate, uh, nearest neighbor search, is that that will actually make it slower because you need to look at a lot more candidates and then find actually what remains after the filtering.

25. 4:26

    There are various optimizations to do that, um, but that's another trick that you can do, depending if you-- how good or how bad you are with filters, that you just tune the right scenario to find your sweet, sweet spot.

26. 4:37

    Um, or that you have just built in some optimizations, and then you're making sure that you have a scenario that hits exactly these optimizations that your competition doesn't have.

27. 4:48

    Um, another thing that I see quite frequently, especially as people, uh, update their benchmarks, they normally always update their own software, but they don't update their competitors'. Um, we see that in various benchmarks, um, where we are also present as a company, that our version is probably eighteen months old, and the competitor has something that came out

28. 5:08

    in the last one or two month. Um, and then obviously they will do quite a bit better there. Um,

29. 5:16

    so yeah, you only updated your own version and the other ones. I can understand, you don't want to keep up with the changes of all your competitors. Uh, you won't-- don't wanna figure out, like, what changed and how are things, uh, configured the right way in the newer version.

30. 5:28

    Um, you're mostly focused on the interesting things that you're doing yourself.

31. 5:33

    It is also convenient for your own performance. Um,

32. 5:38

    there's also, like, a lot of implicit biases. It's like you, you know how your system is built and what it is built for and how it works well. And then you might often pick scenarios, uh, that work well for those scen-- uh, for that use case.

33. 5:50

    And then it might not even be intentional, but then you might pick something that is just not a very good fit, um, for, uh, your competitors. The same goes for defaults.

34. 6:00

    It's like how you split up shard, how you allocate memory, how-- what instance size or instance configuration you pick. Um-

35. 6:08

    You might not even mean it in a bad way or you haven't even benchmarked the competitor against it on that specific machine. Uh, but you might just pick something that you know that works well for you, and then conveniently it's not so great for your competitors.

36. 6:21

    Um, it could be like the shard size, the total data size, what fits into memory, how you compress something, uh, the overall memory settings, the, the general data access patterns.

37. 6:30

    There are a ton of different ways how you could tweak that, uh, one way or another. Um, and then there is

38. 6:39

    obviously cheating. Uh, my, my favorite example for that is, um, [audience chuckles]

39. 6:45

    if you got the, the Volkswagen a while ago, like they, they had some creative ways, uh, to look good in the benchmarks for like how, how much exhaust, uh, they were producing.

40. 6:56

    Um, I think there was even a, a fun project, uh, that was, uh, like a Volkswagen plu- plug-in for UCI server that would just always make your tests pass.

41. 7:06

    Um, which is kind of like the same thing here. And cheating can be many things. For vector search specifically, um, approx-- especially when you do approximate nearest neighbor search, um, where it's depending on how you pick the parameters and like how, how widely you search, um, the, the quality of the results will be different or potentially be

42. 7:29

    different. So precision recall is normally what we do for information retrieval, and if you have different parameters or different implementations and you don't include those in the benchmark, um, you're comparing totally different things.

43. 7:41

    And then it's not totally uncommon to see, uh, that precision and benchmarked are, I'm not sure if intentionally or unintentionally skipped, um, for, for performance. Um, but that you then see wildly different numbers, but the results are actually totally different or the quality is totally different.

44. 7:59

    It's like obviously you can produce crap results very quickly, um, but that is probably not the point of what you do in your benchmarks. We actually had some of those ourselves with people who were, who were less experienced in vector search benchmarks.

45. 8:13

    They forgot about that quality attribute because for other types of searches, for example, if you have a, a Boolean filter where something passes and something doesn't pass, this doesn't exist.

46. 8:22

    Um, that they almost published something without looking at precision and recall, and it did make quite a bit of a difference, um, even in our own benchmarks. Um, I sometimes see creative, um, statistics.

47. 8:35

    Um, so one thing that I've recently seen that was very funny is like there were like I think twenty data points or so, or twenty different things measured, and in eighteen or nineteen, two systems were quite similar.

48. 8:46

    And then one system basically introduced some optimization for one of the use cases and made it like ten times faster than the competitor. Um, and then they found a way to kind of like even out the statistics across all of them, and then they said like, "Overall we're that faster."

49. 9:01

    But they found this one weird edge case. I think it was like instead of descending sorting, it was ascending sorting or something like that, um, on one specific data type, and that one was much faster.

50. 9:13

    And then they evened it out enough that it looked like overall the systems were wildly different. But it was basically like one specific query on one of the many things, uh, you looked at.

51. 9:23

    And then you have like the headline, because that's mostly what you're going for in the benchmarks, right? You want to have this benchmark five times faster than X. Um, again, coming back to the glossy charts, those are the things that you, you do, um, and it's kind of like the, the money shot in, in benchmarketing.

52. 9:41

    Um, and of course, then there are the problems when da- data doesn't reproduce. You don't publish all the pieces that anybody can actually run this to try it out.

53. 9:49

    Um, those don't make it easy either. Um, so how do we make benchmarks better or me- more meaningful? Um, they should definitely be automated and reproducible. Um, so what we internally do, for example, we have a very wide set of-- we call them the nightly benchmarks.

54. 10:06

    That's why up here it says nightly. Um, I just pulled this one out because it was nice. We, we optimized one specific thing, I think like, I don't know, a hundred x or ten x or whatever the ratio was.

55. 10:18

    Uh, but we, we optimized this, which was nice. But the important thing here is we run this benchmark every night, and then we put all the, the changes that we had during the day together to see if the performant change-- performance changes.

56. 10:30

    Why are we doing that? To avoid the slow-boiling frog problem. Who knows the slow-boiling frog?

57. 10:38

    It's a very French, uh, thing. It's like, you know, you throw the frog into the boiling water, and the frog will jump out again. And you put the frog into the cold water and slowly heat it up, and the frog will stay in the water.

58. 10:49

    And the same thing kind of like applies to benchmarks. If you make a change today and it makes something one percent slower, you're probably not going to see it.

59. 10:57

    And next week you make another change where it gets two percent slower, and over time, your performance just decreases a lot. Um, so you're the, the frog sitting in the warming up water and being boiled.

60. 11:08

    So you want to avoid that with your own system, that you don't want to boil your own performance. Um, so you want to have nightly benchmarks where you see how things are changing and evolving over time.

61. 11:18

    Um, second, this is a bit of an unfortunate thing, but you will need to do your own benchmarks. Um, don't trust anybody's benchmarks, and they-- probably nobody has done exactly the scenario that you want to have.

62. 11:33

    It's like you need to know this is my data size, and this is my data structure, and this is the read/write ratio, and this is how exactly my queries look like, and this is the latency that I am willing to accept, and this is the type of hardware that I have.

63. 11:46

    There are so many ways that make the existing benchmarks probably not a hundred percent meaningful for you, that if you want to really be sure, there is no way around doing that yourself.

64. 11:57

    Otherwise, you will buy somebody's, uh, glossy benchmarks, and you will have to trust them and hope for the best that it actually behaves the same way. Um,

65. 12:07

    it's unfortunate because it always means work, um, and you need to put in that work, but there's no easy way around that. Um, we have like A tool we call Rally.

66. 12:15

    It's basically you create a so-called track, and then you run the track where you say, like, "This is the data, this is the query," and then you can optimize that.

67. 12:23

    And then you can tune, like, the settings or the hardware or whatever else. It's only against us and ourselves. This is how we would do nightly benchmarks with ourselves.

68. 12:31

    Uh, but that is how you can then benchmark and figure out, for my hardware, what can I actually get out in terms of performance? And then you will need to take whatever other tools you want to evaluate and do something similar to see, like, for exactly your workload, um, how does that compare?

69. 12:46

    Um, one final thing, and that is also what I need to keep repeating internally for ourselves. If you find a flaw in a benchmark, it's easy to call the entire benchmark crap and then ignore everything that comes out of it.

70. 13:00

    What is much smarter is if y- if you look at it and still figure out, can I learn something from that? Even if it's like somebody pro- produces a, a bad benchmark against a competitor, it can tell you, like, where they think their sweet spot is, or what is the scenario that they pick for themselves, or what

71. 13:16

    they try to highlight. Because even then you can learn, like, what are the potential strengths of a system, uh, where do they shine, uh, what works well and what doesn't work well.

72. 13:24

    So all of that makes it easier to find something useful if you want to look. But it's of course much easier to say like, "All of this is crap, and we'll ignore it, and we'll call it BS."

73. 13:35

    And then you move on. Um, I hope this was useful. Um, I'm afraid you will have to do your own benchmarks to get proper results. Um, otherwise you will have to, uh, believe the benchmarketing.

74. 13:48

    Um, let me take a quick picture with you so I can prove to my colleagues that I've been working today. [laughs]

75. 13:54

    Um, can you all wave and say benchmarking?

76. 13:57

    Benchmarking. [laughs]

77. 14:00

    Thanks a lot. Um, enjoy the heat. And I'll hand over to the next speaker. [upbeat music]