Information Retrieval from the Ground Up

Talk transcript

1. 0:00

   [upbeat music] Let's get going.

2. 0:15

   Audio is okay for everybody? I have some slight feedback, but I'll try to manage. I hope it's okay for you. Um, hi, I'm Philipp. Uh, let's talk a bit about retrieval.

3. 0:26

   I'll show you some retrieval from the ground up. We'll keep it pretty hands-on. Um, you will have a chance to follow along and do everything that I show you as well.

4. 0:35

   I have like a demo instance that you can use, um, or you can just watch me. Um, if you have any questions, ask at any moment. If anything is too small to read, shout, and we'll try to make it larger.

5. 0:47

   Uh, we'll try to adjust as we go along. So I guess we're not over RAG yet, but uh, RAG is a thing, and we'll focus on the R in RAG, the retrieval-augmented generation.

6. 1:00

   We'll just focus on the retrieval. Um, just let's see where we are with retrieval. Um, quick show of hands, who has done RAG before? Okay, that's about half or so.

7. 1:13

   Um, who has done anything with vector search, uh, and RAG?

8. 1:18

   Do I need vector search for RAG or can I do, do anything else?

9. 1:22

   Do anything to crunch stuff in?

10. 1:24

    Yeah. Yeah. So you can do anything. Um, retrieval is actually a very old thing. Um, depending on how you define it, it might be fifty, [REDACTED:age], whatever years old.

11. 1:34

    Retrieval is just getting the right context to the generation. I like-- Ignore all the generation for today. We'll keep it very simple. We'll just focus on the retrieval part of getting the right information in.

12. 1:44

    Um, partially from the old stuff like the classics, uh, but we'll get to some new things as well as we go along. Um, who has done keyword search before?

13. 1:55

    Just-- That is fewer than vector search, I feel like. Um, which is, um, which almost reminds me of like fifteen years ago or so when NoSQL came up, like more people had done MongoDB, Redis, whatever else, um, rather than SQL.

14. 2:09

    That has changed again. Um, I think it will be kind of similar for retrieval. Um, the way I would always

15. 2:17

    vector search, um, is a feature of retrieval. It's only [REDACTED:generic_id] of multiple features or many features that you want in retrieval. And we'll see a bit why and how, and we'll, we'll dive into those details.

16. 2:28

    Um, so I work for Elastic, the company behind Elasticsearch. We're the most downloaded, deployed, whatever else search engine. We do vector search, we do keyword search, we do hybrid search.

17. 2:39

    Um, we'll dive into various examples. Everything that I will show you, um, works well. The query language is Elasticsearch, but if you use anything built on Apache Lucene, everything behaves very similarly.

18. 2:52

    If you use something that is a clone or close, uh, to Lucene, like anything built on, uh, 10tv or a-anything like that, it will be very similar. The founda-

19. 3:03

    keyword search and vector search will apply broadly everywhere. Um, so let's get going. Um, we'll keep this pretty hands-on. Um, who remembers in Star Wars when he's making that hand gesture?

20. 3:14

    Um, what is the quote?

21. 3:17

    Mind tricks.

22. 3:18

    These are not the droids you're looking for. Um, we'll keep this relatively Star Wars, uh, based. Um, feel free to come in and filter on the sides or whatever.

23. 3:28

    Um, I'm afraid we have, I think, [REDACTED:generic_id] chair over there otherwise, and [REDACTED:generic_id] down there. Otherwise, it's getting a bit

24. 3:37

    full. Um, okay. Let's look at this, uh, of what these are not the droids you're looking for, uh, does for search. And I will start kind of like with the classic approach.

25. 3:47

    Keyword search or lexical search is like you search for the words that you have stored, and we want to find what is relevant in our examples. Um, if you want to follow along, um, there is a gist which has all the code that I'm showing you.

26. 4:02

    It's at [REDACTED:username]/ai[REDACTED:username]. Um, there is [REDACTED:generic_id] important thing. It's I have [REDACTED:generic_id] shared instance basically for everybody, so you can all just use this without signing up for any accounts or anything.

27. 4:16

    So this is just a cloud instance, um, that you can use. There is my handle, um, is in the index name. If you don't want to fight and overwrite each other's data, reply-- replace that with your unique handle or something that is specific to you, because otherwise you will all work on the same index and kind of

28. 4:34

    like overwrite each other's data. You can also just watch me. If you don't have a computer handy, that's fine. Um, but if you want to follow along, [REDACTED:username]/ai[REDACTED:username], there will be a gist.

29. 4:45

    Uh, it will have the connection string, like there's a URL, and then the credentials are workshop, workshop. If you go into login, it will say, "Login with Elasticsearch." That's where you use workshop, workshop.

30. 4:56

    Then you'll be able to log in, and you can just run all the queries that I'm showing you. You can try out stuff. Um, if you have any questions, shout.

31. 5:04

    I have a couple of colleagues dispersed in the room, so, um, if we have too many questions, we'll somehow divide and conquer. Um, so let's get going and see what we have here.

32. 5:15

    Um, and I'll show you most of the stuff live. Um,

33. 5:20

    I think this is large enough in the back row. If it's not large enough for anybody, shout, and we'll see how much larger I can make this. Um,

34. 5:29

    and let me turn off the Wi-Fi and hope that my wired connection is good enough.

35. 5:37

    Oh. Let's refresh to see... Ooh. Maybe we'll use my phone after all.

36. 5:59

    Okay. Let's try this again. Okay, this is no good.

37. 6:24

    Out you go. Okay, hardest problem of the day solved. We have network.

38. 6:34

    Um, okay, so we have the sentence, "These are not the droids you're looking for." And we'll start with the classic keyword or lexical search, like what happens behind the scenes.

39. 6:42

    So what you generally want to do is you basically want to extract individual words and then make them searchable. Um, so here I'm not storing anything, I'm just looking at, like, how would that look like if I stored something?

40. 6:54

    I'm using this underscore analyze endpoint, uh, to sh- to see, um, what I will actually store in the background to make then searchable. So I have, "These are not the droids you're looking for," and you see, "These are M, not M, the droids you are looking for."

41. 7:15

    In Western languages, the first step that hap- or every time or everywhere, the s- first step that hap- happens is the tokenization. In Western languages, it's pretty simple. It's normally any white spaces and punctuation marks where you just break out the individual tokens.

42. 7:29

    Um, especially Asian languages are a bit more complicated around that, but we'll gloss over that for today. And we have a couple of interesting pieces of information here. So we have the token.

43. 7:39

    So these, um, is the first token. We have the start offset and the end offset. Um, why would I need a start and an end offset? Why would extract and then store that potentially?

44. 7:51

    Any guesses? Yeah. So you have a reference back to the original text, right? Yes. Especially if you have a longer text, you would want to have that highlighting feature that you want to say, um, this is where my hit actually was.

45. 8:02

    So if I'm searching for these, which is maybe not a great word, uh, but you would very easily be able to highlight where you had actually the match. And the trick that you're doing in search, generally what differentiates it from a database is a database just stores what you give it, and then does basically almost everything at

46. 8:19

    query or search time. Whereas a search engine does a lot of the work at ingestion or when you store the data. So we break out the individual tokens. Um, we calculate these offsets and store them, so whenever we have a match afterwards, we never need to reanalyze the actual text, which could potentially be multiple pages long.

47. 8:37

    Um, but we could just highlight where we have that match because we have extracted those positions. Um, we have a position. Why would I want to store the position with the text that I have?

48. 8:49

    Yeah. For annotations? Uh, yeah. Annotations. So the, the main use case, uh, that you have is if you have these positions, and later on we'll briefly look at if you want to look for a phrase, if you want to look for this word followed by that word.

49. 9:03

    Um, so you could then just look for all the texts that contain these words, but then you could also just compare the positions and basically look for N, N plus 1, et cetera.

50. 9:12

    And you never need to look at the string again, but you can just look at the positions to figure out like this was [REDACTED:generic_id] continuous phrase, even if you have broken it out into the ne- individual tokens.

51. 9:22

    Um, most of the things that we see here is alphanumeric, um, for alphanumeric. An alternative would be synonyms. We'll skip over synonym definition because it's not fun to define tons of synonyms.

52. 9:34

    Uh, but this is all the things that we're storing here in the background. You can also customize this analysis, and that is [REDACTED:generic_id] of the, the features again of full text search and lexical searches that you pre-process a lot of the information to make that search afterwards faster.

53. 9:49

    So here you can see I'm stripping out the HTML because nobody's going to search for this emphasis tag. Um, I use a standard tokenizer, um, that breaks up, for example, on dashes, you will see that.

54. 10:02

    Alternatives would be white space that you only break up on white spaces. Um, I lowercase everything, which is most of the times what you want because nobody searches in Google, um, with proper casing, or at least maybe my parents, but nobody else searches, uh, with proper casing in, in Google.

55. 10:20

    Um, we remove stop words. We'll get to stop words in a moment. Um, and we do stemming with the snowball stemmer. What stemming is, it basically reduces a word down to the root.

56. 10:31

    So you don't care about singular, plural, or like the flexion of a verb anymore, but you really care more about the concept. So

57. 10:40

    if I run through that analysis, does anybody want to guess what will remain of this phrase? So which tokens will be extracted and what, in what form?

58. 10:53

    Not a lot will remain. Um, two? Droid and look maybe. Yeah, close. Um, so we'll actually have [REDACTED:generic_id]. So we have droid, you, and look, and you can see all the others were stop words which were removed.

59. 11:12

    Um, the stemming reduced looking down to look because we don't care if it looks, looking, uh, look. We just reduce it to the word stem. So we do this when we store the data.

60. 11:23

    And by the way, when you search afterwards, um, your text would run through the same analysis that you would have exact matches. So you don't need to do anything like a like search anymore in the future.

61. 11:32

    Um, so this will be much more performant, uh, than anything that you would do in a relational database because you have direct matches, and we'll look at the data structure behind it in a moment.

62. 11:41

    But we, what we get is, uh, droid, you, look, um, with the right positions. So for example, if we searched for droid, you, we could easily retrieve that because we have the positions, um, even though that is a weird phrase.

63. 11:55

    Um, do we start indexing at zero or [REDACTED:generic_id]?

64. 12:01

    Zero. Zero, yes. It's the only right way. Um, [laughing]

65. 12:05

    the- there's no discussion here. Um, so we, we are-- The positions are based, uh, starting at zero, and these are the tokens that are remaining. Um, if you do this for a different language like- You might hear I'm a native German speaker.

66. 12:19

    Um, this is the text in, in German, and you would, uh, if you use a German analyzer, it would know the rules for German and then would analyze the text in the right way.

67. 12:28

    So then you would have remaining droid den such. Um, anybody wants, wants to guess what happens if I have the wrong language for a text?

68. 12:40

    Nothing good, guys.

69. 12:42

    It will go very poorly because the... So how, how this works is basically you have rules for every single language. It's like, what is the stop word? How does stemming work?

70. 12:51

    If you apply the wrong rules, you basically just get wrong stuff out, so it will not, um, do what you want. So what you get here is, um, like the, this is an article, um, but, well, in the-- in English, the rule is an S at the end just gets stemmed away even though this doesn't make any

71. 13:08

    sense. So you apply the wrong rules, um, and you just produce pretty much garbage. Um, so don't do that. Um, just to give you another example, um,

72. 13:18

    French, this is the same phrase in French. Um, and then you see, uh, droid la and rechercher as-- are the words that are remaining in these examples. Otherwise, it works the same, but you need to have the right analysis for what you're doing, otherwise you'll just produce garbage.

73. 13:35

    Um, a couple of things as we're going along. The stop word list by default, which you could overwrite, is relatively short. This is-- linguists have spent many years figuring out what are the right lists of stop words, and you don't want to have too many or too few.

74. 13:52

    Uh, in English, it's-- I always forget. I think it's thir- thirty-[REDACTED:generic_id] or so. Um, this is where you can find it in the source code. It's, I don't want to say well-hidden, but it's not easy to find either.

75. 14:01

    Um, so every language has, like, a list of stop words that are defined, uh, that will be automatically removed for. These are not the droids you are looking for.

76. 14:09

    By accident, more or less, we had a lot of stop words that why not a lot remained here in the phrase. And then for all other languages, you will have a similar list, um, of stop words.

77. 14:18

    Um, should you always remove stop words? Yes, no.

78. 14:28

    Hmm, it can remove context when you have things like not.

79. 14:32

    Yes. That is, by the way, another... Not is a very good-- So I'm not sure if everybody heard that. Um, the, the comment was about not. [REDACTED:generic_id] important thing here, we're talking about lexical or keyword search, which is dumb but scalable.

80. 14:45

    Um, it doesn't know-- understand if there is a droid or there's no droid. It's just defined as a stop word. It does just keyword matching. Um, that is in, in vector search, um, or anything with a machine learning model behind it will be a bit of a different story afterwards, uh, where these things might make a difference.

81. 15:02

    But this is very simple because it just matches on similar strings, basically. Um, it doesn't understand the context. It doesn't know what's going on. That's why the linguists decided not is a good stop word.

82. 15:13

    Um, you could overwrite that if for your specific use case this is not a good idea. Um,

83. 15:21

    always removing stop words, yes, no, maybe? So our favorite phrase is, "It depends."

84. 15:29

    And then you have to explain, like, what it depends on. So what it depends on is there are scenarios, uh, where removing all stop words does not give you the desired result.

85. 15:38

    And maybe you want to have, like, a text with and without stop words. Like, sometimes stop words are just, like, a lot of noise that blow up the index size and don't really add a lot of value.

86. 15:47

    That's why we have defined them and try to remove them by default. Um, but if you had, for example, to be or not to be,

87. 15:54

    these are all stop words. This would all be gone, uh, when, when you run it through analysis. Um, so you-- It is tricky to figure out, like, what is the right balance for stop words or what works for your use case, but you might have unexpected surprises in all of this.

88. 16:09

    Um, okay, we've seen the German examples. Um,

89. 16:14

    let's do some more queries. Um, or let's actually store something. Um, so far we only pretended or we only looked at what would happen if we would store something.

90. 16:24

    Now I'm actually creating an index. Again, if you're running this yourself, um, please use a different name than me. Um, just, uh, replace all my handle instances with your handle or whatever you want.

91. 16:38

    Uh, since this is a shared instance, if we have too many collisions, I might jump to another instance that I have as a backup in the background. Um, but what I'm doing here is I'm creating my-- this analysis pipeline that I've looked at before.

92. 16:49

    Like, I'm throwing out the HTML. I use a standard tokenizer, lowercasing, stop word removal, and stemming. And then I, I call this my analyzer, and then I'm basically applying this, my analyzer, on a field called, uh, quote.

93. 17:04

    This, we call this a mapping. It's kind of like the equivalent. It's a schema in a relational database, but this defines how different fields behave. Um,

94. 17:15

    okay, and somebody did not replace, um, the, the query. Um,

95. 17:23

    by the way, you need to keep user underscore. Um, let me, let me quickly do this, um,

96. 17:32

    myself. Oops. I should have seen this coming. Um,

97. 17:44

    we want to replace, and we'll use... Oops.

98. 18:02

    Please don't copy that. Um, and I want to...

99. 18:13

    Okay. Looks like it worked. Let's try it again. Um,

100. 18:23

     so we're creating our own index. Um, and now I just to double-check, I'll just again, uh, run this _analyze against this field that I've set up to just double-check that I've set it up correctly.

101. 18:39

     And now I'm actually starting to store documents. Um, bless you. Um, so we'll store, "These are not the droids you're looking for." Um, I have two others, um, that I'll index just so we have a bit more to search.

102. 18:51

     Um, "No, I am your father." Any guesses what will remain here?

103. 18:58

     Father.

104. 18:59

     Father, yeah.

105. 19:01

     No, I am your father.

106. 19:03

     Okay. Let's, let's try this out. Let me copy my...

107. 19:06

     You father.

108. 19:07

     Ah. This [REDACTED:generic_id] actually has way fewer stoppers than you would expect. Uh, let's quickly do this. Um,

109. 19:19

     since I didn't do the HTML removal, let's take these out manually. So what you get is, "No, I am your father." And this was stupid because this was not what I wanted.

110. 19:31

     We need to run this against the right analysis. Um,

111. 19:44

     this happens when you copy-paste. Okay. Um, uh...

112. 19:57

     Ah, sorry. And we'll do text. Now I think I've patched this back together. Okay. I am your father.

113. 20:12

     So no is the only stopper in this list, actually.

114. 20:16

     No was on the stopper list, all the others are not. Um, okay, let's try another [REDACTED:generic_id]. Obi Wan never told you what happened to your father.

115. 20:27

     How many tokens will Obi Wan be? Two? [REDACTED:generic_id]?

116. 20:32

     [REDACTED:generic_id].

117. 20:35

     No. Obi Wan will, will be two, like, Obi Wan. Because we used a default tokenizer or standard tokenizer, that [REDACTED:generic_id] breaks up at dashes. If you had used another tokenizer like white space, that would keep it together because that only breaks up at white spaces.

118. 20:49

     So there are various reasons why you want or would not want to do that. I don't want to go into all the details, but there are a lot of things to do right or wrong when you ingest the data, which will then allow you to query the data in specific ways.

119. 21:01

     So for example, if you would have an email address,

120. 21:07

     that [REDACTED:generic_id] is also weirdly broken up. Like, you might use, like, there is a dedicated tokenizer for URL email addresses. So depending on what type of data you have, you will need to process the data the right way because pretty much all the, the smart pieces are kind of like at ingestion here to make the search afterwards

121. 21:25

     easier. Um, so you can easily do that. Um, let's see. Let's index all my [REDACTED:generic_id] documents so that we can actually search for them. Now, if I start searching droid,

122. 21:39

     it should match, "These are not the droids you're looking for," yes or no? Because this [REDACTED:generic_id] is singular and uppercase, and the droid that we stored was plural and lowercase.

123. 21:48

     Will it match yes or no?

124. 21:50

     Yes.

125. 21:51

     Why?

126. 21:52

     Because you've got the stemming.

127. 21:55

     Uh, yes, we had the stemming. We had the lowercasing. And when we search, so we store the text, it runs through this pipeline, uh, or the analysis. Um, and for the search, it does the same thing.

128. 22:06

     So it will lowercase the droid. Um, it has stemmed down the droids in the text to droid, and then we have an exact match. Um, so what the, the data structure behind the scene actually looks like.

129. 22:20

     The magic is kind of like in this so-called inverted index. What the inverted index is, is

130. 22:27

     these are all the tokens that remained that I've extracted. I have alphabetically sorted them, and they basically have a pointer and say, in this document, like with the IDs [REDACTED:generic_id], two, [REDACTED:generic_id] that I've stored, we have how many occurrences?

131. 22:40

     Like zero, [REDACTED:generic_id], um, yeah, nothing had two. Um, and then we also know at which position they appeared. So search for droid now,

132. 22:53

     this is what I have stored. Um, I will lowercase the droid to droid. I have an exact match here. Then I go through that list and see, retrieve this document, skip this [REDACTED:generic_id], skip this [REDACTED:generic_id].

133. 23:03

     Um, and at position four, um, you have that hit. And then you could easily highlight that. So you have almost done all the hard work at ingestion, and this retrieval afterwards will be very fast and efficient.

134. 23:14

     That's the classic data stru-structure for search, the inverted index, where you have this alphabetic list, uh, of all the tokens that you've extracted to do that. Um, and this will just be built in the background for you, and that's how you can retrieve all of this.

135. 23:27

     Um, let's look at a few other, um,

136. 23:32

     uh, queries and how they behave. Um, if I search for robot, will I find anything?

137. 23:43

     No, because there was no robot. Um, there was a droid. Um, we could now define a synonym and say, like, all, uh, droids are robots, for example. Um,

138. 23:56

     who likes creating synonym lists? Nobody anymore. Okay. No-normally, I would have said that's the Stockholm Syndrome because there is sometimes somebody who likes, uh, creating, uh, synonym lists because they've done that for so many years.

139. 24:09

     Um, but it got easier nowadays. Now you can use LLMs to generate your synonyms, so it can get a bit easier to c-create them. But they're still limited because you have always this mapping.

140. 24:19

     Um, so with synonyms, you can expand the right way. Where it gets trickier if you have homonyms, uh, if a word has multiple, um, meanings like a bet- Could be the animal or it could be the thing you hit a ball with.

141. 24:30

     Um, there it just gets trickier because there is, there is no meaning behind the words or no context. So you just match strings, um, and that is inherently limited.

142. 24:40

     But like I said, it's dumb, but it scales very well, and that's why it has been around for a long time, and it does surprisingly well for many things because there's not a lot of things that are unexpected or that can go totally wrong.

143. 24:53

     Um, now, other things that you can do. You could do a, a phrase search where you say, "I am your father." Um, will this find anything?

144. 25:04

     Yes.

145. 25:05

     Yes, because we had, "No, I am your father." Um,

146. 25:11

     what happens if I say, for example, "I am,"

147. 25:17

     uh, let's say, "I am not your father." Yes? No?

148. 25:22

     No.

149. 25:23

     No. Why? So, you're right.

150. 25:27

     Looking for an exact match based on the position. So since it's not, it won't.

151. 25:31

     But not as a stop word. But you're right, because the positions still don't match. Um, so the stop word not would be filtered out, um, but it still doesn't match because the positions are off.

152. 25:45

     Um, that is [REDACTED:generic_id] of the things that sometimes can be confusing. So even if something is a stop word and will be filtered out, uh, it doesn't work like that.

153. 25:53

     Um, [REDACTED:generic_id] thing that you can do is, though, that the factor is called slop, where you basically say if there is something missing, um, it would still work. So, "I am your father," and, "I am father," with slop, um, zero, that's kind of like the implicit [REDACTED:generic_id], um, will not find anything.

154. 26:11

     But if I say [REDACTED:generic_id], then I basically say, like, there can be a [REDACTED:generic_id] off in there, like [REDACTED:generic_id] word can be missing. Um,

155. 26:20

     uh, however, "I am his father," here his would not match, so this still will not work. Um, sl- the slop is really just to skip a word. Yeah.

156. 26:32

     What about, "I'm your father"?

157. 26:35

     I, I am your father?

158. 26:37

     I assume that... No, I, I apostrophe am your father. I assume that won't work-

159. 26:40

     Ah

160. 26:41

     ... but.

161. 26:42

     That will not work.

162. 26:43

     How would you get that to work?

163. 26:45

     Uh, there you might need to do something like a synonym where you say slash, um, M gets replaced by M. Uh, or we'll need to have some more machine learning capabilities behind the scenes to, to do stuff like that.

164. 26:58

     Are there any libraries that would predefine, uh, you know, contractions like that or?

165. 27:03

     So what, what is built in is generally a very simple set of rules. Um, what you will need to do for things like this is normally you need a dictionary.

166. 27:14

     Um, the problem around these is they're normally not available for free or open source. Um, funnily enough, they're often coming out of, uh, university, the s- the dictionaries, um, because they have a lot of free labor, the students. [laughs]

167. 27:29

     That's why the universities have been creating a lot of dictionaries, but they often come out under the weirdest licenses. That why they're ve- not very widely available. But yes, there is a smarter or more powerful approach if you have a dictionary and you can do these things.

168. 27:43

     Um, for example, um, [REDACTED:generic_id] thing to, to show is like--

169. 27:48

     Or maybe that's a good thing to, to also mention. Um,

170. 27:53

     you don't always get words out of the stemming. It's not a dictionary. It doesn't really get what you're doing. It just applies some rules. So for example, uh, Bl- uh, Blackberry.

171. 28:05

     Blackberry, um... Ah, sorry, blackberries. I think that this will be stemmed down differently. Ah, sorry, I need English. Without English, this will not work.

172. 28:19

     So this will stem down to this weird word, blackberry. Um, and it will also stem down the singular to blackberry. So there's a rule that applies this, uh, but it's just a rule.

173. 28:31

     It's not dictionary-based. It's not very smart. Um, and it only has some rules built in, um, that work for this, but you will definitely hit limits. Um, the other thing, by the way, and why I picked blackberry as an example, um, you have some annoying languages like German, Korean, and others that compound nouns like blackberry, um, where

174. 28:52

     you have basically two words. Um, black would never find blackberry in the simplest form because it's not a complete string. There are various ways to work around that, um, that all come with their own downsides.

175. 29:05

     Um, and either you have a dictionary or you extract the so-called n-grams. It's like group of words, and then you mention group of words. Um, but all of those are [REDACTED:generic_id] of the many tools how we try to make this a bit better or smarter, but it all has limitations.

176. 29:20

     I hope that answers the question and makes sense. Um, so there are dictionaries, but they're generally not free or not under an easy license available. For some languages, by the way, um, even the, the stemmers are not freely available.

177. 29:34

     I think there is a stemmer or analyzer for Hebrew. I think that has also, like, some commercial license, or at least, uh, you can't use it for free in commercial products.

178. 29:45

     Um, though licensing with machine learning models is also its own dark secret. Um, yeah.

179. 29:53

     Um, I guess maybe not exactly here because we don't know how to spell your name, but, like, uh, why not just have, like, ex-- much smaller tokens, um, like subword tokens?

180. 30:02

     Um, and then you'd be able to, like, match a lot of things. You're gonna have, like, more false positives, but presumably things which are true positives will have more matches.

181. 30:09

     Yes, um, that is what an n-gram is doing. Let me, let me see if I can, um...

182. 30:18

     An n-gram is normally a, a word group, normally a trigraph. This is way too small. Um,

183. 30:26

     somehow I have weirdly overwritten my command plus, so I can't use that. Let me make this slightly larger.

184. 30:37

     Okay. Um, here, um, we basically use [REDACTED:generic_id] or two letters as word groups, whi- which is way too small, but, uh, just to, um, show the example, and this is very hard to read.

185. 30:50

     Let, let me co-copy that over to my console that there you can, um,

186. 30:57

     there you can... Oops. There you can see this. But this is a great question. Um, so we'll use n-gram for quick fox,

187. 31:07

     and then you can see the tokens that I extract here are the first letter, the first two, um, the second, the second and third, et cetera. And you end up with a ton of tokens that the downside is, A, you have to do more work when you store this.

188. 31:21

     Um, B, it creates a lot of storage on disk because you extract so many different tokens. And then your search will also be pretty expensive because normally you would at least do [REDACTED:generic_id] like trigrams.

189. 31:33

     Um, but even that creates tons of tokens and then tons of matches, and then you need to find the ones with the most matches. And it works, but A, it is pretty expensive in disk, but also query time.

190. 31:45

     Um, and it might also create undesired results or res- results that are a bit unexpected for the end user. It is-- I would call it a-- It's, it's again, it's a very dumb tool that works reasonably well for some scenarios, um, but it's only [REDACTED:generic_id] of many potential factors.

191. 32:01

     Um, what you could potentially do is, and I, I don't have a full example for that, but we could build it quickly. Um, what you would do in reality probably, um, you might store a text more than [REDACTED:generic_id] ways.

192. 32:13

     So you might store it like with stop words and without stop words and maybe with n-grams. And then you give a lower weight to the n-grams and say like, "If I have an exact match, then I want this first.

193. 32:24

     But if I don't have anything in the exact matches, then I want to look into my n-gram list. Uh, and then I want to kind of like take whatever is coming up next."

194. 32:32

     Um, so even keyword-based search will be more complex if you combine different methods. Um,

195. 32:41

     n-grams are interesting, but again, they're a dumb but pretty heavy hammer. Use them with the right or at the right scenario.

196. 32:48

     Sorry, quick question about these n-grams. Is it by default [REDACTED:generic_id] or two?

197. 32:52

     Yes.

198. 32:53

     Oh.

199. 32:53

     But you, you could, uh, redefine that. So we can, uh, we can... Let me go back to the docs. The, the n-gram, you can, uh, say min gram and max gram.

200. 33:04

     If you set both to [REDACTED:generic_id], you would have trigrams, where it's always groups of [REDACTED:generic_id], like [REDACTED:generic_id], two, [REDACTED:generic_id], two, [REDACTED:generic_id], four, et cetera. Um, you could also have something called edge n-gram, where you expect that somebody types the first few letters right, uh, and then you only start from the beginning but not in the middle of

201. 33:19

     the word, which sometimes avoids unexpected results and, of course, reduces the number of tokens, um, quite a bit. Um, so somewhere in here, um,

202. 33:33

     edge n-gram. Let's just copy that over so I won't type and...

203. 33:43

     So here we have edge n-gram with quick, and you can see it only does

204. 33:49

     the first and the first two letters, but nothing else. And in reality, you would probably define this like two to five or more or whatever else you want. Uh, but here we only do from the start and nothing else, uh, which reduces the tokens tremendously.

205. 34:03

     But of course, if you have blackberry and you want to match the berry, you're out of luck.

206. 34:11

     Make sense? Anybody else? Anything else?

207. 34:16

     Yeah. So another question.

208. 34:17

     Sure.

209. 34:18

     So if you're working with a language like English and then Hebrew, where it's right to left, how do you deal with the indexes and stuff like that?

210. 34:26

     Yeah. So if you have multiple languages, um, do not mix them up. That will just create chaos. Uh, because we'll get to that in a moment, but how keyword search works is basically word frequencies, and it-- if you mix languages, it screws up all frequencies, uh, and statistics.

211. 34:42

     Um, so what you would do is either you would have, um,

212. 34:48

     you could do two things, uh, to do, um...

213. 34:53

     If you have English and Hebrew, uh, you could either do field English, and then you would have

214. 35:03

     field, uh, whatever the abbreviation for Hebrew is. Um, Hebrew, uh, oops,

215. 35:12

     and then you would have that. And then you would need to define the right analyzer for that specific field. But, so you break it out either into different fields, or you could even do different indices.

216. 35:22

     Um, and ideally, we even have that built in. Um, we have a language analyzer. Even if you just provide a couple of words, it will guess or not guess.

217. 35:32

     It will infer the language, um, with a very high, uh, degree of certainty. Um, especially Hebrew will be very easy to identify. Uh, if you have your own diacritics, it's easy.

218. 35:43

     Um, but even if you just throw random languages at it, it will have a very good chance, like just with a few words, uh, to know this is this language, and then you can treat it the, the right way.

219. 35:56

     Um, good. Let's continue. Um, so we have done all of these searches. Uh, we have done slope. Uh, [REDACTED:generic_id] more th-thing before we get into the, the relevance.

220. 36:13

     [REDACTED:generic_id] other very heavy hammer, um, that people often overuse is fuzziness. Uh, so let's see. Uh, if you have a misspelling, so I misspelled Obi-Wan Kenobi. Uh, we already know that this is broken out into two different words or tokens.

221. 36:30

     Um, it will still match your Obi-Wan because we have this fuzziness, uh, which, uh, allows edits. It's like a, it's a Levenshtein distance, so you can have [REDACTED:generic_id]-- By default here- You could either give it an absolute value, like you can have [REDACTED:generic_id] edit, which could be [REDACTED:generic_id] character too much, too little, or [REDACTED:generic_id] character different.

222. 36:52

     Uh, you could set it to two or [REDACTED:generic_id]. You can't do more, because otherwise, uh, you match almost anything. Um, and auto will-- auto is kind of smart because auto, depending on how long the token that you're searching for is, uh, will set a specific value.

223. 37:08

     And if you have zero to two characters, auto fuzziness, I think, is [REDACTED:generic_id] from, um, two to-- no, ze-zero to two characters is zero, uh, [REDACTED:generic_id] to five characters is [REDACTED:generic_id], and after that it's two.

224. 37:21

     Um, so you can match these. Um, will this [REDACTED:generic_id] match?

225. 37:31

     Yes, no, and why?

226. 37:36

     Uh, no, because you've got the B and the W.

227. 37:40

     Yes. So we have, we have both of those are misspelled.

228. 37:46

     It still matches. Why?

229. 37:49

     They get tokenized separately, and each [REDACTED:generic_id] has a single Levenshtein distance.

230. 37:53

     Yes. That is a bit of a gotcha. Um, so yes. You need to know the tokenizer. So we tokenize with standard, so it's two tokens, and then the fuzziness applies per token,

231. 38:03

     which is another slightly surprising thing. Uh, but yes, that's how you end up here.

232. 38:08

     Um, okay. Now, we could look at like the, how the Levenshtein distance works behind the f-scenes, but it's,

233. 38:17

     it's basically a Levenshtein automaton, um, which looks something like this. If you search for food and you have two edits, this is how the automaton would work in the background to figure out like what are all the possible permutations.

234. 38:28

     Um, it's a fancy algorithm, uh, that was, I think, pretty hard to implement, uh, but, uh, it's in, uh, Lucene nowadays. Um, okay. Now, let's talk about scoring. Uh, [REDACTED:generic_id] thing that you have seen that you don't have anywhere or in a non-search engine or just in a database is like we have these scores.

235. 38:46

     Like how well does this match? Um, how does a score work here? Um,

236. 38:53

     let's look at the, the details of that [REDACTED:generic_id].

237. 38:58

     Um, so the, the basic algorithm, which is also most of us, or pretty much all of us here, um, term frequency, inverse document frequency or TF-IDF, um, it has been slightly tweaked, like the new implementation is called BM25, which stands for best match, and it's the twenty-fifth iteration of the best match algorithm.

238. 39:17

     Um, so what they look like is, um, you have the term frequency. If I search for droid, how many times does droid appear in the text that I'm looking for?

239. 39:27

     Um, and it's basically the square root of that. So the assumption is, um, if a text contains droid once, this is the relevancy. If I have a text that contains droid ten times, um, this is the relevancy.

240. 39:40

     The tweak between TF-IDF, that [REDACTED:generic_id] just keeps growing. Uh, BM25 says like once you hit like five droids in a text, it doesn't really get much more relevant anymore.

241. 39:50

     Um, so it kind of like flattens out the curve. Um, that is the idea of term frequency. Um, the next thing is the inverse document frequency, um, which is almost the inverse curve.

242. 40:02

     The assumption here is over my entire text body, this is how often the term droid appears. So if a term is rare, it is much more relevant than if a term is, uh, very common.

243. 40:16

     Then it's kind of like less relevant. Basically, the assumption is rare is relevant and interesting, very common is not very interesting anymore. Uh, and then it's kind of like just, uh, works its curve out like that.

244. 40:28

     Um, and the final thing is, uh, the field length norm is like the shorter a field is and you have a match, the more relevant it is. Which assumes like if you have a short title and your keyword appears there, it's much more relevant than if there's a very long text body and your keyword or, and you

245. 40:45

     have a match there. Um, and these are the [REDACTED:generic_id] main components of, uh, TF-IDF. So let's take a look at how this looks like. Um,

246. 40:54

     you can make this a bit more complicated, um,

247. 40:58

     at- for you why something matches. Don't be confused by the-- Or let me take that out for the first try. So I'm looking for father, and I am, uh, "No, I am your father," and, "Obi-Wan never told you what happened to your father."

248. 41:14

     Um, [REDACTED:generic_id] is more relevant than the other. Why is the first [REDACTED:generic_id] more relevant than the second [REDACTED:generic_id]?

249. 41:21

     Term frequency.

250. 41:21

     Yeah. Term frequency is the same. Both contain, contain, uh, father once. The, uh, inverse document fre-quency is also the same because we're looking for the same term. Uh, the only difference is that the second [REDACTED:generic_id] is longer than the first [REDACTED:generic_id], um, and that's why it's, um, more relevant here.

251. 41:41

     Um, so this is very simple. And you can then, if you're unsure why something is calculated in a specific way, you can add this explain true, uh, and then it will tell you all the, the details of like, okay, we have father, um, and it then calculates basically all the different pieces of the formula for you and

252. 42:00

     shows you how it did the calculation. So you can debug that if you need to, um, but it's probably a bit too much output for the everyday use case.

253. 42:08

     Um, and then you can customize the score if you want to. Here I'm doing a, a random score. So my two fathers, uh, this is a bit hard to show, um, they will just be in random order because their score is here, um, randomly assigned.

254. 42:24

     Um, but you could do this more intelligently that you combine like the score and like if you have, I don't know, the margin that on a product that you sell or the rating that you include that in the rating somehow, and you can build a custom score for th-things like that.

255. 42:38

     Um, so you can influence that any way you want. Um,

256. 42:43

     [REDACTED:generic_id] thing that I See every now and then that is a very bad idea, and we'll skip this [REDACTED:generic_id] because it's probably a bit too much. Um, this [REDACTED:generic_id], by the way, is the, is the total formula that you can do, or maybe I'll show you this parts that I skipped.

257. 42:57

     What happens if you search for two terms and they're not the same... They don't have the same relevancy? So what the calculation behind the scenes basically looks like is, let's say we search for, um, father.

258. 43:09

     Father is very rare, that's why it's much more relevant than your. Your is pretty common. And then we have a document that contains your father. It's kind of like this axis.

259. 43:18

     This is like the... This will be the best match. But will a wor- will a document that only contains father be more relevant or only your? Intuitively, the [REDACTED:generic_id] with just father will be more relevant.

260. 43:31

     But how does it calculate that? Um, it basically calculates like this is the relevancy of father, um, this is the ideal document, and this is your, and then it c- looks like which [REDACTED:generic_id] has the shorter angle, and this is the [REDACTED:generic_id] that is more relevant.

261. 43:45

     Um, so if you have a multi-term search, um, you can figure out which term is more relevant and how they are combined. Um, and then you can also have the coordination factor, which basically re-rewards documents containing more of the terms that you're searching for.

262. 43:59

     So if you-- I'm searching for [REDACTED:generic_id] ti-terms like, um, you-

263. 44:04

     I am father, whatever, um, if a document contains all [REDACTED:generic_id], this will be the, the formula that combines the scores of all [REDACTED:generic_id] and multiplies it by [REDACTED:generic_id] divided by [REDACTED:generic_id].

264. 44:15

     If it only contains two of them, it would only have the relevancy of two third and with [REDACTED:generic_id], [REDACTED:generic_id] third. And then you put it all together, and this is the formula that happens behind the scenes, and you don't have to do that in your head, luckily.

265. 44:28

     Cool. We have seen these, um... [REDACTED:generic_id] thing that I or we see every now and then is that people try to translate the score into percentages. Um, like you say, "This is hundred percent, percent score, and this is only like a fifty percent match."

266. 44:46

     Who wants to do that? Hopefully nobody, uh, because the Lucene documentation is pretty explicit about that. Um,

267. 44:56

     you should not think about the problem in that way because it doesn't work. Um, and I'll show you why it doesn't work or how this breaks. Um, let's take another example.

268. 45:06

     Uh, let's say we take this short text. "These are my father's machines." I, I think of a good Star Wars quote to use here, but bear with me. Um, so what remains if I run this through my analyzer?

269. 45:22

     My father machine. These are the [REDACTED:generic_id] tokens that remain. Now, I will store that, you remember, with the [REDACTED:generic_id] tokens that we have stored. Um, and if I search for my father machine, um, you might be inclined, uh, to say, um, this is, um, the perfect score.

270. 45:44

     This is like a hundred percent. Agreed? Because all the [REDACTED:generic_id] tokens that I have stored in these are my father's machines, um, are there. So this must be like my perfect match.

271. 45:55

     So it's [REDACTED:generic_id] point two, that would be a hundred percent. The problem now is every time you add or remove a document, the statistics will change, and your score will change.

272. 46:04

     So if I delete that document, um, and I search the same thing again, um, I don't know what percentage this is now. Is this now the new hundred percent, the best document, or is this a zero point or, I don't know, twenty percent?

273. 46:17

     Um, how does this compare? Um, and then you can ply- play funny tricks where these droids are my father's father's machines, um, and you can see I have a term frequency of two for father here.

274. 46:30

     So if I store that [REDACTED:generic_id] then and then search it, um,

275. 46:38

     is this now a hundred percent? Is this now a hundred ten percent? Um, so don't try to translate, uh, scores into percentages. They're only relevant within [REDACTED:generic_id] query. They're also not comparable across queries.

276. 46:50

     They're really just sorting within [REDACTED:generic_id] query, um, to do that. Um, okay, let me get rid of this [REDACTED:generic_id] again. Now,

277. 46:59

     we've seen the limitations, uh, of keyword search. We don't want to define, um, our, our synonyms. We might want to extract a bit more meaning. So we'll, we'll do some simple examples, uh, to extend.

278. 47:13

     Um, I will add, uh, from OpenAI text embedding small.

279. 47:20

     I'm basically connecting that inference API for text embeddings here in my instance. Um, I have removed the API key. You will need to use your own API key, um, if you want to use that, but it is already configured.

280. 47:33

     Um, so let me pull up the inference services that we have here. I have done, or I have added two, uh, different models. Uh, [REDACTED:generic_id] sparse, [REDACTED:generic_id] dense. Let's go to these.

281. 47:48

     Uh, by the way, if you try to do this, um, with a hundred percent score, don't do this, um,

282. 47:55

     because it will just not work. Um, okay.

283. 48:00

     Not everybody has worked with dense vectors, right? Uh, so I have a couple of graphics coming back to our Star Wars theme, uh, just to look at how that works.

284. 48:08

     So what you do with dense vectors is, um, we keep this very simple. Um, this [REDACTED:generic_id] just has a single, uh, dimension, um, and it has like the axis is pretty much like realistic Star Wars characters and cartoonish Star Wars characters.

285. 48:23

     And this [REDACTED:generic_id] falls on the realistic side, and that other [REDACTED:generic_id] is just cartoonish. And you have model behind the scenes that can rate those images and will figure out like where they fall.

286. 48:35

     Um, now in reality, you will have more dimensions than [REDACTED:generic_id], um, and you will also have floating point precision, so it's not just like, um, minus [REDACTED:generic_id], zero or [REDACTED:generic_id].

287. 48:46

     Um, but You will have more pr- dimensions. So for example, here in human and machine, and in a realistic model, you don't have like the dimensions are not labeled as nicely and clearly understandable.

288. 48:59

     The machine has learned what they represent, but they're not representing an actual thing that you can extract like that. Um, but in our simple example here now, we can say this Leia character, um, is realistic and a human versus, um, I don't know, the Darth Vader, um, is cartoonish and, I don't know, somewhere between human and machine.

289. 49:22

     Um, so this is their representation in the vector space. And then you could have, like I said, you could have floating point values, and then you can have different characters, um, and similar characters, like both of those are human.

290. 49:34

     Um, without the hand, he's only like not quite as human anymore, so he's a bit lower down here. Um,

291. 49:41

     uh, so he's a bit closer to the machines. Um, so you can have all of your entities in this vector space, and then if you search for something, you could figure out like which characters are the closest, uh, to this [REDACTED:generic_id].

292. 49:55

     And again, in reality, you will have hundreds of dimensions. Uh, it will be much harder to say like, uh, these are the explicit things and this is why it works like that.

293. 50:04

     Um, it will depend on how good your model is, uh, in interpreting your data and extracting the right meaning from it.

294. 50:12

     Um, but that is the general idea of dense vector representation. You have your documents, or sometimes it's like chunks of documents, um, that are represented in this vector space, and then you try to find something that is close to it, um, for that.

295. 50:28

     Um, does that make sense for everybody or any specific questions?

296. 50:35

     So it's a bit more opaque, I want to say. It's not quite as easy because you so say like these five characters match these other five characters here. Um, but you need to trust or evaluate that you have the right model to figure out how these things connect.

297. 50:50

     So let's, um, see how that, uh, looks like.

298. 50:55

     Um, I have [REDACTED:generic_id] dense vector model down here. We have OpenAI embedding. Um, this [REDACTED:generic_id] is a very small model. It only has [REDACTED:generic_id] hundred and twenty-eight dimensions. Um, the results will not be great, but it's actually for demonstrating, uh,

299. 51:14

     it actually helpful. Uh, so we'll see that. The other model that we have, and let me show you the output of that. So if I take my text, "These are not the droids you're looking for," this is the representation.

300. 51:26

     It's basically an array of floating point values, uh, that will be stored, and then you just look for similar floating point values. And then you have, "These are not the droids you're looking for" here on the previous [REDACTED:generic_id],

301. 51:39

     uh, uh, dense text embedding. Um, this [REDACTED:generic_id] here does sparse embedding. Sparse is the, the main model used for that is called Splade. Um, our input of Splade is, we call it Elser.

302. 51:53

     Uh, it's kind of like a slightly improved, uh, Splade, but the concept is still the same. What you get is, um, you take your,

303. 52:02

     you take your words, and this is not just a TF-IDF. This is a learned representation where I take all of my tokens and then expand them and say like, "For this text, um, these are all the tokens that I think are relevant."

304. 52:17

     And this number here basically tells me how relevant they are. Um, again, not all of these make sense intuitively. Um, and you might get some funky results, for example, with foreign languages.

305. 52:30

     This currently only supports English. Um, but these are all the terms that, uh, we have extracted normally. Um, yeah, you get like a hundred something or so. Um, so the, the idea is that this text is represented by all of these tokens, and the higher the score here, the more, uh, important it is.

306. 52:52

     And what you will do is you store that behind the scenes. When you search for something, um, you will generate a similar list, and then you look for the ones that have an overlap, and you basically multiply the scores together, and the ones with the highest values will then find the most relevant document.

307. 53:09

     This is insofar interesting or nice because it's a bit easier to interpret. It's not just like long array of floating point values. Um, sometimes these don't make sense. The main downside of this though is that it gets pretty expensive at query time because you store like a ton of different tokens here for this.

308. 53:29

     Um, when you retrieve it, um, the search query will generate a similar long list of terms and at-- if you have a large enough text body, um, a query might hit a very large percentage of your entire stored documents to-- with these OR matches because basically these are just a lot of ORS that you combine, calculate, uh,

309. 53:51

     the score, and then return the most or the, the highest ranking results. Um, so it's an interesting approach. It didn't gain as much traction as dense vector models, but it can be as a first step or an easy and interpretable step.

310. 54:05

     It can be a good starting point, um, to dive into the details here. Um- So do, do, so this is your entire vocabulary? What you're showing- Uh, so- No, right ...

311. 54:16

     this, "These are not the droids you're looking for" is basically represented by this embedding here.

312. 54:24

     So it's like this entire list of, of terms with these, um, yeah, with this relevancy, basically. This is the representation of this string. And then when I search for something, I will generate a similar list, and then I basically try to match the two together, like for what is-- has the most or the highest matches here.

313. 54:47

     Make sense? Yes, we'll do that in a second. Um, it, it's a bit of because, um, with the or, um, like it doesn't tell you exactly this is what matched, uh, but yes.

314. 55:08

     Um, so I will need to create a new index. Um, this [REDACTED:generic_id] keeps the configuration from before, but I'm adding this semantic text for the, the sparse model and the dense model.

315. 55:21

     So I've created this [REDACTED:generic_id]. Uh, and now I'll just put [REDACTED:generic_id] documents I have my other index. Um, as you can see here it says [REDACTED:generic_id] documents were moved over.

316. 55:31

     Um, so we can then start searching here, and, um, if I look at that, the first document is still, "These are not the droids you're looking for." You don't see like for the, for a keyword search, you don't see the extracted tokens here.

317. 55:44

     We also don't show you the, the dense vector representation or the sparse vector representation. Those are just stored behind the scenes, um, for querying, but there's no real point in retrieving them because you're, you're not going to do anything with that huge array of, of dense vectors.

318. 55:58

     Um, it will just slow down your searches. Um, you can look at the, the mapping and you can see, um, I'm basically copying my existing quote field to these other two that I can also search those.

319. 56:12

     Okay. So if I s- look for machine on my original quote, will it find anything?

320. 56:21

     No.

321. 56:21

     No?

322. 56:21

     Yes.

323. 56:25

     No, because it only had, "These are not the droids you're looking for." And this is still a keyword search.

324. 56:32

     So this is just a query from before just to show you that, yes, we are not matching anything because it only had, "These are not the droids you're looking for," but here we're looking for machine.

325. 56:40

     Um, it was still a keyword matching. Doesn't work, shouldn't work. That's exactly the result that we want out of this here. Um, now if I say answer, and I say machine,

326. 56:52

     then it will match here. These are not the droids you're looking for, and you can see this [REDACTED:generic_id] matches pretty well, I don't know, at zero point nine. But it also has some overlap with, "No, I am your father."

327. 57:03

     I mean, it is much lower in terms of relevance, uh, but something had an overlap here. And only the, the third document,

328. 57:12

     um, "Obi-Wan never told you what happened to your father," only that [REDACTED:generic_id] is not in our result list at all. But there was something here, I don't know the expansion.

329. 57:20

     We would need to basically run-- Where was it? We would need to run, uh, this [REDACTED:generic_id] here for all the strings and look then for the expansion of the query, and then there would be some overlap, and that's how we retrieve that [REDACTED:generic_id].

330. 57:36

     Um-

331. 57:37

     So is there like some sort of threshold that you have that-

332. 57:39

     You could define a threshold. Um, it will though depend, um... Let's see. Uh, these are not the droids you're looking for. Let's say if I,

333. 57:53

     if I say-- I'm not sure if this will change anything.

334. 58:02

     I mean, the relevance here is still, it's still ten x or so. Um, but yeah, this [REDACTED:generic_id] still [audio cuts out]

335. 58:11

     Will just have a very low match. It's still [audio cuts out]

336. 58:16

     terms you look for. Um, the score just totally jumps around. So it's a bit hard to define the threshold.

337. 58:23

     Because here you can see in my previous query, we might have said zero point two is the cutoff point, but now it's actually zero point four, even though it's not super relevant.

338. 58:32

     So it might be a bit tricky, uh, or you might need to have a more dynamic threshold depending on how many terms you're looking for and, uh, what is a relevant result.

339. 58:41

     In the, in the bigger picture, the assumption would be if you have hundreds of thousands or even millions of documents, um, you will probably not have that problem that anything that is so remotely connected will actually be in the top ten or twenty or whatever list that you want to retrieve.

340. 58:57

     Um, so for larger proper data sets, this should be less of an issue. With my hello world example of [REDACTED:generic_id] documents, um, it can be a bit misleading. But yes, you can have a cutoff point if you figure out what for your data set and your queries is a good cutoff point, you could define a cutoff point.

341. 59:14

     No, sorry, but you have [REDACTED:generic_id] documents. How come it's only showing two? Is it because of there's a-

342. 59:18

     Be- be-- So the query gets expanded into, I don't know, those hundred tokens or whatever. And then for those two there is some overlap, but the third [REDACTED:generic_id] just didn't have any overlap.

343. 59:29

     Oh, wow.

344. 59:29

     But I-- So we-- What-- Okay, we can, we can do that. It's just a bit tricky to figure out the, the term that has the overlook, uh, uh, the overlap.

345. 59:38

     So we will need to take this [REDACTED:generic_id], machine, uh, "No, I am your father." Let's take this [REDACTED:generic_id].

346. 59:46

     What you need to do is, um, to figure that [REDACTED:generic_id] out... I know actually we

347. 59:51

     should be able-- Let me see. Um, let's see.

348. 1:00:09

     Ba, ba, ba, ba, bum. This is a pretty long output.

349. 1:00:16

     Somehow I was actually hoping that it will show me the term that has matched here. Okay, I see something.

350. 1:00:25

     Okay, there is something puppet that seems to be the overlap.

351. 1:00:30

     How much sense that term expansion for the, the stored text and the query text makes is a bit of a different discussion. Uh, but in here with that explain true, you can actually see how it matched and what happened behind the scenes if you have any really hard or weird queries or something that is hard to explain

352. 1:00:47

     to debug that. But the third [REDACTED:generic_id] didn't match. Um, now if I take the dense vector model with OpenAI and I search for machine, how many results do you expect to get back from this [REDACTED:generic_id]?

353. 1:01:04

     Zero, [REDACTED:generic_id], two, [REDACTED:generic_id]. Yes, [REDACTED:generic_id]. Why [REDACTED:generic_id]?

354. 1:01:09

     There'll always be some result.

355. 1:01:10

     Yes, because there's always some match. That is the other... Or let me run the query first.

356. 1:01:17

     Um, here you... These are not the droids you're looking for. This [REDACTED:generic_id] is the first [REDACTED:generic_id]. Um, I don't think that this model is generally great because here the results are super close.

357. 1:01:27

     It is, I mean, the, the droids with the machines, that is the first [REDACTED:generic_id], but the score is super close to the second [REDACTED:generic_id], which is "No, I am your father," which feels pretty unrelated, and, uh, "Obi-Wan never told you what happened to your father."

358. 1:01:39

     Even that [REDACTED:generic_id] is still with a reasonably close score. Uh, but why do we have those? Um,

359. 1:01:47

     because if we say, "What is the relevance?" Uh, I mean, it's further away, but it's always like there is always kind of like some angle to it, even if the kind of like the angle here or depending on the, the similarity, uh, calculation that you do, but it's still always related.

360. 1:02:04

     There is no easy way to say something is totally unrelated. Um, there is, by the way, [REDACTED:generic_id] good thing about keyword search, where it was relatively easy to have a cutoff point of things that are totally not relevant, where you're not going to confuse your users.

361. 1:02:18

     Whereas here, if you don't have great matches, you might get almost... It's not random, but it's m- potentially it looks very unrelated to your end users what you might return, just because it's very hard to show.

362. 1:02:31

     Yes.

363. 1:02:31

     Is it fair to say then that the, um, like the OpenAI embedding search is worse for this kind of toy example because the magnitude of difference is, is much-

364. 1:02:42

     I, I'm careful with worse because it's really a hello world example, so I don't take this as a quality measurement in any way. Um, I, I am... Yeah, I mean, the, the OpenAI model with 128 dimensions is very few dimensions.

365. 1:02:55

     I think it will probably be cheap but not give you great results necessarily, but don't use this as a benchmark. I think it's just a, a good way to see, um, that you...

366. 1:03:05

     This is now much harder because now you need to pick the right machine learning model, uh, to actually figure out what is a good match. With keyword-based search, it was a bit of a different story.

367. 1:03:15

     There you need to pay more attention to like how do I tokenize and do I have the right language and do I do stemming or not stemming? But most of that work is relatively, I want to say, almost algorithmic.

368. 1:03:26

     Um, and then you can figure that out and you configure it, and then it's very predictable at query time. Um, whereas with the dense vector representation,

369. 1:03:36

     you really need to evaluate for the queries that you run and the data that you have, like is that relevant and is that an improvement or not? Um, it's very easy to get going and just throw a dense vector model together, um, and you will match...

370. 1:03:48

     You will always match something that might be an advantage over the, the lexical search where you don't have any matches, which sometimes is the other problem, that nothing comes back and you would to-- want to have at least some result.

371. 1:04:00

     Um, here it might just be unrelated, so that, that can be tricky. Um, the... You want to have some results. This, by the way, um, a funny story that a European e-commerce store once told me.

372. 1:04:13

     They said they accidentally deleted, I think, two-thirds of their data that they had for the products that you could buy. And then I asked them like, "Okay, so how much revenue did you lose because of that?"

373. 1:04:26

     And they said, "Basically nothing, because as long as you showed some somewhat relevant results quickly enough, people would still buy that." So only if you have no results, that's probably the worst.

374. 1:04:37

     Uh, so for an e-commerce store, you might want to show stuff a bit further out because people might still buy it. Uh, but it really depends on the... I'm, I'm coming to you in a moment.

375. 1:04:47

     Uh, it really depends on your use case. E-commerce is kind of like [REDACTED:generic_id] extreme where you want to show always something for people to buy. Um, if you have a database of legal cases or something like that, you probably don't want that approach, uh, because that will go horribly wrong.

376. 1:05:01

     So it is very domain specific. Um, that's I think also the good thing about search because it keeps a lot of people employed because it's not an easy problem.

377. 1:05:11

     It's, it's almost job security because it depends much on the, this is the data that you have, and this is the query that people run, and this is the expectation of what will happen, and this is, for this domain, the right behavior.

378. 1:05:23

     Um, so there's no easy right or wrong with a checkbox. And the other thing is you might make... If you tune it, you might make it better for [REDACTED:generic_id] case, but worse for 20 others.

379. 1:05:33

     That's why a robust evaluation set is normally very important, though very rare. Uh, a lot of people YOLO it, um, and you will see that in the results. And for the e-commerce store, it probably works well enough.

380. 1:05:45

     Uh, sorry, you had a question.

381. 1:05:46

     Um, can I limit the semantic enrichment to a subset of my index based off of properties of the document? So if I have a very large shared index with a lot of customers and I wanna enable AI for a subset of the index, can I say, "Hey, only do the semantic enrichment if the document has this property

382. 1:06:04

     where maybe it's like an AI customer?"

383. 1:06:07

     Yeah. So the, the way we would do it in our product, um, is that you would probably have two different indices with different mappings.

384. 1:06:16

     Yeah, but then it's not so fun, like the customer upgrades, and I have to migrate them to the new index.

385. 1:06:23

     Uh, Dave, please.

386. 1:06:25

     Yeah. So if you, for example, have an index in Elasticsearch, you can think of it almost like a sparse table, right? So there's no penalty for having a field that is not populated.

387. 1:06:37

     So either in your application or in an ingest processor, you could have an if statement and say-

388. 1:06:42

     Yeah. That's how we do it now.

389. 1:06:44

     Yeah. Only, only move it over. Um-

390. 1:06:45

     I was wondering if this, with this automatic way where you kind of turn it off.

391. 1:06:48

     No. The, the problem is the data structure, like if the field is there... So the data structure that we build in the background is called HN. And either we build the data structure or we don't build it.

392. 1:06:59

     Yeah. So if you had, you know, 10 billion entries in your vector-

393. 1:07:14

     Triggering the inference to create the dense vector to put into there, then it's just gonna be a, you know, a bunch of-- the index is just a bunch of pointers, and none of them head towards the HNSW-

394. 1:07:24

     Right

395. 1:07:24

     ... and it won't show up in search results. The penalty to you is nothing.

396. 1:07:28

     Okay.

397. 1:07:28

     Right? Um, but you're gonna have to manage what does or does not create the vector. You could do that in an ingest processor by just saying, "Hey, we're gonna use the copy command to have two copies of the text, [REDACTED:generic_id] that's meant for non-vector indexing, [REDACTED:generic_id] that's meant for actual vector indexing."

398. 1:07:45

     But you'd have to manage that with some tricky, complex AI technology called if-then-else.

399. 1:07:50

     Mm-hmm.

400. 1:07:50

     Right? Somewhere inside of your ingesting pipeline.

401. 1:07:53

     Right.

402. 1:07:54

     And it would work just fine.

403. 1:07:56

     Yeah. Um, [REDACTED:generic_id], [REDACTED:generic_id] more question. Uh, when we did, uh, HNSW with Elastic, we found that, uh, it was extremely slow at write time, and the community suggested that we freeze our index if we were gonna use, uh, HNSW.

404. 1:08:12

     Force merge? Or... Mm.

405. 1:08:14

     Yeah. I, I think just freeze writes. Um, they said, they said build the index and freeze it. Otherwise you'll put a ton of load on the CPU.

406. 1:08:21

     I mean, yes.

407. 1:08:22

     So, uh, what we found is that some of the default kind of like have been around in Elasticsearch for ten years settings with the merge scheduler are really optimized for keyword search.

408. 1:08:31

     Mm-hmm.

409. 1:08:32

     And for high update workloads on HNSW, we've got some suggestions.

410. 1:08:36

     Okay.

411. 1:08:37

     Um, they, they take a little bit of parameter sweet tuning to go find something right for your IOPS and for your actual update workload, but sometimes it's about the merge scheduler and not doing kind of an inefficient HNSW build when it's not important for your use case.

412. 1:08:51

     Okay.

413. 1:08:52

     Um, the other thing we'd say is that sometimes, uh, friends don't let friends run on Elasticsearch 8.11. Upgrade, upgrade, upgrade. They put a lot of optimization work into here 'cause it should be simple.

414. 1:09:03

     Okay.

415. 1:09:04

     So the, the reason-

416. 1:09:05

     It used to be bad. You are correct.

417. 1:09:07

     The, the reason why that is, uh, it's like merging-- So because you've got the immutable segment structure in Elasticsearch, uh, and HNSW you cannot easily merge. You basically need to rebuild them.

418. 1:09:20

     The [REDACTED:generic_id] trick, I forgot which version it was, uh, I'm not sure, Dave, if you remember it. I think it was even before 8.11. But basically, if we do a merge, we would take the, the largest segment with not deleted documents and basically plop the new documents on top of them rather than starting from scratch from two

419. 1:09:35

     HNSW data structures. There's another optimization somewhere now in 9.0, uh, that will make that a lot faster, so it really depends on the, the version that you have. And there are a couple of tricks that you can play.

420. 1:09:47

     Uh, but yeah, that is [REDACTED:generic_id] of the downsides of like the way immutable segments work and HNSW is built that you can't easily just merge it as easily together as other data structures, uh, because you really need to rebuild the HNSW data structure or like take the largest [REDACTED:generic_id] and then plop the other [REDACTED:generic_id] in.

421. 1:10:04

     Okay.

422. 1:10:05

     Some of the things were just like we fixed it in the next version we'll see-

423. 1:10:09

     Yeah

424. 1:10:09

     ... so I want you to upgrade.

425. 1:10:09

     We, we found like A-ANN was too slow, and then A-ANN broke our CPU, and then we moved to Fine-tune, and then now we're curious.

426. 1:10:18

     Stay curious.

427. 1:10:18

     Yeah.

428. 1:10:19

     Yeah. [laughs]

429. 1:10:20

     Might have been a while ago. Uh, yeah?

430. 1:10:22

     Um, so for like traditional document search, you know, I'm saying like, "Hey, please find me a document that contains my search query," right? Um, for R in the context of RAG, it might be something more like, "Hey, come up with a fun plan for my weekend," right?

431. 1:10:36

     And then the, the documents that we wanna find don't necessarily look like the search query, right?

432. 1:10:41

     Yeah.

433. 1:10:42

     Um, so like [REDACTED:generic_id] approach to that is you just give-- it's an agent, and you give it a search tool, and it searches, right? But like I'm just curious what you-- how you think about that in general.

434. 1:10:51

     Yeah. I feel like RAG has been very heavily abused as or like the mental model I think started off as like you do retrieval, and then you do the generation.

435. 1:10:59

     But you could do the generation earlier on as well, that you do the query rewriting and expanded query. Um, so I-- my favorite example for that is, um, you're looking for a recipe.

436. 1:11:12

     You don't need to have the, the LLM regenerate the recipe. You just want to find the recipe. But maybe you have a scenario where you forgot what the thing is called that you want to cook, and then you could use the LLM, for example, to tell you what you're looking for.

437. 1:11:25

     Um, like you say like, "Oh, I w- I'm looking for this Italian dish, uh, that has like these layers of pasta and then some meat in between." And then the LLM says like, "Oh, you're looking for lasagna."

438. 1:11:36

     And then you basically do the generation first or a query rewriting, um, and then search and then get the results as a very, bless you, uh, e-explicit example here.

439. 1:11:46

     Your example will look very different and probably smarter than my example. Uh, but query rewriting is [REDACTED:generic_id], um, thing. There's also the-- this concept of Hyatt, where your, your documents and your queries often look very different, and that you use an LLM to generate something from the query that looks more close like the doc- the documents that

440. 1:12:07

     you have. And then you match the documents together because they are more similar in structure. Um, so there are all kinds of interesting things that you can do. Um, like I said earlier, the it depends is becoming a bigger and bigger factor.

441. 1:12:19

     Uh, but yeah, your use case is probably might be,

442. 1:12:24

     yeah, maybe a multi-retrieval where you figure out like, oh, you look... I don't know. I've-- I know the example from an, uh, uh, e-commerce store where it's like, "I'm going to a theme party from the 1920s, and give me some suggestions."

443. 1:12:38

     And then the LLM will need to figure out like what am I searching for, and then it can retrieve the right items a-and rewrite the query and then actually give you proper suggestions.

444. 1:12:46

     Uh, but it's, it's not just running a query anymore.

445. 1:12:50

     Uh, yeah?

446. 1:12:51

     Uh, we, we use the destruction, uh, tuning for many models, and we kind of like cur-- give the list because we can still do it in small class. And then like a-along with your query, you can say like, "This is the ki-- like this is the kind of thing I am doing."

447. 1:13:04

     Um, and you can similarly embed the documents this way. Um, so you can like, uh,

448. 1:13:11

     like you can have document query embeddings. We've tried a-

449. 1:13:23

     You want to go through things, go through things,

450. 1:13:27

     and that is requiring a 70-pass process, which means what?

451. 1:13:37

     Yeah?

452. 1:13:38

     How should we be thinking about the number of dimensions in the embedding model? Is, is like a 512-dimensional model necessarily better than a 128?

453. 1:13:46

     Uh, definitely not necessarily. It's, it's an interesting question. It's-- That feels almost like a blast from the past. I remember, like, two or [REDACTED:generic_id] years ago, there was this big debate of, like, how many dimensions does each data store support and, like, how many dimensions should you have?

454. 1:14:02

     And at first it looked like, oh, more dimensions is always better, but then it turned out more dimensions are very expensive. Um, so it really depends on the model and what you're trying to solve.

455. 1:14:10

     Like, if you can get away with fewer dimensions, it's potentially much cheaper and faster. Um, but it-- I don't think there is a, a hard rule. Like, maybe the model with more dimensions can express more, uh, in because it just has more data, and then it will come in handy.

456. 1:14:26

     Uh, but maybe it's not necessary for your specific use case, and then you're just wasting a lot of resources. I don't think there is a, an easy answer to say, like, yes, for this use case, you need at least four thousand dimensions.

457. 1:14:38

     Um, it will depend. But it depends on the model, um, how many dimensions it will output, and then maybe you have some quantization in the background to reduce that again or reduce the, either the number of dimensions or the fidelity per dimension.

458. 1:14:51

     Um, so there are a lot of different trade-offs in that performance consideration, but it will mostly rely on, like, how good does the model work for the use case that you're trying to do.

459. 1:15:06

     How should we go about thinking, uh, or, or building a evaluation framework for vector search, especially the, the search similarity scores that just comes up? And again, it's not, it's not equaling to a percentage, right?

460. 1:15:22

     So...

461. 1:15:24

     Yeah. So the-- That is [REDACTED:generic_id] area. So I want to say historically, what you would do is, um, you would have a golden dataset, um, and then you would know what people are searching for, and then you would have human experts who rate your queries, and then you run different queries against it, and then you see, like,

462. 1:15:41

     is it getting better or is it getting worse? Um, now LLMs open a new opportunity where you might have human experts in the loop, uh, to help them out a bit, but they might be actually good at evaluating the results.

463. 1:15:54

     Um, so you-- Almost nobody has, like, the golden dataset and test against that. Um, but you can either use a-- look at the behavior of your end users and try to infer something from that, um, or you have an LLM that evaluates, like, what you have, or you have a human together with an LLM evaluate, uh, the

464. 1:16:13

     results. Uh, so you, you have various tools, uh, but again, it's--

465. 1:16:20

     And it depends on a really not an easy question of saying, like, "This is the right thing." Maybe you can get away with something simple. So the, the classic approach I want to say is, like, you looked at the clickstream of how your users behaved, and then you saw, like, they clicked on the first or up to

466. 1:16:35

     the third result. The result was potentially good, and they didn't just go back and then click on something else, but they, they stuck on the page. Um, if they don't click on anything and just leave, it might be very bad.

467. 1:16:45

     If they go to the second or third page, it might also not be great. So there are some quality signals that you can infer from that, or you really look into the, the quality aspect and try to evaluate, like, what people were doing and how, how it behaves.

468. 1:16:58

     Uh, but you can make this from relatively simple to pretty complicated.

469. 1:17:12

     Um, what else? What else? Okay. Obviously, if I search for "dad with squid extension," uh, it will find my father example. Um, and this [REDACTED:generic_id] here, um, will still again match my, my droids, um, pretty much like the, the OpenAI exa- AI example.

470. 1:17:36

     [REDACTED:generic_id] thing that I, I wanted to, to show you what is also happening behind the scenes here. This is a very long segment. Like, it's, it's m- a lot of information with different speakers.

471. 1:17:46

     Um, what I have created here, though, is we have, um, created multiple chunks behind the scenes. And if I search for that, um,

472. 1:17:58

     I think looking for murder in the Skywalk, uh, saga works pretty well here. It finds the document that I've retrieved, but it can also highlight. Um, so here I say, "Show me the fragment that actually matched best here."

473. 1:18:13

     Um, and I-- If I search here for murder, it didn't find anything, but I think the term that it found was in this highlighted segment here, it found kill, and it was that [REDACTED:generic_id] that was, uh, expanded here.

474. 1:18:27

     Um, so here I've broken up my long text field into multiple chunks, and there are multiple strategies. You could do that by page, by paragraph, by sentence. Um, you could do it overlapping or not overlapping.

475. 1:18:42

     Um, many strategies. It will depend on, like, how you want to retrieve what works best for your use case. But you want to kind of, like, reduce the context per element that you're matching because there's only so much context that a dense vector representation can hold.

476. 1:18:56

     Um, so you want to chunk that up, especially if you have, like, a full book, you want to break up those individual at least pages, um, and then find the relevant part where the match is, and then you can actually link back to that.

477. 1:19:09

     Um, the point in this query here is also to show you I didn't define any chunks. Um, I didn't say like, "Okay, send this representation of a dense vector there, and then when it comes back, uh, interpret again."

478. 1:19:22

     Um, this is all happening behind the scenes just to make this easier. So the entire behavior here is still very similar to the keyword matching, even though there's a lot more magic happening behind the scenes.

479. 1:19:33

     Um, just to keep that very simple. Um,

480. 1:19:39

     okay. How does everybody feel about long adjacent queries?

481. 1:19:47

     Um, we'll see about alternatives, and maybe we can make this a bit simpler again. Uh, but let me s- show you [REDACTED:generic_id], [REDACTED:generic_id] more way of, of looking at.

482. 1:19:55

     Um, we call them retrievers. They're a more powerful mechanism to actually combine different types of searches. Um, combining different types of searches, let me get from my slides, actually.

483. 1:20:09

     When we talk about combining searches and how this all plays together. Um, this is kind of my, my little interactive map of what you do when you do retrieval or what your, what your searches do.

484. 1:20:21

     So we started here in the, the lexical keyword search, uh, and then we run a match query, and we are matching these strings. Um, this often combined with some rank features, um, are often what we call full text search.

485. 1:20:37

     The rank features could be either you extract a specific signal or it could also be something, um, however you influence that ranking. It could be like the margin on a product, like how many people bought something, what the rating is.

486. 1:20:50

     There are many different signals that you could ex- include, like not just with the, the match of the, the text, but any other signals that you want to combine, uh, for retrieving that.

487. 1:20:59

     And then you have like full text search as a whole. On top of that, I kept it kind of like to the side here. You might have a Boolean filter where you have like a hard include or exclude of certain attributes.

488. 1:21:13

     Um, this does not contribute to the score. This is just like black and white. This is included or excluded. Whereas this here calculates the score for you, how you match.

489. 1:21:24

     And then this was kind of like the, the algorithmic side. And then we have this machine learning, the learn side or the semantic search, uh, where you have a model behind the scenes, uh, split into the dense vector, um, embeddings and the sparse vector embeddings, um, for vector search or learn sparse retrieval.

490. 1:21:46

     I think those are the two common terms. Um,

491. 1:21:49

     and the, the interesting thing is all of these, including the, the sparse [REDACTED:generic_id], these are the, the sparse vector representation in the background, and only this [REDACTED:generic_id] here is the dense vector representation.

492. 1:22:02

     Um, and then when you combine any, any grouping down here to combine for [REDACTED:generic_id] search, this is then what we would call hybrid search. Um, even though there can be big discussions of like what is exactly hybrid search or not.

493. 1:22:19

     I would definitely stick to the definition that as soon as you combine, uh, more than [REDACTED:generic_id] type of search, it could be sparse and dense, or it could be dense and keyword.

494. 1:22:28

     Or maybe if you combine two dense vector, uh, searches, um, then hybrid search because you have multiple approaches, and then you can either boost them together. You could do re-ranking, which is becoming more and more popular.

495. 1:22:42

     Uh, [REDACTED:generic_id] thing that we lean heavily into is RRF, which is reciprocal rank fusion. Uh, that doesn't rely on the score, but it relies on the precision of each search mechanism.

496. 1:22:53

     So it basically says like the, the lexical search had this document at position four, and the dense vector search had it at position two, and then it kind of like evens out the precision and gives you an overall precision by blending them together rather than looking at the individual scores because they might be totally different.

497. 1:23:12

     So this is kind of like the, the information retrieval map overall, and we have-- Okay, we didn't do a lot of filters, but I think filters are intuitively relatively clear that you just say like, "I'm only interested in users with this ID," or whatever other criteria.

498. 1:23:26

     It could be a geo-based filter, like only things within ten kilometers or only products that came out in the last year. Um, like a hard yes or no. Um, all the others will give you, um, uh, a value for the relevance, and then you can blend that potentially together to give you the overall results.

499. 1:23:46

     That is kind of like the, the total map of search.

500. 1:23:51

     Can you give an example of the signal [REDACTED:generic_id], please?

501. 1:23:54

     Uh, yeah. For, for signal, um, so we have our own data structure for, for these rank features. It could be, for example, the, I don't know, the, the rating of a book.

502. 1:24:05

     And then you combine, um, the, the keyword match for, I don't know, you, you search for, um, murder mysteries, uh, but then another feature would be, um, how well they are ranked, and then you would see that.

503. 1:24:22

     Or it could be your margin on the product or the stock you have available, and you would want to show the product where you have more in stock. Uh, there can be-- Or it might even be a simple like a click stream, like what have people clicked before.

504. 1:24:35

     There are a lot of different signals that you could include in all of this searching then.

505. 1:24:45

     Any other questions or everybody good for now? Yeah?

506. 1:24:48

     When you do this like scoring, uh, you said you do like a blended approach, right? Like-

507. 1:24:53

     RRF does a blended approach, yes.

508. 1:24:55

     Those scores, are they kinda standardized or are they different based on the type of search that you're doing? Like the relevancy scores that you're kind of combining.

509. 1:25:03

     Yeah. So they are-- You would have to normalize them. There will be-- So de- depending on like if you have-- depending on the, the comparison that you do for dense vectors, it might be between, uh, zero and [REDACTED:generic_id].

510. 1:25:15

     But you saw that for the keyword search, uh, also depending on how many words I was searching for, it might be a much higher value. There is, there is no really real ceiling for that.

511. 1:25:24

     Um, or you could add a boost and say like, "This field is twenty times more important than this other field." Um, there is no real max value that you would have here.

512. 1:25:34

     You could normalize the score and then basically say like, "I'll take like the highest value in this subqueries-" A hundred percent and then reduce everything down by that factor, and then I combine them.

513. 1:25:46

     Maybe that works well. RF is a, it's a very simple paper. I think it's like two pages. Um, and it really just takes the different positions. I think it's [REDACTED:generic_id] divided by sixty, which is like a factor they figured out made sense, plus the position.

514. 1:26:00

     Uh, and then you add the scores or like the positions for each document together, and then that value gives you the overall position. Um, it really just-- it doesn't look at the score anymore, but it blends the different positions together and like how they are interleaving and what should be first or second then.

515. 1:26:18

     Um, yeah?

516. 1:26:20

     Um, if I just want vector search, um, why should I use Elastic over pgvector or something like that, right? I'm sure there are some trade-offs because if my data is already in a database, every change probably via CDC, change data capture has to go through Elastic.

517. 1:26:39

     So that's [REDACTED:generic_id] extra hop in ingestion versus like, like pgvector, like it's right there, you know, site. So I was just curious what sort of systems maybe people who already have keyword search requirements, like what have you seen in production?

518. 1:26:55

     Why, why, what systems chose pgvector over other versus-

519. 1:26:59

     I mean, pgvector will always be there because like if you are already using Postgres, it's very easy to add. Um, I think then the question is like, does it have all the features, uh, that you need?

520. 1:27:09

     For example, Postgres doesn't even do BM25. Um, it has some matching, but it's not the full BM25 algorithm because I don't think it keeps all the statistics. Um, it will be a question of like scaling out Postgres can be a, a problem, and then just like the breadth of all the search features.

521. 1:27:26

     Um, if you only need vector search, um, I think my or our default question back to that is like, do you really only need vector search? Um, maybe for your use case, but for many use case you probably need hybrid search.

522. 1:27:41

     Uh, [REDACTED:generic_id] area, for example, where vector search will not do great is like if somebody searches for like a brand because there is no easy representation in most models, uh, for this specific brand, and it will be very hard to beat keyword search.

523. 1:27:55

     So there will be very-- And also your users get very angry when they know you have this word somewhere in your documents, uh, or in your data set, but you don't give me the result back.

524. 1:28:04

     Um, so there are many scenarios where you probably want hybrid search or I feel like that's the-- We started two years ago, we started with just vector search, but I feel like the overall trend is coming more to hybrid search because you probably some sort of keyword search.

525. 1:28:20

     Um, and then you want to have that combined probably with some model, uh, for the added benefit and extra context. Um, but you often want the combination. It might also depend a bit on like the types of queries that your users run.

526. 1:28:34

     So if your users run single word queries like I've done in my examples, that's often not really ideal for vector search because you live off of like any machine learning model because you live off extra context.

527. 1:28:46

     Um, so depending on that, I've seen some people build searches where it's like if you search for [REDACTED:generic_id] or two words, they do keyword search, but if you search for more, they might fall over to vector search.

528. 1:28:57

     So it depends a bit on the context what works. Um, if you really only need vector search, um, and pgvector is small enough, uh, to do all of that,

529. 1:29:08

     um, and Postgres is your primary data store, then that's probably where you will do well. Uh, but there are plenty of scenarios where that will or not all of those necessary boxes will be ticked.

530. 1:29:18

     [REDACTED:generic_id], [REDACTED:generic_id] last question. Um, specifically for code, um, let's say you have a file, uh, and it's like you have Git, a Git repository with like thousands of commits, right?

531. 1:29:28

     Um, the file doesn't change across many commits, but you change [REDACTED:generic_id] byte and then technically the, the embedding of the chunk has changed.

532. 1:29:37

     Mm-hmm.

533. 1:29:37

     And so you have two options. Either you like embed at a file level or you embed at a chunk level, right? Like, um, but I don't want to pay the penalty across thousands of unchanged-- Like the file hasn't changed for these thousand commits, but just for this [REDACTED:generic_id] it has changed, right?

534. 1:29:53

     Like, so I kinda need shadow copies of the same thing. And have you seen like what are some, uh, like tips and tricks that people use to not have exploding storage costs and how like, like this might not be just Elastic problem, but a general vector DB and retrieval problem.

535. 1:30:11

     Like how do I just pay the penalty once of storing the same embedding and only when it changes, uh, like I, uh, I re-embed and like, uh, ingest it.

536. 1:30:22

     But so you would create-- So it's [REDACTED:generic_id] data set basically with thousands of files that all are chunked together, and so o-[REDACTED:generic_id] change would invalidate all of them or?

537. 1:30:32

     No, no. So imagine I have a Python repository, right? Like it has like five thousand commits, and there's [REDACTED:generic_id] file, food.py, let's say. It didn't change for four thousand nine nine nine commits.

538. 1:30:45

     But on the five thousandth commit it did change, right? And so if it hasn't changed, I want to only pay the penalty once of the, of the embedding, right?

539. 1:30:55

     But only when I-- the file contents have changed, I need-- I want to re-ingest, re-embed, whatever, right? But for those four nine nine nine times, I don't want to store like, like this ha- like this Git hash has the same embedding, same embedding.

540. 1:31:10

     Like-

541. 1:31:10

     Ah, so-

542. 1:31:11

     I'm not sure if this is a problem you have seen with some of your customers or-

543. 1:31:14

     Maybe so the-

544. 1:31:15

     Some company has solved the problem.

545. 1:31:17

     I, I think the way we might solve it is that if you create the hash of the file and use that as the ID, and you only use the operation create, and it would reject any, any dup-duplicate writes.

546. 1:31:30

     You would at least not ingest and, and then create a vector representation again. Um, you will still send it over again, uh, and it would need to get rejected.

547. 1:31:40

     The doc ID will have to be

548. 1:31:42

     Yes. If you, if you have that doc ID and then you need to set the operation to just create and not, uh, update or upsert, uh, then it would just be rejected and you would only write it once.

549. 1:31:54

     I'm not sure if that is a great use case or if you might want to keep like a, I don't know, an outside cache of like all the hashes that you've already had and deduplicated there, but that would be the Elasticsearch solution of like using the, the hash as the ID and then just writing to that.

550. 1:32:07

     Yeah, I tell you what we did, we just, we just broke it out into chunks and did that only. I was curious if-

551. 1:32:12

     Okay. And although with create on the-- Yeah. That, that is I think the intuitive or most native approach that we could offer for that. Yeah. I think there was some other question somewhere.

552. 1:32:22

     Uh, yeah.

553. 1:32:23

     Uh, yeah, I just wanted to add onto the Postgres question a minute ago. Postgres does have a native full site search engine in it that's pretty good. It's the tsvector extension.

554. 1:32:32

     It's been there for a while. Um, there has also been recently a lot of work, and so there is a Rust-based, um, BM25 search engine and, uh, implementation in Postgres now.

555. 1:32:43

     It's called like tensor4 or something like that.

556. 1:32:47

     Yeah. But, uh, from what I remember, the default Postgres full text search does not do full BM25, but it only does-- It doesn't have all the statistics I think from what I remember.

557. 1:32:57

     Correct. Yeah.

558. 1:33:00

     Yeah. Any other questions? Uh-

559. 1:33:05

     Hey, Philipp.

560. 1:33:05

     Joe, please go ahead.

561. 1:33:06

     Yeah. Do you still plan to cover more on the retriever thing? Like the retriever concept and-

562. 1:33:13

     To, to show now? Yeah. I mean, how much chase do you want to see?

563. 1:33:16

     Well, depends. [laughs]

564. 1:33:18

     Um-

565. 1:33:19

     I, I'm particularly interested in, I mean, you're modeling so you have multiple-- I love the retriever concept. It's a great concept. Uh, I think you have multiple different candidate retrievers.

566. 1:33:29

     What kind of flexibility do you have on top of that? Because I'm not interested in using this. I'm interested in rescoring them. Not necessarily looking at any, but I'm modeling like, uh, like you're just retrieving a bunch of stuff, candidates-

567. 1:33:45

     Mm-hmm

568. 1:33:45

     ... and then you're actually evaluating your real scoring function.

569. 1:33:49

     Yes. Um-

570. 1:33:50

     I'm sure there's some double [inaudible]

571. 1:33:52

     May-maybe for before we dive into that, be-be-- for, for everybody else, like rescoring is like, let's say we have a million documents, and then we have [REDACTED:generic_id] cheaper way of retrieving them, and we retrieve the top, I don't know, a thousand candidates, and then we have a more expensive way but higher quality way of actually rescoring them.

572. 1:34:11

     Then we will run this more expensive rescoring on just the top thousand, uh, to get our ultimate result list of, of results. But the, the rescoring algorithm would be too expensive, uh, to run it across like the million documents.

573. 1:34:24

     That why you don't want to do that. That's why you two-step process, and that's why you might want to have the, the rescoring. So yes, we have a-- You can in Elasticsearch now, you can do rescoring because it becomes more and more popular.

574. 1:34:37

     Um, I don't have a full example there, but we do have like, we do have a rescoring model built in by default now. Let me pull that up. Um, ba, ba, ba, ba, ba, ba.

575. 1:34:50

     Not this [REDACTED:generic_id]. So we have currently it's the, the version [REDACTED:generic_id] re-ranking, but we have a built-in re-ranking, uh, model now as well. So for [REDACTED:generic_id] of the tasks, uh, that we can do, um, you can see here we have the other tasks like, for example, the dense text embedding.

576. 1:35:08

     Now we have a re-ranking task that you can also call.

577. 1:35:13

     You have question?

578. 1:35:13

     How do you express that in the whole-

579. 1:35:15

     Okay

580. 1:35:16

     ... retriever concept? Like if you have-- It might be too detailed.

581. 1:35:27

     No, re-ranking is good. Let me, uh ... Somehow my keyboard binding is broken. This is very annoying.

582. 1:35:40

     Okay. Um, we re-rank results. Let me see. Somewhere here there should be-- So there's learn to rank, um,

583. 1:35:50

     but it should not be the only [REDACTED:generic_id].

584. 1:35:55

     This is what we want. Okay, we have our re-ranking model, um,

585. 1:36:04

     unless Dave you know from the top of your head where we have the right docs for this. [laughs]

586. 1:36:09

     Organization of our docs can't help you with, but retrievers-

587. 1:36:12

     Yeah, retrievers could find them

588. 1:36:14

     ... starting in 8.16 or 8.17-

589. 1:36:17

     Uh

590. 1:36:17

     ... the retri-retrievers API added a specific like parent level, uh, retriever that you, you, it's like you nest it around outside the, the retrievers that go inside of it, and it's specifically called the text re-ranking retriever.

591. 1:36:31

     Ah. [laughs]

592. 1:36:32

     And so if you have a cross encoder, say from Hugging Face, or you're using, uh, the Elastic re-ranker, using [REDACTED:generic_id], [REDACTED:generic_id] of these things that complies with that kind of inference task of taking a bunch of things and re-ranking them against a query, right?

593. 1:36:46

     Taking the full token stream, right, and the full token stream with the context and doing that, it is-- it can target a parent level text field of the documents that are being retrieved. [phone ringing]

594. 1:36:58

     So it works really well for the [REDACTED:generic_id] document per chunk kinda, kinda, uh, re-ranking use case. I've also seen people just do it outside in a second API call, say if you wanted to do it on a highlighted thing, uh, or if you wanted to do re-ranking of sub-document chunks.

595. 1:37:14

     That works pretty well just hitting the inferences API kinda directly as a secondary call. But there's a, there is a text, um,

596. 1:37:23

     re-ranker retriever clause that specifically got added in like 8.16 or 8.17.

597. 1:37:30

     Yeah. So I think this is a simple example. Like we have a standard match, like this is-- will be very cheap, and then we have the text similarity re-ranker, which uses our Elastic re-ranker, um, that falls back to that model behind the scenes

598. 1:37:52

     But you would have like the text re-ranker retriever. Inside of that, you would have the RRF. Inside of that, you would have lexical and KNN as peers, and it works from the inside out.

599. 1:38:03

     Hey, do each of those retrieval methodologies, do like the Venn diagram, find the best results, and then take the full text of those results and run them through the re-ranker.

600. 1:38:14

     Almost like a little mini LLM saying, "Which [REDACTED:generic_id] of these actually answers the question?" [chuckles] And then what out- what out comes is pretty good. The cool thing about the, the re-ranker is you can run it on structured lexical retrievals, right?

601. 1:38:27

     You don't have to run it on a vector search. You can run it on anything you want. So if you don't wanna pay to vector search everything or maybe the, the text is too small to vector search, there's not enough meat in there to actually have the model lock onto stuff, the re-ranker, when you run it on

602. 1:38:43

     just kinda actual customer datasets and say, "How did, how did it do?" They're like, "Yeah, our evaluation score is bumped by 10 points basically for free." It feels like cheating, right?

603. 1:38:55

     So when you run against like a Gemini API and it's like, "Wow, why is this 10 points better than the Amazon [REDACTED:generic_id]?" It's 'cause they threw on their retriever.

604. 1:39:03

     It didn't tell you, right? So there's a lot of black box stuff out there that we're exposing. So don't be, don't be scared that we're showing you how it works inside, but this is what the, the leading retrieval technologies are doing under the hood and then reselling it to you a- as if they're, uh, you know...

605. 1:39:23

     It's all AI, right? It's, it's all just code. You can't trade model on its own.

606. 1:39:28

     Yeah.

607. 1:39:29

     Does that, does that answer the question?

608. 1:39:30

     Yeah, sure. Thanks.

609. 1:39:31

     Yeah. So my wishlist is I wanna be able to do, in the retrievers API, re-ranking on sub-document hits. A lot of my things are about sub-document knowledge retrieval. Right now, I gotta do it outside of the retrievers API, but, uh, I'm bending the ear of the developer to get that.

610. 1:39:51

     Yeah. So the-- Just to give you the example of like, I don't think I have a re-ranking example here, but this [REDACTED:generic_id], um, uses a classic keyword match, uh, for retriever, and then we have-- we normalize here the score.

611. 1:40:07

     I think somebody else asked about the normalize or we had a discussion about the normal- normalizing. We do a min-max, uh, normalizing. We weight this with two, um, and then I use the OpenAI embeddings with a, again, normalized with a weight of 1.5, and then they will get blended together and you, you get the results that won't

612. 1:40:25

     surprise you that these are not the droids you're looking for. If you search for a droid and robot, um, will be by far the highest ranking document.

613. 1:40:34

     You had a question s-somewhere. Yes.

614. 1:40:37

     Um, what-- How much control does Elastic, if you're doing the re-rankers thing, um...

615. 1:40:44

     Currently, we do something similar, but we like do the different steps of the re-rankers at kind of like different levels of the distribution hierarchy. So like you have like your like sharded query processing nodes, and you do some like-- you do some summary in there, and then like once you like rejoin, um, before you return the result

616. 1:40:59

     from the query engine, you will-- that's when you run the final re-ranker, um, just for like cost reasons, uh, like, uh, performance reasons. How much control does the retrievers API give over that sort of thing or is that like internally optimized, um, what parts of re-ranking happen where?

617. 1:41:14

     I'm, I mean, so in the first-- So we would retrieve like X candidates, and you could define the number of candidates, and then we would run the re-ranking on top of those.

618. 1:41:22

     So that it will be a trade-off for you, like the larger the window is, the, the slower it will be, but the potentially higher quality your overall results will be because you will just have everything in your day-to-day data s- data set that you can then re-rank at the end of the day.

619. 1:41:39

     Is that what you meant or you wanted something per node or?

620. 1:41:44

     Yeah. I mean, like right now we like are actually concerned like specifically like where, where are kind of like where in the compute that happens so that we can like fan out more aggressively with the like cheap re-ranking and then do the expensive re-ranking like once.

621. 1:41:59

     But, um, but maybe that doesn't apply in the case of Elastic like in terms of like how the queries and canopy line up and things.

622. 1:42:07

     Yeah. And-

623. 1:42:08

     Maybe not all.

624. 1:42:09

     I don't think that's how we do it. So what, what you can control here is like this is the window of like what you might retrieve, and then we have the minimum score, like a cutoff point, uh, to

625. 1:42:19

     throw out what might not be relevant anyway to, to keep it a bit cheaper. That's what we have here.

626. 1:42:26

     Um, those are the retrievers, uh, and then you could do the RRF that I've explained where you blend results together. All of that is easy. Um, [REDACTED:generic_id] final note.

627. 1:42:38

     If you, if you got tired of all the JSON, um,

628. 1:42:44

     we have a new way of defining those queries as well, um, where here we have a match operator like the [REDACTED:generic_id] we've used all the time, uh, that you can use either on a keyword field, but it could also be either a dense or a sparse vector embedding, and then you can just run a query on that

629. 1:43:00

     and then just get the discourse from that. So it is a query language. It's a bit more like, I don't know, like a shell. Um, but if you don't want to type all the JSON anymore, um, this is how you can do that, and here my screen size is a bit off.

630. 1:43:13

     Uh, but yeah, you get the, the quote that we retrieved, the speaker, and the, the score. Maybe, maybe I'll take out the speaker to make this slightly more readable.

631. 1:43:28

     Now it broke. Oh. Um, uh, this is-- You could write queries with a fraction of the JSON. Uh, this will also support funny things like joins.

632. 1:43:46

     It doesn't have every single search feature yet, uh, but it's getting pretty close. So this is more like a closing out at the end. If you're tired of all the JSON queries, you don't have to write JSON queries anymore.

633. 1:43:57

     Um- This is nice both for like observability use cases where you have like just like aggregations and things like that, but it's also very helpful for full-text search now, um, if you want to write different queries.

634. 1:44:10

     I think the main downside is that the language support in the different languages like Java, et cetera, is not very strong. We have to basically give it strings, and then it gives you a result back that you need to parse out again.

635. 1:44:20

     Uh, so it is not as strongly typed on the client side yet as the other languages. Um,

636. 1:44:29

     any final questions? Yes.

637. 1:44:31

     Uh, you just talk about hybrid search, and I was just curious like what is kind of the recommended best practices to-- Like we also do hybrid search today. What we do is we trigger two Elastic queries, [REDACTED:generic_id] to do like a basic keyword search, the other [REDACTED:generic_id] to do like a key, uh, vector search, then we walk

638. 1:44:47

     through some like, uh, like a coherent reranking API to get the final res- results. But like from the retrievers you just showed, feels like it's, it, it's better like to combine those two queries in [REDACTED:generic_id] retriever and like would that makes the results might potentially be better?

639. 1:45:04

     Like s-sort of sounds like each of the query will have its own stats and normalizer, and it will make things seems like more, I don't know, just like in general sounds better.

640. 1:45:14

     I mean, we can make your life easier.

641. 1:45:16

     Yeah.

642. 1:45:17

     It's, it's just all behind [REDACTED:generic_id] single, uh, query endpoint. So you, you could use the two different methods to retrieve, um, and then you could still re-rank, but all from [REDACTED:generic_id] single query.

643. 1:45:26

     So you don't have to do it yourself. I mean, it's not like we want to stop you, but you don't have to, and we can make your life a bit easier.

644. 1:45:33

     But like I'm just curious, instead of just like simplify the process, would that impact the performance in any ways also?

645. 1:45:42

     I mean, it's only [REDACTED:generic_id] single query that you need to run and like [REDACTED:generic_id] single round trip to the server that you need to do.

646. 1:45:49

     Yeah. But like I was just curious comparing to the two queries methods what we do today. Would that make any difference?

647. 1:45:56

     I, I mean, if you still need to do the retrieval like you, you do the retri-- Like all the individual pieces are still there. If you have two parts of the query, you will still retrieve those if that is the main cost, and then you have the re-ranking.

648. 1:46:07

     Um, so you're not getting out of those completely, but you can just do it in [REDACTED:generic_id] single request that you send. We take care of all of that for you and then send you [REDACTED:generic_id] result set back rather than sending more back to your application.

649. 1:46:18

     So it, it will potentially be a little less work on the Elasticsearch side, but it will mostly be less work on your application side.

650. 1:46:29

     Okay.

651. 1:46:29

     Yeah. You're, you're, you're doubling the utilization of your thread pools. So if your thread pools don't have any problems, you may not notice. But you're, you're running two map reducers, right?

652. 1:46:43

     And when you could be running [REDACTED:generic_id], and you're denying the optimizer the opportunity to do any short circuits to say, "Oh, there's no more results that are better. You should stop."

653. 1:46:54

     Yeah. That's-

654. 1:46:55

     Yeah. So you're, you're potentially gonna be a little bit worse performance-

655. 1:46:59

     Mm-hmm.

656. 1:46:59

     And, uh, resource consumption by hybrid query. Time, it's easy

657. 1:47:08

     separately, profiling the queries separately, which is people-- what people do before they turn it into a whole query.

658. 1:47:17

     Right.

659. 1:47:17

     So if it's not hurting you, by all means, keep going, right? But when you at some point you're gonna start vertically scaling your hardware when you don't need to.

660. 1:47:28

     You could get further if it was just a hybrid search.

661. 1:47:31

     Yeah.

662. 1:47:38

     Perfect. Thank you so much. I hope everybody learned something. I will let the instance running for today or so, so you can still play around with the queries if you feel like it.

663. 1:47:47

     Uh, thanks a lot for joining. If you want stickers, we have stickers up there. We also have a booth the next few days. Come join and, uh, get some proper swag from us there.

664. 1:47:56

     Um, thank you. Uh, see you around. [clapping] [outro music]