The State of AI-Powered Search and Retrieval

Talk transcript

1. 0:00

   [upbeat music] Welcome, everybody.

2. 0:16

   Uh, I wanna thank you for coming to this session today. Um, today I wanna talk about AI-powered search and retrieval. Uh, and for those of you who don't know me, my name is Frank.

3. 0:25

   Uh, I am actually a part of the Voyage AI team, and we recently joined MongoDB, I wanna say probably about three to four months ago. Uh, just a quick introduction to Voyage AI, you know, we build the most accurate, cost-effective embedding models and rerankers for RAG and semantic search.

4. 0:39

   Uh, a lot of the applications, I think that we've seen, that I've seen in particular, actually go beyond that. So some folks use it for classification, others for, you know, a variety of different applications.

5. 0:47

   We've got clustering, so on and so forth. Uh, Voyage is available via the Voyage AI API, Azure, and AWS Marketplace as well. And now we are a part of MongoDB, so- [cheering] Representing MongoDB. [laughs]

6. 0:59

   Yeah, thank you. So we're representing MongoDB here. Uh, and, and, you know, we're all really excited to see what the future will hold. And the reason, you know, I think in, in the past when I would give presentations like this, uh, I'd actually talk very, very specifically about things like evaluating...

7. 1:14

   evaluation for embeddings, right? I would talk about things like, um, you know, how, how do rerankers, uh, play a role in your ultimate retrieval stack? And today, I wanna go a little bit higher level.

8. 1:24

   I wanna talk a little bit more about what I like to call AI search. I know it's a very, very broad term, has a lot of different meanings to different folks.

9. 1:30

   But I wanna go-- I wanna start, use that as a starting point, and then talk about where we are today, and also to a secondary extent, where we're going as well.

10. 1:38

    So a quick, uh, agenda here for the next 10 or, 10 or so minutes. Um, but first I wanna give a quick refresher, you know, uh, a little bit about, uh, about embeddings, uh, about, uh, search and retrieval more broadly in this day and age.

11. 1:51

    Then I'll talk about some real-world applications. And I think each of the applications, there's gonna be three, each of the applications I'm gonna talk about, I'll probably spend about a minute, uh, each there.

12. 1:59

    There's gonna be a, a lesson or something to learn, or, or a key fact to take away from that application. And then what's most exciting, I think, to me, and hopefully to the rest of you as well, is what's to come, right?

13. 2:11

    Where is the future of AI-powered search and retrieval, and, and where are we going from here? So a quick refresher. I'm, I'm gonna blow through these pretty quickly. AI-powered search, at least how I define it, and, and how I hope, uh, this will continue to be defined moving forward, is a search system that finds related concepts even

14. 2:29

    without identical wording. I think this is very important, right? So a lot of folks, uh, you may be familiar with things like TF-IDF or BM25. Uh, you know, AI-powered search goes way beyond that, right?

15. 2:39

    Not only it can understand keywords, it can help you retrieve based on some of these more traditional, uh, information retrieval algorithms, but it can also help you find related concepts.

16. 2:49

    To that point, it also understands the user's intent. So for example, if I am in a, you know, I'm trying to search for some products to buy something to buy, and I say, "My best friend is sick," uh, perhaps it can recommend me some, some, some, you know, get well baskets or something like that, right?

17. 3:06

    It really should be able to understand what my ultimate meaning is, rather than just saying, "Hey, okay, my best friend is sick. Maybe, you know, I'm gonna try to find something that's good for a best friend, uh, more generically."

18. 3:16

    The last thing that I'll mention is that it can perform some level of reasoning and some level of instruction following, and I'll get to that a little bit later, uh, in that last section.

19. 3:25

    But to get right into it, um, you know, a really popular use case of AI-powered search and retrieval is RAG, and I'm gonna go through this pretty quickly. The idea is that without retrieval, you either get probably some sort of hallucination, or in some cases, your LLM is just gonna flat out refuse to respond to you.

20. 3:41

    Or it's gonna give you a really, really generic answer like the one you see up here above. Uh, but with retrieval, with AI-powered search built into it, you get a much, much more grounded response.

21. 3:51

    Uh, again, this use case I think is, is pretty common. These are actually slides that I, uh, I, I took from, from Tengyu's talk that's gonna be happening at 12:15, so you guys should definitely go to that if you're around.

22. 4:02

    Uh, and the idea is that, hey, you generate embeddings, use that for search, and then you give it to your LLM.

23. 4:07

    Last thing I wanna mention is that embedding quality here is a very, very core component of AI-powered search and retrieval. And I think pretty much 95 to 99% of the systems I've seen out there, from what I get- from what I go- from what I gather, use embeddings in some way, shape, or form.

24. 4:25

    The idea is that, again, you have embedding, you have these, this unstructured data. Usually, it's text, you know, PDFs, Word documents, Google Drive files, uh, PowerPoints, et cetera. Uh, and you're able to embed them into a same space such that when you do a search or you do a...

25. 4:39

    you, you try to find a prompt, you have a prompt, you search for the most relevant documents, you're able to pull that information all the way up to the top.

26. 4:47

    So I'm gonna go through these pretty quickly. I'll try to spend about a minute each here. Um, some real-world applications of AI-powered search and retrieval. The first one is chatting with your codebase.

27. 4:58

    And this is actually, if folks have heard of Continue.dev, this is actually their application. A lot of their code is open source. You do... This is a classic RAG, uh, plus re-ranking approach.

28. 5:10

    I think the lesson from this particular application is that there is no one-size-fits-all embedding model. There's no one-size-fits-all LLM. Uh, always do evaluations to see which one is best for your application.

29. 5:24

    Now, in this case, Continue did theirs, and, and they, they found that voyage-code-3 actually performs the best. Uh, again, the reason is because for a lot of chat with your codebase applications, you want to have an embedding model, and also to a secondary extent, an LLM that is really, really good at understanding, well, code, documentation, and developers,

30. 5:43

    so on and so forth. So this is the first lesson from the first application. The second one is that there's, you know, when it comes to... The second one, again, it's also a very domain-specific application here.

31. 5:53

    But the second one that I wanna mention is if you see this blue box here, where there's, there, there's some filtering and then there's some other structured data that's also passed to the search system-

32. 6:04

    If this is the thing that I want, that I wanna highlight for this particular application, is that oftentimes, and, you know, coming from a company that builds embedding models, it's, it's hard for me to say this.

33. 6:13

    I think oftentimes embedding a l- embeddings alone are not enough, right? If you wanna build a really powerful search and retrieval system, you need to have a lot of that structured data that's a part of it.

34. 6:23

    So to give you an example beyond just this particular, uh, domain, you know, if I have, let's say, some legal documents, uh, I embed those legal documents using Voyage Law 2.

35. 6:35

    Uh, but then when I do my search or if I'm building my agent, perhaps I want to understand, I wanna know, "Hey, I, I only wanna find documents that are from a particular state," or maybe I wanna find only official legal documents, or I wanna find documents that have a very, very particular set of details inside of

36. 6:52

    them. This is all, this- these are all things that can be done, uh, at the filtering stage. So this is typically done directly inside the vector store. In some cases, it's done after that, right?

37. 7:00

    So I wanna say, I wanna be very, very clear, there is oftentimes, and the second lesson here is that there's oftentimes other sources of data, other pieces of structured data that you need to include inside of your search system as well, right?

38. 7:12

    Uh, and, and really, you know, just go beyond that. And then the last thing, you know, the last sort of application, these are, these are real world, that these are actually built today.

39. 7:21

    The last thing that I wanna say is when it comes to a lot of agentic retrieval, oftentimes it's a feedback loop. So your AI search system is no longer just input, output, right?

40. 7:34

    Sometimes if you get a query, you might want your LLM to, you know, you know, do some searches, and then you might want your LLM to expand that query, or you might want your LLM to decompose that query.

41. 7:44

    I'll give you a quick example. Uh, if I ask for something like, um, you know, what are Twitter's... Give me Twitter or I guess X now. Uh, you know, give me X's Q4, you know, or 2024 earnings, right?

42. 7:57

    You could, your LLM could decompose that into Q1, Q2, Q3, Q4 earnings, and then send that as four separate queries to these different vector stores, to the different vector databases, so on and so forth.

43. 8:08

    On top of this as well, you also see that we're in the era of agents, right? 2025, I would say, is the year of the agents, maybe even 2026 as well.

44. 8:16

    And a lot of agents, a lot of these agentic applications, they're gonna need to be really, really powerful at conversational data. So you really want embeddings. You really want a search system that's built around that.

45. 8:27

    Now, there's a lot of details that's missing in this particular block diagram, but I think hopefully that goes to show you, uh, sort of the lessons to take away from that.

46. 8:37

    Okay, so this is the most... So, you know, I just covered three, um, existing applications. Uh, and hopefully these applications, I think, they give you a window into where we are today.

47. 8:48

    Um, you know, some of the tips, some of the tricks, just three of them, uh, that, that's being used today in these AI search systems. There's many, many out there, right?

48. 8:55

    There's, I think, a lot to cover here. But I think what's more exciting to me is, is what's to come in AI-powered search and AI-powered retrieval.

49. 9:04

    Uh, I don't know what happened to my arrows here. I apologize. Seemed to sort of disappear in the background. But the future is 100% multimodal. Uh, that is the case for large language models.

50. 9:15

    That is the case for embeddings. For AI search and retrieval overall, that's going to be the case as well. Uh, I think there's no doubt about that. And when I say multimodal, I really...

51. 9:24

    I don't mean multimodal in the sense of, oh, you know, I'm, I'm, I'm an agent that's operating in the real world. I can connect all these different modalities like sight and, and, and, and touch and, and taste together.

52. 9:34

    I'm talking more about modal- multimodality just from a pure, sort of like a foundational perspective. The ability to understand images and text together, or the ability to understand images, text, and audio together.

53. 9:46

    And it's gonna be really important for search systems. It's gonna be really important for embedding models, just as we have a lot of VLMs out there today. This particular example is Voyage Multimodal 3.

54. 9:56

    Um, again, I apologize. I don't know what happened to the arrows here. But the idea here is that it can take text, it can take images, or it can take a combination of text, interleave text and images, and really embed all of those into a single, really powerful semantic space.

55. 10:13

    So it might be a little bit hard to understand exactly what's going on here. But, uh, the query being strong LLMs and then the nearest, uh, the nearest sort of document being, uh, the Claude 3.5 blog post.

56. 10:27

    So I hope that's a little bit clearer here. I know, again, um, uh, I know it might be a little bit harder to understand, but this is, uh, this is one of the things that I wanna get to.

57. 10:35

    The second thing that I also think is particularly exciting is instruction tuning. Instruction tuning into a second, a secondary extent reasoning as well. So right now, if you look at embeddings and you look at embedding models, they just take a query or they take a document, and they give you a vector.

58. 10:51

    I think moving forward, we're going to see situations where in addition to that query or in addition to that document, we wanna be able to steer the vector in a particular direction or in a particular, in a particular way, right?

59. 11:04

    So to give an example, perhaps I want to, you know, in addition to my query, in addition to my prompt, in addition to my search, uh, I also ask, I also give it an instruction to say, uh, "Find documents for me that only dive into detail about this particular aspect," right?

60. 11:19

    And that is where I think instruction tuning, uh, is really gonna play a huge role moving forward for AI search and retrieval.

61. 11:27

    Uh, last thing that I wanna talk about, um, is, uh, and this is sort of a, a buzzwordy kinda term out there, is sort of the agent native database.

62. 11:34

    And I think this is, this is where Voyage joining forces with MongoDB is super exciting for a lot of us. The idea that today a lot of search and retrieval, there's many, many multiple different components that you have to put together, and what you see on the left is actually already a really, really simplified version of that.

63. 11:50

    And the capability to move directly to something that is just a single piece of infrastructure that does the embedding for you, it does the re-ranking for you, perhaps it does some of that query augmentation or query decomposition for you, uh, all of that inside a single data platform, inside a single database, I think that is super exciting.

64. 12:08

    So I think this is something that you'll see more and more of this year and also next year as well, and something to look out for hopefully as well.

65. 12:16

    So with that being said, I've got about three minutes left. Would love to take any questions if you have them. Um, but, uh, I'll also leave this up for a couple more seconds.

66. 12:23

    Feel free to, uh, scan those QR codes, follow us, and I hope to see you sometime else at this conference. [upbeat music]