AI Engineer World's Fair 2025
The State of AI-Powered Search and Retrieval
Read the talk
Search that understands what you mean
A shopping query like “My best friend is sick” should find get-well baskets. Making that work takes more than matching words: it takes retrieval that understands intent and respects constraints.
From a talk by Frank Liu
Before you start: Familiarity with LLMs and the idea of representing text as vectors will help; the article explains how those vectors fit into retrieval.
From matching words to understanding intent
Frank Liu approaches search from the embedding and reranking layer. His team at Voyage AI had joined MongoDB roughly three to four months before this talk. Voyage positioned its models around accuracy and cost for retrieval-augmented generation (RAG) and semantic search, with applications extending to classification and clustering. Liu listed the Voyage AI API, Azure, and AWS Marketplace as access points. Here, he moves beyond individual model evaluations to examine three application lessons and the capabilities search systems may need next.
AI-powered search finds related concepts even when the wording differs. Traditional methods such as TF-IDF and BM25 provide useful keyword retrieval, but conceptual relevance broadens what a system can find. The goal is not to discard keywords; it is to retrieve useful material when the query and the answer do not share them.
Consider the shopping query “My best friend is sick.” A useful result might be a get-well basket, even though the query never names that product. Merely matching “best friend” could produce generic gifts and miss the reason for buying one. Search needs to connect the situation to the user's likely intent. Liu adds limited reasoning and instruction following to this definition, returning to them later as directions for retrieval models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieval gives generation something to work with
RAG is the familiar application of this idea. Without relevant retrieved information, an LLM may hallucinate, refuse to answer, or produce a generic response. Retrieval supplies material from which it can construct a more grounded answer. Liu credits the RAG slides to Tengyu's talk and describes a short pipeline:
- Generate embeddings for the material to be searched.
- Search for information relevant to the prompt.
- Pass the retrieved information to the LLM as context for its response.
Embedding quality matters because it helps determine which information reaches the model. Liu estimates that 95–99% of the search systems he has encountered use embeddings in some form. That is an observation about his experience, with no sample size or survey method supplied. Text from PDFs, Word documents, Google Drive files, and PowerPoints can be represented in a shared vector space so that a query brings relevant documents toward the top of the results. A generator cannot make good use of a document that retrieval never supplies.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose models for the codebase, not the leaderboard
Chatting with a codebase makes the importance of domain fit concrete. Continue, whose code is largely open source, uses a RAG pipeline with reranking. Retrieval supplies candidate code and documentation; reranking helps order that material before it becomes context for an answer.
There is no single best embedding model or LLM for every application. Liu reports that Continue's evaluation found voyage-code-3 performed best for its application. He gives no numerical scores, dataset, or comparison protocol. The relevant requirement is clear, however: codebase chat depends on understanding code, documentation, and developer context. Evaluate model choices against those demands rather than assuming that a general-purpose result predicts performance on your repository.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Semantic relevance still needs structured constraints
In the second application, Liu draws attention to a blue box containing filtering and other structured inputs to search. The lesson is that embeddings alone are often insufficient. A document can be semantically relevant while failing a condition that determines whether the application may use it.
Take legal documents embedded with voyage-law-2. A query might concern the right legal topic but require documents from a particular state, only official documents, or documents containing a specific set of details. These requirements belong in a filtering stage. Liu says filtering typically happens inside the vector store, although some systems apply it afterward.
A small TypeScript example makes the distinction explicit. Suppose a semantic search has returned these three records, and the application requires official California documents:
typescript
type SearchHit = {
id: string;
state: string;
official: boolean;
score: number;
};
const hits: SearchHit[] = [
{ id: "doc-a", state: "NY", official: true, score: 0.94 },
{ id: "doc-b", state: "CA", official: false, score: 0.91 },
{ id: "doc-c", state: "CA", official: true, score: 0.87 },
];
const eligible = hits.filter(
(hit) => hit.state === "CA" && hit.official
);
console.log(eligible.map((hit) => hit.id)); // ["doc-c"]
The illustrative scores rank semantic matches; the metadata determines eligibility. The highest-scoring result does not satisfy the state requirement, and the next result is unofficial. Only doc-c survives. This shows the post-retrieval variant of Liu's filtering stage: structured data adds constraints that similarity scores do not enforce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An agent can search, reconsider, and search again
The third application changes retrieval from a single input-output operation into a feedback loop. An LLM can issue a search, inspect what comes back, and then expand or decompose the query. Retrieval becomes part of an ongoing reasoning process rather than a one-time preparation step.
Liu illustrates decomposition with an X/Twitter earnings request. His spoken example moves from Q4 to full-year 2024; the full-year interpretation explains the resulting plan: query Q1, Q2, Q3, and Q4 separately, sending the four searches to vector stores or databases. The example describes how to break up a request, not a retrieved set of financial results. Each subquery creates an opportunity to gather a distinct part of the information needed for the larger answer.
Calling 2025—and possibly 2026—the era of agents, Liu emphasizes another requirement: search must work well with conversational data. The diagram connects a human and an agent to both a knowledge vectorstore and a conversation vectorstore, with retrieval and return paths. It separates stored knowledge from conversation history while allowing both to inform the interaction. Liu notes that the diagram omits implementation details; its useful architectural point is that future searches can depend on earlier exchanges and retrieved results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Text and images in the same search space
The first future direction is multimodality. Liu expects it to become foundational to search, just as it is becoming foundational to language models. He means joint understanding of text and images, or eventually text, images, and audio—not embodied combinations of sight, touch, and taste. The practical question is whether information in different media can participate in the same retrieval system.
voyage-multimodal-3 provides the concrete text-and-image example. It accepts text, images, or interleaved text and images, embedding them into one semantic space. Audio belongs to Liu's broader vision, not the model inputs described here. In the slide example, the query “strong LLMs” has a Claude 3.5 blog post as its nearest document. Although Liu points out that arrows are missing from the displayed diagram, the intended relationship is straightforward: a text query can retrieve a relevant document represented through multimodal content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tell the embedding what to emphasize
The next direction is instruction tuning, with reasoning as a secondary possibility. The baseline embedding interface takes a query or document and returns a vector. Liu envisions adding an instruction alongside that input to steer what the vector emphasizes. His forecast concerns richer steering and broader use of instruction-aware retrieval.
For example, a user might ask for documents that discuss one particular aspect of a topic in detail. Topic similarity alone can retrieve broad overviews; an additional instruction expresses the desired focus and depth. That differs from the earlier state or official-status filter: it changes the semantic preference the system should represent rather than testing a known metadata value. Liu presents this as a direction for search, without supplying a concrete reasoning implementation or a guarantee that an instruction will enforce every constraint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring the retrieval pipeline into the database
The final direction is what Liu calls an agent-native database, acknowledging the term's buzzword quality. Even a simplified retrieval architecture requires several components to work together. Voyage joining MongoDB makes the prospect of integrating those components particularly relevant to his team.
The proposed shift is from assembling a pipeline to using one data platform that performs its core retrieval operations:
| Operation | Separate components | Proposed integrated platform |
|---|---|---|
| Embedding | Call an embedding model | Platform generates embeddings |
| Vector search | Query a vector database | Platform retrieves candidates |
| Reranking | Call a reranker | Platform reranks results |
| Query transformation | Coordinate additional logic | Possibly augment or decompose queries |
The slide contrasts an embedding model, vector database, and reranker with a single MongoDB + Voyage AI box between query and results. Embedding and reranking are central to the proposal; query augmentation and decomposition are possibilities Liu explicitly leaves open.
For this June 2025 talk, the integrated platform is a forecast, not a deployment recipe. Liu expects more consolidation during that year and the next. The substantive promise is to make the database responsible for more of the retrieval work—representing information, finding candidates, and ordering results—so application builders have fewer separate components to coordinate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains codebase indexing and shows how to configure Voyage and local embedding providers in Continue.
Describes code retrieval training, evaluation, variable embedding dimensions, and quantized storage formats.
Introduces the legal embedding model and reports evaluations across legal and long-document retrieval datasets.
Explains shared representations for text, screenshots, and interleaved images, with multimodal retrieval evaluations.
Further reading
Research on generating task- and domain-specific embeddings by supplying instructions alongside text.
Updates since the talk
Announces Atlas embedding and reranking API availability, the Voyage 4 family, and video retrieval with Multimodal 3.5.
Read the complete timestamped transcript
- 0:00
[upbeat music] Welcome, everybody.
- 0:16
Uh, I wanna thank you for coming to this session today. Um, today I wanna talk about AI-powered search and retrieval. Uh, and for those of you who don't know me, my name is Frank.
- 0:25
Uh, I am actually a part of the Voyage AI team, and we recently joined MongoDB, I wanna say probably about three to four months ago. Uh, just a quick introduction to Voyage AI, you know, we build the most accurate, cost-effective embedding models and rerankers for RAG and semantic search.
- 0:39
Uh, a lot of the applications, I think that we've seen, that I've seen in particular, actually go beyond that. So some folks use it for classification, others for, you know, a variety of different applications.
- 0:47
We've got clustering, so on and so forth. Uh, Voyage is available via the Voyage AI API, Azure, and AWS Marketplace as well. And now we are a part of MongoDB, so- [cheering] Representing MongoDB. [laughs]
- 0:59
Yeah, thank you. So we're representing MongoDB here. Uh, and, and, you know, we're all really excited to see what the future will hold. And the reason, you know, I think in, in the past when I would give presentations like this, uh, I'd actually talk very, very specifically about things like evaluating...
- 1:14
evaluation for embeddings, right? I would talk about things like, um, you know, how, how do rerankers, uh, play a role in your ultimate retrieval stack? And today, I wanna go a little bit higher level.
- 1:24
I wanna talk a little bit more about what I like to call AI search. I know it's a very, very broad term, has a lot of different meanings to different folks.
- 1:30
But I wanna go-- I wanna start, use that as a starting point, and then talk about where we are today, and also to a secondary extent, where we're going as well.
- 1:38
So a quick, uh, agenda here for the next 10 or, 10 or so minutes. Um, but first I wanna give a quick refresher, you know, uh, a little bit about, uh, about embeddings, uh, about, uh, search and retrieval more broadly in this day and age.
- 1:51
Then I'll talk about some real-world applications. And I think each of the applications, there's gonna be three, each of the applications I'm gonna talk about, I'll probably spend about a minute, uh, each there.
- 1:59
There's gonna be a, a lesson or something to learn, or, or a key fact to take away from that application. And then what's most exciting, I think, to me, and hopefully to the rest of you as well, is what's to come, right?
- 2:11
Where is the future of AI-powered search and retrieval, and, and where are we going from here? So a quick refresher. I'm, I'm gonna blow through these pretty quickly. AI-powered search, at least how I define it, and, and how I hope, uh, this will continue to be defined moving forward, is a search system that finds related concepts even
- 2:29
without identical wording. I think this is very important, right? So a lot of folks, uh, you may be familiar with things like TF-IDF or BM25. Uh, you know, AI-powered search goes way beyond that, right?
- 2:39
Not only it can understand keywords, it can help you retrieve based on some of these more traditional, uh, information retrieval algorithms, but it can also help you find related concepts.
- 2:49
To that point, it also understands the user's intent. So for example, if I am in a, you know, I'm trying to search for some products to buy something to buy, and I say, "My best friend is sick," uh, perhaps it can recommend me some, some, some, you know, get well baskets or something like that, right?
- 3:06
It really should be able to understand what my ultimate meaning is, rather than just saying, "Hey, okay, my best friend is sick. Maybe, you know, I'm gonna try to find something that's good for a best friend, uh, more generically."
- 3:16
The last thing that I'll mention is that it can perform some level of reasoning and some level of instruction following, and I'll get to that a little bit later, uh, in that last section.
- 3:25
But to get right into it, um, you know, a really popular use case of AI-powered search and retrieval is RAG, and I'm gonna go through this pretty quickly. The idea is that without retrieval, you either get probably some sort of hallucination, or in some cases, your LLM is just gonna flat out refuse to respond to you.
- 3:41
Or it's gonna give you a really, really generic answer like the one you see up here above. Uh, but with retrieval, with AI-powered search built into it, you get a much, much more grounded response.
- 3:51
Uh, again, this use case I think is, is pretty common. These are actually slides that I, uh, I, I took from, from Tengyu's talk that's gonna be happening at 12:15, so you guys should definitely go to that if you're around.
- 4:02
Uh, and the idea is that, hey, you generate embeddings, use that for search, and then you give it to your LLM.
- 4:07
Last thing I wanna mention is that embedding quality here is a very, very core component of AI-powered search and retrieval. And I think pretty much 95 to 99% of the systems I've seen out there, from what I get- from what I go- from what I gather, use embeddings in some way, shape, or form.
- 4:25
The idea is that, again, you have embedding, you have these, this unstructured data. Usually, it's text, you know, PDFs, Word documents, Google Drive files, uh, PowerPoints, et cetera. Uh, and you're able to embed them into a same space such that when you do a search or you do a...
- 4:39
you, you try to find a prompt, you have a prompt, you search for the most relevant documents, you're able to pull that information all the way up to the top.
- 4:47
So I'm gonna go through these pretty quickly. I'll try to spend about a minute each here. Um, some real-world applications of AI-powered search and retrieval. The first one is chatting with your codebase.
- 4:58
And this is actually, if folks have heard of Continue.dev, this is actually their application. A lot of their code is open source. You do... This is a classic RAG, uh, plus re-ranking approach.
- 5:10
I think the lesson from this particular application is that there is no one-size-fits-all embedding model. There's no one-size-fits-all LLM. Uh, always do evaluations to see which one is best for your application.
- 5:24
Now, in this case, Continue did theirs, and, and they, they found that voyage-code-3 actually performs the best. Uh, again, the reason is because for a lot of chat with your codebase applications, you want to have an embedding model, and also to a secondary extent, an LLM that is really, really good at understanding, well, code, documentation, and developers,
- 5:43
so on and so forth. So this is the first lesson from the first application. The second one is that there's, you know, when it comes to... The second one, again, it's also a very domain-specific application here.
- 5:53
But the second one that I wanna mention is if you see this blue box here, where there's, there, there's some filtering and then there's some other structured data that's also passed to the search system-
- 6:04
If this is the thing that I want, that I wanna highlight for this particular application, is that oftentimes, and, you know, coming from a company that builds embedding models, it's, it's hard for me to say this.
- 6:13
I think oftentimes embedding a l- embeddings alone are not enough, right? If you wanna build a really powerful search and retrieval system, you need to have a lot of that structured data that's a part of it.
- 6:23
So to give you an example beyond just this particular, uh, domain, you know, if I have, let's say, some legal documents, uh, I embed those legal documents using Voyage Law 2.
- 6:35
Uh, but then when I do my search or if I'm building my agent, perhaps I want to understand, I wanna know, "Hey, I, I only wanna find documents that are from a particular state," or maybe I wanna find only official legal documents, or I wanna find documents that have a very, very particular set of details inside of
- 6:52
them. This is all, this- these are all things that can be done, uh, at the filtering stage. So this is typically done directly inside the vector store. In some cases, it's done after that, right?
- 7:00
So I wanna say, I wanna be very, very clear, there is oftentimes, and the second lesson here is that there's oftentimes other sources of data, other pieces of structured data that you need to include inside of your search system as well, right?
- 7:12
Uh, and, and really, you know, just go beyond that. And then the last thing, you know, the last sort of application, these are, these are real world, that these are actually built today.
- 7:21
The last thing that I wanna say is when it comes to a lot of agentic retrieval, oftentimes it's a feedback loop. So your AI search system is no longer just input, output, right?
- 7:34
Sometimes if you get a query, you might want your LLM to, you know, you know, do some searches, and then you might want your LLM to expand that query, or you might want your LLM to decompose that query.
- 7:44
I'll give you a quick example. Uh, if I ask for something like, um, you know, what are Twitter's... Give me Twitter or I guess X now. Uh, you know, give me X's Q4, you know, or 2024 earnings, right?
- 7:57
You could, your LLM could decompose that into Q1, Q2, Q3, Q4 earnings, and then send that as four separate queries to these different vector stores, to the different vector databases, so on and so forth.
- 8:08
On top of this as well, you also see that we're in the era of agents, right? 2025, I would say, is the year of the agents, maybe even 2026 as well.
- 8:16
And a lot of agents, a lot of these agentic applications, they're gonna need to be really, really powerful at conversational data. So you really want embeddings. You really want a search system that's built around that.
- 8:27
Now, there's a lot of details that's missing in this particular block diagram, but I think hopefully that goes to show you, uh, sort of the lessons to take away from that.
- 8:37
Okay, so this is the most... So, you know, I just covered three, um, existing applications. Uh, and hopefully these applications, I think, they give you a window into where we are today.
- 8:48
Um, you know, some of the tips, some of the tricks, just three of them, uh, that, that's being used today in these AI search systems. There's many, many out there, right?
- 8:55
There's, I think, a lot to cover here. But I think what's more exciting to me is, is what's to come in AI-powered search and AI-powered retrieval.
- 9:04
Uh, I don't know what happened to my arrows here. I apologize. Seemed to sort of disappear in the background. But the future is 100% multimodal. Uh, that is the case for large language models.
- 9:15
That is the case for embeddings. For AI search and retrieval overall, that's going to be the case as well. Uh, I think there's no doubt about that. And when I say multimodal, I really...
- 9:24
I don't mean multimodal in the sense of, oh, you know, I'm, I'm, I'm an agent that's operating in the real world. I can connect all these different modalities like sight and, and, and, and touch and, and taste together.
- 9:34
I'm talking more about modal- multimodality just from a pure, sort of like a foundational perspective. The ability to understand images and text together, or the ability to understand images, text, and audio together.
- 9:46
And it's gonna be really important for search systems. It's gonna be really important for embedding models, just as we have a lot of VLMs out there today. This particular example is Voyage Multimodal 3.
- 9:56
Um, again, I apologize. I don't know what happened to the arrows here. But the idea here is that it can take text, it can take images, or it can take a combination of text, interleave text and images, and really embed all of those into a single, really powerful semantic space.
- 10:13
So it might be a little bit hard to understand exactly what's going on here. But, uh, the query being strong LLMs and then the nearest, uh, the nearest sort of document being, uh, the Claude 3.5 blog post.
- 10:27
So I hope that's a little bit clearer here. I know, again, um, uh, I know it might be a little bit harder to understand, but this is, uh, this is one of the things that I wanna get to.
- 10:35
The second thing that I also think is particularly exciting is instruction tuning. Instruction tuning into a second, a secondary extent reasoning as well. So right now, if you look at embeddings and you look at embedding models, they just take a query or they take a document, and they give you a vector.
- 10:51
I think moving forward, we're going to see situations where in addition to that query or in addition to that document, we wanna be able to steer the vector in a particular direction or in a particular, in a particular way, right?
- 11:04
So to give an example, perhaps I want to, you know, in addition to my query, in addition to my prompt, in addition to my search, uh, I also ask, I also give it an instruction to say, uh, "Find documents for me that only dive into detail about this particular aspect," right?
- 11:19
And that is where I think instruction tuning, uh, is really gonna play a huge role moving forward for AI search and retrieval.
- 11:27
Uh, last thing that I wanna talk about, um, is, uh, and this is sort of a, a buzzwordy kinda term out there, is sort of the agent native database.
- 11:34
And I think this is, this is where Voyage joining forces with MongoDB is super exciting for a lot of us. The idea that today a lot of search and retrieval, there's many, many multiple different components that you have to put together, and what you see on the left is actually already a really, really simplified version of that.
- 11:50
And the capability to move directly to something that is just a single piece of infrastructure that does the embedding for you, it does the re-ranking for you, perhaps it does some of that query augmentation or query decomposition for you, uh, all of that inside a single data platform, inside a single database, I think that is super exciting.
- 12:08
So I think this is something that you'll see more and more of this year and also next year as well, and something to look out for hopefully as well.
- 12:16
So with that being said, I've got about three minutes left. Would love to take any questions if you have them. Um, but, uh, I'll also leave this up for a couple more seconds.
- 12:23
Feel free to, uh, scan those QR codes, follow us, and I hope to see you sometime else at this conference. [upbeat music]