AI Engineer World's Fair 2025
RAG in 2025: State of the Art and the Road Forward
Read the talk
RAG in 2025: Better Retrieval, Less Application Machinery
Enterprise AI needs access to private knowledge. Tengyu Ma explains how retrieval supplies it, where embeddings still fall short, and which parts of the pipeline models may absorb next.
From a talk by Tengyu Ma
Before you start: Familiarity with LLM prompts and the idea of representing text as vectors will help; the article explains the retrieval pipeline from there.
How does a model learn what your company knows?
If an out-of-the-box language model already knew MongoDB’s internal information, something would have gone wrong: that information would have leaked. An enterprise assistant needs a deliberate path to proprietary knowledge. Calling the system an agent does not remove this requirement; an agent using an LLM inherits the same information gap.
Tengyu Ma, Voyage AI’s co-founder and former CEO, introduces the topic following the company’s acquisition by MongoDB. Retrieval is the company’s focus: supplying the information a model is missing. The architectural choice is how to supply that missing information: put documents in the model’s context, encode knowledge in its parameters through fine-tuning, or retrieve relevant documents when a question arrives.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ways to use a library
The three approaches place knowledge in different parts of the answering process.
| Approach | Where the documents go | What happens at query time |
|---|---|---|
| Long context | Into the prompt | The model reads the supplied collection |
| Fine-tuning | Into updated model parameters | The model answers from those parameters |
| RAG | Into an external searchable collection | Retrieval supplies a relevant subset |
For long context, Ma imagines supplying a million or even a billion tokens; these are illustrative collection sizes, not claims about a particular model’s capacity. Fine-tuning instead updates the model in advance, then answers without revisiting the documents. Retrieval-augmented generation keeps the documents available externally: the query drives search, and the resulting passages become the context for generation.
Ma’s preference for RAG comes from a library analogy. Long context resembles scanning the entire library for every question. Fine-tuning resembles reading the library beforehand and trying to internalize its contents. He doubts the long-run cost efficiency of the first approach and the suitability of the second for ingesting enterprise knowledge.
Memorization creates more problems than capacity alone. Which books should the model memorize? If a fact must be removed, how do you make it forget that fact cleanly? If different users may access different books, how do you enforce those boundaries once their contents have been mixed into shared parameters? These are knowledge-selection, deletion, and governance problems.
RAG resembles how people actually use a library: find the relevant book or chapter, read it, and answer the question. Ma describes this as a modular, reliable, fast, and inexpensive arrangement. Its underlying advantage is hierarchical information access: keep the large collection outside the model and bring a small relevant portion into the answering process when needed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What better embeddings change
The basic retrieval path has two principal components: an embedding model and a vector database. The embedding model maps documents and queries into vectors that represent their content or meaning. The database stores document vectors and uses k-nearest-neighbor search to find those closest to the query vector. Their associated documents then pass to the LLM for answer generation. A more advanced pipeline adds a reranker, discussed shortly.
Ma places the launch of OpenAI V3 embeddings roughly a year and a half before the talk and describes subsequent progress from Voyage and Cohere. The improvement is not only higher retrieval quality: it is a better quality–cost trade-off. At a fixed parameter count, a model can retrieve more accurately; at a fixed quality level, a smaller model can be cheaper. He attributes these gains to work throughout the training stack, including data curation, data selection, architecture, loss functions, and evaluation.
Ma reports roughly 80% average retrieval performance across about 100 datasets. The displayed chart labels retrieval quality as NDCG@10, a ranking metric; the figure should not be read as the percentage of enterprise questions answered correctly. He describes the remaining roughly 20% as improvement headroom.
The average conceals a wide distribution: Ma estimates that roughly half the datasets reach 90–95%, while others sit around 60%, 20%, or 30%. Common retrieval tasks can therefore already work well even while harder tasks leave substantial room for improvement. An aggregate score alone cannot tell you which situation your application resembles.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Smaller vectors without starting over
Embedding storage has two adjustable dimensions: how many coordinates each vector contains and how precisely each coordinate is stored. Matryoshka learning trains a representation so that a prefix of its coordinates remains a useful embedding. You can retain the first portion of the vector rather than requiring its full length for every deployment.
In Ma’s example, retaining the first 256 coordinates of a 2,048-dimensional embedding incurs an estimated 1–2% retrieval-quality loss. The prefix works because the model was trained to make it useful; arbitrary truncation of an arbitrary embedding is not the mechanism. Quantization during training addresses the other dimension, preserving useful retrieval behavior when coordinates use lower precision.
Ma reports at least 10× vector-storage savings with little quality loss, and roughly 5–10% loss at 100× savings. These are approximate retrieval trade-offs, not reductions in the total RAG bill; his spoken percentages do not distinguish relative losses from percentage-point changes. He also claims that Voyage remains ahead of OpenAI at roughly 100× storage reduction because the models occupy different quality–storage Pareto frontiers. A stronger starting representation can remain competitive even after substantial compression. Domain specialization, introduced next, can improve that trade-off further.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the candidates and the text being searched
Replacing the embedding model is one of the simplest ways to improve retrieval, but it is not the only one. Hybrid search brings together candidates from lexical search and other retrieval methods, then uses a reranker to order them. Ma notes that Voyage also supplies a reranker: candidate discovery and final relevance ordering need not be the same operation.
The next set of changes improves what enters retrieval:
- Query expansion: use an LLM to elaborate a short query into a longer, more informative search input.
- Query decomposition: split a complex question into smaller subqueries, each capable of finding a different subset of documents.
- Document enrichment: attach information that makes an isolated passage easier to interpret and retrieve.
The document side matters because chunking can discard context. A passage may lose its title, section header, category, author, or date even though those fields explain what the passage means. Restoring that global information to each chunk gives retrieval access to it again.
For example, an isolated chunk saying “Requests remain available for 30 days” becomes less ambiguous when its document title and section accompany it. A small enrichment function can preserve the original text while preparing a separate search representation:
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
id: str
text: str
title: str
section: str
def search_text(self) -> str:
return (
f"Document: {self.title}\n"
f"Section: {self.section}\n\n"
f"{self.text}"
)
chunk = Chunk(
id="export-policy-1",
text="Requests remain available for 30 days.",
title="Data Export Policy",
section="Download availability",
)
text_to_embed = chunk.search_text()
original_text = chunk.text
Here, the added fields are existing metadata. Ma also describes generating additional chunk context with an LLM and points to Anthropic’s Contextual Retrieval. That approach makes otherwise isolated chunks more informative before retrieval; he reports good results without giving a numeric improvement in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialize the representation, then add retrieval layers
A general-purpose embedding model does not have to be the endpoint. Ma describes Voyage and MongoDB’s work on code-specific embeddings as improving both retrieval performance and resilience to compression. In the code-domain example, Ma estimates about 5% quality loss at roughly 100× compression, compared with about 10–15% in the earlier comparison. The configuration matters: these estimates describe the examples he presents, not a fixed compression penalty shared by every corpus.
You can also fine-tune the embedding model on your own data. This is a different use of fine-tuning from the earlier proposal to memorize enterprise documents inside a generative model: here, the adaptation changes the representation used to find documents. Graph information and iterative retrieval add further layers over those embeddings, changing how the system explores and selects evidence while retaining the underlying vector representations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The training work moves to the provider
Ma expects retrieval to persist because its basic operation remains useful: hierarchically select a small subset of a large collection, then use it to answer a question or take an action. His forecast for how the implementation changes comes from an earlier transition in enterprise machine learning.
When he began teaching a Stanford machine-learning course with Chris Ré roughly seven years earlier, one slide described a seven-step enterprise workflow. It included collecting data, creating train/test splits, defining loss functions, building models, and iterating. The course still teaches that workflow, but with more qualifications.
Off-the-shelf LLMs changed the starting point. Ma argues that, in many cases, an imperfect pretrained model already performs better than what an enterprise could previously achieve through its own bespoke training process. Private information remains missing, so RAG still has a role. But application teams can begin by connecting capable components rather than first carrying out the entire training workflow themselves.
The training work has not disappeared; responsibility for it has shifted. Providers such as OpenAI, Anthropic, Voyage, and MongoDB do that work, while application builders consume its results. Ma expects a similar transfer of responsibility within RAG.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A larger model layer, fewer workarounds
The current RAG stack separates into several layers:
| Layer | Examples |
|---|---|
| Computing infrastructure | GPUs; CPU-based nearest-neighbor search |
| Models | Embeddings, rerankers, LLMs |
| Application retrieval techniques | Parsing, chunking, recursive search, contextual chunks, graph RAG |
The top layer compensates for limitations below it. If an embedding loses context, the application enriches chunks. If retrieval misses useful evidence, the application searches recursively or follows graph relationships. These techniques are necessary today because the models are imperfect.
Ma predicts that stronger models will absorb more of the performance gains currently supplied by application code. He compares this with applications built around GPT-3 roughly two years earlier: work that once required substantial machinery around the model can increasingly be handled by a stronger model out of the box.
That does not eliminate customization. A general-purpose model cannot infer information it has never been given. In particular, an application may have its own definition of similarity or relevance, which Ma suggests specifying in a prompt. The distinction is between compensating for a model’s general weaknesses and supplying task-specific information the model could not otherwise possess.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the screenshot as the retrieval input
Multimodal embeddings provide a concrete example of moving work into the model. Ma describes Voyage’s ability to embed screenshots directly. A traditional PDF pipeline first extracts text and images, then may embed each separately. Parsing the PDF becomes a substantial part of the system. For video, a text-based pipeline commonly begins by producing a transcript and embedding it.
A screenshot-based path lets PDFs, PowerPoint presentations, and other slide decks use the same basic operation: render a page or slide, embed its screenshot, and search the resulting vectors. Ma extends the idea to consecutive video-frame screenshots, while explicitly calling it an imperfect approach. Searching visual frames does not by itself provide retrieval over the audio or understanding of the video’s temporal structure.
Tables illustrate why this matters. Instead of first deciding how to serialize headers, rows, and cells into text, the system can embed a screenshot of the table. The model receives the visual arrangement directly. Ma reports improvements across evaluations involving document screenshots, tables, figures, and text-only inputs, without giving spoken numeric results. The architectural benefit is a shared ingestion path for sources whose useful information is not confined to plain text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return small chunks with knowledge of the whole document
The final example is a proposed context-aware, automatic-chunking embedding service. Ma first explains why chunking remains necessary. One reason is the embedding model’s context limit: with the stated 32K-token window, a 100K-token document needs at least four chunks. His suggestion that Voyage has the longest window is tentative; the concrete constraint is that the document exceeds the model’s input capacity.
A larger embedding window would solve only that first problem. If retrieval returns the entire document, the generation model must still process a very large input for each question. Ma uses a 100K-token document to illustrate the expense. Smaller retrieval units also focus the answer on a relevant paragraph or page instead of relying on the model to notice information buried in the middle of a long context. The unit you can embed need not be the unit you should retrieve.
At the time of the talk, users had to perform that chunking themselves. Ma’s proposed interface transfers the work to the provider:
- Submit a long document.
- Receive the document’s chunks and a vector for each chunk.
- Use those vectors to retrieve focused pieces of the document.
Each returned vector would represent the detailed content of its own chunk together with coarse information from the rest of the document. This preserves local retrieval granularity without treating every chunk as an isolated fragment. The final diagram shows the intended output: multiple chunk–vector pairs, with both local detail and document-level information represented in a vector.
This is a roadmap announcement in the recording, not a demonstration of an already available automatic-chunking API. Ma closes with a further plan for a fine-tuning API that would let customers adapt embeddings using their own data, without giving a firm launch date. Both proposals follow the same direction: let application builders supply their documents and domain knowledge while the provider takes responsibility for more of the machinery that makes them retrievable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Adds explanatory document context to chunks before embedding and lexical indexing.
Further reading
Historical evaluation across 100 datasets, with retrieval metrics and vector storage trade-offs.
Original research on embeddings that support multiple representation sizes.
Code retrieval evaluation and examples of reducing embedding dimensions and precision.
Explains embedding interleaved text and images, including document screenshots, tables, and slides.
Updates since the talk
- Voyage Context 3 releaseArticle
The July 2025 release of chunk embeddings that incorporate surrounding document context.
Current examples for embedding document chunks and automatically returning chunk text with vectors.
Read the complete timestamped transcript
- 0:00
[on-hold music] Thanks for coming.
- 0:15
Thanks for having me here. Um, I'm Tengyu Ma. I'm, uh, um, I was, uh, uh, the CEO and co-founder of Voyage AI. We just recently got acquired by MongoDB.
- 0:25
I'm also teaching at Stanford as well. So, um, this is about RAG, which is, uh, the main focus of Voyage AI, uh, the startup who is focusing on how to make retrieval better.
- 0:35
So, um, um, but I would just generally talk about, you know, RAG and, and we'll touch on some of the products we make as well very quickly. So I guess, uh, why we are, um, doing RAG or anything like that, right?
- 0:45
So I guess the main reason is that large language model or these days agent, um, which are, you know, using large language models as well. Uh, if they out-of-the-box, they cannot just, uh, uh, have, uh, uh, propriety information from any of the companies, right?
- 0:59
Because if they know anything about what MongoDB, for example, internally has, then the, the data was leaked. So that means that if you want to apply any of this to enterprise, uh, uh, then, um, uh, you need to ingest a lot of data from the, uh, propriety information.
- 1:15
So, um, and, uh, I'm gonna discuss, you know, why, uh, which kind of technologies to, to enable us to ingest the data. I guess there are, uh, a few options: RAG, fine-tuning and long context, which are all ways to ingest data, and I'll focus on RAG for the rest of the talk.
- 1:30
So I guess, you know, for this audience, probably most people knows these technologies, and they are all very simple on, uh, on high level. So for long context, it's just the, the most simple, you just, uh, dump all your documents, uh, uh, to, uh, a large language model's context, and maybe it's like one million tokens, maybe it's
- 1:46
one billion tokens. Um, so and then you have a query, and you just, uh, get a response. Uh, fine-tuning is like you first fine-tune a large language model, you update the parameters, and then you say, "I'm not gonna look at the documents anymore.
- 1:59
Uh, when the query comes, I just use the updated parameters to generate response." Uh, and RAG is, um, um, also pretty simple. So basically what happens is that on the fly, you use the query to retrieve some subset of the documents.
- 2:12
You use a retrieval or search method, uh, um, and then you get some relevant documents. You give this small set of relevant documents to the large language model, and then you generate response based on those contexts.
- 2:23
So this is one-- my one slide kind of like, uh, uh, summary of, you know, how I think about the differences between these technologies. You know, some of the, these are, uh, uh, inspired by some of the research at Stanford when we kind of started to, uh, uh, um, uh, uh, build Voyage.
- 2:38
You know, we kind of like believe in RAG, and one of the reason is that we don't believe that fine-tuning can work. And, and long context, I think, um, um, I also don't really, uh, believe that it can be cost efficient in the long run.
- 2:49
So, so basically, I think the way that I think about this is that, um, I, uh, try to make analogy to how humans, uh, are, uh, learning from or using the additional propriety information.
- 3:01
Um, um, so in some sense, long context is kind of like you scan the entire library to answer any single question, right? Every time you answer a question, you, you need to sc-- go through the entire library, which has like probably one billion tokens.
- 3:14
And fine-tuning is kind of like you read this library in your, in advance. You muscle memorize them. You try to internalize them in your brain, in your neurons, in your synapses, and you update your brain, basically rewire your brain, so that you really know all of those deeply.
- 3:27
Um, the challenge there is that, you know, it's very difficult and, um, somewhat unnecessary, um, um, because, you know, you cannot really memorize all the books in your-- in, in the, in the world and, uh, and, and memorizing a subset of them sometimes is kind of like, you know, which subset you wanna memorize is kind of, uh,
- 3:44
tricky as well. So and another thing is that, um, um, it makes, you know, forgetting the knowledge also tricky because you don't know which part of the knowledge you should forget and how to clearly forget all of them.
- 3:55
And also this makes the, the access, the, the, the data governance also kind of tricky because, you know, uh, maybe there are so many libraries, so many books in the library, and not everyone can access everything and, uh, how to organize those.
- 4:07
And on the other hand, RAG is very, very simple and modularized, uh, as I've shown. So, um, and very reliable and, and, you know, and, and also kind of fast and, and cheap.
- 4:17
So, um, and it's kind of like similar to how humans actually are using the libraries, right? You retrieve the most relevant, you know, book chapters or books or book chapters and then answer the question.
- 4:27
It's kind of a hierarchical way to store information, right? You don't really put all of the information in your brain. You put them in a library and then use them, uh, uh, when you need it.
- 4:36
So, um, that's why I believe in RAG, and, uh, and this is kind of how you implement the retrieval part. So basically, there is a breakdown of two components.
- 4:44
Actually, there are three, you know, if you are advanced. So, uh, so this, uh, embedding models which vectorize the, the documents and query, uh, into vectors, and the vectors are representations of the, uh, this kind of like representations of the content or the meanings of, uh, the documents and queries, and then use the vector database to store
- 5:02
the data and also search, uh, uh, with the k-nearest neighbor search in the vector space. And then you get the relevant documents, and then you can use large language model to generate answers.
- 5:11
So, um, we have seen significant improvements over the retrieval accuracy in the last, uh, two years. Um, when we started Voyage, you know, I think, uh, uh, OpenAI V three was not, uh, yet launched.
- 5:23
I think OpenAI V three was launched one-point-five years ago. Um, and in the last one-point-five years, you know, Voyage, you know, uh, uh, has made significant progress. You know, Cohere also made some progress.
- 5:33
So, uh, we can see that the new model has much better, uh, accuracy and with lower cost. Uh, um, and, uh, uh, generally, we have much better scaling law, right?
- 5:41
So the same number of parameters, the, the quality becomes better or the same quality, the, the, the parameters become smaller, and it becomes cheaper. Um, and all of these are through kind of like, uh, you know, optimizing the research stack, uh, the tuning stack, you know, as much as possible, you know, all the way from like data
- 5:58
curation, data selection, uh, uh, uh, architecture, loss functions, you know, evaluation, so on and so forth.
- 6:04
Uh, and we still, you know, believe that there's a big headroom here because, you know, n- right now you can see that in this plot, you know, we are averaging over about 100 data sets, and the accuracy is about 80%.
- 6:14
So that means that you still have, like, probably 20% of the improvement, um, uh, headroom. But that said, you know, just to be clear, uh, it's not like for every data set you only have 80% accuracy.
- 6:24
For probably half of the data sets, the accuracy is probably 90% or even 95%, and for some of the other ones, it's kind of 60, sometimes 20, sometimes 30, so that's why your average is 80%.
- 6:35
Um, so for-- so basically I'm saying, like, for some of the tasks that are common, uh, I think you can get a-already very high accuracy in the retrieval step.
- 6:43
Um, and another thing that, uh, uh, Voyage and other, uh, uh, uh, companies has, uh, uh, offered is this, uh, so-called matrix learning and also quantization while training. So basically these are two approaches to reduce the, the storage cost, uh, for the vectors.
- 6:59
So basically, matrix learning means that you, uh, make sure that, uh, even you have like a, a high-dimensional, uh, embedding, right? You can use a subset of the, uh, the coordinates.
- 7:10
Uh, it's, it's usually the first, uh... Let's say suppose you have, like, 2048 dimensional vectors, and the first, uh, 256 dimensional, uh, sub, uh, vector is still a reasonable embedding.
- 7:22
The accuracy wouldn't be as high as two, uh, 2048, but it will be almost the same, uh, maybe with, like, 1 or 2% loss. And quantization is kind of this, in a similar vein, so where you are-- even you lower your precision of the vectors, you still get pretty high performance.
- 7:37
Uh, and you can see the, uh, the trade-off on the right of the figure, uh, here. So basically, you can save, you know, 100X, you know, at least 10X without losing much.
- 7:47
If you save 100X in the storage cost, then you start to lose probably five to 10%. Um, but Voyage, you know, is doing a great job here because, you know, you can save, uh, 100X but still doing better than OpenAI.
- 7:59
That's just because the parental frontier is, uh, different. So, um,
- 8:05
um, and, and you can actually see better trade-off, you know, for domain-specific models, which I, I'm gonna discuss in a moment. Um, I have nine minutes here, so I will probably just quickly go through some of the, uh, uh, the techniques that you can use.
- 8:17
Um, so basically the next question is: how do you do better RAG, you know, without-- besides using better embedding models? Using better embedding models is probably one of the simplest way.
- 8:25
Um, so I'm just gonna go through it quickly. So one of them is to use hybrid search and rerankers. You can use, you know, lexical search and other kind of search, and then combine them with a reranker.
- 8:36
Uh, and Voyage provides a, a reranker as well. Um, and another one is you can enhance the queries and documents by the so-called query decomposition and, and document enrichment.
- 8:46
Um, so this is probably the most common one, maybe I spend one minute on it. So it's actually very simple. You just say if you have a query RAG, then you try to improve the query by, uh, making it longer using a large language model.
- 8:56
Uh, you can also decompose the longer query to small sub-queries so that you can have, like, a few different queries and search for different subset of documents. Um, and you can also enrich the document by adding additional meta co-- information in there.
- 9:09
You can add titles, you know, headers, you know, categories, authors, dates. Sometimes you trunk the document so that in the trunk you don't even have this information anymore, so that's why you have to add the global information into each of the trunks.
- 9:21
And some of this global information can be added by large language models. Anthropic wrote a blog post which does chieve-- achieve pretty good results. So basically they use large language models to generate additional context, um, that you can add to the trunks so that you can make the trunks, you know, more, uh, informative, and then the, uh,
- 9:38
it's easier to search, uh, through them. So, um, another one is you can use domain-specific embeddings where, um, you, um, c- you customize embeddings for certain kind of domains.
- 9:48
You know, in, uh, MongoDB or Voyage, we customize it for code, for example, and you can see that, you know, you get much better performance and also, uh, it's a better trade-off in terms of the, the storage cost and the, um, uh, and the, and the accuracy.
- 10:01
So basically, you don't lose as much if you compress the vectors even further. Um, so, so here we lose probably 5% by compressing, uh, for, like, uh, about 100X, um, but before we lose probably 10% or 15%.
- 10:15
Um, fine-tuning is another one. You can fine-tune the embedding models with your own data, um, and you can also use other, um, sometimes I call them tricks, on, on, on top of the embeddings, right?
- 10:24
So these are different type, type of ways to retrieve, um, uh, using additional information like graph, you know, iterative retrieval, so on and so forth. They're all based on embeddings, um, but, uh, you can, uh, use the embeddings in many different ways, uh, uh, as addit- an additional layer.
- 10:38
Um, so, um, I guess I'll use the next probably five minutes to discuss some of the, uh, uh, the, uh, my vision for how RAG will go in the future.
- 10:48
I do believe that RAG will be there forever because this is, as I sh- argued in the first slides, uh, the first set of slides, this is the kind of like very similar to how humans are, uh, using, uh, additional large amount of data.
- 11:00
You retrieve, you hierarchically select some subset, and then you use those, uh, to, uh, uh, to, to answer the questions, uh, or, or, or take some actions. And this is very efficient because you only use a small subset of the data.
- 11:13
Um, and, um, and as oppo-- uh, uh, um, regarding, uh, how, um, how RAG will evolve in from a technical point of view, uh, I'd like to draw some analogy, uh, um, uh, um, from, uh, how the, the AI generally is evolving.
- 11:29
So I think I was reflecting on when I was teaching at Stanford, you know, starting to teach at Stanford about seven years ago. Uh, I started to teach with Chris Ré on this machine learning course, and, uh, one of the slides literally have these seven steps on how do we build ML systems in enterprises.
- 11:44
Um, so this actually, this slide is still in the, the lecture notes. Uh, uh, we still teach them, but just with more kind of like, uh, uh, a- asterisks, uh, around it.
- 11:53
So you can see, like you, you need to go through, you know, many steps, you know, collecting data, you know, train test split, you know, define your loss functions, you know, build models and, and iterate and you repeat.
- 12:03
And then, uh, for the large language model world, it's kind of like this, right? You don't n- need to do any of this. You just take a large language model out of the box and just, uh, you know, you can deploy it in enterprise in most of the cases.
- 12:14
Of course, it's not gonna be perfect, but this is already better than in the old days you do all of these steps in the enterprise using all the enterprise data.
- 12:21
Just out of the box, you are doing already very, very well. Of course, you still have this issue that you cannot-- the out-of-the-box large language model cannot, um, uh, access propriety information, then you can use RAG, uh, for it.
- 12:34
So, but I think the point here is that before, uh, all of these steps have to be done by the kind of like the users or the enterprise or the customers in some sense.
- 12:44
Um, uh, and now, uh, you largely speaking just can take off-the-shelf components and connect them and build your, uh, AI applications very fast without going through these training steps.
- 12:54
The trainings still have to be done, but all of these steps still are done, uh, um, but they are done by OpenAI, Anthropic, or Voyage, MongoDB, uh, the providers of the models, but not the, uh, uh, the, the users, um, uh, the end users.
- 13:08
So, um, and I think for RAG, I, I would say probably the same kind of evolution should happen. Um, um, um, um, so right now what happens is that, um, we have the several different layers, where we have the computing infrastructure layer above the GPUs, you know, uh, or some of the KNs on the CPUs, and there's
- 13:25
also a model layer where, uh, you have the embedding models, the rerankers, the large language models. And then on top of all of this, um, people use a lot of, like I call it, tricks to make RAG, uh, uh, accuracy much better, right?
- 13:37
You can, uh, uh, use all kind of parsing strategies. You can use all kind of trunking strategies, you know, uh, uh, that the, the, the, um, you can do some recursive search, you can do some contextual trunks, graph RAGs, so on and so forth.
- 13:51
Right? That's what happens right now, and it's kind of necessary. These tricks are somewhat necessary because the embeddings and rerankers and large language models, none of them are perfect yet, right?
- 14:00
So, um, and but I do believe that in the future, I think this model layer will grow, uh, uh, and the tricks will be s- smaller. So it's gonna be fewer and fewer tricks, and the models can capture many of the, uh, the, the performance gain, um, by the tricks.
- 14:15
I think we have seen this in the large language model space as well, right? So like, um, um, two years ago, I think you need to do a lot of things on top of the GPT-3 to make your application work, and now even off the, uh, uh, out of the box, you can get the same performance as
- 14:29
before with all of the tricks. Um, of, of course, you still probably need some kind of tricks because some information are not, um, um, um, uh, uh, some information, um, the embedding models and rerankers don't have just, right?
- 14:42
So the, the, the general purpose models or off-the-shelf models don't have certain information, and then you can incorporate those, uh, into your tricks. For example, one thing that, uh, uh, is that, you know, the definition of the similarity metrics, uh, uh, could be something that you should customize in your prompt.
- 14:59
And, um, uh, in this one, you know, um, I think, uh, there are several things that we are developing towards this vision, right? So one of them is, uh, multi-model embedding.
- 15:07
This is to dramatically simplify the workflow so that you don't have to do many things, right? So these days, the multi-model embedding pr- provided by Voyage can just take in screenshots, right?
- 15:16
Before you take a PDF, you have to do the data extraction to turn them into image and text, and then probably do some embeddings for the image and the embedding for the text separately.
- 15:25
Um, and, and, and parsing this PDF is actually complex, you know. And for videos, you have to do, turn them into transcript, um, um, and then use the text embedding, so on and so forth.
- 15:35
Right? Right now, uh, right now we have the multi-model embedding which just takes in screenshot. You can deal with PT, P- PDF, you know, PPT, uh, uh, PowerP- you know, PowerPoint, you know, any of the other kind of slide stack in the same way.
- 15:46
Just take a screenshot and then use the multi-model embedding. We don't have the, uh, we can even do the, uh, same thing for video. Uh, not necessarily the perfect way, but, like, you just take screenshots of the frames, you know, consecutive frames, and you give it to the, uh, multi-model embedding, and you can, uh, turn them into
- 16:01
vectors, and you can search over, uh, on those documents or, or videos or slide stack. So, um,
- 16:08
s- and these are some performance metrics that we have evaluated. You know, we have tried kind of like... Oh, by the way, another one application is tables, right? Now you can just take a screenshot of the tables.
- 16:17
You don't have to think too much about what is the header, what is the row, so on and so forth. And we have done evaluations on many of these document screenshot, you know, table figures and, and also text only, and you can see that it's, uh, improving across the board.
- 16:31
So, um, the, the final one I would like to mention, which is something that we're gonna launch, uh, soon, is that this context aware and auto-chunking embedding. So, uh, right now what happens is that when you have a long document, you do have to chunk the data.
- 16:46
Um, uh, one of the reason is that the co- the context, uh, length of the embeddings is limited, and if you have like 100K tokens, you do have to chunk it into three or four chunks.
- 16:55
You know, even though Voyage AI has the probably, I think it, we have the longest context window, it's still like 32K. So that's one reason to chunk. And another reason to chunk is that sometimes it's a long, the long document, even you don't chunk.
- 17:08
Suppose you can have a way to, uh, uh, put all of them in a context window. Still, when you retrieve, you're gonna retrieve on a document level, then you retrieve a very, very long document, and then you should give this long document to large language model.
- 17:20
It's gonna be very, very expensive, right? If you give 100K, uh, tokens to la- large language model every time you answer any question, if you do some cost analysis, you'll find that that query is very, very expensive.
- 17:30
So that's why you do have to work on a smaller unit, so that you can cut the cost and, and be also more focused, right? So sometimes you give a long document to large language model, it misses some of, some of the context in the middle, and you have to use the retrieval to focus on a paragraph,
- 17:44
a page, so on and so forth. So, so that's what happens right now with the chunking and, uh, but all of these are done by the users. Uh, and w- our vision is that we're gonna do this for you, and also we're gonna, uh, get all the meta information about from other chunks.
- 17:58
So basically, um, uh, in a nutshell, the, the, the, the interface will be that you give us a long document, and we're gonna chunk it for you, and then also we return the chunks and also the vectors for each of the chunk.
- 18:10
And each of these vector is not only representing that chunk but also representing some of the global, uh, uh, meta information from other chunks. So it has all the details, uh, uh, of the corresponding chunk and also has some kind of like a cross-grain information from other chunks so you can get the best of the both worlds.
- 18:27
Um, and, uh, that's what I'm gonna launch, you know, soon. And another one is that we're gonna have some fine-tune API at some point, uh, to make you, uh, uh, uh, so that you can fine-tune, uh, with your own data.
- 18:38
Um, I guess, uh, it's exactly time. Thanks very much. [upbeat music]