RAG in 2025: State of the Art and the Road Forward

If an out-of-the-box language model already knew MongoDB’s internal information, something would have gone wrong: that information would have leaked. An enterprise assistant needs a deliberate path to proprietary knowledge. Calling the system an agent does not remove this requirement; an agent using an LLM inherits the same information gap.

Tengyu Ma, Voyage AI’s co-founder and former CEO, introduces the topic following the company’s acquisition by MongoDB. Retrieval is the company’s focus: supplying the information a model is missing. The architectural choice is how to supply that missing information: put documents in the model’s context, encode knowledge in its parameters through fine-tuning, or retrieve relevant documents when a question arrives.

The three approaches place knowledge in different parts of the answering process.

For long context, Ma imagines supplying a million or even a billion tokens; these are illustrative collection sizes, not claims about a particular model’s capacity. Fine-tuning instead updates the model in advance, then answers without revisiting the documents. Retrieval-augmented generation keeps the documents available externally: the query drives search, and the resulting passages become the context for generation.

Ma’s preference for RAG comes from a library analogy. Long context resembles scanning the entire library for every question. Fine-tuning resembles reading the library beforehand and trying to internalize its contents. He doubts the long-run cost efficiency of the first approach and the suitability of the second for ingesting enterprise knowledge.

Memorization creates more problems than capacity alone. Which books should the model memorize? If a fact must be removed, how do you make it forget that fact cleanly? If different users may access different books, how do you enforce those boundaries once their contents have been mixed into shared parameters? These are knowledge-selection, deletion, and governance problems.

RAG resembles how people actually use a library: find the relevant book or chapter, read it, and answer the question. Ma describes this as a modular, reliable, fast, and inexpensive arrangement. Its underlying advantage is hierarchical information access: keep the large collection outside the model and bring a small relevant portion into the answering process when needed.

The basic retrieval path has two principal components: an embedding model and a vector database. The embedding model maps documents and queries into vectors that represent their content or meaning. The database stores document vectors and uses k-nearest-neighbor search to find those closest to the query vector. Their associated documents then pass to the LLM for answer generation. A more advanced pipeline adds a reranker, discussed shortly.

Ma places the launch of OpenAI V3 embeddings roughly a year and a half before the talk and describes subsequent progress from Voyage and Cohere. The improvement is not only higher retrieval quality: it is a better quality–cost trade-off. At a fixed parameter count, a model can retrieve more accurately; at a fixed quality level, a smaller model can be cheaper. He attributes these gains to work throughout the training stack, including data curation, data selection, architecture, loss functions, and evaluation.

Ma reports roughly 80% average retrieval performance across about 100 datasets. The displayed chart labels retrieval quality as NDCG@10, a ranking metric; the figure should not be read as the percentage of enterprise questions answered correctly. He describes the remaining roughly 20% as improvement headroom.

The average conceals a wide distribution: Ma estimates that roughly half the datasets reach 90–95%, while others sit around 60%, 20%, or 30%. Common retrieval tasks can therefore already work well even while harder tasks leave substantial room for improvement. An aggregate score alone cannot tell you which situation your application resembles.

Embedding storage has two adjustable dimensions: how many coordinates each vector contains and how precisely each coordinate is stored. Matryoshka learning trains a representation so that a prefix of its coordinates remains a useful embedding. You can retain the first portion of the vector rather than requiring its full length for every deployment.

In Ma’s example, retaining the first 256 coordinates of a 2,048-dimensional embedding incurs an estimated 1–2% retrieval-quality loss. The prefix works because the model was trained to make it useful; arbitrary truncation of an arbitrary embedding is not the mechanism. Quantization during training addresses the other dimension, preserving useful retrieval behavior when coordinates use lower precision.

Ma reports at least 10× vector-storage savings with little quality loss, and roughly 5–10% loss at 100× savings. These are approximate retrieval trade-offs, not reductions in the total RAG bill; his spoken percentages do not distinguish relative losses from percentage-point changes. He also claims that Voyage remains ahead of OpenAI at roughly 100× storage reduction because the models occupy different quality–storage Pareto frontiers. A stronger starting representation can remain competitive even after substantial compression. Domain specialization, introduced next, can improve that trade-off further.

Replacing the embedding model is one of the simplest ways to improve retrieval, but it is not the only one. Hybrid search brings together candidates from lexical search and other retrieval methods, then uses a reranker to order them. Ma notes that Voyage also supplies a reranker: candidate discovery and final relevance ordering need not be the same operation.

The next set of changes improves what enters retrieval:

• Query expansion: use an LLM to elaborate a short query into a longer, more informative search input.
• Query decomposition: split a complex question into smaller subqueries, each capable of finding a different subset of documents.
• Document enrichment: attach information that makes an isolated passage easier to interpret and retrieve.

The document side matters because chunking can discard context. A passage may lose its title, section header, category, author, or date even though those fields explain what the passage means. Restoring that global information to each chunk gives retrieval access to it again.

For example, an isolated chunk saying “Requests remain available for 30 days” becomes less ambiguous when its document title and section accompany it. A small enrichment function can preserve the original text while preparing a separate search representation:

from dataclasses import dataclass

@dataclass(frozen=True)
class Chunk:
    id: str
    text: str
    title: str
    section: str

    def search_text(self) -> str:
        return (
            f"Document: {self.title}\n"
            f"Section: {self.section}\n\n"
            f"{self.text}"
        )

chunk = Chunk(
    id="export-policy-1",
    text="Requests remain available for 30 days.",
    title="Data Export Policy",
    section="Download availability",
)

text_to_embed = chunk.search_text()
original_text = chunk.text

Here, the added fields are existing metadata. Ma also describes generating additional chunk context with an LLM and points to Anthropic’s Contextual Retrieval. That approach makes otherwise isolated chunks more informative before retrieval; he reports good results without giving a numeric improvement in the talk.

A general-purpose embedding model does not have to be the endpoint. Ma describes Voyage and MongoDB’s work on code-specific embeddings as improving both retrieval performance and resilience to compression. In the code-domain example, Ma estimates about 5% quality loss at roughly 100× compression, compared with about 10–15% in the earlier comparison. The configuration matters: these estimates describe the examples he presents, not a fixed compression penalty shared by every corpus.

You can also fine-tune the embedding model on your own data. This is a different use of fine-tuning from the earlier proposal to memorize enterprise documents inside a generative model: here, the adaptation changes the representation used to find documents. Graph information and iterative retrieval add further layers over those embeddings, changing how the system explores and selects evidence while retaining the underlying vector representations.

Ma expects retrieval to persist because its basic operation remains useful: hierarchically select a small subset of a large collection, then use it to answer a question or take an action. His forecast for how the implementation changes comes from an earlier transition in enterprise machine learning.

When he began teaching a Stanford machine-learning course with Chris Ré roughly seven years earlier, one slide described a seven-step enterprise workflow. It included collecting data, creating train/test splits, defining loss functions, building models, and iterating. The course still teaches that workflow, but with more qualifications.

Off-the-shelf LLMs changed the starting point. Ma argues that, in many cases, an imperfect pretrained model already performs better than what an enterprise could previously achieve through its own bespoke training process. Private information remains missing, so RAG still has a role. But application teams can begin by connecting capable components rather than first carrying out the entire training workflow themselves.

The training work has not disappeared; responsibility for it has shifted. Providers such as OpenAI, Anthropic, Voyage, and MongoDB do that work, while application builders consume its results. Ma expects a similar transfer of responsibility within RAG.

The current RAG stack separates into several layers:

The top layer compensates for limitations below it. If an embedding loses context, the application enriches chunks. If retrieval misses useful evidence, the application searches recursively or follows graph relationships. These techniques are necessary today because the models are imperfect.

Ma predicts that stronger models will absorb more of the performance gains currently supplied by application code. He compares this with applications built around GPT-3 roughly two years earlier: work that once required substantial machinery around the model can increasingly be handled by a stronger model out of the box.

That does not eliminate customization. A general-purpose model cannot infer information it has never been given. In particular, an application may have its own definition of similarity or relevance, which Ma suggests specifying in a prompt. The distinction is between compensating for a model’s general weaknesses and supplying task-specific information the model could not otherwise possess.

Multimodal embeddings provide a concrete example of moving work into the model. Ma describes Voyage’s ability to embed screenshots directly. A traditional PDF pipeline first extracts text and images, then may embed each separately. Parsing the PDF becomes a substantial part of the system. For video, a text-based pipeline commonly begins by producing a transcript and embedding it.

A screenshot-based path lets PDFs, PowerPoint presentations, and other slide decks use the same basic operation: render a page or slide, embed its screenshot, and search the resulting vectors. Ma extends the idea to consecutive video-frame screenshots, while explicitly calling it an imperfect approach. Searching visual frames does not by itself provide retrieval over the audio or understanding of the video’s temporal structure.

Tables illustrate why this matters. Instead of first deciding how to serialize headers, rows, and cells into text, the system can embed a screenshot of the table. The model receives the visual arrangement directly. Ma reports improvements across evaluations involving document screenshots, tables, figures, and text-only inputs, without giving spoken numeric results. The architectural benefit is a shared ingestion path for sources whose useful information is not confined to plain text.

The final example is a proposed context-aware, automatic-chunking embedding service. Ma first explains why chunking remains necessary. One reason is the embedding model’s context limit: with the stated 32K-token window, a 100K-token document needs at least four chunks. His suggestion that Voyage has the longest window is tentative; the concrete constraint is that the document exceeds the model’s input capacity.

A larger embedding window would solve only that first problem. If retrieval returns the entire document, the generation model must still process a very large input for each question. Ma uses a 100K-token document to illustrate the expense. Smaller retrieval units also focus the answer on a relevant paragraph or page instead of relying on the model to notice information buried in the middle of a long context. The unit you can embed need not be the unit you should retrieve.

At the time of the talk, users had to perform that chunking themselves. Ma’s proposed interface transfers the work to the provider:

1. Submit a long document.
2. Receive the document’s chunks and a vector for each chunk.
3. Use those vectors to retrieve focused pieces of the document.

Each returned vector would represent the detailed content of its own chunk together with coarse information from the rest of the document. This preserves local retrieval granularity without treating every chunk as an isolated fragment. The final diagram shows the intended output: multiple chunk–vector pairs, with both local detail and document-level information represented in a vector.

This is a roadmap announcement in the recording, not a demonstration of an already available automatic-chunking API. Ma closes with a further plan for a fine-tuning API that would let customers adapt embeddings using their own data, without giving a firm launch date. Both proposals follow the same direction: let application builders supply their documents and domain knowledge while the provider takes responsibility for more of the machinery that makes them retrievable.