AI Engineer Summit 2023
Retrieval Augmented Generation in the Wild
Read the talk
Retrieval Augmented Generation in the Wild
Vector search starts the RAG loop, but useful AI memory also needs feedback, updates, task-aware chunking, and a way to recognize when retrieved results are irrelevant.
From a talk by Anton Troynikov
Before you start: Familiarity with embeddings, nearest-neighbor search, and an LLM’s context window will help you follow the retrieval design choices.
Start with the retrieval loop
Take a corpus of documents, embed it, and store the vectors alongside their source documents. When a query arrives, embed that too. Find its nearest neighbors, retrieve the associated documents, and place those documents and the query into an LLM’s context window. The model then generates a response. This is the familiar starting point for retrieval augmented generation, or RAG.
Anton Troynikov, Chroma’s co-founder, calls this open-loop retrieval. Information flows from the corpus through retrieval into generation, but the basic diagram contains no path for learning from the result. Vector search supplies candidates; building more capable applications requires a memory system that can also change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Feedback turns retrieval into mutable memory
The first addition is human feedback. An embedding model trained for general use does not automatically encode what matters for a particular dataset, task, generation model, or user. Relevance judgments provide a signal for adapting the data and embedding model to that application. The memory layer therefore needs somewhere for those judgments to go and a way for them to influence future retrieval.
Agents add another update path: they need to write to memory themselves. Instead of repeatedly searching a fixed corpus, an agent can retain information produced during its work. That makes the dataset continuously mutable. Troynikov’s objection to a search-index-only architecture is that nearest-neighbor lookup alone does not provide the feedback and update behavior these applications need.
An agent with a world model takes this further. It must store its interactions with the environment and revise its working data based on what happened. Human feedback, agent-written memory, and observations of the world then become parts of the same system. At the time of the talk, Troynikov describes this combination as research-grade, while pointing to early applications already using parts of it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remembering skills in Minecraft
Voyager, the Minecraft agent shown in the talk, illustrates why memory can be more than a collection of reference documents. The agent learns skills in its environment and recalls them when it encounters a suitable context again. Retrieval lets a previously acquired capability become available for later work instead of requiring the agent to rediscover it.
Troynikov also characterizes some complex skills as learned through human demonstration and retained in Chroma. That characterization differs from the original Voyager project’s account, which describes autonomous skill acquisition without human intervention. Chroma’s role in retrieval is supported by the public Voyager skill manager, though that source is not a snapshot of the conference demonstration. The useful architectural point is the connection between acquiring a skill, retaining it, and retrieving it in a later context.
The basic RAG loop remains useful for many applications, but this kind of reusable memory asks more of the retrieval layer. Search and recommendation systems provide a long history of information-retrieval techniques. Production AI applications introduce a different consumer of the results: a model that will use the retrieved material as part of its computation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve the relevant information—and exclude distractors
Useful context requires both inclusion and exclusion: retrieve the information the task needs, while keeping irrelevant material out. Troynikov emphasizes that distractors can sharply degrade an AI application’s performance. The talk provides no measured degradation rate, so this is a warning about context quality rather than a universal numerical result.
That requirement exposes three connected decisions:
| Decision | What it determines |
|---|---|
| Embedding model | Which documents appear close to the query |
| Chunking | Which units of information are available to retrieve |
| Relevance assessment | Whether a result helps this task and user |
A provider’s claim that an embedding model is strong on code, English, or multilingual data does not settle its suitability for your corpus. Chunking changes the candidates themselves, and even a close candidate still needs to be relevant to the intended use.
There is no universal recipe for these choices in the talk. The opportunity is to use production data to investigate them: real queries, real documents, and examples of what users actually find useful can expose requirements that academic benchmarks miss.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose embeddings with application evidence
Troynikov describes existing academic embedding benchmarks as largely saturated and insufficiently representative of AI retrieval workloads. He does not identify a benchmark or supply a saturation measurement. His practical response is to bring the evaluation machinery to the application’s own data.
The proposed workflow is straightforward:
- Apply existing open-source benchmark tooling to your datasets and use cases.
- Collect human relevance judgments through a feedback endpoint.
- Build evaluation datasets from the queries and outcomes observed in production.
- Compare candidate embedding models using evaluation tools against those application-specific examples.
Chroma’s relevance-feedback endpoint and support for evaluation vendors are presented here as planned work, not as demonstrated features. The important dependency is that model comparison needs a definition of usefulness grounded in the application.
There is also a more speculative possibility: the initial embedding-model choice might become less binding if representations can be mapped between models. Troynikov’s claim is conditional on models having the same training objective and roughly the same training data. Under those conditions, he says, learned representations tend to be similar up to an affine transformation. Written explicitly, the proposed mapping has the form:
Here A is a learned matrix and b is an offset. The suggestion is to determine such a mapping from your own dataset, potentially projecting one model’s embeddings into another model’s space. This is a research possibility, not a guarantee that arbitrary embedding models are interchangeable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Chunk boundaries determine what the model can see
Chunking begins with a capacity constraint: retrieved material must fit within the LLM’s bounded context. But fitting is only one requirement. A chunk should preserve semantic content, and specifically the semantic content relevant to the task. Document structure helps because text written for people already contains cues to meaning: its natural boundaries can guide where to split.
NLTK, LangChain, and LlamaIndex are named as existing sources of chunking tools. Troynikov then turns to an experimental approach using a lightweight language model: slide a window over the preceding tokens and ask how probable the next actual token in the document is. A token that the model assigns very low probability may indicate that the text has crossed a semantic boundary.
A compact way to express that signal is token surprise. Given next-token probabilities supplied by a model, this Python function returns candidate boundary positions:
python
from math import log
def candidate_boundaries(
next_token_probabilities: list[float],
minimum_surprise: float,
) -> list[int]:
boundaries = []
for position, probability in enumerate(next_token_probabilities):
if not 0 < probability <= 1:
raise ValueError("Probabilities must be in (0, 1].")
if -log(probability) >= minimum_surprise:
boundaries.append(position)
return boundaries
Each probability refers to the actual token at that position, conditioned on its preceding window. Lower probability produces higher surprise. The threshold produces candidates; it does not establish that every surprising token is a useful semantic split. Because a model supplies the boundary signal, Troynikov proposes fine-tuning it so that its boundaries better match the application.
Other approaches complement or replace that signal:
- Information hierarchies: Organize content at multiple levels; Troynikov points to LlamaIndex’s support for this approach.
- Multiple retrieval signals: Bring multiple data sources and signals into reranking rather than relying on one similarity value.
- Embedding continuity: Embed successive sliding windows of a document and look for discontinuities in the resulting sequence of vectors.
The last approach asks where the representation changes abruptly; the token-prediction approach asks where the text becomes unexpectedly hard to predict. Both boundary-detection ideas are described as experiments, without an effectiveness result in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five bird pages cannot answer every fish question
Imagine a corpus containing every English-language Wikipedia page about birds. For each query, the application retrieves the five nearest neighbors and puts them into the model’s context. Then a user asks about fish. In Troynikov’s hypothetical, the system still returns five neighbors, but none is relevant to the query. Being nearest is a relative property; being useful to the task is a separate judgment.
How can the application make that judgment?
- Human feedback: Gather explicit judgments about whether retrieved material was relevant.
- Auxiliary reranking: Use another model and additional signals, including what the user was looking at and what they previously found useful.
- Scoped retrieval: When useful constraints are known beforehand, narrow the search through keywords or metadata filters. Troynikov describes these as supported by Chroma at the time; that statement does not establish a ranked lexical or hybrid-search implementation.
These approaches introduce evidence beyond the fact that a document occupies a position in the nearest-neighbor list.
The more ambitious proposal is to calculate conditional relevance directly. Given the available dataset and what the user is trying to accomplish, a relevance model would distinguish usefulness per user, per task, per generation model, and per individual task instance. That requires understanding both the query’s semantics and the content available in the corpus. Troynikov identifies lightweight open-source language models operating at the data layer as a promising experimental direction, not a demonstrated relevance guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From vector storage to an AI data layer
The closing roadmap moves from retrieval research to infrastructure. Chroma’s next engineering step was a horizontally scalable cluster version, extending the existing single-node system across multiple nodes. Troynikov announced a December target for a Chroma Cloud database-as-a-service technical preview and a January target for hybrid deployments in enterprise clusters. These are targets from the 2023 recording, not statements of current availability.
The planned data layer also extended beyond text. Troynikov anticipated the arrival of the GPT Vision API, possibly at OpenAI’s developer day, and image and voice capabilities in Gemini. Those contemporaneous expectations motivated multimodal retrieval support: if a model can consume more kinds of data, its retrieval system must be able to supply them.
Multimodal data expands the definition of a useful result. Semantic relevance remains important, but qualities such as aesthetic suitability can also matter. A retrieval system may therefore need to evaluate several dimensions of usefulness rather than treating text similarity as the entire problem. Model selection appears as another item of work in progress, without implementation details.
The intended developer experience is analogous to using Postgres in a web application: the database handles the data layer so that developers can concentrate on application logic. Chroma’s ambition is to provide that boundary for AI applications, absorbing the work of storing, updating, and selecting useful information while leaving developers responsible for making the application behave correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original project explanation of Minecraft exploration, reusable executable skills, and retrieval from a growing skill library.
Implementation of persistent skill-description embeddings and retrieval using Chroma.
Further reading
A 2021 study of conditions under which learned representations are identifiable up to a linear transformation.
Arithmetic-reasoning experiments measuring the effects of irrelevant context and ways to mitigate them.
Anton’s April 2023 interview on Chroma's application-memory goals and developer experience.
An implementation of topical text segmentation based on changes in lexical co-occurrence.
Practical examples of splitting documents with ordered separators, size limits, and overlap.
Updates since the talk
Current examples for nearest-neighbor queries, result counts, metadata filters, and document-content filtering.
Read the complete timestamped transcript
- 0:00
[on hold music] Hi, everybody.
- 0:15
As they said as I walked up, I'm Anton. I'm the co-founder of Chroma. I'm here to talk to you about retrieval-augmented generation in the wild, um, and what it is that Chroma is building for beyond just vector search.
- 0:27
So by now, you've all seen versions of this probably a half dozen times throughout this conference. This is the basic retrieval loop that one would use in a RAG application.
- 0:37
You have some corpus of documents. You embed them in your favorite vector store, which is Chroma. You [laughing]
- 0:45
I mean, check the lanyards, man. Um, you embed your, you embed your corpus of documents. You have an embedding model for your queries. You, um, find the nearest neighbor vectors for those embeddings, and you return the associated documents which, along with the query, you then put into the LLM's context window and return some result.
- 1:02
Now, this is the basic RAG loop, but I think of this as more like the open loop retrieval-augmented generation application. And my purpose in showing you all this is to show you that you need a lot more than simple vector search to build some of the more powerful, more promising applications that take RAG in the future.
- 1:20
So let's get into what some of those might be. The first piece to this, of course, is incorporating human feedback into this loop. Previously, you, um-- without human feedback, it isn't possible to adapt the data, the embeddings model itself to the specific task, to the model, and to the user.
- 1:36
Human feedback is required to actually pr-return better results, um, for particular queries on your specific data on the specific tasks that you wanna perform. Generally, embedding models are trained in a general context, and you actually want to update them for your specific task.
- 1:49
So basically, the memory that you're using for your RAG application needs to be able to support this sort of human feedback. Now, the other piece that we've seen, and these, these are currently in the early stages, uh, but they're emerging as something like a capable machine, and I think that one of the ways to make agents actually
- 2:06
capable is a better RAG system, a better memory for AI. And that means that your retrieval system, your memory, needs to support, uh, self-updates from the agent itself out of the box.
- 2:17
All in all, what this means is you have a constantly dynamically updating data set. Something that's built as a search index out of the box is not gonna be able to support these types of capabilities.
- 2:27
Next, of course, we're talking about agents with world models. So in other words, the agent needs to be able to store its interaction with the world and update the data that it's working with based on that interaction.
- 2:37
And finally, you need to be able to tie all of these together. Now, this sounds like a very complex system that's, uh, frontier research, and it is currently research-grade, but we're seeing some of the first applications of this in the wild already today.
- 2:50
This is an animation from, uh, I'm sure some of you are familiar with this paper. This is the famous Voyager paper out of NVIDIA, where they trained a agent to play Minecraft, to learn how to play it by learning skills in a particular environment and then recognizing when it's in the same context and recalling that skill.
- 3:07
Now, the other interesting piece to this is several of the more complex skills were learned through human demonstration and then retrained in the retrieval system, which of course was Chroma.
- 3:16
Um, my point in showing this to you is that the simple RAG loop might be the bread and butter of most of the applications being developed today, but the most powerful things that you'll be able to build with AI in the future require a much more, uh, a much more capable retrieval system than one that only supports
- 3:35
a search index. Now, of course, in retrieval itself, there are plenty of challenges. Information retrieval is, is kind of a classic task, and the setting in which it's been found previously has been in recommender systems and, uh, and in search systems.
- 3:49
Now that we're all using this in production for AI applications in completely different ways, there's a lot of open questions that haven't really been asked quite in the same way or with quite the same intensity.
- 4:00
A key piece of how retrieval needs to function for AI, and anyone who's built one of these is aware of this, is you need to be able to return all-- not just all relevant information, but also no irrelevant information.
- 4:10
It's common knowledge by now, and this is supported by empirical research, that distractors in the model context cause the performance of the entire AI-based application to fall off a cliff if those distractors are present.
- 4:21
So what does it mean to actually retrieve relevant info and no irrelevant info? You need to know which embedding model you need to be using at all in the first place, and we've all, we've all seen the claims from the different API and embedding model providers.
- 4:32
This one is best for code. This one is best for English language. This one is best for multilingual datasets. But the reality is the only way to find out which is best for your dataset is to have a, a effective way to figure that out.
- 4:43
The next question, of course, is how do I chunk up the data? Chunking, uh, chunking determines what results are available to the model at all, and it's obvious that, um, different types of chunking produce different relevancy in the return results.
- 4:57
And finally, how do we even determine whether a given retrieved result is actually relevant to the task or to the user? So let's dive into some of these in a little bit more depth.
- 5:06
So the bad news is, again, nobody really has the answers. Despite the fact that information retrieval is a long-studied problem, there isn't great solution to these problems today. But the good news is that these are im-important problems and increasingly important problems, and we see much more production data rather than sort of academic benchmarks, um, that we can
- 5:23
work from to solve some of these for the first time.
- 5:27
So first, the question of which embedding model should we be using? Of course, there are existing academic benchmarks, and for, for now, these appear to be mostly, uh, saturated.
- 5:35
The reason for that is these are synthetic benchmarks designed specifically for the information retrieval problem and don't necessarily reflect how retrieval systems are used in AI use cases. So what can you do about that?
- 5:46
You can take some of the open source tooling built to build these benchmarks in the first place and apply it to your datasets and your use cases. Um, you can use human feedback on relevance by adding a simple relevance feedback endpoint, and this is something that Chroma is building to support in the very near future.
- 6:02
You can construct your own datasets because you're viewing your data in production. You know what actually matters to you. And then you need the effect-- you need a way to effectively evaluate, um, the performance of particular embedding models.
- 6:14
Of course, there are great evaluation tools coming onto the market now from several vendors. Um, which of these is best, we don't know, but we intend to support all of these with Chroma.
- 6:22
Um, one interesting part about embedding models, and this is again, this is a piece of-- This is something that's been well known in the research community for a while, but has been empirically tested recently.
- 6:32
Embedding models with the same training objective with roughly the same data tend to learn very similar representations up to an affine linear transform, which suggests that it's possible to project one model's embedding space into another model's embedding space by using a simple linear transform.
- 6:46
So the, the choice of which embedding model you actually wanna use might not end up being so important if you're actually able to, um, to sort of apply and figure out those transform from your own dataset.
- 6:57
So the question is how to chunk. Um, s- of course, there's a few things to consider. Chunking, in part, exists because we have bounded context lengths for our LLMs, uh, so we wanna make sure that the retrieved results can actually fit in that context.
- 7:10
We wanna make sure that we retain the semantic content of, uh, of, um, of the data we're aiming to retrieve. And then we wanna make sure that we retrieve-- that we retain the relevant semantic content of that data rather than, um, rather than just semantic content in general.
- 7:28
We also wanna make sure that we're respecting the natural structure of the data because often, especially textual data, was generated for humans to read and understand in the first place, so this inherent structure of that data provides cues about where the semantic boundaries might be.
- 7:41
Of course, there are tools for chunking. There's NLTK, there's LangChain. Uh, LlamaIndex also supports many forms of chunking. Um, but there are experimental ideas here which we're particularly interested in trying.
- 7:52
Um, one interesting thought that we've had and we're experimenting with lightweight open source language models to achieve these is using the model prediction perplexity for the next actual token in the, in the document based on a sliding window of previous tokens.
- 8:05
Um, in other words, you can see when the model mispredicts or has a very low probability for the next actual piece of text as a determinator of where a semantic boundary in the text might be, and that might be natural for chunking.
- 8:17
And what that also means is because you have a model actually predict- predicting chunk boundaries, you can then fine-tune that model to make sure the chunk boundaries are relevant to your application.
- 8:26
So this is something that we're actively exploring. We can use information hierarchies. Again, tools like LlamaIndex support information hierarchies out of the box and, uh, multiple data sources and signals to re-ranking.
- 8:36
And we can also try to use embedding continuity. This is something that we're experimenting with as well, where essentially you take a sliding window, uh, across your documents, uh, embed that sliding window, and look for discontinuities in the resulting time series.
- 8:51
So this is, this is an important question, and I'll give you a demonstration about why retrieval results, uh, uh, being able to compute retrieval result relevance is actually very important in your application.
- 9:01
Imagine in your application you've gone and you've embedded every English language Wikipedia page about birds, and that's what's in your corpus. And in your traditional retrieval-augmented generation system, what you're doing for each query is just returning the five nearest neighbors and then stuffing them into the model's context window.
- 9:16
Now, one day, a user's query comes along, and that query is about fish and not birds. You're guaranteed to return some five nearest neighbors, but you're also guaranteed to not have a single relevant result among them.
- 9:28
How can you, as an application developer, make that determination? So there's a few possibilities here. The first, of course, is, um, human feedback around relevancy signal. The traditional approach in information retrieval is using an auxiliary re-ranking model.
- 9:42
In other words, you take other signals, um, in sort of the query chain. So what else was the user looking at at the time? What things has the user, uh, found to be useful in the past?
- 9:52
And use those as additional signal around the, uh, around the relevancy. And we can also, of course, do augmented retrieval, which Chroma does out of the box. We have keyword-based search, uh, and we have meta da-database filtering, so you can scope the search, uh, if you have those additional signals beforehand.
- 10:08
Now, I-- to me, the most interesting approach here is actually an algorithmic one. So what I mean by that is conditional on the dataset that you have available and conditional on what we know about the task that the user is trying to perform, it should be possible to generate a conditional relevancy signal per user, per task, per
- 10:26
model, and per instance of that task. But this requires a model which can understand the semantics of the query as well as the content of the dataset very well.
- 10:35
This is something that we're experimenting with, and this is another place where we think open source lightweight language models have actually a lot to offer, even at the data layer.
- 10:45
So to talk about, a little bit about what we're building, um, this is the advertising portion of my talk.
- 10:51
In core engineering, we're of course building a horizontally scalable cluster version. Single-node Chroma works great. Many of you have probably already tried it by now. It's time to actually make it work across multiple nodes.
- 10:59
Um, by December, we'll have a database-as-a-service technical preview up and ready so you guys can try Chroma Cloud. [lip smacks] In January, we'll have our hybrid deployments available if you wanna run Chroma in your enterprise cluster.
- 11:10
And along the way, we're building to support multimodal, um, data. We know that, um, GPT Vision's API is coming very soon, probably at OpenAI's developer day. Um, Gemini will also have image understanding and voice.
- 11:24
That means that you'll be able to use multimodal data in your retrieval applications for the first time. So we're no longer just talking about text. So these questions about relevancy in other types of data become even more important, right?
- 11:35
Because now you start having questions about relevancy, aesthetic quality, all of these other pieces, um, which you need to make these multimodal retrieval-augmented systems work. And finally, we're working on model selection.
- 11:47
Chroma-- Basically, Chroma wants to do everything in the data layer for you so that just like a modern DBMS, just like you use Postgres in a web application, everything in the data layer for you as an application developer should just work.
- 12:01
Your focus should be on the application logic and making your application actually run correctly, and that's what Chroma is building for in AI. And that's it. Thank you very much. [audience applauding] [upbeat music]