When all context matters: Extended Cache Augmented Generation (ECAG)

Talk transcript

1. 0:00

   [outro jingle] Hi, I'm Luis Romero-Sevilla, and I'm the VP of AI at Orbis Operations. I'm on a mission to solve knowledge representation when all context matters.

2. 0:19

   So let's start with a very specific example. Let's say you have a large number of documents, and all documents represent an event, and all documents in the collection are relevant to answer a set of questions that the user has.

3. 0:34

   Not only that, there's one more challenge. The document in the collection becomes obsolete very fast, and all documents get replaced with new information.

4. 0:45

   Let's start with the simplest approach. We could start with a simple RAG. For that, we just need a vector database and an embedding model.

5. 0:53

   An embedding model takes the documents and turns them into a learned numerical representation vector. Now we take those vectors, and we store them in a database optimized for performing operations with vectors.

6. 1:07

   Perfect. Now we can take all of our questions, turn them into vectors, and then look for vectors that are similar to the initial query.

7. 1:18

   Those vectors that are within the tri-- similarity threshold are retrieved, and we can pass them to the LLM to answer the question.

8. 1:27

   Inserting to a vector database, it's relatively fast. So whenever a collection becomes obsolete, we can just replace it with a new one.

9. 1:36

   We still have one problem with our very specific scenario.

10. 1:41

    All the documents in the collection are relevant for us to answer the question, so we can't just take all the documents in the collection and pass them to LLM, and that's just one of the many limitations with this approach.

11. 1:55

    Now let's get a bit more sophisticated. All documents are relevant to answer a global question. Therefore, there must be some connections and relationship between the details within a document in the collections.

12. 2:08

    For us to map out those relationship, we're going to need a knowledge graph, and one implementation we could try is GraphRAG. GraphRAG has many steps, but basically it uses an LLM to read through all the documents and extract key entities and relationship between them.

13. 2:27

    It constructs a network, a knowledge graph, where all those connections and details are tied together. Then when the question is asked, it navigates this graph to synthesize a complete answer drawn across the entire collection.

14. 2:39

    If your collection of documents isn't changing very often, GraphRAG is an excellent approach for finding those relationships within details to answer the user's question.

15. 2:51

    However, our very specific scenario states that our data is not only deeply interconnected, but also the data gets replaced very often. Recomputing a knowledge graph every time the data gets replaced is computationally very expensive, and it takes relatively long time.

16. 3:08

    Okay, what if we take an even simpler approach, and we continue to build on top of it? If we were to use something like GraphRAG, each document needs to pass through an LLM for the entity and relationship extraction anyway.

17. 3:21

    Why can't we just throw all the documents into context? This approach will look something like Cache Augmented Generation, CAG, where we use a model with a large context window, load the documents into the context, and cache the context by storing the model's KV matrix.

18. 3:38

    The problem here is that the context window is limited, and if you fill the context window too much, the quality of the answer gets degraded too.

19. 3:46

    The solution: what if we use more CAGs in parallel and distribute the documents across different context buckets?

20. 3:54

    Now each CAG can answer questions regarding its content, and now we just need something to ask the right questions to the right buckets. So for this, we can use a smarter model to interrogate each bucket and eventually synthesize an answer.

21. 4:11

    How do we distribute the documents? It sounds tempting to organize the documents by domains and tell the supervisor, "Hey, here are the different categories." But in practice, with very dense relationship between documents, the supervisor tends to ignore domains that at first glance seem irrelevant.

22. 4:30

    For this reason, all documents are distributed in no particular order. The only requirement is to balance the number of documents in a way that the least amount of documents are needed.

23. 4:40

    Then the supervisor model start exploring the buckets and progressively build its internal understanding. And if it finds something interesting, it can ask a specific bucket follow-up questions. Because all caches can be loaded in parallel,

24. 4:56

    the knowledge-building process is significantly faster than GraphRAG while providing more accurate answers than a simple RAG.

25. 5:06

    And you're probably thinking, "KV cache can be pretty expensive," and you're absolutely right. But there are ways to reduce that cost by optimizing how long each cache lives. And at the end, there are many retrieval strategies, and all of them have their trade-offs, whether it's compute, cost, speed.

26. 5:26

    Currently, there is no one solution fits all. So ECAG is our solution to our very specific problem. Thank you for watching and for questions or continuing this conversation, gonna leave my details here. [outro jingle]