When all context matters: Extended Cache Augmented Generation (ECAG)

Suppose you have a large collection of documents describing an event. Answering the user's questions requires information from every document. Then add a second constraint: the collection becomes obsolete quickly, and the whole set must be replaced. A system must preserve access to the entire collection without making each replacement prohibitively expensive. This is the knowledge-representation problem Luis Romero-Sevilla, who introduces himself as VP of AI at Orbis Operations, sets out to solve.

Start with ordinary retrieval-augmented generation, or RAG. An embedding model and a vector database provide a straightforward pipeline:

1. Convert each document into a learned numerical vector.
2. Store those vectors in a database designed for vector operations.
3. Embed the user's question into the same representation space.
4. Retrieve documents whose vectors meet a similarity threshold, then supply their content to the LLM to answer the question.

Romero-Sevilla describes vector insertion as relatively fast, making it practical to replace an obsolete collection with a new one. But the retrieval step creates a mismatch with this workload: it selects material by similarity when the answer needs information from the whole collection. A document can matter to the global explanation without looking sufficiently similar to the initial question. Simply passing everything to the LLM would bypass that selection problem, but it would also move the burden into the model's context window.

If all the documents contribute to a global question, relationships between their details become useful structure. GraphRAG builds that structure by having an LLM read the documents and extract entities and relationships into a knowledge graph. Answering can then draw on connections across the collection instead of relying only on similarity to the question.

Romero-Sevilla describes the answering process as navigating the graph to synthesize a complete response. More specifically, Microsoft's GraphRAG global search uses pregenerated community reports, produces partial answers, and aggregates their useful content into a final answer. The graph supports a representation of the collection that can serve global questions.

That preprocessing is attractive when documents remain useful for a while. Romero-Sevilla recommends GraphRAG for collections that change infrequently. In the opening scenario, however, the documents are both densely interconnected and frequently replaced. Repeatedly extracting entities and relationships and reconstructing the graph makes each refresh computationally expensive and relatively slow. The useful structure has a short life in which to repay its construction cost.

Graph construction already requires each document to pass through an LLM. What if that pass retained the document context directly, instead of extracting a separate graph? Cache Augmented Generation, or CAG, loads the documents into a model with a large context window and caches the computed key/value inference state for that document prefix. Subsequent questions can reuse this state. The KV cache contains intermediate inference computations, not changes to the model's learned weights.

The limit is the context window itself. The collection must fit, and answer quality can deteriorate as the amount of context grows, depending on the model, task, and context length. Caching avoids repeatedly computing the same prefix; it does not create unlimited capacity or guarantee that the model will use every detail effectively.

Extended Cache Augmented Generation, or ECAG, distributes the collection across multiple CAG instances. Each context bucket holds part of the collection and answers questions about its own contents. A more capable supervisor model asks questions of the buckets and synthesizes their responses into an answer. The collection no longer has to fit inside one document context.

Partitioning by domain sounds convenient: give each bucket a category and let the supervisor choose where to look. But Romero-Sevilla reports that, with dense relationships across documents, the supervisor tends to ignore categories that appear irrelevant at first glance. Those skipped categories may contain precisely the connections needed to answer the global question.

The proposed allocation therefore uses no particular document order and balances document counts across buckets. A simple TypeScript allocation helper illustrates that rule without introducing domain labels:

function distributeDocuments<T>(
  documents: readonly T[],
  bucketCount: number,
): T[][] {
  if (!Number.isInteger(bucketCount) || bucketCount < 1) {
    throw new RangeError("bucketCount must be a positive integer");
  }

  const buckets = Array.from(
    { length: bucketCount },
    (): T[] => [],
  );

  documents.forEach((document, index) => {
    buckets[index % bucketCount]!.push(document);
  });

  return buckets;
}

const buckets = distributeDocuments(
  ["event-a", "event-b", "event-c", "event-d"],
  2,
);
// [["event-a", "event-c"], ["event-b", "event-d"]]

Here, each document goes into one bucket, and the counts are balanced without inspecting subject matter. This handles allocation only: choosing a bucket count still requires ensuring that each bucket's document content fits its model's usable context. Equal document counts do not imply equal token counts.

The supervisor progressively explores the buckets and builds its understanding from their responses. When something interesting appears, it can return to a specific bucket with a follow-up question. This makes answering an iterative process: an initial response can reveal which detail to investigate next, rather than forcing all relevance decisions into the first query. All the document caches can be loaded in parallel.

For this workload, Romero-Sevilla claims significantly faster knowledge building than GraphRAG and more accurate answers than simple RAG. He supplies no latency measurements, accuracy metric, corpus size, model configuration, hardware, or concurrency conditions for that comparison. Parallel cache loading explains the proposed reduction in preparation time, while bucket exploration is intended to recover information that a similarity-based retrieval step might omit.

Keeping multiple KV caches available can be expensive. Romero-Sevilla identifies cache lifetime as a way to reduce that cost: optimize how long each cache remains alive. In a workload where documents quickly become obsolete, the useful life of their cached state matters alongside the speed of creating it. The talk leaves the cache provider, expiration policy, and pricing model unspecified.

ECAG is a response to a particular combination of requirements: whole-collection reasoning, dense relationships, and frequent replacement. Its supervisor and cached buckets exchange graph-building work for cache retention and repeated model interrogation. Compute, cost, and speed remain trade-offs; which architecture makes sense depends on how long the knowledge stays useful and what the questions require.