GraphRAG methods to create optimized LLM context windows for Retrieval — Jonathan Larson, Microsoft

What would an LLM need to explain a game: its source files, or an understanding of how those files work together? Jonathan Larson, who introduces himself as leading Microsoft Research’s GraphRAG team, approaches that question through structured LLM memory. The GraphRAG paper and public repository had already inspired derivative work, including Neo4j’s card-game explanation. Here, the progression moves from repository understanding to agents that act on it, then to evaluating retrieval quality with BenchmarkQED and LazyGraphRAG.

The first example is a terminal game: jump over an obstacle and earn points; collide with one and lose points. Larson describes it as roughly 200 lines across seven files, with code the LLM has never seen. That makes it small enough for a person to understand completely, yet the team found that placing all its code directly into a context window still produced poor understanding. Context capacity alone did not solve the problem.

The team asks an ordinary RAG system, “Describe what this application is and how it works.” Its answer identifies a game initialized through a main function, a configuration file, and separate classes and functions for the screen and logic. Those details describe code organization, but they leave the reader unable to picture the game.

With the same repository and question, GraphRAG for Code gives a more behavioral description: a curses-based terminal game, a player that jumps vertically, obstacles moving horizontally, a static background, and the Spacebar as the jump control.

This is a global query: answering it requires connecting behavior across the repository. A local query can focus on a narrower piece of code; explaining the application requires assembling those pieces into a coherent account. The demonstration makes the distinction visible in the answer itself.

Once the system can explain the game, the next test is whether it can preserve its behavior in another language. The team takes the same Python repository—four main source files within seven files total—and asks for a Rust translation. The demonstration first runs the Python game, establishing the behavior the translation must retain.

The baseline puts all the source into an LLM. Larson says the team tried a variety of prompts aimed at translating the application as a whole. The model produces Rust, but attempting to compile that output reveals problems throughout it. Producing plausible source text has not produced a working application.

The GraphRAG for Code path instead uses the repository’s graph structures during translation and generates a collection of Rust files. Larson then shows the translated game running natively in Rust. The useful comparison is between two end states: generated code that fails to compile, and a translated application that runs. The talk does not identify the model, enumerate trials, or establish a translation success rate beyond this demonstration.

The next repository is Doom, which Larson describes as approximately 100,000 lines across 231 files and about 30 years old. Its public history creates a different challenge from the unseen terminal game. Larson initially expects models to know the code from training; his team’s tests instead suggest broad familiarity without the implementation detail needed to make meaningful modifications. Recognizing a repository is not enough to change it reliably.

The team starts by generating documentation. At the global level, that means explaining entire modules rather than summarizing files independently. A description of Doom’s sound system, for example, must connect responsibilities spread across roughly 20–30 files. Local queries then let the reader drill into individual files beneath that module-level account. Global understanding and local detail serve different parts of the same task: first establish how a subsystem works, then inspect the code that implements it.

Documentation, Q&A and translation lead to the next experiment: feature development. Original Doom does not let the player jump, so the team chooses to add that capability. It requires coordinated changes across files—the kind of task where an agent can make a convincing edit in one place while breaking another. Repeatedly repairing those breakages can leave the repository in a cycle of incompatible changes because the system lacks a sufficient understanding of how its parts fit together.

After showing the original game, the team connects the GitHub Copilot coding agent, introduced at Build, to GraphRAG for Code. The workflow is short:

1. Create an issue against the Doom repository asking to add player jump capability.
2. Add a couple of sentences describing the request and assign the issue to the coding agent.
3. Let the agent consult GraphRAG for Code to develop a holistic, top-down plan before making the changes.

GraphRAG supplies repository context to the agent; the agent turns that context into a plan and edits.

Larson reports that the agent changed many files and the feature worked out of the box, whereas the other agents his team tried failed on this task. The resulting demonstration shows jumping in Doom. This is the concrete payoff of pairing structured memory with an agent: repository understanding becomes a resource for coordinating action. The comparison remains a reported demonstration, without a controlled agent benchmark or detailed trial accounting.

The coding demonstrations raise a measurement problem: how should a system be evaluated when some questions target a specific fact and others require understanding the whole dataset? Larson introduces BenchmarkQED, announced as newly open source during the talk, to address both local and global answer quality.

BenchmarkQED separates three responsibilities:

This separates the questions a benchmark asks from how it judges responses and how it prepares the underlying data.

AutoQ organizes questions along two axes: local versus global scope, and data-driven versus persona- or activity-driven intent. Data-driven questions start from information in the dataset. Persona- and activity-driven questions adopt the perspective of someone working in the domain and ask what that person needs to know.

The examples come from an AP News dataset focused on medical events:

• Data-local: “Why are junior doctors in South Korea striking in February 2024?” The country, profession, event and date give retrieval specific targets. Larson expects ordinary RAG to perform well here.
• Activity-global: “What are the main public health initiatives mentioned that target underserved communities?” This asks for a synthesis of initiatives across the dataset. A relevant passage may help, but retrieving one passage does not establish the main patterns across the corpus.

AutoQ generates questions across both dimensions so that evaluation does not quietly reduce overall quality to performance on precise lookup questions.

Once AutoQ supplies the questions, AutoE judges the answers. Its composite combines comprehensiveness, diversity, empowerment and relevance. The first three carry forward the criteria used in the original GraphRAG work; relevance is the added dimension. These are judgments of answer quality, not a factual-accuracy score.

Larson uses this framework to compare LazyGraphRAG with vector RAG using 8K, 120K and one-million-token context windows. A bar above 50% means the judge favors LazyGraphRAG over the corresponding baseline.

The companion evaluation article describes 1,397 health articles, 200 queries, six trials per query per metric, and a GPT-4.1 judge, with ties counting as half a win. Its GPT-4o comparisons precede a separate GPT-4.1 million-token experiment; these should not be read as one fixed-model context-length sweep.

The tie rule can be expressed directly in Python when aggregating pairwise verdicts:

from collections.abc import Sequence
from typing import Literal

Verdict = Literal["win", "tie", "loss"]

def preference_rate(verdicts: Sequence[Verdict]) -> float:
    if not verdicts:
        raise ValueError("At least one verdict is required")
    scores = {"win": 1.0, "tie": 0.5, "loss": 0.0}
    return sum(scores[verdict] for verdict in verdicts) / len(verdicts)

This computes preference for a chosen system from supplied verdicts; the four judging criteria determine what those verdicts reward.

For data-local questions, Larson reports the following LazyGraphRAG win rates under the composite judging criteria:

The local results surprise him because these are the questions expected to suit ordinary retrieval. He reports strong performance across the question categories, with a somewhat larger advantage on global questions.

Increasing the vector-RAG context window did not materially close the judged-quality gap in these tests. Larson had expected longer contexts to help with global questions, but the presented comparison still favored LazyGraphRAG. He also reports that, in a separate off-slide test, LazyGraphRAG cost one-tenth as much as the one-million-token-context comparison. The talk does not specify that cost test’s model configuration, pricing or whether indexing costs were included. The result points toward the value of selecting and organizing context, rather than assuming that a larger window alone supplies the needed understanding.

LazyGraphRAG had first been discussed in a November blog post the previous year. At the time of this talk, Larson describes Azure Local integration as forthcoming following a Build announcement, and integration into Microsoft Discovery as underway. Discovery extends the application from code to scientific work: hypothesis leads to experiment, experiment produces learning, and learning becomes knowledge for the next cycle.

The closing demonstration shows a copilot reasoning over scientific knowledge, with GraphRAG and LazyGraphRAG powering the underlying graphs and answers. The same relationship seen in the Doom example reappears in another domain: structured memory gives an agent connected knowledge to reason with, and the agent uses that knowledge to carry work forward. Larson ends on that pairing—memory with structure, amplified by agents—and invites further questions outside the session.