Benchmarking semantic code retrieval on Claude Code

Should a coding agent discover a repository by repeatedly searching and reading files, or should it start with a searchable index of the code’s meaning? That is the practical difference behind this experiment. Kuba Rogut, a deployed engineer at turbopuffer—a serverless full-text and vector search database built on object storage—starts with Claude Code’s choice. At the time of the recording, it relied on agentic filesystem search by default. Rogut cites Boris Cherny’s account that early versions used a local vector database, but iterative search proved simpler and worked better for Claude Code.

Cursor made a different investment: it indexed codebases into turbopuffer to support semantic retrieval. In Cursor’s semantic-search evaluation, Rogut highlights roughly 24% relative answer-accuracy improvement for Composer, with about 12.5% averaged across models. These results predate Composer 2; they do not establish its performance. He also recommends Cursor’s account of indexing large codebases.

Cursor’s separate online A/B test measured user outcomes rather than benchmark answer accuracy. Its report gives 2.6% higher code retention with semantic search for codebases containing at least 1,000 files. It also reports 2.2% more dissatisfied follow-ups when semantic search was disabled. The direction of that second comparison matters: it is not exactly the same percentage as a reduction measured from the disabled condition. Aggregate effects also include requests that have little use for semantic search, so a modest overall change can conceal larger gains on the requests that need it.

Consider the question: how does metadata filtering work in this codebase? An agent using filesystem search starts with candidate terms, reads matching files, and searches again when those files do not answer the question. A shell search can begin broadly and then request nearby context:

rg -n -i 'metadata|filter' .
rg -n -i -C 12 'metadata.*filter|filter.*metadata' .

The agent still has to interpret the matches and decide where to look next. Rogut’s illustrative trace spends about 6,000 tokens on this discovery process. That is one example, not a benchmark-wide average. The same work can recur in another session or in another agent working on the same repository.

The indexed route moves some of that work earlier:

1. Split the codebase into chunks.
2. Embed those chunks and store them in a searchable index.
3. At runtime, use the metadata-filtering question to retrieve relevant chunks.

Embeddings are cached compute in this sense: they preserve a reusable representation of meaning that can shorten later discovery. They do not cache a complete answer. The agent still needs to read and reason about the returned code. Nor does indexing eliminate upkeep; changed code requires the index to be maintained. The complete slide compares repeated grep-and-read discovery with the upfront indexing work and subsequent retrieval.

The economic argument becomes stronger across repeated sessions and concurrent agents. A small saving on one question may be unremarkable, but paying to rediscover the same area of a repository repeatedly can accumulate. Whether indexing pays off depends on how often the representation is reused relative to the cost of building and maintaining it.

TurboGrep implements a small version of that pipeline. It walks the filesystem, uses tree-sitter to parse and chunk code, generates embeddings with a Voyage code model, and uploads them to turbopuffer. Claude Code then gets a search tool backed by that index. At recording time, Rogut described V1 as open source and V2 as awaiting release; the public repository should not be assumed to reproduce the exact demonstrated integration.

The demonstration uses the Django repository. Claude Code invokes the TPuff search tool with a query for a password reset token generator, receives the relevant content, and uses it to explain the implementation to the user. The useful handoff is from a description of the desired behavior to code the agent can inspect. A successful trace shows that the integration works in that example, but does not yet establish whether it improves retrieval across tasks.

To move beyond a convincing demonstration, the evaluation asks what context the agent discovers while attempting a task. Rogut distinguishes Cursor’s internal context benchmark from the public ContextBench work. The latter uses human annotations identifying files, lines, and symbols that an agent should inspect to complete a task well. This evaluates the retrieval trajectory, rather than simply whether the final code solves the issue. Retrieval precision and recall are not answer accuracy or task completion.

The experiment compares three conditions:

The read limit matters because reading an entire thousand-line file can sweep in annotated context without showing whether the agent located it precisely. Large reads made the measurements noisy and obscured differences between retrieval strategies. The windowed condition provides a more informative baseline for assessing the added semantic tool.

Precision asks how much of the retrieved context belongs to the annotated target set:

If the agent reads ten files and eight are required, file precision is 80%. The same idea applies to lines and symbols. Rogut reports these approximate results:

The intermediate windowed condition also improves precision. Consequently, the entire increase from stock Claude Code to the combined condition cannot be attributed to semantic search alone; restricting reads contributes too. These are Rogut’s reported experimental results, not a general guarantee for other models or repositories.

Rogut describes default Claude Code as highly exploratory: it tries to read broadly before settling on the relevant implementation. His rounded interpretation is that files outside the labeled target set account for about one in three baseline reads, one in five windowed reads, and one in eight reads with semantic search added. That makes the selectivity improvement tangible. Calling those reads wasted is shorthand for being outside the benchmark’s annotations, not proof that every exploratory read was useless.

Recall changes the denominator: how much of the required context did the agent find?

Suppose a task requires ten files. The agent reads five files, but only three belong to the required set. Its file recall is 30%. Reading fewer irrelevant files can improve precision without improving this coverage measure.

Stock Claude Code has the highest file recall in the reported comparison. Broad exploration helps it encounter more target files. Its line recall is weaker, however: reaching the right file does not necessarily mean inspecting the right parts of it. Windowed reads and windowed reads plus semantic search have roughly similar aggregate recall, with some decreases when semantic search is added. The aggregate covers 50 tasks in Rogut’s experiment, not the full public benchmark. Higher precision therefore coexists with a mixed coverage result.

Splitting the tasks by which condition performs better reveals differences that the aggregate hides:

• Semantic search: Useful when relevant files implement related behavior but use different keywords. Rogut recalls an example involving related handling across several ORMs or libraries: lexical search missed some implementations, while semantic search found their behavioral relationship.
• Windowed grep: Useful when an early search finds a good identifier and the remaining work follows explicit connections such as imports. Once the agent has the right names, keyword search can trace the relevant files directly.

These are different ways to navigate a repository. One follows similarity of behavior; the other follows names and explicit code relationships. The task breakdown supports choosing tools according to that structure, rather than declaring one search method universally better.

The experiment improves precision, but the agent also has to decide when and how to use the new tool. Rogut’s interpretation is that Claude Code is oriented toward grep: attaching a semantic-search tool does not automatically give it a reliable policy for choosing that tool, phrasing a query, or deciding when to return to lexical search. He contrasts this with Composer’s integrated retrieval behavior.

That is a plausible explanation for why an added tool might fall short of an integrated system, rather than a causal result isolated by this experiment. Cursor’s answer-accuracy evaluation and Rogut’s context-retrieval evaluation also measure different outcomes, so their gains cannot be compared directly. His closing design principle is to provide lightweight, complementary ways of finding context. Even very large context windows still benefit from selecting the relevant information before asking a model to reason over it.

The audience first asks for an exact definition of semantic search. In this implementation it means vector similarity search: embed the code, embed the query text, and search the stored vectors in turbopuffer to retrieve code chunks. The embedding model is voyage-code-3. This is a concrete, relatively simple retrieval setup, not a claim that every system described as semantic search uses the same architecture.

Rogut names Cursor, Anthropic, and Notion as turbopuffer customers. A company’s use of the database does not identify the retrieval architecture of each of its products; in particular, that customer relationship does not establish semantic code search inside Claude Code.

What happens on poorly written code? Rogut does not give a controlled benchmark answer. Instead, he describes a pattern from reviewing trajectories, including using Claude to help examine them: examples with useful function comments and inline documentation tended to retrieve better. Those comments supply context about behavior that both the embedding model and the coding agent can use.

The difficult part is not merely producing a vector. It is making the chunk’s meaning available to the representation. A function’s syntax may show what operations it performs without clearly expressing why it exists or which user-facing behavior it supports. Good documentation can help connect those levels. This remains a qualitative observation from the trajectories, not a measured rule that comments always improve retrieval.

A follow-up question points to a representation mismatch: a natural-language query and raw code may express the same behavior in very different forms. The audience proposes a parent–child arrangement in which a query-like description points to a child containing the actual code. Rogut clarifies that this experiment did not apply that preprocessing; it embedded the raw code chunks.

Authentication flow is the audience’s example of a human-level concept that may not appear directly in implementation text. A description at that level could provide a useful retrieval target, while the linked code remains the evidence the agent needs to inspect. Rogut mentions Cursor’s own embedding model as another way to address the translation between code and human queries, but says he cannot explain its internal implementation.

Another audience member suggests that Cursor generates comments and embeds them with code. That explanation is unverified in the discussion. Rogut agrees that comment enrichment could work; he does not establish that it is Cursor’s actual method. The concrete distinction is between the raw-code experiment presented here and possible enrichment methods that would need their own evaluation.

The final question raises long-term memory and reducing database size. Rogut answers in terms of workload placement, without specifying when to encode memories or how to compact them. If all relevant information can be downloaded to a local filesystem, grep is convenient and requires little setup. That simplicity is valuable, even though the agent’s repeated searching and reading still consumes compute.

Shared workloads change the calculation. A Notion-like knowledge base contains relationships across information that may be awkward to gather and search as local files. A shared vector index can make those semantic relationships available to multiple agents. Multimodal data extends the case further: grep does not directly recover the semantic content of raw video, audio, or images. Matching a filename is different from finding what the media contains.

The choice therefore depends on the data, the workload, and the amount of repeated discovery. Local lexical search remains useful when the relevant material and identifiers are readily available. As agents share larger or more varied information spaces, storing reusable semantic representations can offload work they would otherwise repeat. That is where the cached-compute argument reaches beyond code search.