We Cut 94% of Our AI Coding Tokens With a Local Code Index. Here's the Architecture.

Rajkumar Sakthivel and his friend Faz were building a project with Claude Code, Cursor, Copilot and Codex. One month their AI bill was manageable; the next it jumped. They had kept the same project and tools, but used them more. Investigating the increase led them to the code being sent with each request: irrelevant context was traveling alongside the material the model actually needed.

Raj reports that a typical query on their project sent 45,000 context tokens, of which about 5,000 mattered. The other 40,000 tokens were still paid for on each request. His analogy is ordering one pizza and paying for nine extra pizzas that nobody eats. The optimization target was therefore context selection before the model call.

Their first three experiments targeted the model’s response rather than the process assembling its input.

Raj reports a 75% reduction in output, with output accounting for only about 10% of their stated cost split. That made shorter answers useful but insufficient.

The next slide assigns 90% of cost to input and 10% to output. Under that split, reducing output by 75% saves 7.5% overall—approximately the 8% Raj gives. He also claims that reducing input by 94% saves about 61% overall, but that does not follow from the same assumptions: with unchanged pricing and output, the calculation would be 90% × 94% = 84.6%. The percentages cannot all describe one consistent bill calculation. The architectural conclusion remains useful: reduce unnecessary input before paying to process it.

The team built a search layer between the repository and the coding assistant. Instead of loading whole files by default, the assistant searches an index and receives selected pieces of code. The architecture has five stages:

1. Chunk by code structure. Split source into functions, classes and methods rather than arbitrary text windows. The slide identifies Tree-sitter as the chunking component.
2. Retrieve through two searches. Run semantic and keyword searches concurrently, then combine their results.
3. Compress when appropriate. Retain a function’s name and description instead of its complete implementation. Raj illustrates this as shrinking a 50-line function to five lines.
4. Follow code relationships. Track which functions call which, so a retrieved function can lead to connected code.
5. Gate by relevance. Score candidates and withhold results that fall below the acceptance threshold.

The slide shows local SQLite files as the storage layer. Raj’s local-only description applies to indexing and search; the downstream coding assistant can still send selected context to a hosted model.

Semantic search and keyword search fail in complementary ways. A semantic query for an authenticate-user function can retrieve a different authentication function because both express a similar idea. Keyword search has the opposite problem: a query for login flow can miss code described as sign in.

Raj reports that either search alone misses roughly one in four results, while their combination misses roughly one in ten. He does not define the evaluation set for these particular rates. The mechanism is clearer than the aggregate number: exact matching protects names, while semantic matching bridges vocabulary.

Finding candidates does not establish that any candidate is useful. A search can return ten results and still miss the requested code entirely. Feeding those results to a model risks a confident answer grounded in the wrong implementation. The team first tried having AI judge the retrieved results; Raj reports that this added two to three seconds per query.

A fixed score threshold was cheaper but rejected short questions even when their matches were good. The replacement combines three signals:

The acceptance threshold then adjusts to the current result rather than remaining fixed across all questions. The talk supplies the weights but not the normalization or threshold-adjustment rule. Raj reports 0.4 milliseconds for this approach without additional AI calls. That timing needs a narrow distinction: the published benchmark’s 0.4-millisecond figure measures median retrieval latency after warm-up, not the scoring formula in isolation.

The published FastAPI benchmark covers 53 files and 20 developer questions. Raj presents rounded context counts:

The reported 94% reduction compares retrieved code with the full-file baseline. Compression is a further step after retrieval has already removed most of the context.

Raj describes the retrieval quality as finding the right code 90% of the time and points viewers to a public test command. More precisely, the report’s 0.90 Recall@10 measures mean expected-file coverage among the files represented in ten retrieved chunks. It is not generated-code accuracy or the percentage of answers that are correct. The benchmark implementation makes that distinction explicit; the published results were inspected, not independently rerun.

The headline comparison uses a deliberately expensive baseline: reading full files every time. Raj chose it because it could be measured consistently, while acknowledging that tools such as Claude Code already select context more intelligently. The 94% figure is not measured savings against normal Claude Code behavior. In the inspected runner, the baseline sums complete files represented by the retrieved chunks; it is not necessarily the entire repository on every request.

Repository structure also changes retrieval quality. Raj reports that recall fell almost to zero on a larger, 396-file project. His practical observation is that files with one responsibility work better than files mixing many responsibilities. That is a useful warning about retrieval dilution, but the talk does not establish file responsibility as an isolated cause.

The search model introduces another tradeoff. The team chose a small, fast model rather than a larger model that might find more relevant code. Raj reports re-indexing in under one second, without specifying the update workload; this should not be read as a promise about initial indexing of any repository. The broader implementation follows the same preference for modest infrastructure: a small database, two complementary searches and local operation.

The team’s workflow spans several assistants: Claude Code for hard problems, Cursor for quick edits and Copilot for small completions. Without shared context, each tool starts over, and the developer explains the same repository repeatedly. Connecting them to one index gives them a common search layer and the same retrieval results.

Persistent memory extends that arrangement beyond source search. When one tool learns something about the project, the knowledge remains available to another tool in a later session. The index supplies code; memory preserves accumulated project context. Together they reduce both repeated retrieval setup and repeated explanations by the developer.

For a real project, Raj reports 247 queries, 12.4 million tokens saved and nearly 186 in avoided spending, without specifying the currency. He attributes 84% of the savings to search and the remainder to compression.

The tool records each query, compares the context that would have been sent with the context actually sent, and multiplies the difference by the model price. Although Raj describes this as measured rather than estimated savings, the distinction is between logged token counts and a counterfactual bill: the money figure is a price-based estimate, not demonstrated invoice savings. Its interpretation depends on the baseline, applicable prices and caching treatment. His practical invitation is to run the tool on your own project for a week and inspect your own numbers.

The closing comparison is familiar: developers debate Opus versus Sonnet, while the amount of code supplied to either model can remain unexamined. Raj loosely assigns 30% of cost to model choice and 70% to what users feed it; those figures have no defined relationship to the earlier input/output split. The actionable point is to make context selection an explicit part of the coding workflow, then measure whether it preserves useful retrieval while reducing input.

The free, open-source tool is Code Context Engine, or CCE. Raj closes with a one-command invitation and a QR code. The repository’s current documented entry point is:

uvx --from "code-context-engine[local]" cce init

That is the current installation instruction, rather than a transcription of the historical slide command. Trying it on a real repository makes both sides of the tradeoff visible: how much context is removed, and whether the code needed for the task is still found.