We Cut 94% of Our AI Coding Tokens With a Local Code Index. Here's the Architecture.

Talk transcript

1. 0:01

   Hey, I'm Raj. I want to tell a story. Me and my friend Faz, we are building project together. We are using AI coding tools every day. Claude Code, Cursor, Copilot, Codex, normal stuff.

2. 0:15

   One month, our AI bill was fine. Next month, huge. We did nothing different. Same project, same tools, just more of it. We panicked. We looked what was happening, and we found something surprising.

3. 0:31

   Most of the money was not the AI thinking. Most of it was

4. 0:39

   sending too much context, files the AI don't need. Context is important. Code that was not relevant, sent anyway every time. So me and my friend Faz, we started to building something to fix it.

5. 0:57

   In this talk is about what we built and what we learned.

6. 1:05

   Every AI coding tools, uh, does same thing. It send your code to the model as a context, and the tools thinks more context is better. We measure typical query on our project.

7. 1:22

   It was sending forty-five thousand tokens of context, but the part of actually mattered is about five thousand only. Other forty thousand tokens are not useful, but we paid for them every single query.

8. 1:38

   It's, um... that's like, uh, ordering a pizza and paying for extra nine pizzas you don't eat every time.

9. 1:49

   We tried three things, uh, before we found what works. First, we changed our prompt. Be short, only show relevant code. Sounds good, but it does not work. The model already got forty-five thousand tokens before it's read the prompt.

10. 2:07

    Cost already happened. Second, we change model setting like a max token, temperature, same problem. This changes the output, not the input. Uh, money is in the input. Third, uh, output compression.

11. 2:22

    This one actually works. We told the model to write short answers. It cut the output seventy-five percentage, but output only, uh, about ten percentage of the cost. So seventy-five percentage of a small number, still small number, not enough.

12. 2:42

    We need to fix the input. This is the most important slide. Ninety percentage of your AI cost is input, files, search results, context you send in. Only ten percentage is output, the code, uh, the AI writes back.

13. 3:00

    So if you cut the output by seventy-five percentage, you can save about eight-- eight percentage total. But if you cut input by ninety-four percentage, you can save about sixty-one percentage total.

14. 3:13

    Same math, but different result. Fix the input. That's where your money goes.

15. 3:20

    We built a local search layer. It sits between your code base and the AI. Instead of sending whole files, the AI search and index, it gets back only small piece of code actually it needs.

16. 3:34

    Here how it works, five steps. Step one, we read the code and, uh, break into small pieces, functions, classes, methods, not a random chunks, proper piece that makes sense.

17. 3:47

    Step two, we run two searches at, at the same time. One search find the code by meaning, one search find the code by exact words, then combine the results.

18. 4:00

    This is the big saving comes from. Steps three, we can shrink the results even more. Keep only the function name and the description. Cut fifty-line function down to five lines.

19. 4:14

    Step four, uh, we track the connection with the fa-- which function call which. So if you find one piece of code, you can find everything connected to it. Step five, every results get score.

20. 4:29

    If the score is too low, we don't send it. No bad context. Everything runs on your machine. Nothing goes to the cloud. This is the beauty.

21. 4:40

    Why do we run two searches instead of one? Because each one has a weakness. Meaning-based search is good at finding related ideas, but it misses exact names. You search for, um, authenticate user function, and it might show you different auth function instead because they are similar meaning.

22. 5:06

    The word-based search is good at exact names, but it misses related ideas. You search for login flow, and it misses everything that says sign in. By themself, both searches miss about one in four results.

23. 5:23

    Together, they miss about one in ten. They fix each other weakness spots.

24. 5:31

    Here is the hard part Faz and I spend most of the time on. The search finds results, but they actually relevant? Sometimes search results returns ten results, and none of them are right.

25. 5:45

    If the AI use the bad results, it gives confident wrong answer. The worse than no answer. [chuckles] We tried asking AI to judge its own results. Too slow. Add two, three seconds every time.

26. 5:59

    We try to fix the score limit. Too simple. Short questions score low even when the match is perfect. It worked. Some simple formula, fifty percentage me-meaning score, thirty percentage keyword score, twenty percentage how the recent code is.

27. 6:17

    And the limit adjusts based on the current result. It runs, uh, pi-- zero point four milliseconds. No extra AI calls needed. The lesson we learned, simple formula beats the complex model most of the time.

28. 6:34

    We need numbers, not just stories. So here are ours. We tested an open source, uh, real project, FastAPI, fifty-three files, twenty real questions a developer would ask. Without using our tool, eighty-three, uh, K tokens per questions.

29. 6:53

    With our tools, uh, four point nine K tokens per questions. That is ninety-four percentage less. With the extra compression on top, uh, five hundred and twenty-three tokens per question.

30. 7:06

    And the accuracy still, um, find the right code, ninety percentage of the code. These numbers are real. Test is public. You can run it yourself. The command is on screen.

31. 7:22

    I want to be honest about the limits. The ninety-four percentage, again, the worst case, reading full files every time. In a real life, the tools like a Claude Code already smarter than, uh, that.

32. 7:37

    Real savings are lower than ninety-four percentage. We use full, uh, file base because it is the only one we can measure the same way every time. Big mixed code base are hard.

33. 7:51

    We tested on a large product, uh, with, uh, three ninety-six files. The recall dropped almost zero. If your files each do one thing, it works well. If your files do many things, it struggles.

34. 8:07

    Uh, we use a small fast model for search. It's quick. Re-indexing takes under a second. But the bigger model would, uh, find more, which was speed over perfection. Simple choices work better than complex one.

35. 8:24

    Small database instead of big infrastructure. Two searches instead of one fancy one. Local instead of cloud. Simple one.

36. 8:36

    Here is something Foss pointed out early on. We use many tools, Claude Code for hard problems, Cursor for q-quick edits, Copilot for small, uh, completions. Each tools starts, uh, fresh every time.

37. 8:51

    They do not share ev-anything. You explain the same code base to three different AIs. We built one shared index, so all your tools connect to it. Same search, same results for all of them.

38. 9:08

    Uh, and we added memory. Uh, when one tools learn something about your project, that knowledge stays. Next session, different tool, same project, the context is already there. We explain the code base once, every tool remembered.

39. 9:29

    This is the saving report on real project. Two forty-seven queries, twelve point four million tokens saved, nearly one eight six not spent.

40. 9:41

    Most of the savings, eighty-four percentage, came from search layer. The rest of them, compression. This is not an estimate. The tool tracks every query. It compares what would have been sent against what was sent, then it multiplies by the model price.

41. 10:00

    Run it on your project for a week. See your own number.

42. 10:06

    Me and Foss built this because we had a huge bill and no good answer. The answer was not a better model. The answer was sending less. We argue about which model is best, Opus or Sonnet, but the models may be thirty percentage of the cost, but other seventy percentage is what you feed it.

43. 10:25

    Fix the input. The model choice matters less than you think. One command to try it, CCE. It's free, open source, QR code on screen. Try it. See the number.

44. 10:38

    Tell us what you saved. Thanks. Happy coding.