AI Engineer World's Fair 2024
Embeddings are Stunting Agents: How Codeium Breaks Through the Ceiling for Retrieval
Read the talk
Retrieving the Code a Coding Agent Actually Needs
A React contact form exposes the limits of similarity search: useful generation needs several complementary sources, an evaluation that measures their recovery, and affordable compute to rank them.
From a talk by Kevin Hou
Before you start: Familiarity with React components and the basic idea of supplying retrieved context to a language model is helpful.
A contact form that belongs in the repository
How do you generate a React contact form that actually belongs in an existing codebase? For Codeium, this is a product problem spanning autocomplete, chat, and search inside an IDE. Kevin Hou describes a plugin with over 1.5 million downloads, supporting 70 languages and 40 IDEs at the time of the talk. He also cites strong marketplace ratings and a leading developer-tool rating in the Stack Overflow survey. Enterprise customers add requirements beyond plausible code: security, licensing, attribution, and output that can make it into production. These constraints lead to the central question of context awareness: what must the model know before it generates?
The basic retrieval-augmented generation flow is straightforward: receive a query, gather context from several sources, supply it to the language model, and generate a response. The contact form shows why the gathering step is difficult. A generic answer might produce a valid React component while missing the repository’s existing buttons and inputs. Even if it finds those components, it still needs examples of nearby forms to understand how the project uses them.
Styling introduces another dependency: a project using Tailwind needs a form that follows its configuration and visual conventions. Local documentation and external dependency documentation supply further constraints. The slide makes this concrete with internal components, other forms, styling guidance, and documentation for packages such as react-hook-form and zod. The retrieval target is a useful set of complementary context, not one similar document. The engineering problem is to collect and rank that set quickly enough for interactive code generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ways to supply repository knowledge
Long context makes the interface to this problem easy: put more material in the prompt. But input capacity does not remove the cost of processing that input. Hou cites a historical Gemini example taking 36 seconds to ingest 325K tokens, without specifying the model version or test conditions. He estimates roughly one million tokens for 100K lines of repository code and describes enterprise code estates exceeding a billion tokens. His practical concern is the mismatch between the amount of available code and the latency and cost an IDE interaction can tolerate.
Fine-tuning moves customer-specific information into model weights. In the per-customer setup Hou considers, keeping those weights aligned with changing code requires continuing updates and expensive computation. Embeddings offer a cheaper way to compute and store searchable representations, but introduce a different question: whether their representation captures all the distinctions needed to select several relevant items.
| Approach | Mechanism | Constraint emphasized in the talk |
|---|---|---|
| Long context | Supply more input at generation time | Input latency and cost |
| Fine-tuning | Adapt weights to customer data | Updates and per-customer computation |
| Embeddings | Store compact searchable vectors | Relevance across multiple items |
These approaches place the work in different parts of the system. Enlarging the prompt, updating the weights, and retrieving compact representations each change the economics of supplying context; none makes the contact form’s collection problem disappear.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Similarity is not the same as task relevance
Embedding search starts by choosing searchable units: functions, documents, or other chunks of code. An embedding model converts each unit into a fixed-dimensional array of numbers. Those vectors make it possible to search a large collection efficiently by similarity, without reading every original item through a language model for each request.
Hou observes that the embedding results displayed in the talk cluster within approximately plus or minus five score points. He interprets this convergence as a ceiling and offers a hypothesis: a fixed vector cannot preserve every distinction that might matter across all possible queries. That is Codeium’s explanation for the observed pattern, not a demonstrated universal bound on embeddings. The product question is narrower and more actionable: does semantic distance between vectors reliably identify the functions needed for a particular coding change?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure recovery of the whole relevant set
Hou characterizes the evaluation mismatch as finding one needle when the task requires several. For the contact form, retrieving a relevant form example does not compensate for missing the design-system inputs or the styling rules. His criticism concerns what a coding product needs to measure: successful recovery of multiple complementary documents, rather than success at locating a single relevant instance.
Codeium therefore uses Recall@50: the fraction of the ground-truth relevant set present among the first 50 retrieved items. Let G be the set of relevant items and R₅₀ the set of the first 50 results:
The denominator is the number of relevant items, not 50. Dividing by the number of returned results would measure precision instead. A retrieval system can return many individually plausible files and still have poor recall if it misses the dependencies the task needs.
A small TypeScript implementation makes the distinction explicit. Here the contact form’s teaching set contains component files, an existing form, styling configuration, and dependency documentation. Retrieving three of those five items yields 0.6, even when the returned list also contains an unrelated file.
typescript
function recallAt50(
rankedIds: readonly string[],
relevantIds: ReadonlySet<string>,
): number {
if (relevantIds.size === 0) {
throw new Error("Recall requires a nonempty relevant set");
}
const retrieved = new Set(rankedIds.slice(0, 50));
let hits = 0;
for (const id of relevantIds) {
if (retrieved.has(id)) hits += 1;
}
return hits / relevantIds.size;
}
const relevant = new Set([
"src/ui/button.tsx",
"src/ui/input.tsx",
"src/forms/signup.tsx",
"tailwind.config.ts",
"docs/form-validation.md",
]);
const ranked = [
"src/forms/signup.tsx",
"src/ui/button.tsx",
"src/pages/home.tsx",
"src/ui/input.tsx",
];
const recall = recallAt50(ranked, relevant); // 0.6
This measures coverage of the labeled set. Whether that set accurately represents what generation needs is the next problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn commits into retrieval tasks
A useful metric needs queries and relevance labels that resemble the product’s workload. Codeium builds these from pull requests:
- Break a pull request into its commits.
- Extract each commit’s English description.
- Pair that description with the files modified by the commit.
- Use the description as a retrieval query and the modified-file set as relevance labels.
This creates a scalable mapping from an expressed change to a set of files involved in implementing it. It approximates the situation in which a developer asks a coding assistant to make a change across a repository.
The labels are product-shaped, but their meaning matters: modified files identify where a change landed, not necessarily every unchanged file or document needed to understand it. For the contact form, an existing button implementation might be necessary context without needing modification. The commit construction is therefore an approximation to useful generation context, consistent with Hou’s goal of getting closer to the end-user distribution.
Hou reports that publicly available models performed worse on this commit-based retrieval task and that Codeium’s approach improved retrieval for its chat and autocomplete workflows. The talk does not supply numerical Recall@50 results or a reproducible evaluation protocol for that comparison. The substantive shift is the evaluation target: measure how well retrieval supports an English request over a real codebase, then optimize against that target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make more retrieval computation affordable
Codeium’s response is to spend more compute on retrieval. That immediately raises a serving question: how can an interactive product afford richer judgments on every request? Hou connects the answer to vertical integration, with three parts:
- Custom models: train models around the workflows the product actually serves.
- Custom infrastructure: control the serving stack, drawing on Codeium’s origins as the ML infrastructure company Exafunction. Hou credits that control with lower costs and higher serving efficiency, including competitive advantages he asserts in the talk.
- Product feedback: judge shipped features through end-user behavior, rather than treating a local benchmark gain as the final result.
The infrastructure investment matters because it changes how much model computation the product can spend on deciding what to retrieve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
M-Query judges candidate items in parallel
M-Query takes a query and a collection of N codebase items, then makes parallel language-model calls to reason about individual items. Hou illustrates the operation as asking a model whether an item is relevant and receiving a yes-or-no answer. That explains the kind of judgment involved; it is not a disclosure of an exact production prompt or output schema. The key change is that relevance can be assessed with the actual query and item together, rather than inferred only from their positions in a fixed vector space.
Those judgments feed a ranking that can also account for the active files, neighboring directories, recent commits, and the ticket currently being worked on. The system selects the top-ranked documents and supplies them to generation, which can then stream code or a chat response. For the contact form, this means evaluating whether a component, form example, or style document helps satisfy the request—not merely whether it resembles the words in the request. The description establishes a model-based relevance stage; it does not establish that embeddings have been removed from every part of the retrieval system.
Hou claims Codeium’s computation costs one-hundredth as much as competitors’ computation, allowing 100 times as much compute per user. He attributes that economics to owned models and infrastructure rather than external API calls; no comparison workload or measurement setup is disclosed. The design commitment is nevertheless clear: spend the resulting compute budget on more thorough context selection because it can improve the generated result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test retrieval in the interaction it serves
The product still has to satisfy three requirements: speed, correct generations, and an understandable interaction. Hou reports that M-Query runs thousands of LLM executions in parallel and can begin streaming code within seconds, sometimes milliseconds. The talk does not give a latency distribution or define the timing boundary. Parallel execution addresses the waiting time associated with many judgments, while the infrastructure economics address their aggregate cost. Usability adds another requirement: developers should be able to understand what context the tool is using.
Codeium rolled the system out to a small percentage of users. Hou describes a download base exceeding one million, but that is not the experiment’s participant count. He reports selecting relevant context across thousands of files in monorepos and remote repositories. The demonstration asks for usage of an internally modified shadcn/ui Alert Dialog, then retrieves the relevant source. That local modification is the point: generic knowledge of a public component cannot establish how a repository’s own version should be used. The linked documentation now describes the Base UI variant, not Hou’s modified component from the recording.
Hou reports more thumbs-up on chat messages, more accepted generations, and more code written for users after the experiment. No effect sizes are supplied. These outcomes connect retrieval quality to the behavior the product ultimately cares about: whether developers find the answer useful enough to accept and use it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A context engine and a production feedback loop
Once context selection becomes a shared engine, it can support more than one generation surface. At the time of the talk, Hou names autocomplete, chat, and search as current uses. He proposes documentation, commit messages, code reviews, and code scanning as future applications. Figma-to-UI generation extends the contact-form problem: converting a design should produce an interface built from the repository’s own components. These are proposed uses of the retrieval foundation, not a claim that every workflow had already shipped.
The development loop starts with the user problem and returns to it through production:
- Build the evaluation from the task. Construct datasets and local metrics that reflect what users ask the product to do.
- Apply infrastructure and compute. Spend additional computation on improving results for paying and nonpaying users.
- Deploy the change. Hou describes pushing changes overnight to obtain a production pulse check.
- Use the feedback in the next iteration. Thumbs-up and thumbs-down signals help determine what to revisit.
M-Query is the example of this loop in action: define a retrieval problem, improve the mechanism, deploy it, and examine whether the resulting interactions are better.
The context engine also depends on less visible work: modeling, serving infrastructure, retrieval, AST parsing, indexing large collections of repositories, knowledge graphs, documentation parsing, and gathering website context. Hou presents these as pieces developed through the same iteration cycle, rather than a single retrieval algorithm that solves the entire problem. Their value compounds when they improve what the product can understand and deliver.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let shipped products test the compute argument
Hou closes by drawing on his experience in autonomous driving. He recalls optimistic predictions in 2015 and contrasts them with the field as it stood in 2024. Earlier systems used sensor fusion, lower polling rates, and off-board models to work around limited onboard compute. In his account, more available computation made larger models, more sensors, and higher-frequency processing practical; Waymo rides in San Francisco provide his concrete contemporary example. The analogy motivates a design direction rather than proving a retrieval result.
Applied to coding tools, the argument is that embedding-based retrieval may be a heuristic shaped by compute constraints. If richer model judgments become affordable, a product can spend more computation on the relevance decision itself. Hou’s final standard is practical: ideas about better retrieval must survive implementation, deployment, and use. The case for more compute is made through the working product it enables.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current installation and usage examples for the alert-dialog component family used in Hou's repository-context example; this page documents the Base UI variant.
Further reading
- Kevin Hou explains M-QueryArticle
Hou's companion explanation of parallel relevance judgments, custom infrastructure, and the retrieval argument behind the talk.
The original benchmark paper evaluates embeddings across eight task categories, including retrieval, clustering, and reranking.
Updates since the talk
A December 2024 interview with Hou about vertical integration, MQuery, internal product testing, and the move to the Windsurf editor.
Documentation covering indexed repository context, context pinning, remote repositories, and M-Query under the current Devin Desktop branding.
Read the complete timestamped transcript
- 0:00
[on hold music] Um, so my name is Kevin, and I'm going to be talking about how embeddings are stunting AI agents.
- 0:20
Uh, so I'm gonna let you in on some secrets about how we build the product, uh, and exactly what we're doing behind the scenes to improve your code gen experience.
- 0:29
So at Codeium, we are building AI developer tools, and we're starting with an IDE plugin. And as, uh, as mentioned before, we've been downloaded over a million and a half times.
- 0:39
Uh, we're one of the top-rated extensions across the different marketplaces, and to rede-- reiterate, we offer free unlimited autocomplete, chat, and search across seventy different languages and forty different IDEs.
- 0:50
So we plug into all the popular IDEs.
- 0:54
Uh, we are the highest-rated developer tool as voted in by developers in the most recent Stack Overflow survey. Uh, and you'll note that this is even higher than tools like ChatGPT and GitHub Copilot.
- 1:06
And importantly, we are trusted by Fortune five hundreds to deliver high-quality code that actually makes it into production. And we do this with top-grade security, licensing, attribution for some of the largest enterprises on the planet.
- 1:18
Our goal at Codeium is to empower every developer to have superpowers both inside of the IDE and beyond.
- 1:26
And today, I'm gonna let you in on some secrets about how we've been able to build a tool like this and why ch-- uh, users choose us over the other AI tools on the market.
- 1:35
And the short answer is context awareness. So here's a quick overview about what context looks like today. Uh, we're all familiar since we're at an AI conference with the basics of retrieval-augmented generation.
- 1:47
The idea being that a user puts in a query, um, you accumulate context from a variety of different sources, you throw it into your LLM, and then you get a response, whether that be a code generation or a chat message.
- 2:01
Um, here's a concrete example about how retrieval can be used in code generation. So let's say we wanna build a contact form in React. Um, now you could go to ChatGPT, you could ask it to generate a contact form, but in reality, on a moderately large codebase, this is really not going to work.
- 2:15
It's not gonna give you, uh, things that are personalized to you. Uh, and this is really where context retrieval comes in. We need to build a contact form that, you know, is in line with our design system components.
- 2:26
Let's say you already have buttons, uh, and inputs. It has to be able to, uh, pattern match with, uh, with local, um, local instances of other forms inside of your codebase.
- 2:36
It has to ingest your style guide. For example, if you're using Tailwind, you have to be able to detect and make the form look and feel like every other thing on your site.
- 2:44
Uh, and then of course, there's documentation both locally and externally, um, for packages and other dependencies. So the question becomes: how do you collect and rank these items so that our code generation can be both fast and accurate for your use case?
- 2:59
So to dive into a couple of different methods about how people are tackling this today, there's really three main pillars. The first one is long context. So this is the idea that if you expand your prompt window in your LLM, it can read more input and therefore be a bit more personal to what you're trying to put--
- 3:15
uh, what you're trying to generate. This is very ergonomically easy to use, right? You just shove more items into your prompt. But this comes at the cost of latency, uh, latency and financial cost.
- 3:25
So one of the most recent examples was Gemini. Um, Gemini actually takes thirty-six seconds to ingest three hundred and twenty-five K, uh, tokens. To put this into perspective, a moderately sized or even small repo is easily over one million tokens, uh, and that accounts to about a hundred K lines of code.
- 3:43
So in this instance, most enterprises have over a billion tokens of code. It's simply not feasible to be throwing everything into a long context model. The second method is fine-tuning.
- 3:52
So for those that are familiar, fine-tuning is the idea of actually tweaking the weights of your model to reflect the distribution of the data that your consumer expects, right?
- 4:02
And so this requires continuous updates. It's rather expensive computationally. You have to have one model per customer, and it's honestly prohibitively expensive for most applications. And finally, we have embeddings.
- 4:13
And for all of you, hopefully you're familiar, this is a relatively proven technology today. Um, it's pretty inexpensive to compute and store. Uh, but the difficulty that we're about to dive into is that it is hard to reason over multiple items.
- 4:26
It also has a low dimensional space, and I'll, I'll talk about that shortly.
- 4:31
So to dive deeper into embeddings, the whole concept is that you take your objects, you throw it through an embedding model, and then you end up with some sort of vector, some sort of array of numerical values, and this is in a fixed dimension.
- 4:43
And so by mapping and chunking code, we can map it to an embedding, and that allows us to quickly search over our functions, our documents, whatever you decide to chunk by, um, and this is what embedding search is called.
- 4:58
Uh, embedding search, like I said, is not a new concept. There's a bunch of model-- models that have tried to optimize, and in this example, we're looking at, uh, one of the kinda North Star, uh, eval benchmarks.
- 5:07
Um, it's become increasingly popular, and the question becomes: how do we fit millions of lines of code into an LLM model so that we can actually generate useful results?
- 5:16
And so it's evident through the years that we're actually hitting a ceiling on what is possible using these traditional, uh, vector embeddings. And over time, even the biggest models, uh, are approximating to around the same level of performance.
- 5:28
As you can see, everything's kind of within plus or minus five. And, uh, at Codeium, we kinda believe that this is because fundamentally, we cannot distill all the, the dimension space of all possible questions, all possible English queries down into the embedding dimension space, uh, that our vectors are gonna occupy.
- 5:47
And so at Codeium, we've thought very critically about what retrieval matters to us. Are we measuring the right things? And does semantic distance between these vectors really equate to things like function relevance in the concrete example that I showed earlier?
- 6:01
And so what we landed on is that benchmarks like the one that I showed you before heavily skew towards this idea of needle in a haystack. It's the idea that you can sift through a corpus of text and find some instance of something that is relevant to you.
- 6:16
Note, it is only one single needle. So in reality, code search requires multiple different needles, right? We showed that slide earlier. When you're building a contact form, you need all these different things in order to actually have a good generation.
- 6:28
And these benchmarks really don't touch that. And so we decided to use a different metric, and it's called Recall @ fifty. The idea and its definition is that, um, it's what fraction of your ground truth is in the top fifty items retrieved.
- 6:40
So the idea being now we have multiple documents, and we're now looking at the top fifty documents that we retrieved, how many of those are part of our ground truth set?
- 6:48
So this is really helpful for understanding document-- multi-document context, especially again, for those large, large codebases.
- 6:57
And now we actually have to build a dataset around this. And so this is where we did a little bi- ma- little bit of magic. We wanted to make the eval as close as possible to our end user distribution.
- 7:06
So we had to compile our own dataset. So what we did, this is a PR that I put out, um, a few months ago, we looked at PRs like this.
- 7:14
It's broken down into commits. Those commits we can extract and actually match them with the modified files, right? So now we have this mapping from something in English to a list of files that are relevant to that change.
- 7:26
And you can imagine we can hash this in many different ways. But ultimately, the point I'm trying to make is we are creating a eval set that mimics our production usage of something like a codegen product.
- 7:39
And so this message serves as the backing for this new type of eval, where now we can run at scale this idea of product-led benchmarks. It gets us closer to the ground truth of what our users are actually experiencing and what retrieval-- tweaks in retrieval actually mean to the end product.
- 7:57
And so we threw some of the, uh, currently publicly available models at this notion of retrieval, this idea of using commit messages, and we found that there is reduced performance.
- 8:08
Um, they're unable to reason over specifically code, but then also specifically this kind of real-world notion of, of English and, and commits, right?
- 8:18
And so at Codeium, we've been able to actually break through the ceiling. This is something that we've worked very hard at. We have to, to, to redefine exactly how we are approaching retrieval in order to be kind of in our class of our own, so that when you are typing in your ID, when you're chatting with our
- 8:32
assistant, when you're generating autocompletes, we're retrieving the most relevant things that are, are for your, your intents. So now the question becomes: how do we actually get this kind of best-in-class retrieval?
- 8:45
And so I'm here to give you the very short and sweet answer, which is we throw more compute at it, right? But of course, that can't come with, uh, absurd, absurd, uh, uh, cost, right?
- 8:55
Financial cost. Uh, so how do we do this actually? In, in production, how do we actually do this without recurring an unreasonable cost? And so this goes back to a little bit of Codeium's secret sauce, right?
- 9:06
We are vertically integrated, and what this means is that we train our own models. So number one, we train our own models. This means that these are custom to our own workflows.
- 9:15
So when you're using our product, you're touching Codeium's models. Number two, we build our own custom infrastructure. This is actually a very important point and actually connects to the whole Exafunction to Codeium pivot that we discussed earlier.
- 9:26
Exafunction was a ML infrastructure, uh, company. And so what we've been able to do is build our own custom infrastructure down to the metal. This means that our speed and efficiency is unmatched by any other competitor on the market, so that we can serve more completions at a cheaper cost.
- 9:41
And finally, we are product-driven, not research-driven. Now, what this means is we look at things like actual end user results. When we actually ship a feature, we're looking at real-world usage, and we're always thinking about how does this impact the end user experience, not just some local benchmark tweaking.
- 9:58
And so we could spend all day talking about, you know, kind of why Codeium has done this and yada yada, but that's a talk for a different time. So I'm gonna talk about something that I find very cool, and this is the reason why we've taken this vertical integration approach and been able to turn it into something
- 10:13
that we call M-Query. So M-Query is this way of taking your query, so similar to that idea of taking your retrieval query. You have your codebase, and let's just say you have N different items.
- 10:26
And because we own our own infrastructure and train our own models, we're now making parallel calls to an LLM to actually reason over each one of those items. We're not looking at, at vectors, we're not looking at small dimension space.
- 10:38
We're literally taking models and running them on each one of those items so that you can ensure... You can imagine, you know, you run ChatGPT and tell it to say yes or no on, on an item, for example.
- 10:48
That is gonna give you the highest quality, highest dimension space of reasoning. This leads into very, very high confidence ranking that we can then take into account things like your active files, your neighboring directories, your most recent commits.
- 11:01
Um, you know, what, what, what is the ticket that you're working on currently? And we can compile all this to give you, you know, the top N, uh, documents that are relevant for your generation so that we can start streaming in higher quality generations, higher quality chat messages, uh, things of that nature.
- 11:18
And the reason behind this is, again, it's that vertical integration. It's that idea that our computation is one one-hundredth of the cost of the competitors. We are not using APIs, and as a result, our customers and our users actually get one hundred x the amount of compute that they would on another product.
- 11:36
And so we're willing to do that. We're willing to spend more compute per user because it leads to a better experience.
- 11:43
And so, like I mentioned earlier, I lead our product engineering team. So we always wanna anchor ourselves around these three different things. One, that we have to build a performant product.
- 11:52
It has to be really fast. For those of you that have used the product, you can probably attest to this. M-Query runs thousands of LLMs in parallel, so the user can start streaming in code within seconds, not minutes, not hours, seconds, and oftentimes milliseconds.
- 12:07
It has to be powerful, right? None of this matters if the actual quality and the actual generations that you're building are wrong, right? And finally, it has to be easy to use.
- 12:16
We're building an end user product for people today that's in the IDE. Tomorrow it might not be in the IDE. How do we actually build something that is intuitive to understand, that people can grapple with and see exactly what my model is thinking?
- 12:30
And so because we have the benefit of distribution, uh, we were able to roll this out to a small percentage of our users. And by small percentage, we're dealing in the order of, you know, a million plus downloads.
- 12:40
This actually reached a surprising number of people. And what we've been able to see is that, um, we were able to successfully reason over these thousands of files in people's monorepos and people's remote repos and select what was relevant, right?
- 12:53
We can very accurately deem which files are relevant for the generation that you're trying to have. And the result, as you can see, this is a real time GIF, is both fast and accurate.
- 13:04
So I'm asking for usage of an alert dialogue. It's going through, and I think I pan down here. Um, this is kind of a Shad CN component that I've modified internally.
- 13:13
We're, we're pulling in basically the source code of, of what is relevant for our generation. Um, and ultimately, the results of this experiment were that users were happy. They were thumbs...
- 13:24
They had more thumbs up on chat messages. They were accepting more generations, and we were able to see that ultimately we were writing more code for the user, which is the ultimate goal.
- 13:32
It's that idea of how much value are we providing to our end users.
- 13:38
And so we built this, this context engine, right? This, this idea of M-Query, this idea of ingesting context and deciding what is relevant to your query to give you coding superpowers.
- 13:48
And so our users will generate... Today they're generating autocompletes. They're generating chats, search messages. But in the future they're gonna generate documentation. They're gonna generate commit messages, code reviews, uh, code scanning.
- 13:59
They're gonna take, you know, Figma art boards and convert them into compo- uh, into UIs that were built by your own components. The possibilities are endless. But what it starts with is this bedrock, this very hard problem of retrieval.
- 14:13
And it brings us to, again, one of the reasons why Codeium is approaching this problem a little bit differently. Our iteration cycle starts with product driven data and eval.
- 14:22
So we're starting with the end problem. We're building a product for millions of people. How do we start with what they're asking for? And how do we build a data set and eval system locally so that we can iterate on the metrics that matter?
- 14:34
Secondly, because we're vertically integrated, we're taking that massive amount of compute, and we're gonna throw it at our users, you know, paying or not paying. We're gonna throw it at our users so that they can get the best product experience and the highest quality results.
- 14:48
And then finally, we're actually gonna be able to push this out to our users in real time overnight and be able to get a pulse check on how this is going.
- 14:56
You know, this is what we did for, for M-Query. And when we evaluate in production, we can say, you know, thumbs up, thumbs down, and then hit the drawing board again, back to that same cycle, repetition.
- 15:08
And so you can start seeing how these pieces of compounding technology come together, right? We've alluded to some of them today, modeling, infrastructure, being able to retrieve, but then it also includes things like AST parsing, indexing massive amounts of repos, knowledge graphs, parsing documentation, looking at websites online.
- 15:27
The list can go on and on and on, but we're confident that we're solving these problems one piece at a time using that same iteration cycle, that same idea that we're gonna take the, the distribution and knowledge that we have and that additional compute that we're willing to afford each user to solve each one of these puzzle
- 15:44
pieces. And, um, I wanna leave you with, uh, a parallel analogy. So in my past life, I had experience in the autonomous driving industry. So to bring over a metaphor from that industry, in twenty fifteen, TechCrunch boldly predicted that that was gonna be the year of the self-driving vehicle.
- 16:03
Uh, i-it was largely, uh, you know, now we're in twenty twenty-four, so we can look back in hindsight, largely untrue, right? We were doing things like sensor fusion. We were decreasing our polling rates.
- 16:14
We were running off-board models, all this in the effort of making heuristics that would compensate for the lack of compute that was available because consumer graphics cards were not as popular or not as, uh, powerful as they are today.
- 16:26
Fast-forward today, we're seeing hundred X the amount of compute available to a vehicle. You can take a Waymo around San Francisco, which I encourage you to do. It's a wonderful experience.
- 16:34
Um, but that means that we're actually able to throw larger models at these problems, right? More sensors, higher frequency. And now twenty twenty-four, TechCrunch has released another article that said, "Will twenty twenty-four finally be the year of the self-driving vehicle?"
- 16:48
And we can now look at this pattern and say driving performance was substantially better by throwing larger models, being able to handle more and more data.
- 16:59
And so at Codeium, we believe that this embedding based retrieval is the heuristic. We should be planning for AI first products, throwing large models at these, at these problems so that AI is a first class citizen.
- 17:12
We're planning for the future. And finally, we also believe that ideas are cheap. You know, I could sit up here and tell you all these different ideas about how, you know, we're gonna transform coding and the way that the, the, the, the theory behind, uh, possible solutions.
- 17:26
But what we believe at Codeium is that actually shipping, actually showcasing this technology through a product is the best way to go.
- 17:34
And so if you agree with these beliefs, you can come join our team. We're based in San Francisco, and you can download our extension. It's free. I'm not obviously, uh, uh, what's it called?
- 17:45
I'm not advertising, uh, the core product nearly as much. We're kinda talking about the technology, but you can experience this technology firsthand today by downloading our extension. It's available on all the different plugins, uh, VS Code, JetBrains, Vim, Emacs, uh, and you can see how this infrastructure and the way that we've approached product development has shaped the
- 18:04
experience for you as a user. And then of course, you can reach out to me on Twitter. Uh, I put my handle up there. I'll be kind of floating around outside, so if you have other questions or are interested in what I had to say, um...
- 18:15
But I hope that you learned something today. I hope that, you know, you use Codeium, you try it out and see what the magic can do for yourself. Thank you. [upbeat music]