The AI emperor has no DAUs: why most devs still don't use code AI

[upbeat music] Great to see you all. Again, I'm Quinn Slack.

I'm, uh, CEO and co-founder here at Sourcegraph, and yes, I was at Bleacher Report. Think I learned I'm better coding than doing sports, so I'm in the right role here.

Uh, I wanna talk to you about why most devs still do not use code AI. It's probably a mind-boggling fact to all the people in this room, but it's true, and we'll have a good discussion about that.

Uh, I'm a coder. I code all the time. Been building Sourcegraph, which is code search and code AI for devs since 2013. I merged four PRs earlier today, but I wanna talk mostly about the product and, and what to build and how to get it in the hands of users.

Because if you don't do that right, then all the code that you write is for nil. So

three points I wanna make. One is that most devs don't use code AI.

I want this to scare you. I want to scare you in this talk, and then I wanna share some of the mistakes we made and some of the tips from our experience doing all the things wrong and then trying to figure out how do we do it right at Sourcegraph as we've been building Cody.

All right. Um, first just show of hands here. Who uses... Who in the last week has used any kind of code AI tool while coding?

All right. Man, I wish the whole world was like this room, but we are in the heart of it in San Francisco, and the whole world is very different.

Most devs do not use code AI, and I'm gonna give some numbers here. Uh, I'm privy to a lot of private information and stuff that I've heard as well, and all of this is in line with what I've been hearing.

So first number. Oh man, this is a great statistic. 92% of devs use code AI tools at work. Wow, that means these tools must have tens of millions of users.

Uh, this was a study from GitHub, uh, 500 people. Well, it turns out this is only in the US, and they had a very broad definition of what it means to use a code AI tool.

So here's a case where the hype kind of outstrips the reality, a common pattern.

This is an infographic from GitHub. 70% of developers see a benefit to using AI coding tools at work. Surely, if there's a benefit, they must all be using these things, right?

I mean, if only it worked that way.

All right, some numbers for the broader universe. Uh, there are 1.3 million paid subscribers to GitHub Copilot. This was what Microsoft reported in January 2024. 1.3 million is a lot, but if you're just paying for something, are you actually using it?

And also, 1.3 million is not a lot compared to all the people in the world that write code. And there's actually a way to get an even finer point on that number.

GitHub released a study saying, uh, if you look at the fine print, that in this time range, this year, 935,000 devs received a suggestion, not even accepted it. So there's gonna be some that saw that little ghost text in their editor and never actually accepted it, and also that's yearly active users.

I don't know of any companies that go and cite yearly active users. Uh, surely the monthly active users and daily active user figures are much lower than that. So these numbers are, you know, pretty...

You could fill, you could fill a city with these people, but it's not that many people when you compare it to the 26 million professional developers in the world.

And I think this is an undercount. Um, I will say that, you know, how do you get this number? I googled a bunch of different stats, and they all were around this.

Um, so, you know, that's the Vibes citation there. There's a lot more people that touch code in some way or another. I... Again, it's really hard to get a number on this, but I'd say probably 100, 150 million people in the world touch code in some way, and that includes students and other people and so on.

So, you know, 935K, best case, what is that in MAU? That's tiny compared to the number of people that interact with code. So this is a scary thing. My best case estimate is that around 5% of professional developers use code AI.

These are people that are paid a lot of money to write code, and they do it all day, every single day, and all of us here in the room, we know that there's this amazing new alien technology that's dropped on Earth, and that it changes how we code, and 5% are using it.

That's crazy. That means in a room of 20 people, 19 are not using this. It just absolutely boggles my mind, and that's, you know, a huge problem. And it's an even bigger problem because if you look at what they're actually using, the vast majority of code AI use is just autocomplete.

It's the ghost text. I mean, yeah, that's, that's good, but anyone who has used anything beyond that knows that AI can do so much more when coding. It can write entire files, it can answer questions, it can fix bugs and all of that stuff.

So this usage is absolutely tiny, and we need to realize that we here in this room, we are all the freaks, and for us to be successful requires us to change a lot of minds out there in the world.

So how do we change those minds? Well, you know, we've been building Cody. Cody is a code AI tool. It does all of these things, and so we encounter a lot of reasons why people are hesitant at first, and I've taken the reasons from our own internal tools and ranked them.

Why don't most devs use code AI? The first one is just, eh, they don't have a good reason. Uh, there's some people in the world, believe it or not, that have not used ChatGPT, that have not heard about it, even devs.

Uh, there's a lot of devs who are kinda grumpy, and they say, "Well, it's not perfect. I tried it and it gave me a wrong answer." Yeah, no shit, but there's a lot of other people that have figured out how to get the right answer out of it.

Uh, a lot of people say, "I don't need it. It didn't help me that much." You know, again, you gotta figure out how to make it useful for you.

A lot of people's company hasn't adopted it. It's too expensive or... You know, one thing we're seeing a lot less of now is the security, privacy, and legal concerns.

I think it's a situation where every... enough people are using it so that if it's illegal to use code AI, then we're all fucked, so it's mutually assured destruction.

But there's a lot of reasons why people don't use it yet.

And this should really scare us because there's a lot of companies that are relying on a lot of people at a lot of enterprises using code AI. There's a lot of people's retirement accounts or option trading accounts or whatever that are also relying on that.

And if you think about how technology makes money, I mean, you know, the, the money that gets deposited into your bank account and your paycheck, where do those dollars come from?

It takes, you know, kind of a long route. Uh, just to explore the market a little bit here, you know, you've got these foundation model companies like OpenAI and Anthropic at the top, and then you have the AI infra companies.

And the AI infra companies, you know, they will send-- If they're serving and doing inference, they'll send some of the money back to the foundation model companies. In some cases, they're the same companies.

And you know what drives usage of that? You have some experimental usage, people being the looky-loos, going and trying it out. That will turn into some experimental revenue. But all of that can happen without any real, actual usage.

Uh, but that gives the perception of, "Holy shit, this stuff is hot. This stuff is working." And then you start to get some real usage, some people using it, uh, you know, maybe paying twenty bucks a month on their credit card.

You get some devs using it at work. And then the Holy Grail, the way that software makes money, is from enterprises where there's recurring revenue from real usage. That first year contract, it doesn't matter.

It's gotta be JPMorgan pays you a million dollars the first year, and then they renew for two million dollars the next year. And all of the money in our paychecks ultimately comes from that.

If you're working at an AI infra company, if you've raised money to go and sell some AI product, it's ultimately because someone out there thinks that five years from now, JPMorgan is gonna be paying you two million dollars a year.

So that's the only thing that matters. Everything else is downstream of that. And it turns out there's just not that much of that going on. And also, this is an even more lopsided pyramid than I've depicted here.

We got Nvidia and chip makers at the top, and so much investment riding on it. So, you know, this is where we ultimately need to get to. We need this person to be using our software, and maybe it's not the coolest thing out there, but that is the reality of how technology makes money.

And wanna put some numbers to this. I estimate, based on, uh, my internal information, some of the information that GitHub has shared at this conference, that the total recurring revenue from code AI usage is around three hundred million ARR.

Now, look, that's a lot of money. If you had a company that was doing that, then it could go public on its own. So that's good. But how much of that money actually goes to these other companies?

Not that much. Uh, from our own data at Sourcegraph, where we spend a lot of money on AI inference, we spend less than ten percent of our revenue. So if you take that number-- And by the way, we're not even doing it in that optimized of a fashion.

If you take that three hundred million number, ten percent of it, that's thirty million dollars going back up to these foundation model companies and AI infra companies. It's a long way from where we are today, that amount of revenue, to them making the kind of revenue that's gonna justify these massive multi-billion dollar valuations.

Also, another way to look at it is Salesforce. Salesforce's annual revenue is thirty-six billion dollars. All of this stuff, all this hype, all this usage of code AI, it's, it's amounting to a tiny, tiny fraction, one one-hundredth and twentieth of Salesforce.

So we have a long way to go. Usage needs to grow a lot. Or I could be wrong and maybe the doomers are right. Maybe not for the reason they thought.

Maybe we should never have done this, and this is all just gonna be one massive hype bubble, and it's gonna pop, and we're all gonna be miserable. But I think because I use code AI every day, obviously, and you do too, this stuff is real.

It's just, it's really early, and usage needs to grow a lot. So if you're building a product, keep that in mind.

This is what I'm betting on. And let's talk about some of the lessons from our experience building Cody at Sourcegraph. I wish I had this mindset going into it.

Uh, just quick background to establish, you know, why would you even listen to me? Maybe I should have done this in the first slide, but hey, here we are.

Uh, Sourcegraph has been around, as Brittany said, for ten years. We started out with code search, and then we found out that if we ha-- We had this tool that all the devs used and had all the code in a company, and it turns out that was amazing context to build a code AI on top of.

And so we took that and we built Cody. It's got really great autocomplete and inline edits, generating unit tests and chat. And chat is where we really differentiate because that's where you can make the best use of context.

So you can figure out why is this broken, or how do I change this, or where should I start on this? That's where we excel.

Uh, we've got a lot of really big customers. We have four of the top five biggest banks in the country. We have most of the FAANG or MAÑANA companies or whatever, and a lot of other great customers, including some that are presenting today.

And we are the number two code AI company in revenue, second only, of course, to GitHub Copilot. Uh, but because we're number two and because we're a startup and we're scrappy, we try harder.

All right, so what have we learned? One is that hype fools everyone. Hype fools us. Hype fools all of you. Hype even fools your customers. And the second thing is that AI code completion, the thing that autocompletes the rest of the line or the next few lines, that's like the freakish kind of feature that comes along like

one time in a hundred years. And we kinda got spoiled. It is so perfect. So I'll get to that. But first, just, you know, on the hype, how do you get away from being fooled by the hype?

I'll share a few tips that have worked for us. The first, and this is the most important thing, if you're building a product and you are not using it every single day, it is not gonna make it.

There is zero hope. If you're building it on a team and the people all building it don't use it every day, your customers will not. But they might say it's really awesome, and that's the dangerous thing about this hype.

Customers don't know what they want. Every customer for about a year said, "I want fine-tuned models." They said that all the time. Our salespeople would say, "Hey, we need to build fine-tuned models."

Everyone was talking about it. GitHub Copilot has it as a bullet. You know, all of our competitors have it as a bullet that's, like, italic and it's, like, coming soon.

It's some bullshit thing. Really, what they just want is they want it to work well, and they use some of these terms. So don't listen to them, 'cause what they're saying is probably downstream of them sitting at, you know, some conference like this and getting some cool ideas.

They want the product to work, and if you can't describe why it works and why it's great without using the word AI, then you're probably not gonna make it.

Uh, this is a really existential question. So we make a product that makes developers a lot more productive, like, 20 or 30% more productive. If it's so damn good, why are we selling a product?

It's like the people that have the stock picks and sell newsletters. Why don't we go and buy a software outsourcing firm and prove it and monetize our product by actually capturing that value directly?

That's a damn good question, and I wish that we had done that a year ago, and we're looking into doing that now. And, uh, if you don't have the confidence to go and do that, to put, you know, a few million bucks behind that if you've raised a bunch of money, then I'd say your product's probably not

ready yet. And then finally, I think we've all seen this. Um, this is actually not even one of the worst ones. This is, you know, a tweet that's got a lotta, a lotta activity, right?

Um, just keep in mind that that does not translate into DAU, so don't feel shitty about yourself all the time. I wish that Elon would add something like this that would actually tell you the DAU of the products that are getting all this hype.

But until he does that, you can imagine it.

All right. I talked about autocomplete being this freakishly good feature. Here's how we think about features at Sourcegraph, this, uh, kind of four box. You know, you wanna be in the top right, and that's where it's an AI feature that doesn't take a ton of time to see is that correct, and it's, you know, really easy, and

it's used often. AI autocomplete is great because it's literally every keystroke, and you can, like, glance at it in th- in a few seconds or even milliseconds, 100 milliseconds, you can figure out, is this correct?

So that's a really good property of a feature. And it just so turned out that the first feature of code AI happened to be smack dab in the top right.

It's pretty amazing. But most other code AI features do not have this level of product market fit, and we need to realize that. Now, we can work on it, but, you know, where's chat?

Not used nearly as often, and it's harder to vet that long response. Edits, you know, similar. I know a lot of people love doing the inline edits like, uh, you know, Meta K, Option K in Cody and Cursor and things like that.

It's great, but we gotta be prepared for what actually works being a very different form from what exists today. Then there's a lot of other stuff like the agentic stuff.

I mean, look, it's obviously the future, but it's just not there yet. Who here has used a code AI agent to actually merge a PR in the last week?

All right. Cool. Thank you for helping us, you know, push the world forward, but it's not there yet.

And, you know, other features, you really gotta go for that top right and drive any feature you make to the top right, and if not, it's just not gonna make it.

So what are we doing at Sourcegraph to address this? Well, we are searching for that next great code AI modality. What's the next autocomplete? There's some ideas like the next edit suggestion or, like, Copilot Plus Plus.

You know, I'm, I'm skeptical of that. Uh, chat. Steve Yegge, who is at Sourcegraph, he just wrote this great blog post about chat-oriented programming, CHOP, where it turns out there's a lot of devs who have a chat session running in, like, Cody or even ChatGPT or something, like, all the time while they're coding, and it's just one

ongoing conversation. It's a totally new way of coding, and what's weird is it's, like, the Gen Zs who do it, and it's, like, the [REDACTED:age], you know, disgruntled programmers who do it.

So I love that it's got kinda, you know, uh, some usage in both generations. Um, there's new ways that people are gonna be using code AI that we have not even thought about, and it's so early to the previous point.

We always gotta remind ourselves to build the manual and explicit thing first. We got spoiled by autocomplete. It's automatic, and, you know, it's, uh, it, it just, like, triggers every keystroke.

No. First, you gotta make something work in manual and explicit mode. If you have chat, you gotta make it so people manually @mention the context they want before magically inserting the context.

If you've got an agent, put it in the editor and make it work in the editor so that if it's wrong, the dev can just change it right in their editor instead of having to go into Codespaces or some other totally different UI.

So make it manual and explicit first and really easy for the dev to go and fix it, and then you can add the magic. Probably, [laughs] you know, you're gonna be bogged down enough with, uh, stuff to make it a great product, and you won't actually get to adding the magic for a while.

When you're pitching, always explicitly de-hype and set expectations low. No matter how much you de-hype, the person receiving it, they will hype it up in their minds. So you gotta remove the hype.

Do this so aggressively. And, uh, here's an example from Sam Altman. This is a tweet, I don't know, like, a year ago when ChatGPT had probably tens of millions of DAU.

So he's right. We know he's right, and if he's saying this and his product has way more validation than your product, well, you probably should be saying it, like, 10 times more and 10 times more intensely.

Also, it's kinda cool marketing, and it, you know, makes you seem cool.

Um, yeah, you gotta become... Here. It's covered up. Uh, you gotta become a DAU yourself or kill the product. And then the last thing is just that as a ecosystem, all these foundation model companies, all these AI infra companies, all these AI applications, we all live or die together.

Anthropic and OpenAI and Mistral and Fireworks and all these great companies that we use are not gonna get paid if we're not in business making a shit ton of money on Cody.

So we all need to work together and realize that we all have so much to benefit from this stuff actually being used, from turning down the hype and building great products that get actual devs using it all the time, and that starts with you if you're the one building the product.

So thank you, and if you, uh, wanna reach out to me, uh, there's my contact info, and happy coding. [upbeat music]