Move Fast and Don't Break Things: Scaling Databases for the AI Era — PlanetScale

Hey everybody, how's it going? Hope you're having a good last day of AI Engineer. Uh, what I'm gonna talk about today is how to move fast and not break things. So the, the phrase move fast and break things, you've probably heard this before, and what this, uh, refers to is it's a phrase from early Facebook engineering culture days, right? And the idea was we wanna be able to move fast and ship fast and not be constrained by, you know, databases going down or breaking certain features because the priority is there's so much value in

shipping fast. And this was 20 years ago, right, before we had AI agents writing code for us and shipping things and deploying things on our behalf. So if anything, the idea of doing this is more true than ever, but in my opinion, we can actually achieve both, right? I wanna be able to ship fast, build great features, get things out to my customers, but also not break the databases, the infrastructure, the application servers that are powering what I'm doing. And we've probably all seen something that looks approximately like this, and I'm

not calling out any particular company. There's a number of companies, right? That due to the kind of insane scale that we've seen over the past even six months, year, two years, right? Where we've had, like, daily active users double, triple, 10X, 100X, a lot of that powered by, like, the demand of AI agents, um, where the infrastructure struggles to keep up. And some of this is simply because it's a very hard problem, right? This is not meant to call anybody in particular out. It's hard to scale things to tens of millions or hundreds of millions of users.

But what we'd like to have is great products that scale well for lots of users and have our status pages look a little bit more like this one, right? Three or four nines of availability, uh, maybe five nines of availability, where our databases are healthy, our applications are healthy, and ultimately, our users are happy, right? So this is kind of the idea behind what I wanna talk about, which is a little different than probably most of the talks you've been at. A lot of the talks are probably talking about inference or specifically how to do things with agents or different kinds of automations.

This is more of an infrastructure talk, but how you can do that and ship fast both with the demand of AI agents as well as the y- you being able to use AI agents to make all of these processes more optimized and smoother. So there's three parts to this that I wanna talk about. The first one is, how do you even build systems, regardless of the number of users you have, that aren't going down all the time? And then tying that into, well, okay, if you're gonna build the next hit application that's gonna scale to millions of people

using it, how do we scale that using these same principles? And then finally, how can we take these two things and let AI agents do them for us, or at least in part work with them in doing this?

So I should probably introduce myself a little bit too. The reason why I'm talking about this is I'm Ben, and I'm from PlanetScale, and we do databases. This is our whole thing. And we work with a lot of these AI companies that are scaling extremely fast. We work with companies like Cursor, for example, and power databases for them. And so this is something that we as a company take very seriously. How do we scale and support companies that have millions of users, uh, on their platforms? And some of what I'm talking about comes from a article that we published last year about

kind of our philosophy of how we operate databases. I would encourage you to go Google that and read that. Um, but a couple of these principles, let's go over some of these. So the, the first and one of the most important is isolation, right? When you are running infrastructure, one of the things you wanna make sure you do is have a setup where if something fails, it doesn't take down the whole system, but every component is very isolated. So what I'm showing here is data plane. That basically means where are your databases? Where does your most critical data live that you need to

be using for your users to log in and interact with your application? But there's lots of other components, right? There's your control planes. There's your observability pipelines. There might be your analytics frameworks. There's your apps. There's your MCPs, and there's all these different components that can potentially fail, and in fact, assuming you're using a good database platform, the database is hopefully low likely to fail. But these other things you're shipping and deploying to more frequently so are more likely to fail. So we wanna make sure that we've designed things where if I deploy a bad ship to the control plane and

take it down, nothing else is impacted and can still access the database, right? And anything in my data plane. So this is one of these kind of core principles that we take very seriously. And another thing is redundancy. And if any of you have worked with databases before in the past, you've probably seen a diagram that looks something like this, and similar things apply to your app infrastructure and your other infrastructure. Although in the case of application servers, usually redundancy is a little bit easier to

solve because it's a stateless workload, right? There's lots of sort of auto-scaling, Cloudflare Workers, and Vercel functions, and AWS Lambda that can scale and have redundancy very easily because it's not storing state. But in the database world, you have to store state reliably. You can't lose a single byte of data, and so how do you do this? Um, a very common approach which costs more money and is more complex, but it's the way that you scale reliably, is by having a primary node. So this is where most of your traffic goes to, but you always have replica servers

or follower servers standing by in different data centers or different availability zones ready to take over because the fact is servers do fail sometimes. Uh, and on a small scale, when you've got an app that has 100 users and you have one database server, you might only experience a failure once every couple of years. But at a large scale, when you have thousands of servers, a once or twice-a-year failure becomes a weekly or monthly failure, and so you need to have systems in place to deal with this very well.

Okay, so we've talked about, uh, what the principles of a- of scaling reliably are or of, um, having reliable systems are. But then the question is, what I've just told you here- We could write a whole book on, right? So there's lots more that, you know, there's people who spend their entire careers focused on this, so we're going a little bit quickly through it. But these principles apply no matter what point of this user growth chart you're at, right? You could be at 100 users, and in this case, uh, it's

still good to have redundancies, to have isolation, to have failure even for that smaller set of users, and you wanna apply those same principles when you are all the way up here. I don't know if you can see that line very well. But at millions of daily active users. But there is a point where you actually have to incorporate different technologies to scale efficiently and effectively. So, uh, let's talk about how we scale those systems for a few minutes, and probably the part you're most excited about is the AI agents part, which we'll get to at the end. Um, so one of

the key things, and this is what we do a lot of at PlanetScale, is sharding. Um, I just realized my mouse is on the screen, so let me get that off the screen here. See if that will go away eventually. Um, so database sharding. What is this? What you do is instead of relying on that single node, which works fine when you have ten gigabytes of data, it doesn't work so well when you have ten terabytes of data, or 100 terabytes of data, or a petabyte of data. So what do you have to start doing? Well, what you need to do is spread your data and your

queries out across many servers, and so that's what all of these shards down here below are. And typically you put some kind of proxy, and we'll talk more about what VTGate means later, in between that intelligently handles distributing all of the user requests or the query requests across all of these servers. Uh, and this is exactly what we do a lot of for some large customers at PlanetScale. We do sharding through Vitess for MySQL and Neki for Postgres. Um, and so this is a very fundamental way of doing sharding. But the other thing is you don't just use

sharding by itself. If we look at this and we see, for example, shard A, b- within that box you actually would want a redundant system. So you'd have a primary and several replicas in there. So if one of those servers fail, that single shard can do self-healing and get back online very quickly if there's a problem. So this is one of the ways that we see and is probably the most common and performant way to scale databases. But really the same principle of, hey, let's take queries and data and spread them out across many servers, many

nodes, many drives, uh, applies to lots of other areas of the stack that you're building. Backpressure. So this is another one that is sort of a, a mix of being a part of scalability but also a part of reliability. And what we mean here is there's a lot of systems like MySQL and Postgres and SQLite, these very popular databases that people run, that don't do a good job at handling being overloaded. The moment that they have too many queries or using too much CPU or they run out of RAM for the

cache, in many cases if not taken a lot of care for, they'll just crash. And now your server is down, and now all of your users are impacted and they can't use your service that the database is powering. What backpressure is, is it's designing a system to be able to push back and say, "When I detect that I'm above a certain threshold of resources, I can actually say I wanna push back and start failing requests, denying query, uh, denying connections, right, so that I can keep the existing users or some of the traffic healthy while gracefully degrading some

of the other traffic." And then the rest of your stack has to be built to support that. So this is a principle that applies in lots of areas of the stack, but in the database it's arguably the most important. Uh, let's see here. Where are the clicker? There we go. And then the last component of this that I wanna talk about before we move into how to do this all with AI is decoupling. Uh, there's a lot of people that you'll talk to, and when it comes to databases, will argue that you should probably have all of your services combined into one because there's some convenience to

that. Your Postgres database, your OLTP workloads combined with what you're doing for analytics, combined with your, your queuing and your job system, your caching. There is convenience and some technologies try to do this where we put this all in one, but that does one, add a lot of complexity, but it also breaks some of the fundamental principles from earlier like failure isolation, right? If one service goes down, that actually means everything is going down. And then you also have to deal with contention of resources. So if you suddenly have a burst of queue jobs

and you need to scale up your queues, but maybe nothing else really needs to scale, it would be nice if we can independently scale the amount of compute resources that we've given to each of these parts of our backend service powering your favorite application, right? So this is not a complete list. What did we go through so far? We went through sharding, we went through backpressure, and now we're talking about decoupling. There's many other aspects to scalability and again, you could spend your entire career becoming an expert in scalability, uh, but

this is a lot of what we think about a lot at PlanetScale because our customers rely on us to be reliable and to scale. So finally, let's go to the last part of this. We-- let's say we have all of these principles in place. We are doing things reliably where we have fault tolerance. We're doing things that are very scalable. But these days, in a lot of organizations, and we talk to a lot of people that feel this both at the small scale and at very large organizations, how can we allow

agents to, whether in part or in full, have control over what is arguably the most important infrastructure of your company, right? The databases, the queues, the things that are powering every other layer of your stack. Um, and this is something that can be a little bit intimidating because it's pretty much the standard these days, right, to write code with AI. In some cases, people are not even writing any lines of code anymore, right? AI is actually writing all of your code, reviewing all of your code, um, and you're just sort of guiding these agents to doing that.

But it is quite different to say, "I'm gonna now also give my agents the ability to deploy changes to the database or make changes to how things are configured there or to shard my database," right? Because that's something if it gets wrong, it's generally not as simple as clicking a button to roll it back, right? It's taking down your entire application and every user notices. So we wanna do this effectively and safely. So I'm gonna revisit a couple of the concepts that I previously talked about and think about how do we use agents to solve this. So

going back to sharding. What I'm gonna talk a little more specifically about is Vitess, and this is an open source project. Uh, we operate Vitess databases for our customers, and it works with MySQL to do sharding. So what you can see here is, like, all of those shards down there, those would be MySQL databases. And the VTGate, that's the proxy. That is basically an intelligent proxy that takes MySQL queries, parses them, figures out which shard they need to go to, and routes all of those requests in a very sophisticated way. So the software is fairly

complicated, but the interface that we present for specifying how do you want all of your queries and your data sharded is actually quite simple. It's what we call a VSchema, and it's literally a JSON file where you say, "Here's the column I wanna shard on, and here's the way I wanna do sharding." And you can specify certain tables to go on certain shards and have lots of control over where and how things are configured. And the great thing is, as we all know, AI agents are excellent at working with things like text files and JSON files. So you can give your agent and say,

"Hey, I'm struggling with scalability of my database. Here's my schema. I want you to come up and help me prototype a good VSchema," right? And once you've developed and designed something that it has thought through and made decisions about the most efficient ways to shard when you are on a platform like PlanetScale, you literally will plug this in and you make a deploy, and you, you c-- we give you an interface for creating new sharded systems and scaling up your database. And a lot of this can happen with either the help of AI or letting a-AI take some control over these processes, uh, given our API and our

command line interface that we run. So this is one of those things, right? Very complex infrastructure in reality under the hood that is scaling your systems, but you provide the right interfaces for it, like simple configuration files, and those things become much more accessible to any company trying to scale. Um, another thing, so this is getting into the, uh, concept of back pressure that we talked about earlier, right? So when a database is overloaded, and this, as you might imagine, probably happens all the time at some of these companies, right? I think there was a,

a tweet a couple of months ago from GitHub that basically said, like, their traffic had, like, two or three or four x from what already was an extremely high load, and pretty much all of that was attributed to AI agents, right? So they already had gigantic infrastructure 'cause they're the most popular place in the world to develop, and then they had multiple, uh, uh, traffic increases, especially over, like, the holidays, right, due to these, uh, overloaded systems. So how do we deal with this? One of our solutions and the way that we like to think about it is we give users a

system called Traffic Control, which allows you to essentially categorize and tag all of the traffic coming into your database, and then you create resource budgets to say, "Hey, this segment of my traffic cannot exceed this amount resources of CPU usage or of backend processes." And again, we do graceful degradation when that happens. So what I'm showing here is this is a graph of I have a budget set up, um, to say, "Don't exceed these amount of resources, and warn if I do start exceeding these resources." So it's showing me, uh, this is the amount of

queries that are exceeding your budget over time. And we're, we're, we're sending warnings back to the client like, "Hey, I'm overloaded. You need to slow down." And we can even make it more strict where we actually kill those queries that are executing, right? And this is one of those things that, like, okay, it's not ideal to have to deny requests, but it's much better than your whole application going down and being offline for all of your users. Uh, and then finally, one other thing I w- I wanna talk about, and I know I'm showing this is the PlanetScale UI here, but this is not

even so much specifically about PlanetScale, but it's about the philosophy of how we do these things with agents, which is most developers are used to using a Git-like flow when you're building, and this applies to your agents as well, right? You have a main, you create a branch, you make code changes, you do merge that in, and you deploy that dynamically to wherever your application is deployed. What we have done, and this was long before agents even existed, is we wanted to take that same UX and mirror that for your database. 'Cause a lot of the times when you're building new

application features, you're also making changes to your schema. You're adding tables, you're adding columns, you're deleting things, you're, you're adding indexes, you're doing all of this stuff to your database to keep it in sync with your code base. So what we give you, powered by that same Vitess earlier, is the ability to branch your database schema, make changes to that schema in an isolated environment, and then merge that back into production with what we call a deploy request, all with no downtime. And not only that, but actually the ability you might see here at the bottom, the ability to, if you

deploy a schema change and you see a problem, we give you the ability to one-click revert that and go back to the old state of your database without data loss. Uh, and so this is something that was great for humans, but we also have the APIs and the CLIs to be able to let agents automate this. So when they are doing things and automating code and branching and making pull requests and doing code review, there can actually be a mirrored database state that is working in sync with all of those things, right? And that can either be fully automated, it can be

semi-automated. Obviously, many of us are working in a human-in-the-loop fashion where humans are still reviewing all of these changes that AI agents are making, which is a good thing. Um, but we have the primitives in place for building these, uh, systems. And there's a number of other things that I didn't get to, to talk about this, but kind of the, the ending point that I wanna make is we're a company, and I know a lot of companies out there like to focus on the developer experience. And when I say developer experience, I don't just mean, like, light and dark mode

switches and good keyboard shortcuts and all of this, but I mean the actual experience of, hey, if you're a developer that's responsible for your company's database or responsible for some piece of infrastructure, do we give you the tools that you need to not mess up, to do deploys safely, to make your database faster, to get better performance? Uh, and we've been building these things for years, and it does kinda turn out that when you really focus on that core of the developer experience, what you actually end up with is also very good primitives for AI being able to work with a database

safely and reliably and helping it scale. Uh, and so because we expose a lot of these things through our MCP server, through command line interfaces, obviously there's plenty of debate on which of those is better when working with your AI agents. But because we expose those things to both parties, it's actually a great experience for operating your database and working well with AI agents. So that is all that I have to say. Thank you guys for coming. I'm Ben, and, uh, have a great rest of the conference.