Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
2:49:48How are we doing?
2:49:51So this is a talk about compute markets, but it's kind of more of a talk about why to make
2:49:57a compute market.
2:49:59If
2:50:00you don't know me, I'm Evan.
2:50:01We run a company called San Francisco Compute.
2:50:04We make a physically settled order, physically settled forwards market.
2:50:08So if you were here for the last talk, the last talk was basically why all the operators of GPU clusters don't really work very well.
2:50:17When we say physically settled, we mean we're the operator.
2:50:21We are both the operator of the hardware in the same way that a cloud is the operator
2:50:25of the hardware, and then we are also the market at the same time.
2:50:29It's kind of like a grid for compute.
2:50:31So the general gist of this is that in order to run and operate a supercomputer, we think
2:50:38problems start right at the design of the supercomputer, right in the design of the
2:50:43bill of materials.
2:50:44It's control of the backplane fabric as well.
2:50:48It's also control of bare metal orchestration.
2:50:50It's also services on top, VMs and so on.
2:50:53There's a lot of different parts of the stack that you have to do effectively.
2:50:57Otherwise, the folks who have a data center and would like to make more money on that
2:51:04data center by becoming the GPU cloud end up failing for the customer.
2:51:10So that's what we do.
2:51:12We offer what's called supercomputing as a service.
2:51:15We go to folks who want to become a cloud.
2:51:16that's like folks who want to become what NVIDIA would call an NCP,
2:51:21or venture firms that want to have clusters for their portfolio
2:51:24and want to be able to reallocate that capacity
2:51:27when a company like Instinct is taking off,
2:51:30or AI companies who would like the residual value on the compute
2:51:33because they've realized that's really important to their financials
2:51:36but don't want to operate a supercomputer.
2:51:38So that's physical settlement.
2:51:40And then on top of that, we run a live order book
2:51:42for capacity that you can buy reserved,
2:51:44served, either right now or in the future.
2:51:49This together is not at all what we thought we were going to be doing.
2:51:54Originally, we were this company called JunLark that was trying to make kind of like Suno or
2:51:59UDO, but prior to Suno or UDO.
2:52:01At the time, what you needed to do to realistically start a lab was you needed to raise a big
2:52:08deposit, a huge one, which was $10 million at the time.
2:52:12We thought that was a crazy amount of money, so we thought, no, that's stupid.
2:52:16We're going to raise 500k and obviously what we're going to do is rent from somebody else.
2:52:21Obviously today it's not $10 million, it's a billion dollars.
2:52:27And nonetheless, we had raised our 500k, we buy a big cluster, and then we're underwater
2:52:33on our cluster every month, and if we didn't sublease our cluster, we would have gone bankrupt.
2:52:38If you have heard of GPU brokers, we were the first GPU broker, so the first year the
2:52:44company was us trying to sell our cluster, otherwise we would go bankrupt.
2:52:47So I would wander around with these actual pieces of paper at NeurIps, which both say
2:52:53H100 and A100 at the same time, because that's how I made the posters so quickly.
2:53:01And so this talk is not really about necessarily how a compute market works, it's how a compute
2:53:08market gets to exist.
2:53:11We like to think about it like there's this pamphlet, and on the pamphlet, on the front
2:53:15side, it's like, man, starting a Neolab sucks, there's all these different financial and
2:53:19economic problems.
2:53:20On the back of the pamphlet, it says compute markets.
2:53:24It's not super clear how one gets to the other until you open the pamphlet and then
2:53:29And holy shit, there's so much stuff in this pamphlet.
2:53:31And it keeps going.
2:53:32There's so many folds.
2:53:33It's a difficult and challenging problem.
2:53:35Because you need to solve two miracle problems to get working.
2:53:39You have to both make a market, which is a uniquely difficult problem because it has
2:53:43a cold start problem.
2:53:44You also have to build a cloud.
2:53:45And you have to not become a cloud.
2:53:47Which is a whole separate challenge.
2:53:50But these are the problems that the labs are facing.
2:53:53And it's about to be the problems that everyone is about to be facing.
2:53:55So if you work at a start-up, these are your problems now.
2:53:58Right now, everybody is sold out of compute.
2:54:01This is probably not going to change for a while.
2:54:05So that means you're going to have to buy a new cluster, a net new one, and there's
2:54:09a lead time.
2:54:10You have no way to pay for that cluster to come online now.
2:54:13You're going to pay for it to come online later.
2:54:14There's nothing in the market that can realistically get that cluster online for you now.
2:54:19Meanwhile, you're going to pay your entire bank balance as a deposit.
2:54:23The reason you're paying your entire bank balance as a deposit
2:54:25is because the lender has no idea how to underwrite AGI.
2:54:29How in the world are you supposed to do that?
2:54:31You've got two-person startups in San Francisco, or maybe New York,
2:54:35who say, I'm going to build superintelligence.
2:54:39No lender knows how to underwrite superintelligence.
2:54:42So they have no idea what the asset is worth.
2:54:45Meanwhile, your cluster is very likely to be delayed.
2:54:48That's because the lender needed to underwrite against the credit risk of the customer,
2:54:51and so they need to know who is the customer first
2:54:54before you deploy the cluster.
2:54:56So right when the customer signs
2:54:57is the riskiest point in the entire operation.
2:55:00You wanted it to be the case
2:55:01that you built the cluster first and then you sell it.
2:55:04This also helps reduce all sorts of different types of problems
2:55:07where clusters break.
2:55:09Meanwhile, most of the people who are setting up clusters
2:55:11are people who have the actual asset
2:55:13that makes the thing cheap.
2:55:14Like you've got the data center space already.
2:55:17What that doesn't get is supercomputing expertise.
2:55:20as we saw in the last talk, if you've watched that one.
2:55:25Meanwhile, you're going to run into the exact same problem
2:55:27that Junelark did,
2:55:28which is you're going to either way overbuy
2:55:30or you're going to way underbuy.
2:55:32That's because AI doesn't have margins
2:55:34that look like SaaS companies.
2:55:35And so you don't have very much margin for error.
2:55:37And so the chances that you either underbuy
2:55:40and somebody else wins because they did it correctly
2:55:42or you overbuy and then you just become stability AI
2:55:45or you blow up, you end up like character AI,
2:55:48are really high.
2:55:50So, these are all the same problem. They're all just different types of risk.
2:55:53They're technical risk, they're financial risk, they're credit risk.
2:55:56They're all one big bundled risk problem, which is the problem that we end up solving,
2:56:00and it's also the problem that we think compute markets broadly, the entire industry solves.
2:56:05So we'll walk through each one of these and walk how that gets to a compute market,
2:56:08because if you don't understand these, it's kind of unclear why you would want a compute market,
2:56:12and they almost seem like this problem that somebody thinks,
2:56:15oh, wow, compute markets are cool, so I'm going to go start one.
2:56:17We think they should be derived from the actual problems that people face,
2:56:21which is, first, everyone is sold out.
2:56:23Why are they sold out?
2:56:24Because the lender couldn't finance the cluster
2:56:26unless you had a long-term off-take agreement.
2:56:28In CPUs, this wasn't the case,
2:56:30because CPU buyers are SaaS companies.
2:56:34SaaS companies have high margins,
2:56:35and because they have high margins,
2:56:37you can build a product that rewards you with high margins for your CPUs.
2:56:48Think about it this way.
2:56:50If you're a SaaS company and your underlying cloud doubles their prices, you don't go bankrupt.
2:56:55You hate your cloud, you're mad, but you don't go bankrupt.
2:56:58If you're an AI company and your underlying cloud doubles their prices, you go bankrupt.
2:57:03And what that means is that you're actually price sensitive, and because you're actually
2:57:07price sensitive, there's nothing keeping you at any particular cloud, so you can't sell
2:57:12on demand because your customers will immediately leave to whoever is cheaper.
2:57:16And so that means that in order to finance GPU clusters, you have to have long-term take-or-pay
2:57:22contracts, hopefully with someone who has investment-grade credit.
2:57:26However, if you don't have liquidity on those, if you're selling reserved and your customer
2:57:30can't get out, then that also means that some other customer who wants the capacity right
2:57:34now because they're like instinct and their product is taking off, can't pay for anything.
2:57:38They have to finance a new cluster which has a lead time because the physical world just
2:57:41exists and it's going to take a while.
2:57:43Sometimes you even have to build the damn data center.
2:57:46And then, because you're financing a new cluster, you're going to pay a deposit.
2:57:49Because almost nobody in AI has any reasonable credit.
2:57:53And because this is a brand new technology, all these companies are new.
2:57:56Meanwhile, even as you start to scale up that ladder, you're still going to run into credit
2:58:01issues all the way through.
2:58:02It's the number one bottleneck of AI at the moment.
2:58:04It's not power.
2:58:05You hit credit way before you hit power.
2:58:11So, what that means is every inference company, every neo lab, they are bottlenecked by how
2:58:16much cash they have in the bank account.
2:58:18So they go back to the venture capitalist who says, oh, I'm going to fund at the model
2:58:22layer, the application layer, the inference layer, because I don't want to fund all of
2:58:25that capex stuff.
2:58:27And so the venture capitalist then funds all the capex stuff by giving the new neo lab
2:58:34or new inference company, the check that they then send to the cloud, which is the deposit
2:58:40for the cluster, and that's how you underwrite the customer.
2:58:45So the clusters then are delayed because the cloud couldn't actually deploy the cluster
2:58:52until they had the customer in hand, and so that was the riskiest point.
2:58:56So therefore, what should have happened was the cloud had bought the cluster ahead of
2:59:02time, and then waited until it was landed in the data center, and then sold it.
2:59:07And that means you have way less likelihood of delays.
2:59:10Meanwhile, the cluster is very likely to break when you get the cluster as the customer because
2:59:15you don't have liquidity.
2:59:16And so if the failure rate is higher than what you expect, and you don't have the ability
2:59:20to transition a node in, then you don't have anywhere you can pull nodes from.
2:59:24Which means that if you need new nodes to make up for the failed ones, you have to RMA
2:59:28them.
2:59:29which means you have to get a new node.
2:59:30And then that gets us back to several steps before
2:59:33where you have the lead time problem again.
2:59:36And so everything breaks for everybody all the time.
2:59:40Once you have gotten through all of that as a new AI lab,
2:59:42your problems are you, you know, you, nobody was sold,
2:59:46like everyone was sold out.
2:59:48You then had to wait a lead time.
2:59:49You paid this big deposit.
2:59:51The thing was delayed and then it broke.
2:59:52And finally you got through all of that.
2:59:54And the likelihood that you predict your demand correctly
2:59:56is very low.
2:59:58And so you're either going to be way
3:00:00over-provisioned or way under-provisioned, and you have no way out.
3:00:02There's no way to sell your capacity to somebody else, and there's no way to buy more capacity if you need it.
3:00:06So, compute markets.
3:00:09To solve these problems, you need to solve a bunch of different problems all at once.
3:00:13You need to underwrite the value of the GPU cluster rather than underwriting the credit risk of the customer.
3:00:20This is how it could work.
3:00:22It's not guaranteed to work like this.
3:00:24In other industries, what happened is everything just consolidated.
3:00:28You got one big credit-worthy customer that does the whole damn industry.
3:00:32That's what the path of AI is currently on track to do.
3:00:36It's instead of going to be a liquid market where anybody can participate,
3:00:40it's going to look like TSMC.
3:00:42You're going to get one buyer or a few buyers that are the clouds.
3:00:46You're going to get a few model labs that are the AI labs.
3:00:49And then you won't get other startups.
3:00:52This is what we think is going to happen.
3:00:54And this is why three years ago, when we were doing our AI lab, we had a chance.
3:00:58We could have said, no, we're not going to stick it out for several years just trying to keep our company alive.
3:01:06We're just going to go do a new AI lab.
3:01:08But we saw this problem.
3:01:10And we said, oh, shit.
3:01:12If we don't do anything about this, then we will just hit this three years from now.
3:01:17And here we are three years from now hitting this problem and where everyone is going to hit this problem.
3:01:22If you watched the last talk, it's very hard to solve this problem unless you are operating
3:01:28both the market and the clusters at the same time.
3:01:33This is Dylan Page.
3:01:34Dylan is the co-chair of the CNCF infrastructure group.
3:01:36That's the organization that makes Kubernetes.
3:01:38He formerly worked at Lambda doing infrastructure there.
3:01:41And he joined San Francisco Compute because GPU Compute isn't sufficiently reliable for
3:01:48the market operator to be divorced from the physical infrastructure.
3:01:50It doesn't work if you aren't the one who is buying the thing.
3:01:53Sorry, it doesn't work if the person you are buying from isn't operating it.
3:01:56To give an example of why this doesn't work is I'm sure some of you are flying in from
3:02:00overseas or out of town and you have a hotel room.
3:02:06Wouldn't it be great if you could sell that hotel room to somebody else?
3:02:10I go to you and I want to buy that hotel room and I say, can I get room service?
3:02:16Can I get my room cleaned?
3:02:18I've got a problem with my room, who do I call?
3:02:21If you don't operate the hotel, you can't fix any of those problems.
3:02:25That's why brokering clusters doesn't work.
3:02:28You have to operate the cluster like the hotel.
3:02:30You have to run the whole damn thing.
3:02:32Otherwise, you can't actually make a market.
3:02:34It doesn't work.
3:02:35You can't do physical settlement.
3:02:37So why does nobody do physical settlement?
3:02:40The problem is there's a very strong incentive to never do physical settlement.
3:02:44That is because if you build out physical settlement for compute, what that literally
3:02:48means is you are building out the entire operating capacity of a GPU cloud.
3:02:52And if you are building out the entire operating capacity of a GPU cloud, you should just be
3:02:57a GPU cloud.
3:02:58Because it looks to be the case that you will make an ungodly amount of money.
3:03:02You will sign billion dollar deal after billion dollar deal.
3:03:06And so, fuck it.
3:03:08Let's not make a GPU cloud, or let's not make a compute market.
3:03:10Let's instead just become a GPU cloud, which is what basically everybody does, because
3:03:15the numbers are astronomical and it's a very reasonable thing to do.
3:03:18The reason why we ended up building physical settlement was because we knew that if we
3:03:22didn't start this, we would then not be able to start an AI lab, which was the real damn
3:03:26thing that we wanted to do in the first place.
3:03:27We wanted to make a lab.
3:03:28We didn't get to make a lab unless this exists.
3:03:32So when we started, it was 10 million as the deposit that you had to pay to start a new
3:03:38lab.
3:03:38This is how you get over the credit risk problem.
3:03:42Then the cluster sizes got bigger, the sort of minimum ante into the space got bigger,
3:03:47then it was 100 million, now it's a billion.
3:03:49We don't think it can continue.
3:03:51We don't think there is a world in which you can get a $10 billion round right out the
3:03:55bat before you have revenue or traction or your product or anything.
3:04:00And what that means is that if you don't solve this problem, we are currently on track to
3:04:05not have any start-ups in AI.
3:04:07Now, one of the things that you could think could happen is maybe there can be new inference
3:04:13companies and new model labs.
3:04:15And the way that they solve the credit problem is that instead they just ship the risk to
3:04:19the application layer.
3:04:20They say, look, I need to get over my credit problem, so what I'm going to do is I'm going
3:04:28to only sign customers who are investment grade credit, you know, Fortune 500, I'm going
3:04:33to only sign customers on five-year contracts for tokens.
3:04:37I'm going to only sign customers for tokens that require a big deposit up front.
3:04:44That is the end of everything.
3:04:48you have inference layer getting bottlenecked by credit, what happens is you can't get tokens
3:04:54unless you have credit or you have a big deposit.
3:04:57the problem of that is that tokens are used by the whole damn economy.
3:05:01There is no rule that says that startups have to exist.
3:05:05If every startup is suddenly bottlenecked by credit in order to access intelligence,
3:05:11what happens is existing firms just ship everything.
3:05:14They themselves get to make the product because they've got credit,
3:05:19and then they make the revenue from that product, and then it makes their credit better.
3:05:22And that's a flywheel that makes consolidation happen throughout the entire economy.
3:05:27If you don't fix this bottleneck, there aren't any startups.
3:05:30This is why we started doing this.
3:05:32Because at one point, what we think is going to happen is you're going to end up in three
3:05:38to five years, you're going to look out and say, I want to start a new startup.
3:05:42To get there, you need to buy so many tokens in order to compete with the existing players.
3:05:50You need to buy $100 million of tokens.
3:05:54We think this is what you're going to have to do in order to start a new company because
3:05:57other people are spending hundreds of millions of dollars of tokens.
3:06:00If you just draw the line out, this is what happens.
3:06:02So you need a big giant deposit because you're two people who have no idea what they're doing
3:06:07and so nobody trusts you.
3:06:09So you're going to go and raise $100 million from your venture capitalist.
3:06:13Meanwhile, nobody wants to work with you because you're a two-person startup.
3:06:17Then you're going to be asked to pay for that hundred, or you're going to be asked to pay
3:06:21for a five-year contract on that hundred million if you're not paying it all as a deposit.
3:06:25And then you just don't get new model labs, you don't get new start-ups.
3:06:28You get a consolidation in the industry.
3:06:31So we, at one point, were this little lab.
3:06:36And what we did is we lived in San Francisco and we knew all the other people who were
3:06:41also just two little folks who had started a start-up and wanted to be able to participate
3:06:46in the most important technology of our time.
3:06:50So I want these people to exist.
3:06:52I want this person to exist and I want this person to exist.
3:06:55I want this startup to exist and this startup to exist and this startup to exist.
3:06:59This one and this one and this one and this one and this one and this one and this one and this one.
3:07:03If you get these companies going, maybe they'll become big folks that build your cancer cures
3:07:09or they build something that nobody thought of or something that wasn't just sort of AI slopped out.
3:07:15This is kind of the problem that compute markets solve.
3:07:18And if you don't solve it, we don't think there's going to be any startups.
3:07:22Cool.
3:07:24That's it.
3:07:25That's it.
3:07:25Yeah.
