Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
3:15:09All right, I think we can get started here.
3:15:12Hey, everyone.
3:15:13My name is Affaan, and I'm a lead for compute trading at Jump Trading, which is a proprietary
3:15:18trading firm, alongside my co-lead over here, Alejandro.
3:15:23So today we're going to be looking over some of the market structure as it applies to compute
3:15:28trading, what compute trading even is.
3:15:31what we can see it becoming.
3:15:34And I think we'll start with some macro,
3:15:37look at microstructure,
3:15:38and then talk a little bit about
3:15:40where we see things in policy.
3:15:49So the title of the talk is
3:15:52The Dark Pools of Compute,
3:15:53but some of you guys are in finance,
3:15:54so you might be confused as to why I'm using this terminology.
3:15:57What do I mean by dark pool here?
3:15:59Essentially what I'm referring to
3:16:00is an unlit market. Right now, most compute trades in blocks, physical compute here. And
3:16:07these are private quotes, we have a lot of brokers, and these are long bilateral contracts.
3:16:11Now, at the end of the day, these are really commercial agreements. So we're using the
3:16:16term trading here to kind of like look at how it's transacted between parties. These
3:16:20commercial agreements tend to be some mixture of an MSA, an SLA, and an order form, cluster
3:16:26specific with hardware specific language in each order.
3:16:34But when we're looking at pricing and looking at some of these other metrics, a lot of the
3:16:38time most of the talk is around the SKU, so what the chip itself is, the baseboard.
3:16:43So H100, H200, V300, etc.
3:16:47And so a lot of the negotiation here happens kind of outside the public view.
3:16:53There's no exchange for physically traded compute that's widely adopted and used for everyone goes there to take a look and trade compute.
3:17:03These are done in chats.
3:17:06They're out of the public view.
3:17:07There's public marketplaces.
3:17:09There's price references.
3:17:10There's a lot of on-demand
3:17:12places. However, for these actual tenured, so on a given term for a fixed amount of capacity
3:17:19with a certain total contract value, these trade over Slack messages, X messages, Telegram group
3:17:25chats, so a bunch of OTC kind of channels that are all over in these kind of, I want to call them
3:17:30like Neo chat. And some of you might be familiar with commodities and how a lot of, you know,
3:17:36There's chatting happening on Ice Chat, Bloomberg Chat, etc.
3:17:40This is kind of where the markets here lie.
3:17:45And so I want to look a little bit here on the macro market structure
3:17:48and a little bit on the players here as well.
3:17:51And so this was a recent kind of A16Z.
3:17:55Some of you guys might have seen this, so I'm putting this out there
3:17:57because you can go on A16Z's actual X and take a look at this graph yourself.
3:18:02What we're looking at is that the cost of a given capability
3:18:04has fallen sharply.
3:18:07The bill, of course, still depends on the model usage, capacity,
3:18:10and other factors, such as speed,
3:18:12is a big thing right now with the new ChatGPT models.
3:18:17Cheaper tokens unlock more usage,
3:18:19and a team can do more work for the same budget.
3:18:21So what we're seeing is that a lot of teams
3:18:24are simply just using way more tokens.
3:18:26So we have this latent demand that hasn't really been hit yet,
3:18:28and people are just going to use more with the same budget
3:18:31that they were using before when models cost more.
3:18:34and they're just spending more tokens.
3:18:37So applications that were too expensive become possible.
3:18:40However, as we're kind of talking about compute here,
3:18:42the capacity underneath still has to be built and paid for,
3:18:46even as the cost of a particular task comes down.
3:18:48And so the important distinction is between the price of the intelligence,
3:18:52the number of tasks people want to run,
3:18:53and the amount of hardware needed to serve them.
3:18:58And so this is where we get to kind of the Javon's paradox
3:19:01and a little bit of some of the stuff that we've been looking at lately.
3:19:04where we had a very interesting thing where, you know,
3:19:08intuitively you'd assume that compute and inference
3:19:11kind of like rises together, correlates very, you know,
3:19:15very closely, as you can see.
3:19:16You can see there's a little bit of a decoupling that happened,
3:19:19and token prices declined further,
3:19:21but compute prices remain robust.
3:19:23And so, like, what can be an explanation for this now?
3:19:25There's a lot of, like, potential explanations.
3:19:28One here is the whole Javon's paradox,
3:19:31is that even though token prices and inference
3:19:33is getting extremely efficient to serve
3:19:35on these new baseboards and GPUs,
3:19:38people are just consuming a lot more.
3:19:40And so the demand for compute rises
3:19:42and people just consume a lot more tokens,
3:19:44even if the cost per million for input and for caching
3:19:49and for output becomes a lot more efficient.
3:19:52And so making the tasks cheaper,
3:19:55encouraging the additional use
3:19:55to outweigh resource saving for tasks.
3:19:57For inference clouds, that's going to be more applications,
3:20:00serving more tokens, bigger market share. And for AppLayer individuals, that means you guys can serve to a lot more customers, anyone who's utilizing the token infrastructure.
3:20:11structure.
3:20:13And so what really goes into the macro structure, now like this is very bare bones here, but
3:20:20you've got your chips, which we've been talking about, you have power, you have buildings,
3:20:25grid connections, and then of course you need a lot of capital to get these build outs going
3:20:29and that's kind of been the whole talk here.
3:20:32And so chips, power, buildings, money, they all have to come together before you can even
3:20:37get to an hour of actual usable GPU per hour compute.
3:20:41And so the IEA April update projects that the global data center electricity is going
3:20:45to rise from 485 terawatt hours in 25 to 950 in 2030.
3:20:51And so a lot of the talk right now is around build outs about getting the power grid set
3:20:56up.
3:20:56Will we have enough power grid here?
3:20:58Is there other energy sources that we can tap into?
3:21:01Where are we able to get the land to actually be able to do this?
3:21:05And so, co-location sites and data center operators and people who are doing the financing, the build-outs, or the lenders themselves, are all kind of looking into this.
3:21:11And that type of everything that's upstream, essentially, of the compute is going to then flow down into the compute costs and also usage.
3:21:21And so, what does this, like, actual macro map look like underneath a quote?
3:21:25Chip and system vendors supply the GPU service and networking.
3:21:29Sites and operators supply the power space assembly and management.
3:21:34right here, GPUs, memory service, networking.
3:21:37Capacity providers, these are
3:21:38hyperscalers, neoclouds, and any
3:21:40other operator that's operating similar
3:21:42to how a neocloud would operate, reselling
3:21:44bare metal capacity, and then the users.
3:21:46So who's the actual end user? The startups,
3:21:48the inference clouds,
3:21:50labs, neolabs, and also anyone
3:21:52on the app layer, and enterprises
3:21:54for anyone doing open-weight
3:21:56stuff as well.
3:21:58And these boxes,
3:22:00I've kind of boxed them out here, but
3:22:03there's people
3:22:04People can be all of the above.
3:22:06A chip and system vendor, someone could do a full end-to-end build out and also be the
3:22:10end user of everything.
3:22:12And so there's a little bit of variation that occurs here.
3:22:23And so as I was kind of mentioning about the inference clouds before, they might rent a
3:22:28cluster and they'll serve tokens off of it and they'll use that revenue to pay off the
3:22:34And so that might be a use case for the actual inference cloud themselves.
3:22:37However, for hyperscalers, frontier labs, and larger NeoClouds,
3:22:40they can make substantial commitments
3:22:41because they have the ability to get into the large equity,
3:22:44they have the ability to take on more debt,
3:22:47and they have the ability to do long horizon kind of deals.
3:22:51So 10-year deal, open AI, contract it out, et cetera.
3:22:55So of that nature.
3:22:56However, anyone in the middle market isn't actually able to do this.
3:22:59Who's underwriting a Series B startup
3:23:01with enough capacity to get in a $200 million plus contract
3:23:06over three years for compute.
3:23:08How many equity rounds will they have to raise, right?
3:23:10Will they have to use their entire equity round
3:23:12to fund just the actual cluster
3:23:14and then they don't have any remaining capital
3:23:15left to use for actual costs?
3:23:20And so we're looking into a little bit of the financing chain here
3:23:22and how that plays in.
3:23:23So lenders are financing the build-out
3:23:26against contracted revenue even in many cases for inference clouds.
3:23:29And so they want to finance the build-out.
3:23:30They want confidence that their loan will get repaid.
3:23:33I mean, like, how are you going to underwrite a startup,
3:23:35like a private company that doesn't have, like,
3:23:37much else going for them right now?
3:23:38Like, what are the, you know, they're not publicly traded.
3:23:40There's no that much cash flows, balance sheet, income statement.
3:23:42They might even be pre-profit, pre-revenue, et cetera.
3:23:45And so what they're looking at is, like,
3:23:46can the person get contracted revenue over a certain scale?
3:23:50And so Core Reef's public filings give large-scale example
3:23:52of infrastructure financing supported by
3:23:54take-or-pay customer contracts.
3:23:56And so this diagram kind of shows the smaller,
3:24:00illustrated example here
3:24:01in the mismatch is that
3:24:03hardware financing, they want very long terms,
3:24:05but customers are only able to afford
3:24:07or don't really want to get into a long-term
3:24:09financing over 36 months.
3:24:11They have to put up on after 15% to 25%
3:24:13is what CoreWeave has quoted
3:24:15that they've seen their median
3:24:17prepayments to be, 15% to 25%
3:24:19up front
3:24:21on a 100 million TCV cluster.
3:24:23That's $15 to $25 million
3:24:25that a startup has to put up front.
3:24:27That might be their entire round.
3:24:30And so this is kind of a look at what I'm
3:24:33referencing here. It's like you're raising $40 million, you need a $100 million contract
3:24:37over three years, you put down 30% prepayments, this is an illustrative example,
3:24:41you've got $10 million left to run the business, and so you might have this
3:24:44kind of step function as you can see here, so that's your contract at capacity
3:24:49and what you can actually, and the demand is below that.
3:24:52And so now every time that you have to get into a new cluster, it's like tied to your
3:24:56equity round or whatever else it may be, but you might have to cap out and turn away customers.
3:25:00So what are your options? You either turn away customers until you can get into your next
3:25:05equity round so you can put down another 30% prepayment so you can get
3:25:08the next little bit of capacity going, or do you use on-demand to close the area
3:25:12under the gap and bridge the shortfall? That's like an
3:25:17option now. Now you have to look into the economics. What does that look like? What's the difference between
3:25:21maybe instead what they do is they go to a lender to finance the prepayment component,
3:25:24But if the yield on the prepayment component that they're getting financed is less than them just buying it out on demand at that time, if they're able to get into that amount of capacity, then does the economics make sense there?
3:25:41And so here's some of the operating models that we've seen.
3:25:44It's like you can build and operate the full stack.
3:25:47Hardware, site, your team is going to arrange power permit, the assembly, the management.
3:25:52You can also own servers in a co-location facility with more or less assembly and operation
3:25:57handled by a partner.
3:25:58On the other hand, an off-take contract buys defined capacity and the provider is going
3:26:03to own and operate the asset and they might not give you complete root access to the bare
3:26:07metal or there might be levels to how much access you get depending on if it's tenant,
3:26:11multi-tenant, et cetera type of configurations that can also affect the pricing.
3:26:17And so some businesses are going to value the physical access.
3:26:20Some businesses are going to value confidential compute.
3:26:25They might value certain topology changes
3:26:27or the ability to reconfigure the system.
3:26:28And so if that is something for your business,
3:26:30then they might actually want to have more and more control
3:26:33and build and operate the stack.
3:26:34And the economics for them might make sense
3:26:36given the opportunity costs they would have to give up
3:26:38if they were to just take rental off-take contracts.
3:26:41So they might be willing to pay more there.
3:26:50And so here's how, like, ownership and rental, this is a very, like, crude graph, how ownership
3:26:54and rental can compare as utilization arises.
3:26:57And so what we're looking at is that usually, you know, an off-take contract or these off-take
3:27:03contracts are getting to the cost of actually almost building out and financing your own
3:27:07cluster.
3:27:08And it's not, because, I mean, if you're spending $100 million over a three-year period,
3:27:11then maybe if the economics makes sense, it would have made more sense to simply buy out
3:27:15the compute operator yourself outside of the time cost and then obviously the operational
3:27:19costs that might exist. So there's an equilibrium that exists that has to be followed where
3:27:26the ownership costs outweigh the rental costs, and the rental costs should sit below the
3:27:35ownership costs. But if you're utilizing at such a rate and in such variable kind of increments,
3:27:40it might be best to just own the stack outright.
3:27:47And so we're seeing this new dynamic
3:27:49that the compute procurement
3:27:51affects how much runway a growing company actually has.
3:27:55And so this also gives investors
3:27:57an interest in compute procurement.
3:27:59So we're kind of seeing this.
3:28:00Access and payment terms affect how much runway
3:28:02their port codes are going to have
3:28:04from a VC perspective.
3:28:05So helping a company buy a suitable capacity
3:28:08can improve the underlying investment.
3:28:10And they're very connected, right?
3:28:11So they're able to procure
3:28:12and they're able to actually find compute capacity
3:28:15for their port codes and fund that.
3:28:17And so that also brings another interesting aspect
3:28:20to the macro structure.
3:28:21Into the micro structure is that
3:28:22who helps place the capacity
3:28:24and how does this offer get compared
3:28:26and what risks these contracts can transfer.
3:28:32So we have a lot of different players
3:28:34around this GPU hour
3:28:35that are helping kind of shape this market here.
3:28:37You've got your price discovery individuals.
3:28:39So people with indexes, data,
3:28:42benchmarks that we're all looking at right now and referencing which would be
3:28:46used to you know in the future launch the futures based on the benchmark it's
3:28:51you know talked about in the associated contracts per spec per pending approval
3:28:58and then there's the physical matching venues auctions aggregators so anyone
3:29:02who can match capacity we're looking at essentially decentralized marketplaces
3:29:07for on-demand compute. You're looking at liquid pools of compute that offer liquid pool services
3:29:13and part it out themselves and have their own standardized contract. You're looking at bare
3:29:17metal resellers. You're looking at neoclouds themselves doing larger capacity sales. And
3:29:22then you're looking at individual brokers and individual operators here as well on the bare
3:29:25metal reselling. You have destined brokers. They're looking at the bilateral quotes and execution.
3:29:30They might know a lot of people. They might be able to move fast. They might be able to see quotes
3:29:35have very similar specs in one place
3:29:36and know that there's a buyer
3:29:38that is extremely inelastic to price right now,
3:29:41needs to compute immediately,
3:29:42and they can get the RFS in time
3:29:43so they can charge a decent spread on it.
3:29:46And then what they do is they're going to have the source
3:29:48negotiating, and then they're going to be carrying
3:29:49and placing the risk.
3:29:51They're going to have to operate that cluster for the person,
3:29:52but it has to be worth that spread.
3:29:55And you have looks in financial markets,
3:29:57the exchanges that are coming that are based on the GPU
3:30:00hour, the futures and derivatives.
3:30:02People are going to be using it to take a look at how to maybe hedge out and look into more financial structures.
3:30:07So hedge out risk, so hedge out balance sheet risk, maybe get into more interesting agreements.
3:30:13We're seeing kind of agreements such as prepayment, payment facilities are coming online.
3:30:18we're seeing people utilize futures
3:30:20and are interested in utilizing futures
3:30:22and other kind of
3:30:25options to be able to
3:30:27there's demand for this
3:30:28to be able to hedge out the risk that they currently have
3:30:30on the books. We have buy side
3:30:32individuals looking for flex contracts
3:30:34flexible contracts like hey
3:30:35is there ability that if I lock myself into a 3-5 year
3:30:38contract that I'll be able to
3:30:39at least upgrade one of the new chips that comes out
3:30:41or maybe I'll be able to roll it over or maybe let me sell it back to you
3:30:45and then there's
3:30:46credit and financing so equipment lenders, leasers
3:30:48specialist credit providers.
3:30:51And so the way we're seeing it right now
3:30:52is that every deal is bespoke.
3:30:55There's different spec sheets, there's formats.
3:30:57The bill of materials may not match
3:30:59the actual delivery,
3:31:01or there might be a subject to change
3:31:03if a part is missing or a part is out of stock,
3:31:05and there's very long lead times
3:31:07that some of these OEM and ODM
3:31:09kind of providers have.
3:31:11And the same term can mean
3:31:13different things when you're negotiating very quickly.
3:31:14If someone's talking about memory,
3:31:16are we talking about, you know,
3:31:18VRAM, are we talking about the CPU memory,
3:31:20are we talking about additional storage here?
3:31:22Like, what did you exactly mean there?
3:31:23It affects quite a bit what type of loads can be run
3:31:26and, you know, the actual end users' needs,
3:31:29and also with pricing and delivery dates.
3:31:33So there's a lot of back and forth
3:31:34before anyone can really compare,
3:31:35and this is where the concept of, like,
3:31:36non-fungibility kind of comes into place
3:31:38and how to partition that out,
3:31:40what parts can be fungible,
3:31:41what parts hold the most of the weight,
3:31:43such as the skew, et cetera,
3:31:44and what parts are still, like,
3:31:46traded as if it was a product.
3:31:47e-commerce, et cetera, where it's only for that particular cluster that this price applies.
3:31:54And so prices can update daily or faster.
3:31:56Quotes are still a good starting point.
3:31:57Bids can get shopped.
3:31:58Gradwarding has changes of what's on offer.
3:32:00There's all these kind of latent factors, hidden factors as well,
3:32:03why we're calling it a dark market.
3:32:05These are some public H100 references to NAP-chotted today.
3:32:09I updated them.
3:32:10So take a look at what the public...
3:32:11So those are the references for if you wanted to go on their websites to see.
3:32:15This is just an example here.
3:32:16of what a public on-demand or spot pricing looks like for an H100 today.
3:32:23You can see various prices here.
3:32:26People might use this or they might use their own internal,
3:32:29what they've seen, to actually assume or use it to charge
3:32:34what they want to charge on their actual cluster specification,
3:32:37taking into account every other kind of factor that goes into it.
3:32:42Why are these new structures needed and what are these microstructures?
3:32:45Interesting.
3:32:46It's mainly to solve for these mismatches, right?
3:32:48Bridging capacity, more flexible commitments,
3:32:50resale rights, and protection against price moves.
3:32:54You know, you're selling a fixed-price service
3:32:56from an inference cloud point of view.
3:32:57You'd be selling a fixed-price service
3:32:59while the GPU bill floats,
3:33:00and that creates price exposure.
3:33:02And so locking capacity for years can solve a part of that,
3:33:05but it'll create idle hour exposure elsewhere as well,
3:33:08unless you're always using it.
3:33:10Reselling spare capacity only helps
3:33:12if someone actually can buy the hours.
3:33:13A financial hedge can address a reference price,
3:33:16but can't deliver the cluster right now in terms of cash settlement
3:33:18or guarantee a utilization.
3:33:20So the useful structure has to start with the particular risk
3:33:23that the business has and needs to manage.
3:33:29And I'll go pretty fast on this policy section
3:33:32because we're running out of time.
3:33:35Everything that Affaan has said sits inside of a structure
3:33:38that is regulated and is affected by policy.
3:33:46In various ways. One way is export controls. One of the examples, in January, the U.S. Commerce Department moved qualifying H200 class license reviews for China and Macau to case-by-case review with conditions.
3:34:01furthermore open weight models connect policy back to the u.s economics we started with the
3:34:07u.s ai action plan explicitly encourages open source and open weight ai and recommends improving
3:34:13the financial market for compute so that startups and academics can access large-scale computing
3:34:19power so i'm going to tie this out right here but where's this opportunity like we have this
3:34:26growing middle market. We need balance sheets that can bridge buyer and supplier needs. There needs
3:34:31to be some level of standardization, a standard capacity contract, common specs, common terms,
3:34:35and definitions. And so growing companies whose needs and commitment do not fit the contracts,
3:34:40the hyperscalers, and Frontier Labs consign. Those companies have real workloads. We want them to
3:34:45grow. They still face this financing problem, a timing problem, a specification problem. A supplier
3:34:49might want the long-term commitment for the build-out while the buyer wants to expand in
3:34:52smaller blocks as demand becomes clearer, as it's so volatile, front of curve. The buyer needs a way
3:34:57to avoid paying for capacity it can't use yet, and so participants with strong balance sheets
3:35:01that are uncorrelated with the market and the ability to assess those risks with a strong
3:35:05credit profile can help bridge the gap between what buyers need and what suppliers offers.
3:35:10That requires more than quoting the actual GPU hour. It means understanding the delivery,
3:35:14workload, all of that, and so there's very few kind of players in the market, and a very well-suited
3:35:19player, can be a firm that's uncorrelated with market, has a strong balance sheet,
3:35:23strong credit rating, and incredible amounts of hardware talent that sits inside for high-performance
3:35:29computing.
3:35:29And so, just making a CTA here, let's build this together.
3:35:32Thank you guys so much for attending our talk.
