AI Engineer World's Fair 2024
Compute & System Design for Next Generation Frontier Models
Read the talk
Serving Frontier Models Is a Compute, Memory, and Coordination Problem
Larger models demand more than faster GPUs: economical inference depends on batching, workload isolation, and cache reuse, while frontier training exposes failures and stragglers at enormous scale.
From a talk by Dylan Patel
Before you start: Familiarity with tokens, model weights, GPU memory, and the distinction between training and inference will help; the serving mechanisms are explained as they appear.
Why better models can still feel like stagnation
Why can models keep improving without delivering an obvious leap in capability? Dylan Patel opens with a distinction between improving an existing model generation and deploying a substantially larger one. He characterizes GPT-4 Turbo and GPT-4o as smaller models trained longer, extending the generation represented by GPT-4. He makes the same comparison between Claude 3.5 Sonnet and Claude 3 Opus: a smaller model can become better through more training. These are Patel’s explanations of proprietary model development; their parameter counts and training histories are not publicly established by the launch material.
The deployment implications become clearer with his GPT-4 estimates. Patel estimates GPT-4 at 1.8 trillion parameters and almost 600 GFLOP of computation per token. He also mentions 200 billion parameters without explaining its relationship to the larger figure. The practical question for this audience is how to run the next generation: most application builders will not train a frontier model, but they may need to serve one.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One request, two resource bottlenecks
Inference begins with prefill, which processes the prompt. Patel estimates roughly one petaFLOP of work for a 2,000-token GPT prompt and 20 petaFLOP for a 32,000-token prompt. These quantities describe computation performed, not a processing rate. Before the user receives an answer, the system may already have done substantial arithmetic.
Then comes decode: generate a token, feed it back into the model, and repeat. This sequential process has a different bottleneck.
| Phase | Work | Dominant pressure in Patel’s account |
|---|---|---|
| Prefill | Process input tokens | Compute throughput |
| Decode | Generate successive tokens | Memory bandwidth |
During decode, the accelerators must read model weights from memory. In Patel’s mixture-of-experts example, a sufficiently large batch activates all experts across its requests, requiring the full weight set to be read for each generation step. That is a claim about the batch collectively; it does not mean every individual token uses every expert.
Patel’s serving example requires approximately 60 TB/s of memory bandwidth for 64 users receiving 30 tokens/s each, compared with approximately 3 TB/s for an H100. He treats 30 tokens/s per user as a minimum practical target, while some users want hundreds. The estimate illustrates the bandwidth gap; its exact derivation depends on weight precision, expert activation, cache traffic, and the H100 variant, which he does not specify.
This helps explain why relatively little arithmetic does not make output tokens cheap. Patel describes input prices at roughly one-third or one-quarter of output prices and uses $5 per million input tokens versus $15 per million output tokens as a contemporary example. He groups GPT-4o and Claude 3.5 Sonnet together, but Sonnet’s launch pricing was $3 per million input tokens and $15 per million output tokens. These are historical prices, not current API rates. The next deployment challenge he introduces is the then-upcoming Llama 3 405B, bringing a much larger model within reach of open-model builders.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Admit new requests while existing requests keep running
Access to model weights does not automatically produce an economical service. Patel contrasts a straightforward llama.cpp deployment with serving engines such as TensorRT-LLM and vLLM. TensorRT-LLM targets NVIDIA GPUs; vLLM is his example of an open-source engine spanning AMD and Intel hardware as well. One correction matters here: although he calls TensorRT-LLM closed source, NVIDIA’s public-release announcement already described it as open source.
The first essential serving mechanism is continuous batching. Running one request at a time can be reasonable on a personal device, but rented cloud GPUs change the economics. Patel estimates that batch-size-one cloud inference can cost 10–100 times as much as serving at high batch sizes. He supplies no workload, hardware configuration, or latency target for that range; the useful engineering point is that multiple concurrent requests can share expensive model-weight reads.
Here, batch size means the number of concurrent users being served. Requests do not arrive together or finish together, so a useful scheduler must admit work between iterations:
- A first user submits a request, and the model starts generating its answer.
- Five seconds later, a second request arrives while the first remains unfinished.
- The scheduler brings the second request into ongoing inference rather than waiting for the first answer to finish.
Patel urges builders to implement this capability or contribute to an engine that provides it. His criticism of llama.cpp describes the talk’s deployment context; the current server documentation explicitly supports continuous batching, enabled by default. That documentation does not establish when the capability appeared.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate prompt processing from token generation
At higher volumes, disaggregated prefill, also called disaggregated batching in the talk, separates the two inference phases into accelerator pools. The user still experiences one operation—submit input, receive tokens—but the infrastructure can schedule compute-intensive prompt processing separately from bandwidth-intensive generation. Patel says Google had publicly discussed doing this; he describes OpenAI and Anthropic deployments as his belief, and Together and Fireworks as having hinted at the approach.
Consider his hypothetical Llama 405B worker occupying four chips. Scaling the service need not mean repeatedly adding complete four-chip replicas. Instead, one pool can process prompts and hand the resulting request state to another pool for decoding. Separation does not require different chip models: Patel describes deployments using the same accelerator type for both roles, primarily H100s, while also mentioning A100s and L40s. The distinction is the work assigned to each pool.
The immediate benefit is protection from noisy neighbors. Patel warns that certain query patterns can slow shared inference services, including deliberately disruptive traffic. Users experience the interference through time to first token and uneven generation speed. His example of output jumping from 100 to 30 and back to 100 tokens/s illustrates why average throughput alone does not describe a usable service. Predictable latency and output pacing belong in the service guarantee.
The disruptive customer need not be malicious. A legitimate customer might submit LLM queries for every row in a database, sending 10,000 rows at once. That burst can harm other users sharing the service. Disaggregated prefill is one technique for containing interference while still serving the bulk customer. Patel argues against relying on annoying rate limits as the answer: once a service is public, it must handle demand that its operator does not control. Separation is a scheduling and isolation tool, not a claim that the incoming work disappears.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse a document instead of paying to process it repeatedly
Another way to make strong models useful is to avoid retraining them. Patel describes fine-tuning Claude 3.5 Sonnet as unavailable in the talk’s context and estimates that fine-tuning Llama 405B would require dozens of GPUs. He introduces context caching, then newly offered by Google, as a way to supply application data to a powerful model without changing its weights. He describes Google as the only provider offering it at that point.
Gemini 1.5 Pro provides his concrete example: put the relevant data into its two-million-token context rather than fine-tuning a smaller model. Google’s June 2024 announcement documents expanded access to that context window and caching for both Pro and Gemini 1.5 Flash. Patel contrasts this route with more accessible fine-tuning targets—Llama 7B, Mixtral, and Llama 70B—whose capabilities he regards as below the strongest closed models. Supplying context and changing weights are different operations, but context can meet the application’s need to make its data available to the model.
The economic obstacle is repeatedly processing the same long input. Under Patel’s illustrative $5-per-million-input and $15-per-million-output rates, a million-token document costs $5 in input processing before any answer is generated. If the answer is short, input dominates the bill. A small calculator makes that asymmetry explicit:
python
def request_cost(input_tokens: int, output_tokens: int) -> dict[str, float]:
input_cost = input_tokens * 5 / 1_000_000
output_cost = output_tokens * 15 / 1_000_000
return {
"input_usd": input_cost,
"output_usd": output_cost,
"total_usd": input_cost + output_cost,
}
# A million-token document and an illustrative short answer.
print(request_cost(input_tokens=1_000_000, output_tokens=200))
Legal research, contract review, and other enterprise document queries often have exactly this shape: a large source document followed by a small answer. Reusing the source context therefore targets the dominant recurring expense. Patel expects longer-context open models and broader caching deployment to make this option more widely available.
The mechanism is to retain the KV cache, the attention keys and values computed for the reusable context, instead of recomputing them on every request. That creates a storage tradeoff: long contexts produce large caches. Patel proposes keeping persistent caches in CPU memory or storage rather than occupying GPU memory continuously, then making the state available when needed. This describes the proposed serving mechanism, not a disclosure of Google’s physical cache placement.
Patel points to vLLM as building this capability and invites contributions or following the project. He contrasts the anticipated long-context models with then-common open-model windows of 4K, 8K, and 32K tokens. As reusable inputs grow, avoiding repeated prefill becomes more valuable—but the system must also accommodate the larger stored state.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beyond the GPT-4 training generation
The talk now moves from serving techniques to forecasts about frontier infrastructure. Patel estimates GPT-4 training used 20,000 chips for 90–100 days and consumed 38 GWh. He names OpenAI, xAI, and Anthropic among the organizations building 100,000-chip clusters, and estimates that a newer cluster of that scale could train GPT-4 in three days. These are his infrastructure estimates, rather than a disclosed training-run comparison.
The change is not just the number of chips. H100 improves on A100, and subsequent NVIDIA hardware adds another generation of capacity. Patel expects those larger systems to support models with tens of trillions of parameters. He places GPT-4 training at roughly 2 × 10²⁵ FLOP, then describes future training budgets as “ten e twenty-six” and “ten e twenty-seven.” His next inference-bandwidth estimate is “two hundred gigabytes or terabytes a second,” leaving the unit unresolved. The forecast is a substantial increase in both training work and serving demands; the spoken figures do not support a precise future hardware budget.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The physical footprint of frontier compute
Patel makes that scale tangible with a displayed image of Microsoft data centers in Arizona. He identifies the site as a GPT-5 training location with approximately 100,000 GPUs and 150 MW of power demand. His loose comparison to tens or hundreds of thousands of homes emphasizes that frontier training is also a power and facilities problem. The site identification and capacity figures are his account of the project.
He then contrasts Musk’s then-current 100,000-GPU cluster construction with a discussed successor containing 300,000 GPUs. Patel estimates the proposed 300,000-GPU successor’s electricity cost at approximately $500 million per year. Power becomes a major recurring expense alongside the cost of building the cluster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A large cluster must keep training while components fail
At smaller scales, it is convenient to think of a training cluster as a connected fabric in which every GPU can communicate with every other GPU at some available bandwidth. Patel argues that this picture becomes increasingly difficult to realize at extreme scale. The 100,000-GPU clusters he describes already span buildings, with plans for multiple such clusters the following year. Connecting them makes network topology and communication capacity central design constraints.
Those connections involve electrical-to-optical conversion, fiber, and transceivers. A component that lasts years can still contribute to frequent failures when the fleet contains enormous numbers of components. Patel describes optical components with a roughly five-year failure timescale and estimates a failure somewhere every five minutes in clusters containing 100,000–500,000 GPUs. He does not derive that frequency or provide a recovery architecture. Instead, he poses the systems question: how does a long-running training job continue making progress when failures are routine?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A functioning GPU can still slow the whole job
Hardware does not have to fail outright to cause trouble. Nominally identical H100s can run at different speeds—the silicon lottery, familiar from gaming-GPU comparisons. In a large synchronous training job, variation matters because participants repeatedly process data, exchange gradients, and update weights together. A slower participant can leave the others waiting at synchronization points.
Patel illustrates the bottleneck by saying that one participant running 10% slower can make the synchronized workload 10% slower. He then recounts an unnamed ByteDance paper in which one GPU was associated with a 25% slowdown in a 20,000-GPU cluster. In his account, the GPU technically worked and NVIDIA regarded it as acceptable, but it was still a training straggler. The slide compares Sync SGD with Hierarchical SGD and marks stragglers with orange arrows.
Removing the slow GPU substantially improved performance, Patel reports. The operational lesson is to measure participation in the distributed workload, not merely whether each device passes a basic health check. The exact paper behind his anecdote is not identified here; the related ByteDance and Peking University paper MegaScale does not establish the complete single-GPU, 25%, 20,000-GPU account.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build for the models these systems might enable
These infrastructure questions eventually return to application design. Much of today’s LLM scaffolding compensates for hallucinations and other capability limits. Patel describes the hope among AGI proponents that increasing compute by 100 times will reduce those problems and enable new applications. His closing scale example combines a cluster costing more than $10 billion with roughly $500 million in annual electricity. The expected improvement in hallucinations is a motivation for that spending, not a demonstrated result in the talk.
The immediate work remains concrete: make models at the Llama 405B scale economical and reliable to serve through capabilities that Patel sees at frontier labs but not yet broadly implemented in open-source systems. At the same time, those labs are building beyond that capability level. Application builders therefore face two moving targets: the serving machinery needed to use the next available model, and the product assumptions that may change when a substantially stronger model arrives.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
NVIDIA's library for optimizing and serving large language models on NVIDIA GPUs.
An open-source inference and serving engine for large language models.
Further reading
Google's June 2024 announcement of expanded Gemini 1.5 Pro context access and reusable context caching.
- MegaScalePaper
ByteDance and Peking University research on large-scale LLM training, fault recovery, observability and straggler diagnosis.
Anthropic's historical announcement documenting Claude 3.5 Sonnet's pricing, context window and reported evaluations.
Updates since the talk
Current server instructions covering multi-user decoding and continuous batching.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] Couple different things, right? Like, you know, people have been talking about stagnation, um, and, uh, it's-- I, I don't think anyone else, anyone here sees that.
- 0:21
But a lot of people have been talking about stagnation of models, and a lot of the, a lot of that has to just do with the fact that we haven't seen a big capabilities leap, uh, in the last bit.
- 0:31
Uh, but that, that comes really from, uh, models that we're using today are largely the same as the models that were trained in 2022, right? GPT-4, 4 Turbo, 4o, those are just smaller models that are trained for longer, so similar quality, right?
- 0:45
Um, you know, 3.5 Sonnet came out recently, but again, that's actually smaller than Opus, but it's somehow better because they trained it for longer, right? But we haven't seen a extremely large model come out yet and, and but we will soon.
- 0:57
Uh, but one interesting thing, right, is GPT-4 is like one point eight trillion parameters. It's crazy, crazy expensive to run, right? Uh, two hundred billion parameters. Uh, e-each token requires, you know, fif-- almost six hundred gigaflops.
- 1:10
Uh, but that, that, that's almost gonna be considered a last generation model, right, in, in a year from now. Um, so there's a couple of things that I wanted to talk about regarding that, right?
- 1:19
And, and mostly on the inference side, because I don't think, you know, anyone here is gonna try and train that kind of next generation model, but definitely we're-- we need to be able to run it.
- 1:27
Um, and so, you know, a few things, right? So just, just gonna break down inference, uh, in detail, right? Uh, you know, uh, you know, there's two parts of inference, right?
- 1:36
There's prefill, there's decode. Prefill is the prompt processing, right? And the interesting thing is if you have a two K prompt, two K, uh, context length prompt, right, two thousand tokens you input into GPT, um, that's, that's a petaflop itself, right?
- 1:50
Um, and then, you know, if you have thirty-two thousand prompt that you enter, it's twenty petaflops actually. So it's, uh, an incredible amount of compute, uh, that's required to just process the prompt.
- 2:00
Um, and, and, you know, while, while prefill is, is very compute intensive, right, it's actually the opposite of decode, right? Decode is actually generating each token iteratively, right? So you, you process the prompt, then you ge-generate a token, you feed it back in, and you keep going iteratively.
- 2:16
Right? Um, and decode is extremely memory bandwidth intensive, right? Um, you have to load the whole model from the weights, the, the entire-- all the weights into the, uh, chip, right, or chips, uh, for decode.
- 2:29
Um, and the big challenge here is that, you know, hey, if you have one point eight trillion parameters, if you're running at a reasonable batch size, you're activating all the experts, you need to re-- load all one point eight trillion parameters every single token generation, right?
- 2:44
Even if you're serving multiple users at once, that means you're, uh, you, you need, you know, a one point eight, uh, you need terabytes a second of memory bandwidth.
- 2:52
You wanna do thirty tokens per second. I think that's like a minimum bar for most people, right? Uh, a lot of people want hundreds of tokens per second. But even if you want thirty tokens per second per user, sixty-four users, you need sixty terabytes a second of memory bandwidth.
- 3:05
Uh, if you, if-- Even if you look at an H100, it has like three, right? So this is a extremely challenging systems problem. Um, more, you know, decode, while it is very bandwidth intensive, it's actually quite cheap on the compute, which is why, uh, if you look at like OpenAI pricing or Claude pricing, you see a three
- 3:22
or four-to-one ratio between prefill versus decode pricing, right? Uh, so the input tokens cost, you know, one-third that of the output tokens, um, or one-fourth that. So, so, you know, today the best models I think, uh, 4o and, and 3.5 Sonnet are, uh, I wanna say it's fifteen dollars per million tokens, and then it's five dollars per
- 3:43
million tokens for input, uh, fifteen for output. Um, so five for prefill, fifteen for decode. Um, and, and soon we're gonna have, you know, in the, in the open source, you know, so what everyone here can touch is, is Llama 3, 405B, right?
- 3:56
And that's, that's gonna be a real capability sort of unlock, um, for the, you know, the open source market as well as, you know, builders here, right? And I think, I think there's a couple things that, uh, people really need to be able to implement, right?
- 4:10
Like, you can't just run llama.cpp on Llama 405B, right? Like, it's just not gonna work. Um, so there's a bunch of stuff that people have to work on, um, you know, whether it's using, you know, closed source libraries like TensorRT-LLM, uh, that only work on NVIDIA or like vLLM, which is an open source library that works, uh,
- 4:28
on AMD and Intel and, and soon other people's chips as well. Um, you know, there's, there's a lot of stuff that people need to figure out. One, one of those is, is continuous batching, right?
- 4:37
Uh, 'cause you're gonna get, you know, running inference at batch size one is horrendously expensive. Um, you know, it's great to run it on-- if you're running it on your own personal devices.
- 4:46
But if you're running it in the cloud, right, you're renting GPUs, you're running batch size one, you're, you're gonna cost yourself ten X more. You know, ten X is a low bar, right?
- 4:55
It's actually could be ten X to a hundred X more than running at a high batch, right? So you have to figure out how to run high batch sizes.
- 5:01
Batch size is how many concurrent users you're serving. Um, and so one of those things that makes it difficult is that users' requests come in at different times, right?
- 5:10
Uh, one person might send a request now, and then another person sends in a request five seconds later, uh, but the first person's request is not done. So you need to be able to do continuous batching, i.e., sub-- uh, be able to run through the model iteratively, uh, every time, right?
- 5:23
Um, and, and bring in new users. So continuous batching is one of the things that you have to have to have support of, and, and a lot of software today like llama.cpp doesn't have support for that.
- 5:32
So either you need to build it yourself or, um, you know, contribute to an open source project that, that builds this, um, to, to enable
- 5:41
low cost inference, right, for, you know, models like Llama 405B, right? Um, another one of those is, is, uh, disaggregated, uh, prefill or disaggregated batching, right? Depends on what you call it.
- 5:54
Um, but you know, if you go back to earlier, I was discussing, uh, prefill is very compute-intensive, decode is very, uh, bandwidth-intensive. These are two different workloads, but when you ser-- when you're serving a user, right, whether it's, uh, you know, in your own app or you're using an API, what have you, right?
- 6:10
Like, these users, uh, don't care that it's two different workloads, right? It's one workload to them. Uh, I get tokens out, right? I submit something to you, and I get tokens back.
- 6:19
Uh, but, but for anyone running the infra themselves, uh, they need to, they need to be keenly aware that these are two different workloads. Um, so one thing that a lot of people have, uh, started to do, um, Google's publicly said they're doing it, I believe OpenAI and Anthropic are also doing it.
- 6:33
Um, you know, uh, other firms like Together and Fireworks have hinted that they're doing this, uh, is, is disaggregated prefill, right? So once your inference volumes are high enough, you don't just run inference...
- 6:46
You know, you don't just replicate the model across however many chips you have, right? Uh, say, say it takes four model-- four chips to serve Llama 405B, right, in the future.
- 6:54
Um, you wouldn't just ha-- You know, if you have so many-- if you have enough users, you don't just go four and then eight, 16, whatever, right? You don't just replicate that across the world.
- 7:03
You actually do this thing called disaggregated prefill. You have one set of accelerators do the prefill, which is very compute-intensive, and then you hand it off to the other set of accelerators to do decode.
- 7:13
Now, today, everyone just uses the same accelerator for that, right? H100 or A100 or, you know, maybe, maybe L40 or something, but mostly H100. Um,
- 7:23
but there's a, there's a reason you do this, right? And, and, and that big reason is that you have a lot of noisy neighbors, right? Um, so if you've ever worked in like CPUs or on anything in cloud computing, noisy neighbors are a huge, huge issue.
- 7:35
Um, and actually, like, there's-- it's very trivial to dramatically slow down most inference providers' services, uh, if you, if you just, uh, send queries in a certain way, like in a, in a sort of malicious way.
- 7:46
Um, you can, you can just slow down people's, uh,
- 7:50
service, right? Whether that's, you know... And, and that'll, that'll impact the user's time to first token, right? Um, and I think that's a huge issue, right? If time to first token is too long, people will just quit, right, using your service.
- 8:01
Um, if, uh, you know, the tokens per second varies a lot, right? For a moment, you're getting a hundred tokens per second, and then it drops down to like 30, then it drops-- it goes back up to a hundred.
- 8:11
That's gonna be really annoying to the user. So, so there's a lot of things around, you know, SLA and, and reliability and all these things that you have to guarantee.
- 8:19
And so disaggregated prefill, uh, is, is one of the techniques to do that, right? Um, and, and so you don't wanna have someone submit, you know, for example, "Hey, I have a database, and I wanna sub-- I wanna run an LLM query across every single row in that database, and I'm just gonna submit it to you, my
- 8:36
service provider, because you have this cool model or what have you that's fine-tuned on some data set." And, well, whatever it is, right? If I submit ten thousand rows to you at once, that's gonna kill everyone else's performance, right?
- 8:47
So, so this is one of the techniques that people have for, uh, making it so, you know, that, that person who you definitely wanna serve, uh, doesn't impact everyone else's usage.
- 8:57
Uh, because once you open up your service to the real world, you're not gonna be able to control who's submitting what, and rate limits are the most annoying thing ever, so that's not the correct way to go about it.
- 9:06
Um, another thing is context caching, right? So Google launched this recently. Uh, they're the only one offering this today, but I think this is a really big deal, uh, 'cause when people talk about fine-tuning, right, of models, that's great.
- 9:19
Uh, but in reality, the best models are really expensive to fine-tune or impossible to fine-tune, right? I can't go fine-tune 3.5 Sonnet, or fine-tuning Llama 405B is gonna take, you know, dozens and dozens of GPUs, right?
- 9:33
So, so instead of that, the, the, uh... Or, you know, and, and in closed source models generally. So Google only does closed source models mostly for the big ones, right?
- 9:40
So Gemini 1.5 Pro, they offered this-- they, they brought this recently, right? Which is context caching. So instead of, you know, fine-tuning your model, why not, you know, just fill out a context length of...
- 9:51
You know, they, they offer, I think, two million now today, right? Two million context length. Um, why not fill it out with your data there, right? Um, you know, and, and there's a couple, you know, advantages to that.
- 10:02
One is you can use the best models, right? In the case of fine-tuned models, you really are focused on like the Llama 7B or Mixtral or Llama S-- uh, you know, 70B.
- 10:11
It's, it's kind of lo-- much lower quality models than what's available in the closed source world. Uh, so one of the things you can do is you can, um, implement what Google has called context caching.
- 10:21
In the, in the open source world, we'll, we'll have super long context models soon enough. But, uh, economically, right, you know, we talked about fifteen dollars token-- per million tokens output, um, and five million per to-- million tokens input.
- 10:34
If you were to have, uh, on, on, you know, the best o-- closed source models today, if you were to submit a prompt of like, you know, a million tokens, and, and most, most of the times you're looking at a document, you get a query back, right?
- 10:45
You- your, your output is very small. Almost all of the cost is just se-sending them that document, right? So that's, that's gonna really, really hurt you. So for people, you know, targeting maybe like a legal AI or like, um, you know, some sort of other contract review AI, a lot of these enterprise use cases, uh, prefill is
- 11:02
gonna dominate your cost if you're using APIs. Um, and so Google has this context caching and, and open source will have it so models you can run yourself and, and others will deploy over time.
- 11:12
Uh, but basically you don't recompute the KV cache, right, the, the context length every single time. Instead, you cache it. Uh, but the problem is to save, save that takes an, an incredible amount of memory.
- 11:25
Um, so you don't save it in the g-GPU's memory, right? You save it on the CPU's memory or storage. Um, and so, uh, vLLM, uh, which is an open source library for inference, is contributing-- is building this currently.
- 11:39
So if you're interested in contributing to that, uh, check that out. Um, or if you're interested in using it, just start the project, right? Um, because, you know- Well, most of the models we have in the closed source today are like only like 32 or 8K or 4K context length.
- 11:52
They're coming with longer, um, and being able to, you know, dramatically reduce your costs, um, by caching the context, um, is, is very-- is gonna, is gonna dramatically reduce cost, right?
- 12:06
Um, so now I'm just gonna talk about like head-in-the-cloud stuff instead of like real usable things, which is, um, you know, what's coming down the pipeline, right? Which is, you know, GPT-4 was like twenty thousand chips for ninety to a hundred days, um, used, you know, thirty-eight gigawatt hours.
- 12:20
Very, very expensive. Cool. Um, but, you know, what's-- what is, what are they building now, right? Uh, OpenAI, xAI, um, Anthropic, many others are building hundred thousand chip clusters, right?
- 12:31
And it would train GPT-4 in three days, right? So it's kind of irrelevant. Um, you know, and, and, uh, I'll, I'll skip over this part, uh, because it's not really, uh, too relevant.
- 12:41
Um, but, you know, what, what, what's a modern system capable of, right? Like H100 is, is pretty, uh, pretty fast relative to A100 and, and coming down the pipeline is these...
- 12:50
the new NVIDIA chips. But what, what, what's come-- you know, what's coming down with these hundred thousand GPU clusters, right? Um, it's not gonna be a one point eight trillion parameter model.
- 12:58
It's actually gonna be, you know, it could be in the tens of trillions of parameters. Um, you know, the, the training FLOPS, right? I talked about GPT-4 is it's roughly two e twenty-five FLOPS, right?
- 13:08
Which is, uh, you know, a number that's not really relevant or two e twenty-five FLOP. Um, but with a hundred thousand GPU cluster, you can do ten e twenty-six, ten e twenty-seven FLOPS.
- 13:19
Uh, and to run that model is gonna require two hundred gigabytes or terabytes a second of memory bandwidth, right? Um, but what does, what does that like-- what does that look like, right?
- 13:26
So, so this is a... On the top right is an image of, uh, M-Microsoft's data centers in Arizona where they're making GPT-5, right? Um, they have about a hundred thousand GPUs here.
- 13:39
Uh, it's a hundred and fifty megawatts, right? Like the average home does not consume... You know, that's like, that's like, like tens of thousands if not hundreds of thousands of homes of power consumption, right?
- 13:49
It's, it's kind of insane. Um, Elon's talked about his next generation cluster. He's building a hundred thousand GPU cluster today, uh, but he's talked about his next generation cluster is three hundred thousand GPUs.
- 13:58
That is kind of insane. But the, the power cost for that alone would be like five hundred million dollars a year, right? So it's like, you know, people are, people are kind of insane, but it's pretty cool.
- 14:08
Um, but, you know, the, the, the interesting thing here is, you know, on training, we, we, you know, when, when you, when you try and train a model today, people just talk about fully connected clusters.
- 14:19
Uh, every GPU is connected to every other GPU at some speed and you, you know, you have to do, you know, all your operations. But that's not really possible when you go to these super large clusters, right?
- 14:29
Um, so the hundred thousand GPU clusters, those are being built this year, and then next year they're planning to build multiple hundred thousand GPU clusters. Already you can see that it exists across multiple buildings, right?
- 14:40
Um, and so there's a lot of complicated networking, uh, going on, right, to connect these data centers together. Um, and, and one other thing I, that, that I think is just like kind of interesting to, again, head in the clouds just to think about is, um, when you connect these chips together, there's a lot of optics, right?
- 14:57
Uh, you know, you convert from electrical to optical, uh, and then, you know, over fiber optics to connect between chips, transceivers, et cetera, right? Uh, these are extremely unreliable, right?
- 15:07
Uh, they tend to have a failure rate around five years. Um, and so what's interesting is if you're talking about a hundred thousand GPU cluster, um, or if you're talking about a five hundred thousand GPU cluster, you're gonna have something fail like every five minutes, right?
- 15:22
Um, which is insane, right? How, how do you even deal with something in your cluster failing every five minutes when you're training a model, right? Um, so, you know, this is, this is again more of like a hardware-oriented thing but, uh, you know, the, the other thing that's interesting is like when you get chips, they're not all
- 15:38
the same speed. You know, an H100 is not an H100. Um, they're stragglers. Uh, so if you get a large distribution of chips, um, what we call it in the industry is, is called the silicon lottery, um, in that like, you know, you, you can buy, for example, a, a gaming GPU and, and compare it to other
- 15:55
people's gaming GPUs on the forums, and they're actually like percentages difference in performance. But when you do a massive training cluster, um, you end up with... You know, training is a synchronous workload, right?
- 16:06
You know, you, you, you update the weights, you then you pass the gradients around, right? Um, and then you, you know, then you again run through a bunch of data, uh, update the weights or pass the gradients around, update the weights, right?
- 16:18
Um, so it's, it's a synchronous workload. So if one of them is ten percent slower, then everything is ten percent slower. And ByteDance had a cool paper where actually they saw a twenty-five percent decrease in speed just because one random GPU they got, uh, while it did technically work, um, and NVIDIA...
- 16:33
and, and, and according to NVIDIA it was fine, it was like twenty-five percent slower than, uh, what they wanted, right? So they're... You know, this is like, this is on like a twenty thousand GPU cluster even, right?
- 16:43
Um, so, so it's, uh, it's, it's quite interesting that, you know, that, that's... These are the problems people are running into at scale, right? So they pulled that GPU out, um, and then you, you can sort of see their performance dramatically uplifted, right, um, during, during training.
- 16:59
Um, and then again, this is ByteDance on a twenty thousand GPU cluster. So it's, it's, um, it's a, it's a big, big issue. Um, and I think, I think some of the other stuff in this presentation is not really relevant.
- 17:11
Uh, but I think, I think what do these next generation systems look like is a very, um, important question to ask yourself, right? Um, you know, and what, what do I, what do I, what do I do when I deal with that, right?
- 17:24
Like I think a lot of the scaffolding that people are building, uh, today for LLMs are dealing with, you know, is, is dealing with hallucinations and things like that.
- 17:33
And, and the hope that everyone has, or at least a lot of the AGI people have is that, you know, when I, when I hundred X the compute, um, you know, when I build a cluster that takes five hundred million dollars of electricity and I train a model with it, it's gonna make something that, uh, uh, you
- 17:47
know, yearly ele-electricity costs and make a model with it. And then the cluster itself costs over ten billion by the way, right? Uh, it's, it's gonna get rid of a lot of these, um, hallucinations.
- 17:56
It's gonna let us do a lot of interesting things. Um, yeah. So, so I think that's, that's basically all for the talk. I just wanted to, you know, uh, mention, you know, sort of a reasonable thing which is how do you run Llama 405B kind of some strategies that people need to implement that aren't necessarily implemented yet,
- 18:12
uh, in the open source that are implemented at the labs. Um, but then also like, you know, what are they doing, right? 'Cause they're not worried about, you know, Llama 405B capable models. [outro music]