AI Engineer World's Fair 2025
How fast are LLM inference engines anyway?
Read the talk
How fast are LLM inference engines anyway?
Open models make self-hosting plausible, but useful speed depends on workload, precision and latency targets. Charles Frye walks through how to measure those tradeoffs.
From a talk by Charles Frye
Before you start: Familiarity with language-model tokens, GPU serving and the difference between latency and throughput will help you follow the benchmark comparisons.
When does running your own model make sense?
How fast can you run an open language model—and is it capable enough to run your application? For years, building useful AI applications largely meant calling proprietary APIs. Access to weights offered the freedom to modify and experiment, but that freedom mattered less when the available models could not do the job. The Llama, Qwen and DeepSeek families changed the practical question: many applications that previously required proprietary models became feasible with open weights.
Serving those models efficiently takes more than wrapping a transformer in torch.nn.Module. KV caching is only the beginning; paged attention, multi-token prediction and speculative decoding make the implementation increasingly specialized. Engines such as vLLM, SGLang and TensorRT-LLM provide that serving machinery without requiring every application team to build it.
Together, better models and better engines broaden the case for self-hosting. Air gaps, government requirements and a strong commitment to open source were already reasons to run your own models. Frye's opening claim is that self-hosting now makes practical sense for a wider set of applications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Applications need sufficient capability
In a 2023 presentation, Frye proposed a conditional argument: if an application's capability requirements saturate, open models can catch up and eventually dominate that use case. Operating systems, databases and programming languages provide the analogy. Once a collaborative project clears the capability threshold users actually need, its other properties can become decisive. The application need not use the most capable system available; it needs one capable enough for its task. Frye treats the subsequent progress of open models as support for that prediction.
The serving ecosystem also changed. Looking back at his earlier list of inference libraries, Frye characterizes TGI as having fallen away while vLLM remained relevant. That is his retrospective assessment of the landscape, not a claim that every library on the old list had formally shut down.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn repeated performance questions into a database
The operational questions came from people building code completion in editors, enriching databases with large backfill jobs, or serving models they had trained themselves. Cost or model customization could make a hosted model provider unattractive, leading them toward generic infrastructure such as Modal. But choosing infrastructure immediately raised a more specific question: how fast would an eight-billion-parameter Llama model run on SGLang with 128 input tokens and 1,024 output tokens? A model name alone could not answer it.
Producing a trustworthy number first required getting packages working together and installing suitable engine versions. Frye reports that answering a configuration question went from a couple of days to one or two hours, then roughly 15–20 minutes with benchmarking software. Once the same questions kept recurring, the next optimization was to compute the answers before anyone asked.
The resulting benchmark covered roughly ten models, three engines—vLLM, SGLang and TensorRT-LLM—and about ten context lengths. Publishing those results turned repeated investigations into a lookup problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the results as configurations, not engine rankings
The LLM Engineer's Almanac Engine Advisor makes the results browsable. Alongside the interface are the benchmarking methodology, open-source benchmark code in Stopwatch, and LLM Engines: An Executive Summary. Frye describes speculative decoding, multi-token prediction and quantization as topics for future additions to the Almanac. The live walkthrough begins with an audience request for Qwen3.
Selecting any engine does not mean every engine has a result. For the tested eight-bit Qwen3 configuration, Frye reports problems with SGLang and believes only vLLM results are available. This is a gap in that experiment's configuration coverage, not a permanent statement about SGLang's Qwen3 support. The full search space spans many combinations, and some are missing because they did not work or because a working setup was not yet clear. Contributions can fill those gaps.
These are initially out-of-the-box results, not fully tuned performance ceilings. Optimizing every configuration would be a substantial separate project. Frye particularly invites optimized TensorRT-LLM implementations because it exposes many tuning controls. A result therefore describes a model, engine and configuration together; it does not isolate an engine's ultimate capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a latency target before reading throughput
The first filter is time to first token, or TTFT: how long the user waits before output begins. Frye starts with a one-second service-level objective, then tightens it to 300 milliseconds for a more interactive experience. His reference to the Doherty threshold is informal; the useful engineering step is choosing an explicit latency target for the application.
With the first-token filter set to 300 milliseconds, Frye reports about one request per second for the selected Qwen3 mixture-of-experts model on vLLM, using 128 input tokens and 1,024 output tokens. That throughput belongs to the selected workload and filter, rather than being a general speed rating for Qwen3.
The interface also supplies a launch snippet. In the demonstration, the command uses uvx modal run and requires a configured Modal token. After saving the supplied snippet as serve.py, the command has this form:
bash
uvx modal run serve.py
Use the generated configuration that accompanies the result; the command alone does not specify the model or server. Frye estimates roughly five minutes to load the weights and start the server. That startup time is separate from the first-token latency of requests to the running server.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model size does not determine serving speed
The next audience selection is Gemma 3 27B. Frye first relaxes the first-token filter, which had been too restrictive for the displayed results. This configuration uses BF16, the easiest numerical format to get working in the initial experiments; he recalls subsequently getting eight-bit variants working too. The comparison has therefore changed both model and precision, as well as the latency filter.
Frye describes Gemma as roughly ten times smaller in total weights than the selected Qwen3 model, with roughly similar active parameter counts and approximately comparable throughput at 128 input and 1,024 output tokens. A mixture-of-experts model's total parameter count and its active parameter count describe different serving burdens. Here, the differing precision and filters also prevent treating the result as a controlled test of model size.
The displayed Gemma results show a pronounced gap between vLLM and SGLang. Frye interprets this, along with the Qwen and Llama results, as evidence that optimization effort varies across model and engine combinations. That is a plausible explanation for the observed differences, not a measured attribution of which team's work caused them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reverse the workload and watch throughput change
Keeping Gemma selected, Frye starts with a short prompt and a long response. At 128 input tokens and 1,024 output tokens, the displayed Gemma configuration delivers about one request per second with roughly 400 milliseconds to first token. He then reverses the lengths: 1,024 input tokens and 128 output tokens. The total token count stays the same, but most of the work moves from generation into prompt processing.
He loosely calls the first shape reasoning-like and the second RAG-like. These are workload analogies, not tests of reasoning quality or retrieval quality. The distinction that matters to the engine is decode versus prefill:
- Decode: Output tokens are generated autoregressively. Each new step depends on earlier generated tokens.
- Prefill: Input tokens are already known, allowing the engine to process their positions in parallel.
Moving tokens from output to input therefore changes the available parallelism even when the combined length is unchanged.
The displayed Gemma comparison is:
| Workload | Input tokens | Output tokens | Approximate throughput |
|---|---|---|---|
| Decode-heavy | 128 | 1,024 | 1 request/s |
| Prefill-heavy | 1,024 | 128 | 4 requests/s |
For this Gemma workload swap, Frye reports roughly four times the throughput with vLLM. Prefill can use large matrix–matrix multiplications, loading weights and reusing them across more arithmetic. This favors the GPU's compute capacity relative to the cost of moving memory or communicating. The gain belongs to this particular workload comparison; it is not a universal multiplier for inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Lower precision could widen the gap
The Gemma example uses BF16. Frye briefly questions its Tensor Core support, then immediately corrects himself: BF16 does use Tensor Cores. Lower-precision formats can offer greater arithmetic throughput on hardware that supports them, but support depends on the GPU generation—FP8 on Hopper and FP4 on Blackwell are distinct cases.
Frye invokes quadratic scaling with bit width to explain why shorter numbers can make multiplication cheaper. That arithmetic intuition should not be read as a quadratic law for hardware throughput or end-to-end serving speed. He predicts that an FP4 experiment on Blackwell could show an even larger separation between these workload shapes, but no such benchmark is demonstrated here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Try useful context before adding generated reasoning
An application cannot always choose its input/output balance: users bring their own queries, and some tasks inherently require long answers. But when the product goal is better quality under a latency constraint, the benchmark suggests an option worth testing. Consider adding useful context before immediately asking the model to generate more reasoning. Parallel prompt processing may make that route easier to fit within the latency budget. The benchmark does not establish that context always substitutes for reasoning or improves quality.
Frye also observes nearly unchanged time to first token after increasing the input from 128 to 1,024 tokens. Those lengths represent an eightfold increase in input, despite his informal description of it as tenfold. The observation is useful because extra context did not impose a proportionate first-token penalty in this example; it is not a guarantee that arbitrary amounts of context are free.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Take the measurements back to your own workload
The interface is a starting point for further investigation. Frye offers access to downloadable raw results and points to the open-source code for running benchmarks independently. The methodology explains how to interpret the measurements, while the executive summary provides material for discussing open-model serving with leadership. These serve different needs: inspect the data, reproduce the experiment, or explain why the infrastructure decision matters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the throughput ceiling without confusing it with latency
The closing question asks how far request throughput can rise beyond the displayed result before latency bends upward. Frye first clarifies the unit of deployment: the displayed Gemma example measures one replica on one H100. That identifies the scope of this example, not the earlier Qwen3 configuration. Higher aggregate throughput eventually requires more replicas. His hypothetical 400-QPS target illustrates a reason to scale out, not a demonstrated fleet result.
To establish an upper throughput bound, the benchmark submits roughly 1,000 requests together, waits for all of them to finish, and divides the completed request count by elapsed seconds:
This exposes substantial parallelism to the engine. It estimates saturation throughput on the assumption that the engine schedules the available work effectively; it does not establish that requests at that load have acceptable latency.
The remaining procedure separates capacity from responsiveness:
- Use the burst result as a capacity estimate. Sustained arrivals above the server's service rate cause queues to grow, so latency rises rather than throughput increasing indefinitely.
- Measure the low-load endpoint. Send one request, wait for its response, then send the next. This estimates the minimum-latency behavior of the running server, not its maximum throughput.
- Sweep request rates between the endpoints. Measure the latency achieved at each load to obtain the operating points shown in the Advisor.
The useful deployment target is therefore the request rate a replica can sustain while meeting the application's latency objective. Once that per-replica operating point is understood, scaling out addresses the larger traffic requirement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore inference configurations by model, workload and latency objective.
Explains throughput bounds, latency measurements and request-rate sweeps for individual serving replicas.
Open-source tools for provisioning and benchmarking vLLM, SGLang and TensorRT-LLM on Modal.
Discusses self-hosting decisions, engine selection and the limitations of default benchmark configurations.
Further reading
Technical background on H100 Tensor Cores and supported numerical formats.
- Introducing NVFP4Article
Explains Blackwell's NVFP4 format, scaling factors and accuracy considerations.
Updates since the talk
- TGI maintenance status and migration guidanceDocumentation
Official documentation describing TGI's maintenance mode and migration options.
Read the complete timestamped transcript
- 0:00
[upbeat music] Thanks everybody for coming.
- 0:16
Um, yeah, wanted to talk about some work I've done recently on trying to figure out, uh, just how fast these inference engines are when you run open models on them.
- 0:29
Uh, so the kinda ... Been talking at AI Engineer since it was AI Engineer Summit two years ago. Um, and the, for a long time, it's basically been the, like, OpenAI wrapper conference, right?
- 0:46
It's like, 'cause just 'cause, yeah, what am I gonna do? Am I gonna run an agent with BERT? Probably not. Um, and that was, like, it was exciting to talk about all these cool new technologies, see people building stuff like Cursor on top of them, or Devin.
- 0:58
Um, but for me as somebody coming from, like, having trained my own models a lot, it was like, oh man, I want, I wanna, I wanna touch the weights.
- 1:05
I wanna play with them. I want to, like, hack them. And the quality, but the quality just, like, wasn't there yet to do, like, some of the interesting stuff.
- 1:14
Um, and that's changed. You've got the Llama series, we've got the Qwen series, we've got DeepSeek. And so now we're, like, catching up to, like, maybe even literally catching up with the frontier labs, which would be pretty crazy.
- 1:26
Um, but we're at the very least, like, at the point where a lot of things people have been talking about at AI Engineer for years are possible with open-weights models where they weren't before.
- 1:35
Um, and at the same time, there's also the development of the software stack on top of that. Um, also as somebody coming from, like, writing my own, writing and running my own PyTorch models, I was like, how hard could it be to run a language model?
- 1:45
Like, you just, you torch.nn.Module, wra-wrap a class around it. And like, yeah, I mean, having done stuff with Transformers before, it's like, oh yeah, there's def- It's a little more complicated with Transformers.
- 1:55
Training looks weird, inference is different. Um, but you know, pretty quickly the state of play has advanced a lot on how to run a transformer, right? KV caching is like, you know, just the first thing, and then it's this, you know, um, page detention.
- 2:08
Now multi-token prediction, speculative decoding, all this stuff. That's, like, pretty hard to write yourself. And so, you know, you want software to do that for you probably. Um, and so these open source engines are available now.
- 2:20
You got vLLM and SGLang and TensorRT-LLM. So the combination of those two things has, like, kind of flipped the, uh, playing field around to where there's like, you'd need a really good reason to run your own models, like you're the US government or something.
- 2:34
Um, or you wanna run on an air gap system. Um, or you, like, or you had to, like, believe it in your heart. You know, you had to want open source models like the, um, like, Noose or Prime Intellect.
- 2:47
You had to be like a decentralized crypto bro or whatever to want to run your own op-open models. Um, but yeah. Now the, the like, uh, the situation has changed, which is really exciting.
- 2:57
It sort of finally makes sense to self host.
- 3:00
Um, so, uh, the ... Just, like, wanted to do this really quickly. I don't ... Was anybody here at this AI Engineer Summit 2023? The, like, first one? Anybody?
- 3:11
Okay. Did anybody come to the AI Engineering 201 workshop the day before, um, uh, in the [REDACTED:location_address]? Maybe not. I talked to one or two people who were there.
- 3:22
Um, so I gave a talk on, on like, you know, how to, how to do your own AI stuff back then. Just wanted to pull out like a couple of slides.
- 3:31
Um, so like, the, one of the main ... This is like 2023. We're like, who's gonna win, open models or closed models? And the key statement, uh, in the talk was, like, if capabilities requirements saturate, open models will catch up to proprietary models and then dominate for those cases.
- 3:48
Inspired by what you see with operating systems, databases, programming languages. Like, as soon as there's, like, a, a sort of like, you know, um, capability level that everybody, like ...
- 3:59
That you don't need the absolute best thing. You just need something that's good enough. Open, uh, like, collaborative projects tend to catch up and, and then have better properties, and that's happened with open models.
- 4:11
So, check. Um, uh, yeah. So that, um, that was mostly around, like, capabilities. Um, and you know, at the time there was only Llama, but now we got a lot.
- 4:22
Um, the other one was, yeah, talked a little bit about LLM inference libraries at the time. And most of them are gone. TGI, RIP, um, uh, for example. But vLLM was good then, it stuck around now.
- 4:34
Um, yeah. So, uh, I don't know what the slides in 2027's AI Engineering, uh, Conference are gonna look like, but, um, the very least, like, three of my slides from two years ago were right.
- 4:46
Um, so yeah. Um, hopefully it's not those three slides again in two years. Um, but yeah. All right. So what does the LLM engine landscape look like two years later?
- 4:56
Um, let's, uh, take a look. So what, uh, what ... You know, we were advising a bunch of people on how to run ... Like, people were coming to us, like, "I wanna run my own code completion editor, uh, in editors.
- 5:08
I wanna run like big backfill jobs to f- to, like, um, enrich data and databases with language models." Uh, it's too expensive or, uh, to run it on OpenAI or, "I train my own model," so it's too expensive to run it on a, like, a provider like Fireworks.
- 5:24
They wanna run it on an more generic infrastructure like what we have at Modal. Um, and so they would ... People would come and they'd be like, "All right.
- 5:31
Well, how fast can you run an eight billion parameter Llama model with SGLang, uh, at like, with 128 tokens in, 1,024 tokens out on a Tuesday when Mercury's in retrograde?"
- 5:43
Um, and that would, like, take ... At first it took a couple days to like re- you know, figure out how to make sure all the packages are working, that we've got the fastest versions installed and, um, like, uh, that we can give like a, you know, trustworthy number.
- 5:57
Got that down eventually to like, you know, like an hour or two, um, and then built some benchmarking software so we could get it done in about, like, 15, 20 minutes.
- 6:06
But then, like, you know, the people ended up asking a lot of similar questions. So we decided just, uh, you know, fifth law of, uh, fifth mantra of performance is, uh, do it when they're not looking.
- 6:18
Um, so like compute the thing ahead of time and store it. Uh, so we ran a giant benchmark over, um, like 10 or so different models on vLLM, SGLang, and TensorRT-LLM on about 10 different context lengths, um, and put that all up on the internet.
- 6:35
So let's take a look at that
- 6:37
Let's see. I'll drop into this one. So this is a live version of it. So you find this at modal.com/llmalmanac. Um, the idea is that this is just one page in the like, you know, your almanac, your little book that has the useful things you need to know to be an LLM engineer.
- 6:55
So to start, we've got our benchmarking results, ah, benchmarking methodology in detail, the open source code for it, and a little executive summary. Um, hope to put more stuff up there as we accumulate the things people need.
- 7:07
Um, things about like speculative decoding and multi-token prediction and quantization. Um, but yeah, so to start off, we got this little interface here. So, uh, yeah, anybody, what's a model people wanna see results for?
- 7:21
Any-- if, uh, hopefully that's legible to people. Anybody got a favorite? Qwen three. Hmm?
- 7:27
Qwen three. Qwen three? Ministrel. Ministrel? Okay. Uh, excuse me, uh, that... Oh, man, that's, that's my boss. I'm not gonna do that one. Um, okay, we'll do any engine here.
- 7:37
Oh yeah, we didn't do, uh, so this was fun. SGLang's Qwen three support is a little buggy for the f- um, for the eight bit quant that we ran.
- 7:46
So I think we only have results for vLLM. We'll stick with any engine there. Oh yeah, by the way, if you, if you try this thing out, um, like you'll see, like if there's a giant tensor of configurations, right?
- 7:57
And we'd love to have that full tensor, but it doesn't alway-- it's like either not always possible or, um, like it's not clear how to do it. So there's a place where you can contribute configurations so we can build up a nice big database of like how to run these models.
- 8:11
We also haven't like carefully optimized any of these things. We started with out of the box performance for all the engines, which is like optimizing a hundred configurations is gonna take some time.
- 8:19
So we'd love, uh, like contributions of optimized implementations, especially TensorRT-LLM, which has like a ton of knobs. Um, and they have names like user buffer. Like what is that?
- 8:31
Um, yeah. Um, okay, yeah. So first token under one second. Let's say this is a, this is a pretty common SLO. Like you want one second's a nice round number.
- 8:41
People feel like that's like a good amount of time to wait. I'd say three hundred milliseconds is a tighter one. That's more like interactive. That's your, uh, Doherty threshold if you're a fan of Halt and Catch Fire.
- 8:51
Um, made up number, but like if somebody repeats a made up number enough, it's a real number. Um, so yeah, three hundred milliseconds. Okay, so we can get a throughput of about one request per second on Qwen three in the mixture of experts model on vLLM for a hundred and twenty-eight tokens in, thousand twenty-four tokens out.
- 9:10
Um, so that's we got. Over here, we got like a little, uh, code snippet here. So it should be the case that you can UVX Modal run this, and if you have a token, that should just work immediately.
- 9:19
Uh, and by immediately, I mean after five minutes of loading the model weights and, and spinning up the model server, but that's as immediate as it gets. Um, cool.
- 9:27
All right, so that's one result. Uh, who's asking for Qwen? Are you satisfied? Yeah. Okay, great. Um...
- 9:36
Gemma three twenty-seven B. Gemma three twenty-seven B. All right, on this one, let's do any engine here. Oh, right, this was... Oh, right, sorry, we got a tight filter on this.
- 9:44
I'm gonna put the first token filter up. Oh yeah, this one we're doing the BF16 quant is the only one that we could get working at first. So I think we eventually got the eight bit quants working.
- 9:56
I don't think that's a hard blocker, but it was the easiest one to get going was the BF16. So these are definitely slower. Um, so you'll see twenty-seven billion parameter models, so like ten X smaller in, uh, model weights.
- 10:09
Roughly the same number of active parameters as Qwen three, but we're getting like about the same, um, like throughput and request per second on the same load, one twenty-eight in, ten twenty-four out.
- 10:20
Um, so yeah. So it's interesting, you can see sort of which ones have had a more optimization work on them. I think the Qwen three models and the Llama model series, you see a lot more optimization.
- 10:31
Um, this is also one of the ones where you saw the biggest gap between, um, SGLang and vLLM, the Gemma one. So it looks like the vLLM team spent a little bit more time, or Google's contributed a little bit more to vLLM on, uh, getting good results.
- 10:45
Oh yeah, let's go. Yeah. So the other thing you'll see is you generally... Like, so I, so I just switched. Sorry, I should say what I'm doing. So this is a hundred and twenty-eight tokens in, ten twenty-four tokens out.
- 10:55
Uh, and you can see we're getting about one request per second on this guy. Um, let's just end it, and the first token comes back in four hundred milliseconds.
- 11:03
Let's flip it to ten twenty-four in and a hundred and twenty-eight out, right? So this is going from like a reasoning workload to like a RAG workload. Big scare quotes on that, but it's just the difference between whether you're dominated by decode time, like more of your tokens are decode or more of your tokens, uh, prefill.
- 11:22
Uh, and what you'll see very consistently in these, in these results is that you get much higher throughput if you have more like tokens in the context as opposed to tokens being generated.
- 11:32
Very straightforward. If, I mean, if you know your transformer architecture, it's like auto aggressive versus parallel. Like yeah, one of the first things you would learn if you looked at the, you know, kind of the implementation of the architecture, but it's nice to see it like nice and, you know, very cleanly.
- 11:45
More of an empiricist than a rationalist myself, so I like to see data, um, and not like, uh, chalkboard stuff. Um, so yeah. So, uh, what I'm getting at here is that the, um, the request per second that we're seeing here is about four requests per second for vLLM on the same workload, but n- with like context
- 12:05
instead of, uh, um, generation as wh- where the majority of the tokens are. Um, so this is, uh, yeah. I gave a talk on GP, like GPUs a little bit earlier today, and like one of the big takeaways there is find things that like are throughput oriented and involve a lot of arithmetic and not like moving me-
- 12:25
memory around or communication, and that's exactly the difference here. You have like big matrix, matrix multiplications, load the weights one time, use them a bunch, um, and that's exactly the difference here, and it's a four X improvement.
- 12:38
And that's using BF16, which Does not have Tensor Core support? No, no, BF16 has Tensor Core support, but it's the slow Tensor Core support compared to FP8 or FP4 on Hopper and Blackwell.
- 12:50
And so you're, like, you're ... The real win there is the shorter, like, shorter numbers, faster multiplication. It's actually quadratic in the bit width, so you get a big win as you go down.
- 13:00
So this sh- like, if we were to run some results with FP4 on Blackwells, you would see an even bigger gap than just this, like, 4X improvement. 4X is, like, barely enough to wake up for, you know?
- 13:11
Um, but yeah. So that's a little ... Like, not every application can you, like, change that. Um, like your users might be bringing queries to you, so you don't have control.
- 13:21
Um, but the, um, uh, it can ... It's more the sort of thing where you, like, a product person is like, "We should improve the quality." You're like, "Oh, how can I improve the quality without tur- like, killing our latency?"
- 13:36
Don't have it ... Like, don't immediately reach for reasoning, reach for context instead, 'cause it's gonna be cheaper and you're gonna get better perform ... Like, you're, you're gonna m- find it easier to hit your latency SLAs.
- 13:46
I forgot to point this out, but the latency is, like, almost identical in time, in time to first token, even though we're doing 10 times as many tokens. Basically a free lunch.
- 13:55
Um, yeah, okay, so that's, uh, that's sort of, like, how I envision people using this interface and the data. Um, there is a, there's a l- URL somewhere where you can just download the raw data.
- 14:06
Um, if you're interested in that, hit me up. The code is also open source if you wanna run some of these benchmarks yourself. Um, I think I've ... I'll close there.
- 14:14
Lots of other stuff to talk about, like our benchmarking methodology, which is written up here, the, uh, executive summary, which you can share with your, um, with your leadership, um, uh, on, like, running open models.
- 14:27
Um, but I'll take a question or two before we close out. Yeah.
- 14:31
I mean, how f- how far can you drive the, uh, max throughputs inside of ... You had it set to four.
- 14:36
Yeah.
- 14:36
Like, how far can you go out before it starts to inflect up?
- 14:39
Yeah, so that ... So one thing I'll say is, like, this is throughput per replica, right? So this is one GP- This is one GPU? Yeah, one H100. So, like, what you ...
- 14:49
The way you solve your, like, total throughput is by scaling out rather than scaling up, right? Um, but, um, so if you want, if you want 400 QPS or whatever, like, eventually you're just gonna have to scale out.
- 15:02
But to your question of like, yeah, how do you know, like, you know, where ... Like, why are we saying this is the highest throughput you can get?
- 15:10
Exactly.
- 15:10
Yeah. So the answer is, like, goes to our benchmarking methodology. What we do is first we dump, like, a, you know, a thousand requests and wait for them all to come back, calculate the, like, thou- uh, seconds divided, uh, requests divided by seconds, thousand divided by how long it took.
- 15:26
That's like a maximum throughput, right? 'Cause we gave it maxim- We exposed the maximum parallelism to the engine, so presumably they knew how to ... They were smart enough to handle that.
- 15:35
Gives you a maximum RPS. Any more than that, you should expect from queuing theory that the latency will blow up, right? Then the other side is, um, like, you send one request at a time, wait for it to come back, send another.
- 15:47
And that gives us our, like, that's the fastest you could possibly run the server, and we sweep between to get the numbers that are here. But yeah, cool. All right, I'll, uh, gotta move on to the next talk.
- 15:56
Uh, I'll be outside if you, uh, have any questions. Thank you very much. [audience applauds] [upbeat music]