Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod
Slide chapters
20 slide and demo groups · Timestamps are approximate.
Read the talk
Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod
Chris Vela walks through BenchPod, a dashboard for comparing vLLM configurations, and demonstrates why a batch-token change can speed up long generations without noticeably changing the first-token wait. The deployment decision starts with workload and concurrency—and ends by asking whether the largest model is necessary at all.
From a talk by Chris Vela
At a glance
Ideas worth remembering
Choose the workload, context requirement, and expected concurrency before choosing a serving configuration. A successful single request does not answer the capacity question.
BenchPod combines vLLM configuration experiments, benchmarks, live metrics, and API access. Applying the demonstrated configuration still involves a service restart.
In the demonstrated comparison, increasing the maximum batch-token count barely changed time to first token but reduced inter-token latency, helping long generations finish sooner. The talk does not establish a universal optimal setting.
Measure cold starts as well as serving latency, especially before moving to serverless workers that must start under demand.
A frontier model may be unnecessary for the task. Compare useful capability against hardware cost, and account for tools, context access, and governance when self-hosting.
A working recipe still leaves production questions unanswered
A team wants the latest open model, rents a GPU, and starts vLLM with a published recipe. The model may run, but the team still cannot explain whether that configuration suits its users. Chris Vela, a forward deployed engineer at Runpod, opens with this recurring customer problem: getting an instance online is easier than showing a manager or staff engineer that a change is actually better.
The first configuration question is what the model needs to accomplish. A coding agent carrying out a long task and a service producing basic summaries do not automatically need the same model or context window. Vela pushes back on choosing an enormous context window before establishing that the application needs it. Start with the job, then choose the capacity.
Concurrency adds a second constraint. Finding the cheapest, smallest GPU allocation does not settle whether the service can support the expected number of simultaneous users. Vela specifically calls out KV-cache use as something to consider alongside context length and concurrent demand. The practical question is whether the chosen configuration still works under the traffic it will actually receive, rather than whether one request succeeds.
Before tuning performance, the software must support the model. New model releases can arrive before the necessary serving implementation is mature; an incompatible vLLM version can fail because required model layers are missing or still under development. A Dockerized GPU environment lets a team bring its own image, but it does not resolve that compatibility problem. Version selection belongs in the deployment process, not in a last-minute response to startup errors.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
BenchPod brings configuration and measurements together
Vela would not put a newly released model into production the next day simply because it can be launched. He raises output reliability as another concern, then introduces BenchPod, his dashboard and metrics system built on vLLM. At the time of the recording, the Runpod template was not yet released. Its purpose is to bring configuration changes, benchmarks, and operational visibility into one place; vLLM already supports benchmarking.
The immediate pain is waiting between experiments. In the existing pod workflow, changing a configuration can mean restarting the image, downloading model weights, and loading them into GPU memory. Vela describes waits of roughly thirty minutes for the Kimi setup. That makes even a small configuration comparison expensive in elapsed time. BenchPod aims to make applying and comparing experiments faster; the later demo still describes a restart when an experiment is applied, rather than establishing restart-free reconfiguration.
The packaging is a Docker image based on vLLM’s API-serving setup, with an added FastAPI application. Models can be specified through the system, and agents can use its API. Vela also describes a watchdog that helps the service recover from failures. These are related capabilities with different jobs: the API makes the service and its information accessible, while the watchdog addresses recovery.
The recovery example comes from Runpod’s internal deployment. A Kimi instance went down over the preceding weekend and recovered; Vela says four nodes were running the service and internal users did not notice a problem. That is a useful operational anecdote, not a general availability guarantee: the recording does not explain the traffic-routing or failover mechanism in enough detail to reproduce it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Advertised context and usable context are different deployment choices
The demonstration uses Kimi K3 on a B300 setup. Vela describes the model as supporting a million-token context window, but says this particular one-B300 setup can use only about half of that. His team nevertheless finds that capacity sufficient for long coding-agent tasks. The useful distinction is between a model’s advertised context capability and the context a particular deployment can serve.
The dashboard includes GPU metrics and the CUDA version because compatibility extends below the serving library. In the demonstrated Kimi environment, Vela reports that CUDA 12.8 did not work and CUDA 13.0 did. Treat that as a constraint of his setup, rather than a universal compatibility rule for every Kimi deployment. Exposing the runtime version makes it possible to distinguish a software mismatch from a performance-tuning problem.
BenchPod is intended to answer basic operational questions before the endpoint is attached to a coding agent: does it work, how long does it take to start, and what happens under actual traffic? Startup measurements and serving measurements describe different parts of the experience. A responsive model after initialization can still be awkward to deploy if bringing up a worker takes too long.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The live demo starts with capacity, errors, and recovery
The deployment walkthrough encounters a practical constraint immediately: no B300 capacity is available in the selection interface, although Vela already has an instance running. The setup otherwise consists of choosing the Docker image, adding a storage volume, and deploying. He also describes the option to deploy when capacity becomes available. A configuration can be ready before the hardware is.
Once the running instance is open, the question becomes: what information helps explain its behavior? BenchPod exposes an error scan and live metrics, with the data backed by storage. Those views are also available through FastAPI calls, so an agent can retrieve the information without using the dashboard. The UI gives a human an “eagle’s-eye view”; the API lets software inspect the same operational information.
The failure history supplies a concrete startup measurement: Vela reports a cold start of about nine minutes following the GPU problem. That should not be read as a measured reduction from the earlier thirty-minute example, which included downloading weights and loading them into memory under unspecified conditions. The nine-minute figure describes this recovery, and shows why startup time belongs beside steady-state serving metrics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Change the batch-token budget, then test concurrent demand
The tuning experiment asks a narrow question: what changes when the maximum number of batch tokens increases? BenchPod’s serving interface accepts the vLLM YAML configuration, lets an experiment be applied, and restarts the service. This is a serving-configuration experiment, not a change to the model’s weights. The configuration view matters because it identifies the input being varied before any speed claim is considered.
The benchmark runs were completed before the presentation to avoid spending roughly twenty minutes watching them execute. They test concurrency levels of 1, 8, 64, and 192 users, and BenchPod can compare multiple runs. Testing beyond one user matters because the opening deployment question was about shared demand: a configuration that looks acceptable in an isolated request may behave differently when many requests compete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The first token barely changes; the rest arrive faster
The comparison answers a more useful question than whether one configuration is simply “faster”: which part of a request improves? The reported changes separate the initial wait from the pace of generation.
- Time to first token: Increasing the maximum batch-token count does not change the first-token wait very much in the compared runs.
- Inter-token latency: The delay between generated tokens decreases substantially in Vela’s comparison. He connects that reduction to long jobs finishing faster than with the base vLLM recipe.
The distinction explains why a long coding task can benefit even when its first visible response arrives at about the same time. The initial wait happens once; the intervals between tokens recur throughout generation. Reducing those intervals can shorten a long response without noticeably improving the beginning of the interaction. A first-token-only comparison would miss the reported benefit.
The supported conclusion is specific to this comparison: a larger batch-token budget improved inter-token latency in the demonstrated setup. The spoken results do not supply the before-and-after token budgets, numerical latency values, or prompt and output lengths, so they do not establish a reproducible speedup factor or an optimal setting for another workload. The experiment earns a reason to compare configurations, not a rule to maximize the setting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the metric before choosing the winner
After comparing configurations on a pod, the deployment can move toward a multi-node cluster or serverless workers, depending on how it needs to scale. Serverless makes startup behavior particularly consequential: Vela sees teams copy a base configuration and then struggle while waiting for workers to become ready. The benchmark should inform that deployment choice rather than stop at a steady-state latency result.
A brief live request asks the running Kimi instance what BenchPod is. Vela describes roughly fifty internal users and says many teammates are moving from Claude to this service. That is adoption context, separate from the controlled concurrency tests. He also mentions instances running GLM, DeepSeek, and Qwen, with smarter routing among hosted models as a future direction rather than a demonstrated routing system.
The experiment can be tailored to the expected audience. If the intended service will have 500 people using it continually, that is a demand scenario to test, not a capacity result established by the earlier runs. Vela’s deployment sequence is concrete: pick a GPU machine, load the model, compare configurations, test with the agent, optimize for the metric the application needs, and “ship that winner.”
The vLLM recipe remains a starting point. What it cannot decide is whether this team values the first-token wait, the time between tokens, or some other requirement most. The batch-token example makes that choice tangible: the same change can leave one metric largely unchanged while improving another. Production configuration starts when the team can say which outcome matters and show what happened under its chosen load.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Do you need the frontier model at all?
The ending returns to model selection, now with an infrastructure bill attached. Vela estimates that the B300 node he is discussing costs about $40,000 per month on Runpod, while noting that the price can move. That is his estimate for the node in the recording, not a current quote or a per-GPU price. Powerful hardware can make a deployment possible without making it the sensible choice for a particular team.
His own usage distinguishes the jobs rather than declaring one model the universal winner. A DeepSeek deployment on four H200s does a good job for his work; Kimi is reserved more for long, detailed tasks. The recording does not provide a matched quality or cost comparison between them. The practical decision is whether a smaller model can do the required work well enough before paying for the largest available option.
Self-hosting also leaves responsibilities beyond serving speed. Vela calls out the necessary tools, access to context such as wikis, and AI governance. Governance is described as a next step for Runpod, not a completed BenchPod feature demonstrated here. A fast endpoint still needs the surrounding capabilities that let an agent do useful work.
BenchPod closes as a forthcoming template, with release expected at the end of the month at the time of the talk. Its demonstrated value is making a specific deployment question easier to investigate: change a configuration, inspect startup and serving behavior, and compare the result under concurrent demand. The final question remains whether that model and hardware are worth deploying for the job in the first place.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
A practical starting point for investigating the Pods, Serverless, and Clusters deployment options discussed in the talk; not a BenchPod template link.
- Runpod pricingReference
Check deployment costs when evaluating model and hardware choices rather than treating the recording’s approximate monthly node estimate as a current quote.
Related talks
- Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA
Extends the benchmarking discussion with Docker isolation, warm-up runs, GPU telemetry, and comparisons of throughput and time to first token across local model deployments.
- Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI
Shows inference parameter sweeps in a large-scale generation workload, alongside checkpointing, GPU-failure recovery, and resource scheduling.
Read the complete timestamped transcript
11 slide groups · 9 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate.
Platinum Sponsors
Black slide listing Oracle AI Database, Sonar, WorkOS and exa as platinum sponsors.
Text in image
PLATINUM SPONSORS ORACLE AI Database Sonar Makers of SonarQube WorkOS exa
- 0:12
My name's Chris Vela. Uh, I work at Runpod. I'm a forward deployed engineer. Uh, today I'm gonna be talking about, um, benchmarking vLLMs, uh, kind of like the struggle that people have when like... Especially like a new model comes out. Um, just some like learnings that we've had, like internally at Runpod and with like customers. Uh, so today I'm gonna be showing you, um, s- essentially like a dashboard that, uh, is built on top of like vLLM. Um, it's using our like pods actually. So to give you like a little more context of like what Runpod does, is, uh, we're in
- 0:42
Neocloud. Um, we make it really, really easy for you to rent GPUs, um, starting from like R&D to like more serverless production, uh, workflows. Uh, a lot of times what I see with customers, especially with vLLM, is like someone wants, you know, the latest model, they don't know how to like spin it up, or they, they wanna start going to production and they don't know the, the knobs to like touch. They, they're just using whatever, like the base config. And I'm guessing a lot of you've probably seen like vLLM recipes before. Um, you know, it, it's a good start, uh, especially
- 1:12
in like... But like, especially in this world, we're always moving fast. Um, so how do you get to more a production level, uh, style of code? And at the same time, not just like vibing it, it's more like, how do you actually have data? How do you show, you know, your manager or like your staff engineer that, "Hey, this is actually better. This is actually working correctly"? So
The Problem: LLM Serving Configurations Ship on Guesswork
The slide illustrates the cost of guessing vLLM settings: discovering problems at concurrency 40, in production, at 2am. Vela urges developers to size context limits and GPU resources around the task and expected concurrent users.
Text in image
The problem LLM serving configs ship on guesswork. Wrong vLLM version, wrong max-model-len, wrong parallelism - and you find out at concurrency 40, in production, at 2am. Optimization takes unnecessary time. © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 73°F Tue Oct 13 3:21 PM
- 1:35
really what I'm trying to say is like, I've seen a lot, everyone's just guessing. Like, you know, you're... A lot of times I see people are using the wrong vLLM version. They might-- They're probably not using like the nightly version. Um, they have the wrong size for their context window. Um, it, it really depends. Like, like I, I take it always a step back. It's like, what are you trying to achieve with your, your LLM model? Are you-- Is it gonna be a coding agent? Is it more kind of like you're just, um, you know, doing summaries, like very, very basic tasks? That's usually the first question I would be
The Problem: LLM Serving Configurations Ship on Guesswork
The slide illustrates the cost of guessing vLLM settings: discovering problems at concurrency 40, in production, at 2am. Vela urges developers to size context limits and GPU resources around the task and expected concurrent users.
Text in image
The problem LLM serving configs ship on guesswork. Wrong vLLM version, wrong max-model-len, wrong parallelism - and you find out at concurrency 40, in production, at 2am. Optimization takes unnecessary time. © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 73°F Tue Oct 13 3:21 PM
- 2:05
asking before you, uh, you know, getting Kimi K3, having a, you know, a million, uh, long context. Like, that's not always necessary. Uh, and at the same time it's like, how many users are gonna be using this? Um, a lot of times people are like going... They're-- Everyone's trying to always get like the, the smallest amount of GPU for the, the cheapest price. But if you're gonna be going to have more concurrent users, you need to start thinking about like, how am I gonna be utilizing the KV cache? Uh, so all this is... There's
- 2:35
all these little like optimization knobs that you need to like figure out before you go to production, and a lot of times we're trying to move fast. So, um, yeah, like, like I said, like Runpod, like we like have very much like, you know, we make it very, very easy. It's like a VM, uh, it's Dockerized. Uh, you can bring your own image, but at the end of the day, like, um, like who here has used like vLLM before? Okay. So, you know, like when you're looking for like, like especially like Kimi K3, like if you, you
- 3:05
use like a version of like, you know, 0.27, it's not gonna be working. Like the, like we're moving so fast, the code is not getting as fast. Like you need to have the right version. And a lot of times that turns into kind of like a CDI-- CI/CD process that, um, especially if people who don't understand what's happening, uh, just start struggling. They, they start seeing errors and they're like, "Why is this not working?" You know, it-- The, the layers within it are not... They haven't been written or they're... It's very in development code. Uh, and at the same time that's like actually kind of a problem, is like
- 3:35
Kimi K3 comes out, you know, the next day I would not put that on production just yet. Uh, you're gonna be seeing a lot of probably hallucinations. Uh, it's just not ready in that state, I would say. Um, so what I propose, um, so, uh, I created this thing called, it's called BenchPod. It's gonna be a template. It hasn't come out yet. Um, so it's essentially built on top of like, uh, vLLM. Um, it's a whole metric system. Uh, you can benchmark very easy, and a lot of you who are familiar with like vLLM, you can
Move Fast...with Receipts
Vela pitches BenchPod as an unreleased template for A/B testing on one pod, workload and dashboard. The slide promises 30 minutes from raw model to trusted metrics, with zero infrastructure to manage.
Text in image
Move fast...with receipts 30 min Time from raw model to trusted metrics in one pod A/B One pod, one workload, one dashboard. Infinite tests. 0 Infrastructure you manage to find the truth © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 73°F Tue Oct 13 3:23 PM
- 4:05
benchmark already. But this kind of, um, centralizes everything so you can be able to see everything, be able to change the config very quickly. Uh, that's like a very important thing that I, I notice is that, uh, especially with people who use like our, uh, our pods, is you set up a config, but then you have to restart the image. You have to wait for cold starts to happen. It's all this loading and you're not moving as fast. You're waiting like thirty minutes, like for Kimi K3 to like download the weights, load into
- 4:34
GPU. Uh, it's kinda wasted time. I mean, if you have an agent, you know, you really don't care, but, um, that is a pain point, especially when we're trying to move fast. And at the same time is like the cool thing about Runpod is you can find these machines. So if you have that question, it's like, "I wanna run Kimi K3 on a B300," we have B300s. And you can start going from like, not just like reading about it, you can actually start testing it and seeing if, you know, there's truth to all these metrics. So yeah.
How BenchPod works
BenchPod puts serving, monitoring and self-healing in one eight-GPU pod. The demo changes only max-num-batched-tokens, then repeats the same workload ladder to compare latency tradeoffs without changing pods.
Text in image
How BenchPod works One RunPod Pod (8 GPUs, tensor-parallel, TP=8) shim-<vllm-tag> image stock vllm-openai base, entrypoint cleared boot payload pulls payload at boot, installs packages boots watchdog self-healing: relaunches vLLM on crash / config change vllm serve Kimi K3, 1.45 TiB weights KV fp8 - max-len 524,288 apply: write config, kill, heal dashboard :8090 Serve tab / Metrics tiles - TTFT, tok/s, KV%, per-GPU nginx :8000 OpenAI API (served model endpoint) /workspace persistent volume HF weights cache - vLLM compile cache - benchpod-state (configs, runs) The Demo (18 min) 1 Pod boots idle nothing downloads until a config exists 2 Paste Config A -> Apply batch 16384 - watchdog relaunches serve (9 min load) 3 Bench: workload ladder c1 / c8 / c64 / c192 - fixed seed - 3 repeats - cache reset 4 Same pod: apply Config B ONLY max-num-batched-tokens: 16384 > 32768 5 Same ladder again same pod, same workload, provenance hashed 6 Compare side by side TTFT +19% worse - ITL tail -60% smoother © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 73°F Tue Oct 13 3:25 PM AI Engineer NEW YORK PRESENTED BY arize runpod OCTOBER 13, 2026 EXPO SESSION McCarthy Room
- 5:04
So like benchm- like, like I said, like BenchPod is more like, um... It's, it's built like of, of, um, a Docker image of, um, um, vLLM Open API. We're gonna be having every version already. Um, we just have a, a little like, like payload that like pretty much creates this whole FastAPI application. Uh, you could use it for your agents. Uh, the cool thing is you can serve, uh, you can pretty much specify which, uh, models you wanna have. We have a whole watchdog too, so it like self-heals too. So, uh, if you are actually starting to go to more
- 5:34
production levels, uh, you can actually, uh, you don't have to worry about it. And like I'll show you, uh, like this past Sunday, uh, Kimi went down on our instance and it, uh, self-healed. And, you know, we have, we have right now like four nodes running it, so none of like our internal users saw any problem. It was production ready, um, didn't have to like think about it too much. And at the end of the day, yeah, like it's, it, it makes it much more sim, uh, simple process to, uh, have a
- 6:04
production worklo- workflows. So I'm guessing a lot of people... I'm, I'm gonna be showing Kimi K3 because we've been using that a lot. Um, I'm guessing a lot of people here are very familiar with Kimi K3. Um, so, you know, one, 1 million in context. Uh, one thing that's interesting on one B300, uh, can't get to the 1 million, you gotta use half of it. But one, one thing I've noticed especially with our team, like we're using it a lot for our coding, uh, agents, um, that is very much, um, enough
Why Kimi K3
Kimi K3 has a 1-million-token design, but this deployment serves 524,288 tokens on one eight-B300 pod. The slide pairs that long context with fp8 KV cache to support more concurrent long-context sessions per GPU.
Text in image
Why Kimi K3 1m-token design Serving today 524,288 context on one pod. Long context is where knobs bite hardest Foundation Model Kimi fits on one 8xB300 on RunPod fp8 KV Cache More concurrent long-context sessions per GPU model: moonshotai/Kimi-K3 tensor-parallel-size: 8 trust-remote-code: true load-format: fastsafetensors enable-prefix-caching: true enable-auto-tool-choice: true tool-call-parser: kimi_k3 reasoning-parser: kimi_k3 host: 0.0.0.0 port: 8000 kv-cache-dtype: fp8 attention-config: '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' max-model-len: 524288 max-num-batched-tokens: 16384 enable-prompt-tokens-details: true © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 73°F Tue Oct 13 3:26 PM AI Engineer NEW YORK PRESENTED BY arize runpod OCTOBER 13, 2026 EXPO SESSION McCarthy Room- 6:34
to do long tasks. Um, it's utilizing KV cache, uh, very well too. So, um, in terms of speed of like time to first token, um, it is very, very, very quick. And like in the demo, I'll show you too with like my coding agent running this Kimi, uh, model on a Runpod. So this is like our dashboard. Um, has a lot of metrics. I know it can, it look, looks like a lot, but uh, essentially, you know, we're always thinking about like the GPU, like, you know, have a lot of like GPU, uh,
The Dashboard
The vLLM-based dashboard combines host-machine and live serving metrics with benchmarking, configuration management and a chat playground. Users can test a deployment before connecting a coding agent.
Text in image
The Dashboard Our Dashboard is built on top of vLLM Key Features • In Depth Metrics to the Host Machine • Live Metrics from vLLM • Benchmark Tab • Serving Tab • Config Management • Self Healing • OpenAI Chat Playground Embedded dashboard screenshot: runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out STATE SERVING MODEL moonshotai/Kimi-K3 CONFIG SOURCE custom [illegible] ENGINE 0.29.0 [illegible] SERVE AGE 25.8h BENCH idle GPUS GPU 0 [illegible] GPU 1 [illegible] GPU 2 [illegible] GPU 3 [illegible] GPU 4 [illegible] GPU 5 [illegible] GPU 6 [illegible] GPU 7 [illegible] STARTUP CACHES refresh sizes [illegible] CACHE PATH PERSISTENT SIZE HF_HOME [illegible] VLLM_CACHE_ROOT /workspace/.cache/vllm [illegible] WEIGHTS CACHE (HF) refresh [illegible] CACHED REPO SIZE moonshotai/Kimi-K3 [illegible] © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 73°F Tue Oct 13 3:26 PM AI Engineer NEW YORK PRESENTED BY arize runpod Engineering the future of AI
- 7:04
metrics, knowing what CUDA version you're on, which is very, very important. You know, if you're running Kimi, it doesn't work on a 12.8 CUDA, but it works on 13.0. Uh, and really it's... What we have is like the bench- We can, we can look at benchmarking, look at all our data. We can like play around with it. So like s- before you even put it into your like coding agent or, uh, just hitting the OpenAI, uh, response, you can start testing real quick if this is working. And just a lot of like more CI/CD processes of like, you know, how long are my cold starts
- 7:34
happening? Um, how are, how are my metrics looking, um, through actual traffic? So I'm gonna show you guys the demo, so...
- 7:45
So really like, I'm gonna show you real quick, like Runpod itself. So, um, this will-- So we don't have this templated up yet. Like I said, um, it'll be up probably at the end of the month. Uh, but really all we have is just like a Docker image. We can be able to, uh, put that in right there, if it loads correctly. But, uh, yeah, so right now, uh, there's not B300s, but, uh, I have one running right now. That's the fun thing about working in this industry. Uh, you know, uh, supply can be, uh,
- 8:16
dependent. Uh, but- But yeah, s- In, in the sense of that, like, you know, we make it really, really easy. You can put storage volume. Uh, you deploy it, you know, uh, we have a part two, so you can deploy when you're available. So, you know, if you were seeing the screen like this, it will deploy whenever, uh, it get, it gets started. Uh, so that's like literally the process. You click that button, BenchPod like starts up. Um, and I can show you like right here.
- 8:47
I have it on the wrong one, but I'll show you guys here. So like, this is like, this is a- our actual like Kimi running currently. Um, so a cool thing is like, you know, you, you start... Like it has a whole error scan, so you have actual data to know like what is happening. Um, cool thing is we have live metrics too, as like things are running. Uh, all of this is also like backed up in like storage data, so like it's all, um, um, backed up no matter what. Um, it's all face- FastAPI calls, so your agent could easily just
Live metrics: six-hour view
BenchPod’s six-hour view shows brief bursts of cache use, requests and prompt throughput, with a 100% prefill / 0% decode split. The visible error-scan categories all show zero.
Text in image
LIVE METRICS Timeframes: 1h, 6h, 24h KV cache usage Legend: kv cache % Y-axis ticks: 0, 2, 4, 6, 8, 10, 12 Timeline: 2:28:59 PM, 2:36:41 PM, 2:44:19 PM, 2:51:57 PM, 2:59:34 PM, 3:07:12 PM, 3:14:50 PM, 3:22:28 PM Requests Legend: running, waiting Y-axis ticks: 0, 0.4, 0.8, 1.2, 1.6, 2.0 Timeline: 2:28:59 PM, 2:36:41 PM, 2:44:19 PM, 2:51:57 PM, 2:59:34 PM, 3:07:12 PM, 3:14:50 PM, 3:22:28 PM Token throughput · window split 100% prefill / 0% decode Legend: prompt tok/s, gen tok/s Y-axis ticks: 0, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000 Timeline: 2:28:59 PM, 2:36:41 PM, 2:44:19 PM, 2:51:57 PM, 2:59:34 PM, 3:07:12 PM, 3:14:50 PM, 3:22:28 PM TTFT (per-interval mean) Legend: TTFT ms Y-axis ticks: 0, 200, 400, 600, 800, 1,000, 1,200, 1,400 Timeline: [illegible] TPOT (per-interval mean) Legend: TPOT ms Y-axis ticks: 0, 0.2, 0.4, 0.6, 0.8, 1.0 Timeline: [illegible] Prefix cache hit rate Legend: hit rate % Y-axis ticks: 0, 20, 40, 60, 80, 100 Timeline: [illegible] CUDA error 0 – CUDA out of memory 0 – Engine died 0 – ERROR log lines 0 – NCCL warning/error 0 – Sequence preemption 0 – Timeout 0 – Python traceback 0 – Interface text: runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out Chrome File Edit View History Bookmarks Profiles Tab Window Help Benchpod Presentation.pptx My Pods RunPod · vLLM BenchPod runpod/vllm-benchpod: [illegible] Benchpod Presentation.pptx [illegible]-8090.proxy.runpod.net Ask Gemini Work 72°F Tue Oct 13 3:29 PM AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room
9:03Demo milestone
Live metrics: 24-hour view
BenchPod’s 24-hour view lines up bursts in requests, KV cache usage and token throughput. The window split is 99% prefill / 1% decode, with a prominent spike in time to first token also visible.
Text in image
LIVE METRICS 1h 6h 24h Selected range: 24h KV cache usage Legend: kv cache % Y-axis ticks: 0, 5, 10, 15, 20, 25, 30, 35, 40 Timeline: 3:28:30 PM, 6:31:29 PM, [illegible], 12:38:41 AM, [illegible], [illegible], 9:47:00 AM, 12:50:09 PM Requests Legend: running; waiting Y-axis ticks: 0, 2, 4, 6, 8, 10, 12 Timeline: 3:28:30 PM, 6:31:29 PM, [illegible], 12:38:41 AM, [illegible], [illegible], 9:47:00 AM, 12:50:09 PM Token throughput · window split 99% prefill / 1% decode Legend: prompt tok/s; gen tok/s Y-axis ticks: 0, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000 Timeline: 3:28:30 PM, 6:31:29 PM, [illegible], 12:38:41 AM, [illegible], [illegible], 9:47:00 AM, 12:50:09 PM Tooltip: 10:28:30 AM prompt tok/s: 0 gen tok/s: 0 TTFT (per-interval mean) Legend: TTFT ms Y-axis ticks: 0, 1,000, 2,000, 3,000, 4,000 Timeline: [illegible; clipped] TPOT (per-interval mean) Legend: TPOT ms Y-axis ticks: 0, 0.2, 0.4, 0.6, 0.8, 1.0 Timeline: [illegible; clipped] Prefix cache hit rate Legend: hit rate % Y-axis ticks: 0, 20, 40, 60, 80, 100 Timeline: [illegible; clipped] CUDA error 0 – CUDA out of memory 0 – Engine died 0 – ERROR log lines 0 – NCCL warning/error 0 – Sequence preemption 0 – Timeout 0 – Python traceback 0 – Recording overlay: AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:29 PM Benchpod Presentation.pptx My Pods RunPod · vLLM BenchPod runpod/vllm-benchpod: Run[illegible] Benchpod Presentation.pptx Ask Gemini [illegible]-8090.proxy.runpod.net Work runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out
9:07Demo milestone
- 9:17
call all this data. You can actually just serve everything. Like you don't even have to use the UI. I, I think I'm a big proponent of like, as we go to agents, it's like I still wanna have like some awareness of what's happening, and having this gives me a good like, like eagle's-eye view of like what is happening with my, uh, Kimi K3, K3 instance. Uh, but yeah, like you can see here, like I was gonna show you, like, uh, this past weekend we had a failure with a GPU. Uh, we were able to fix it, but, uh, being able to just do a cold start time of
Restart history and latency trend
Vela highlights roughly nine-minute cold starts after a GPU failure. BenchPod separates startup time from total outage—one row shows 8m 58s versus 22.7h—and plots recurring spikes in hourly time-to-first-token.
Text in image
Watchdog activity {"t":1791897693,"event":"recovered","downs":0} {"t":1791896773,"event":"restart_issued","downs":2} {"t":1791896773,"event":"restart_begin","downs":2} {"t":1791896753,"event":"unhealthy","downs":1} {"t":1791827693,"event":"recovered","downs":0} {"t":1791826773,"event":"restart_issued","downs":2} {"t":1791826773,"event":"restart_begin","downs":2} {"t":1791826753,"event":"unhealthy","downs":1} {"t":1791826753,"event":"started","downs":0} {"t":1791825921,"event":"restart_issued","downs":2} {"t":1791825921,"event":"restart_begin","downs":2} {"t":1791825901,"event":"unhealthy","downs":1} Server log refresh full logs [illegible] (APIServer pid=57165) INFO: 127.0.0.1:58920 - "GET /metrics HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58926 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58932 - "GET /version HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58942 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58944 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58946 - "GET /version HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58962 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58968 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58972 - "GET /version HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58984 - "GET /metrics HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58990 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:57972 - "GET /health HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:57988 - "GET /version HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:57998 - "GET /metrics HTTP/1.1" 200 OK (APIServer pid=57165) INFO: 127.0.0.1:58006 - "GET /health HTTP/1.1" 200 OK HISTORY median cold start 8m 58s · min 8m 49s · max 37m 5s · 10 restarts recorded · 90 failed Restarts / cold starts TIME (UTC) | COLD START | OUTAGE | SOURCE | RESULT 10-13 13:15 | 8m 54s | 9m 29s | exp:base | up 10-12 17:48 | 8m 58s | 22.7h | exp:base | up 10-12 17:26 | – | – | custom | failed 10-12 17:07 | – | – | custom | failed 10-12 16:51 | – | – | custom | failed 10-12 16:35 | – | – | custom | failed 10-12 16:20 | – | – | custom | failed 10-12 16:04 | – | – | custom | failed 10-12 15:48 | – | – | custom | failed 10-12 15:33 | – | – | custom | failed 10-12 15:17 | – | – | custom | failed TTFT trend (hourly, all-time) Legend: TTFT ms (hourly mean); requests finished/h Left Y-axis: TTFT ms Left Y-axis ticks: 0, 4,000, 8,000, 12,000, 16,000 Right Y-axis: req/h Right Y-axis ticks: 0, 200, 400, 600, 800, 1,000, 1,200, 1,400 Timeline: 09-30 10:00, 10-01 00:00, 10-01 14:00, 10-02 04:00, 10-02 18:00, 10-03 10:00, 10-04 00:00, 10-04 14:00, 10-05 06:00, 10-05 20:00, 10-06 10:00, 10-07 00:00, 10-07 14:00, 10-08 06:00, 10-08 20:00, 10-09 10:00, 10-10 00:00, 10-10 14:00, 10-11 06:00, 10-11 20:00, 10-12 10:00, 10-13 00:00, 10-13 18:00 Interface text: runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out Chrome File Edit View History Bookmarks Profiles Tab Window Help Benchpod Presentation.pptx My Pods RunPod · vLLM BenchPod runpod/vllm-benchpod: [illegible] Benchpod Presentation.pptx 0snz68816fsiiv-8090.proxy.runpod.net Ask Gemini Work 72°F Tue Oct 13 3:29 PM AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room9:43Demo milestone
- 9:47
like nine minutes on it, like K3 is pretty, pretty good. Um, and the cool thing here is that we have the serve. So you can start out experiments and essentially like apply them. It'll restart real quick. Um, you can... It's literally just like the config YAML file from like vLLM. And um, so like for this like little demo, what I tested was we were just ch- changing the max number of batch tokens, uh, that were sent through. Uh, and I kinda wanted to show you guys like, like, you know, what, what kind of like speed
Baseline serving configuration
BenchPod shows the baseline vLLM YAML with max-num-batched-tokens set to 16384. Vela opens the saved-experiment menu to prepare a batching comparison; the baseline remains staged, with an apply-and-restart control below.
Text in image
Serve configuration The YAML handed to vllm serve --config. Pick a saved config from the source list, or upload/paste your own: any edit saves as a custom staged config. “Apply & restart” writes the serving sidecars and lets the watchdog fast-heal relaunch vLLM. Source ✓ base (ui) base-increase-max-tokens (ui) custom (edited) Delete UI experiment Extra CLI args (appended after --config; CLI wins) --max-num-seqs 256 Save as staged Apply & restart serving Config YAML staged: exp:base model: moonshotai/Kimi-K3 tensor-parallel-size: 8 trust-remote-code: true load-format: fastsafetensors enable-prefix-caching: true enable-auto-tool-choice: true tool-call-parser: kimi_k3 reasoning-parser: kimi_k3 host: 0.0.0.0 port: 8000 kv-cache-dtype: fp8 attention-config: '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' max-model-len: 524288 max-num-batched-tokens: 16384 enable-prompt-tokens-details: true Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:29 PM Benchpod Presentation.pptx My Pods RunPod · vLLM BenchPod runpod/vllm-benchpod: Run[illegible] Benchpod Presentation.pptx Ask Gemini [illegible]-8090.proxy.runpod.net Work runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room9:51Demo milestone
BenchPod serving configuration: increased batched tokens
Chris selects a saved experiment with max-num-batched-tokens set to 32768. BenchPod shows the staged vLLM YAML beside save and restart controls; this demo tests how changing the batch-token limit affects speed.
Text in image
Serve configuration The YAML handed to vllm serve --config. Pick a saved config from the source list, or upload/paste your own: any edit saves as a custom staged config. “Apply & restart” writes the serving sidecars and lets the watchdog fast-heal relaunch vLLM. Source base-increase-max-tokens (ui) Save as new experiment... Upload config file... Delete UI experiment Extra CLI args (appended after --config; CLI wins) --max-num-seqs 256 Save as staged Apply & restart serving Config YAML staged: exp:base-increase-max-tokens model: moonshotai/Kimi-K3 tensor-parallel-size: 8 trust-remote-code: true load-format: fastsafetensors enable-prefix-caching: true enable-auto-tool-choice: true tool-call-parser: kimi_k3 reasoning-parser: kimi_k3 host: 0.0.0.0 port: 8000 kv-cache-dtype: fp8 attention-config: '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' max-model-len: 524288 max-num-batched-tokens: 32768 enable-prompt-tokens-details: true Interface text: runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out Chrome File Edit View History Bookmarks Profiles Tab Window Help Benchpod Presentation.pptx My Pods RunPod - vLLM BenchPod runpod/vllm-benchpod: Run[illegible] Benchpod Presentation.pptx [illegible]-8090.proxy.runpod.net Ask Gemini Work 72°F Tue Oct 13 3:29 PM AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room10:01Demo milestone
Show 2 builds
- 10:01
Serve configuration page with an increased-token experiment selected and staged YAML displaying a maximum batched-token setting of 32768 beside save and restart controls.
Text in image
Serve configuration The YAML handed to vllm serve --config. Pick a saved config from the source list, or upload/paste your own: any edit saves as a custom staged config. “Apply & restart” writes the serving sidecars and lets the watchdog fast-heal relaunch vLLM. Source base-increase-max-tokens (ui) Save as new experiment... Upload config file... Delete UI experiment Extra CLI args (appended after --config; CLI wins) --max-num-seqs 256 Save as staged Apply & restart serving Config YAML staged: exp:base-increase-max-tokens model: moonshotai/Kimi-K3 tensor-parallel-size: 8 trust-remote-code: true load-format: fastsafetensors enable-prefix-caching: true enable-auto-tool-choice: true tool-call-parser: kimi_k3 reasoning-parser: kimi_k3 host: 0.0.0.0 port: 8000 kv-cache-dtype: fp8 attention-config: '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' max-model-len: 524288 max-num-batched-tokens: 32768 enable-prompt-tokens-details: true Interface text: runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out Chrome File Edit View History Bookmarks Profiles Tab Window Help Benchpod Presentation.pptx My Pods RunPod - vLLM BenchPod runpod/vllm-benchpod: Run[illegible] Benchpod Presentation.pptx [illegible]-8090.proxy.runpod.net Ask Gemini Work 72°F Tue Oct 13 3:29 PM AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room - 10:02
BenchPod Serve tab with a staged YAML configuration, a maximum batched-token setting, and controls to save or apply and restart serving.
Text in image
Serve configuration The YAML handed to vllm serve --config. Pick a saved config from the source list, or upload/paste your own: any edit saves as a custom staged config. “Apply & restart” writes the serving sidecars and lets the watchdog fast-heal relaunch vLLM. Source base-increase-max-tokens (ui) Save as new experiment... Upload config file... Delete UI experiment Extra CLI args (appended after --config; CLI wins) --max-num-seqs 256 Save as staged Apply & restart serving Config YAML staged: exp:base-increase-max-tokens model: moonshotai/Kimi-K3 tensor-parallel-size: 8 trust-remote-code: true load-format: fastsafetensors enable-prefix-caching: true enable-auto-tool-choice: true tool-call-parser: kimi_k3 reasoning-parser: kimi_k3 host: 0.0.0.0 port: 8000 kv-cache-dtype: fp8 attention-config: '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' max-model-len: 524288 max-num-batched-tokens: 32768 enable-prompt-tokens-details: true Interface text: runpod vLLM BenchPod Overview Serve Playground Benchmarks SERVING admin Sign out Chrome File Edit View History Bookmarks Profiles Tab Window Help Benchpod Presentation.pptx My Pods RunPod - vLLM BenchPod runpod/vllm-benchpod: Run[illegible] Benchpod Presentation.pptx [illegible]-8090.proxy.runpod.net Ask Gemini Work 72°F Tue Oct 13 3:30 PM AI Engineer NEW YORK PRESENTED BY arize AI Engineer
- 10:01
- 10:16
ups would happen with this. Um, I'll go right here. So I ran these before because we're not gonna spend, uh, like 20 minutes just looking at, uh, benchmarks. So these benchmarks are running like concurrency, so they're, they're running, um, you know, starting with like just one concurrent user and then eight, 64, 192. Um, and the cool thing is like we can be at-- we can actually compare multiple runs against each other. Um, you can
Benchmark results across concurrency levels
As concurrency rises from 1 to 192, the increased-token run’s throughput grows from 114.9 to 3,131.6 tok/s, while median time to first token rises from 211.8 to 4,529.2 ms: more output, longer initial waits.
Text in image
Run a benchmark Drives bench/run.sh on the pod against the currently serving config: the workload ladder × 3 repeats + 1s DCGM/engine capture. One session at a time; serve restart is locked out while a session runs. base Run benchmark Runs EXPERIMENT | RUN ID | VLLM | CACHE RESETS | RESULTS base-increase-max-tokens | ui-20261008T231255Z | 0.29.0 | yes | ✓ | view base | ui-20261007T073649Z | 0.29.0 | yes | ✓ | view base-increase-max-tokens · ui-20261008T231255Z vllm 0.29.0 · cache resets: yes · 816180 /usr/bin/python3 /usr/local/bin/vllm serve --config /workspace/benchpod-state/experiments/base-increase-max-tokens/config.yaml --port 8000 RUNG | CONC. | DONE | ERR | OUT TOK/S | TTFT MED (MS) | TPOT MED (MS) | ITL P99 (MS) c1 | 1 | 48 | 0 | 114.9 | 211.8 | 8.4 | 24.2 c8 | 8 | 144 | 0 | 548.1 | 541.8 | 13.9 | 33.5 c64 | 64 | 384 | 0 | 2,038.7 | 2,348 | 30.4 | 77.8 c192 | 192 | 768 | 0 | 3,131.6 | 4,529.2 | 47.5 | 338.8 Output throughput (tok/s) Y-axis ticks: 0, 500, 1,000, 1,500, 2,000, 2,500, 3,000, 3,500 X-axis labels: c1, c8, c64, c192 TTFT median (ms) Y-axis ticks: 0, 500, 1,000, 1,500, 2,000, 2,500, 3,000, 3,500, 4,000, 4,500, 5,000 X-axis labels: c1, c8, c64, c192 TPOT median (ms) Y-axis ticks: 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 X-axis labels: c1, c8, c64, c192 Interface text: Chrome File | Edit | View | History | Bookmarks | Profiles | Tab | Window | Help 72°F Tue Oct 13 3:30 PM Benchpod Presentation.pptx My Pods RunPod · vLLM BenchPod runpod/vllm-benchpod: [illegible] Benchpod Presentation.pptx Ask Gemini [illegible]-8090.proxy.runpod.net Work runpod vLLM BenchPod Overview | Serve | Playground | Benchmarks SERVING admin Sign out AI Engineer NEW YORK PRESENTED BY arize Engineering the future of AI
10:27Demo milestone
- 10:46
kinda see like even just increasing the, the match, the, the number of batch tokens doesn't really change that much the like time to first token. Uh, but interesting thing that I did notice of increasing that is like the inter-token latency actually decreases quite a bit when you increase that, um, uh, batch tokens. Uh, which means it's like the request itself, if it's especially running like a long, um, uh, like job essentially, uh, it's actually much
Baseline versus increased batched-token results
At 64 and 192 concurrent users, increasing batched tokens cuts p99 inter-token latency by 60.1% and 47.7%, while median time to first token rises about 20%. Throughput stays nearly flat.
Text in image
Compare runs — tick two checkboxes in the runs table (results runs only) comparing base/ui-20261007T073649Z ↔ base-increase-max-tokens/ui-20261008T231255Z METRIC / RUNG | BASE · UI-20261007T073649Z | BASE-INCREASE-MAX-TOKENS · UI-20261008T231255Z | Δ (A→B) c1 · out tok/s | 113.12 | 114.9 | +1.6% c1 · TTFT med (ms) | 216.59 | 211.83 | −2.2% c1 · TPOT med (ms) | 8.41 | 8.44 | +0.4% c1 · ITL p99 (ms) | 20.29 | 24.23 | +19.4% c8 · out tok/s | 583.42 | 548.07 | −6.1% c8 · TTFT med (ms) | 542.45 | 541.85 | −0.1% c8 · TPOT med (ms) | 13.44 | 13.94 | +3.7% c8 · ITL p99 (ms) | 32.31 | 33.55 | +3.8% c64 · out tok/s | 2,004.03 | 2,038.66 | +1.7% c64 · TTFT med (ms) | 1,966.71 | 2,348.04 | +19.4% c64 · TPOT med (ms) | 30.93 | 30.36 | −1.9% c64 · ITL p99 (ms) | 195.08 | 77.78 | −60.1% c192 · out tok/s | 3,121.47 | 3,131.58 | +0.3% c192 · TTFT med (ms) | 3,781.46 | 4,529.15 | +19.8% c192 · TPOT med (ms) | 47.56 | 47.47 | −0.2% c192 · ITL p99 (ms) | 648.18 | 338.8 | −47.7% Left chart: Y-axis ticks: 0, 500, 1,000, 1,500, 2,000, 2,500, 3,000, 3,500 X-axis labels: c1, c8, c64, c192 Middle chart: Y-axis ticks: 0, 500, 1,000, 1,500, 2,000, 2,500, 3,000, 3,500, 4,000, 4,500, 5,000 X-axis labels: c1, c8, c64, c192 Right chart: Y-axis ticks: 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 X-axis labels: c1, c8, c64, c192 Interface text: runpod · vLLM BenchPod Overview · Serve · Playground · Benchmarks SERVING · admin · Sign out Chrome · File · Edit · View · History · Bookmarks · Profiles · Tab · Window · Help Benchpod Presentation.pptx · My Pods · RunPod - vLLM BenchPod · runpod/vllm-benchpod: Run[illegible] · Benchpod Presentation.pptx [illegible]-8090.proxy.runpod.net Ask Gemini · Work 72°F · Tue Oct 13 3:31 PM AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room
10:55Demo milestone
- 11:16
faster finishing the job than the, the base like config that like you find on vLLM Recipe. Um, the cool thing is, is like you can add this to an agent and just start going crazy with it, and you can start getting those answers much, much quicker, uh, have the data, and move from this. If you have it like on a pod, you can start moving to, you know, a multi-node, uh, cluster. Uh, you can move to serverless too. So if you're gonna be like scaling up and down, um, that, that--
- 11:46
this is a perfect way of like actually getting data. And just to give you like a context on our serverless, like serverless, a lot of people will do the same thing. They, they just put whatever the base config is and then are struggling as like they're waiting for workers to start up. Um, this actually gets you the real data. You're now ready to go to production. Um, makes it really, really simple, um, and just gives b- It's... I, I like I always say, it's basing everything in facts. And, um, I can like show you guys right here, like this is
BenchPod runtime and benchmarking overview
Chris shows Kimi explaining BenchPod: a watchdog restarts serving after dashboard config changes, while benchmark sessions leave serving untouched and compare repeated runs using medians and spreads.
Text in image
5. Otherwise it preflights weights (hf download only if not in the HF cache), records /run/benchpod-active-config, and execs scripts/serve.sh — a flock’d vllm serve self-healed by scripts/watchdog.sh. Serving surface and the restart contract • 8000 = OpenAI API direct · 8001 = same API behind nginx (friendly 503 while loading) · 8090 = dashboard (nginx → FastAPI on 8099). Auth everywhere is Bearer $VLLM_API_KEY; the template default renders to sk-<pod-id> per pod. • The dashboard never launches vLLM or touches GPUs. "Apply & restart" writes the sidecars (/run/benchpod-active-config + /run/benchpod-extra-args) and kills vllm serve; the watchdog's fast-heal relaunches serve.sh from the sidecars, skipping weights preflight. Restart is refused while a bench session runs. Dashboard (8090) — one FastAPI process plus a static SPA, no added dependencies. Metrics (5s sampling of vLLM /metrics: KV-cache %, queue depth, token rates, TTFT; per-GPU tiles; 1h/6h/24h charts), Serve tab (edit or pick a config, extra args, staged → apply, save-as-experiment), History (boot journal with cold-start/outage timing, hourly TTFT trend, OOM scan of the server log), Bench tab (runs the harness with a live console, side-by-side run comparison, built-in quickstart workload, in-tab docs), Playground (chat proxy — the Bearer key is injected server-side; thinking traces fold), and Storage (cache dirs + whether they survive restarts, error scan). Bench harness • Drivers: bash /opt/benchpod/bench/run.sh <name> [run-id] on the pod, or bench/session.sh from a laptop (POD_ID + RUNPOD_TEAM_API_KEY; the run dir is pulled back locally). • The workload is a ladder of bench params: every non-underscore key passes through to vllm bench serve, _benchmark_name is unique per rung; saved experiments can carry their own workload.json. • Per rung × 3 repeats: one vllm bench serve, a /metrics snapshot, a prefix-cache reset — all sampled at 1s (dcgm-exporter DCP counters, nvidia-smi dmon fallback, engine gauges). Output lands in the state-dir experiments tree as results.json (per-rung medians + spreads) plus a serving.json provenance record, rendered in the dashboard; never committed to git. • Hard rules: a session is health-gated and refuses a broken server; it never pauses, restarts, or reconfigures serving. Comparing configs = serve A, run it, serve B, run again — a delta is real only when it exceeds both runs' spreads. No automated winner selection; analysis is a human(+assistant) reading. State, tests, CI • Everything persistent lives under /workspace/benchpod-state/ on the pod volume — staged config, experiments, metrics/boot journals, run index — and the pod never writes back to git. Stop keeps the volume (warm reboot); terminate wipes it. The template env also pins weights (HF_HOME) and vLLM's compile caches (VLLM_CACHE_ROOT) onto /workspace so restarts skip re-download and re-compile. • Tests: offline unit tests in tests/unit; the tier-1 integration suite runs from outside against a live pod (BENCHPOD_BASE_URL/BENCHPOD_API_KEY), with BENCHPOD_HAS_TOOLS=1/BENCHPOD_HAS_REASONING=1 gating parser-dependent suites. • CI: build.yml publishes payload + shim on push, shim-matrix.yml keeps a shim per upstream release daily, support-tag.yml builds any single requested tag. CI holds only Docker Hub credentials; HF_TOKEN exists only as pod env. In one line: BenchPod is the run-any-vLLM-model-on-a-pod template whose whole design — shim for bootability, payload for iteration, dashboard for control, bench for comparison — exists to make "whatever model + whatever vLLM tag" a template choice instead of a build. Interface text: kimi (kimi-code) #1 [illegible]presentation (-zsh) #2 can you explain Runpod? (kimi) #3 + > Kimi K3 (Runpod) thinking: max ~/Projects/vllm-benchpod master [+1 -1] ctrl+o expand context: 23% (92.8k/420k)
12:14Demo milestone
- 12:16
actually like Kimi running. So it's just like, what is a BenchPod? Misspelled it, sorry
- 12:24
But that, this is little Kimi instance running through it. Um, yeah, and, um, you know, it is pretty, pretty quick. Like m- like, I would say, like, there's like probably like 50 users right now on our internal one, and we... most of us are getting off Claude and just using this. And, uh, we have other instances with like GLM, uh, DeepSeek, uh, Qwen 3.8. Um, but yeah, like, w- I think in the future, we're looking to more at like smart routers, like how, how do we host all this without being reliant
Concise BenchPod explanation
Kimi answers a BenchPod question in the terminal: one GPU pod combines model serving, a dashboard and benchmarks. Its explanation separates the shim that boots a vLLM tag from the payload that supplies the runtime.
Text in image
gauges). Output lands in the state-dir experiments tree as results.json (per-rung medians + spreads) plus a serving.json provenance record, rendered in the dashboard; never committed to git. • Hard rules: a session is health-gated and refuses a broken server; it never pauses, restarts, or reconfigures serving. Comparing configs = serve A, run it, serve B, run again — a delta is real only when it exceeds both runs' spreads. No automated winner selection; analysis is a human(+assistant) reading. State, tests, CI • Everything persistent lives under /workspace/benchpod-state/ on the pod volume — staged config, experiments, metrics/boot journals, run index — and the pod never writes back to git. Stop keeps the volume (warm reboot); terminate wipes it. The template env also pins weights (HF_HOME) and vLLM's compile caches (VLLM_CACHE_ROOT) onto /workspace so restarts skip re-download and re-compile. • Tests: offline unit tests in tests/unit; the tier-1 integration suite runs from outside against a live pod (BENCHPOD_BASE_URL/BENCHPOD_API_KEY), with BENCHPOD_HAS_TOOLS=1/BENCHPOD_HAS_REASONING=1 gating parser-dependent suites. • CI: build.yml publishes payload + shim on push, shim-matrix.yml keeps a shim per upstream release daily, support-tag.yml builds any single requested tag. CI holds only Docker Hub credentials; HF_TOKEN exists only as pod env. In one line: BenchPod is the run-any-vLLM-model-on-a-pod template whose whole design — shim for bootability, payload for iteration, dashboard for control, bench for comparison — exists to make "whatever model + whatever vLLM tag" a template choice instead of a build. ✦ What is benhpod The user asks "what is benhpod" — a typo for "benchpod" (letters transposed: b-e-n-h-p-o-d). Given the conversation context, they're asking about benchpod again. This is the third time: "also explain benchpod in a concise way", then "explain benchpod", now "what is benhpod". … (6 more lines, ctrl+o to expand) BenchPod (spelled "benchpod" — the repo vllm-benchpod) is a public Runpod template: one GPU pod that can serve and benchmark any Hugging Face model with vLLM, with no repo checkout needed. In practice a user picks a shim-<vllm-tag> image on a Runpod pod template, deploys, and gets on that pod: • vllm serve behind an OpenAI-compatible API (watchdog-supervised, weights + compile caches persisted on /workspace), • a dashboard (port 8090) for metrics, serving-config control, and a playground, • a benchmark harness that runs health-gated sessions against whatever is currently serving. Mechanically it works via a shim + payload split: the shim image makes any upstream vLLM tag bootable as a pod, and the boot payload delivers this repo's runtime (dashboard, bench, scripts) as a small scratch image, so nothing is ever cloned or rebuilt per model. The two earlier answers above cover the same thing in short and long form if you want depth. Interface text: kimi (kimi-code) [illegible]1 …presentation (-zsh) [illegible]2 can you explain Runpod? (kimi) [illegible]3 + > Kimi K3 (Runpod) thinking: max ~/Projects/vllm-benchpod master [+1 -1] ctrl+o expand context: 23% (93.3k/420k)
12:27Demo milestone
- 12:54
on someone else? Um, yeah, so let me go back to our, uh,
- 13:02
thing real quick. So some like important data that we were just talking about if anyone wants to screenshot that. But, uh, like I said, like, you can, you can specify the experiment yourself too. So if you were like, "I'm gonna have like 500 people using this all the time," like, "I wanna like see how well this is gonna be." Um, but at the end of the day, like I said, is how do we go from R&D to production? And I think this is where like BenchPod will make it a lot more easier for people to use.
One pod. Two configs. Proof.
Longer batch tokens trade higher first-token latency for smoother streaming at c64: median TTFT rises 19.4%, while p99 inter-token latency falls from 195 to 78 ms. Output throughput increases only 1.7%.
Text in image
One pod. Two configs. Proof. Metric | Config A - baseline | Longer Batch Tokens | Delta Median TTFT @ c1 | 216.59 | 211.83 | -2.2% Median TPOT @ c8 | 13.44 | 13.94 | +3.7% TTFT med @ c64 (ms) | 1,966 | 2,348 | +19.4% Output tok/s @ c64 | 2,004 | 2,048 | +1.7% ITL p99 @ c64 (ms) | 195 | 78 | -60% smoother © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:33 PM
The pattern to take home
Benchmark on one pod, compare configs on the same workload, then deploy the winner to Serverless. Choose the winner for the latency or throughput you need, record its configuration, and repeat when the model or vLLM version changes.
Text in image
The pattern to take home 1. Bench on a pod Pick the model + vLLM tag. Serve a config, measure TTFT / TPOT / throughput. 2. Compare configs Same workload, side by side. Provenance recorded: cmdline, vLLM version, config. 3. Ship the winner The winning config goes to Serverless with our vLLM template. Measure on one GPU, serve on fleet and scale. 4. Repeat Every new model, every new vLLM version. Thirty minutes, one pod, no CI. © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize AI Engineer Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:33 PM
Show 2 builds
- 13:32
Four-step workflow along a purple gradient timeline: benchmark on a pod, compare configurations, ship the winner to Serverless, and repeat for new models and versions.
Text in image
The pattern to take home 1. Bench on a pod Pick the model + vLLM tag. Serve a config, measure TTFT / TPOT / throughput. 2. Compare configs Same workload, side by side. Provenance recorded: cmdline, vLLM version, config. 3. Ship the winner The winning config goes to Serverless with our vLLM template. Measure on one GPU, serve on fleet and scale. 4. Repeat Every new model, every new vLLM version. Thirty minutes, one pod, no CI. © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize AI Engineer Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:33 PM
- 13:49
Four-step workflow: benchmark on a pod, compare configurations, ship the winner to Serverless, and repeat for new models and vLLM versions.
Text in image
The pattern to take home 1. Bench on a pod Pick the model + vLLM tag. Serve a config, measure TTFT / TPOT / throughput. 2. Compare configs Same workload, side by side. Provenance recorded: cmdline, vLLM version, config. 3. Ship the winner The winning config goes to Serverless with our vLLM template. Measure on one GPU, serve on fleet and scale. 4. Repeat Every new model, every new vLLM version. Thirty minutes, one pod, no CI. © 2026 Runpod Inc. All rights reserved. Confidential and proprietary. AI Engineer NEW YORK PRESENTED BY arize Engineering the future of AI Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:33 PM
- 13:32
- 13:32
Um, so like at the end of the day, like what I want you guys to like know is like, um, you know, pick a pod, get like a GPU machine. You have your model, compare configs. Don't just start with like the vLLM recipe. The vLLM recipe is proven, but it's not production ready. Um, and then test. Have your agent test it, and then whatever, you know, based on what you need, if it's ITL, if you need TTF, TPOT, like metric, like optimize on
- 14:03
that, and then ship that winner. Go to serverless, scale out, and now you can like, um, now you have a working LLM model that, uh, can be used internally, can be used for your customers. Um, yeah, it's, it, it just makes it very, very simple. Um, but yeah, um, one thing I would say, especially with like Kimi that I've noticed is, um, it's not always necessary for a team. Like, like a
- 14:32
big thing that I think like most people don't think about is that, um, you can use a smaller model. You know, it, it's a B300 right now, like a node that's... Like, on our, on our platform, like that's gonna cost you 40K a month, essentially. And it might go up, it might go down a little, but you know, it's a very powerful machine. And I think when you're thinking about like how you're gonna be, um, serving stuff if you wanna be doing this internally or for yourselves is like
Join the Runpod Discord
Runpod points builders to its Discord for help along the way and connections with others building with Runpod.
Text in image
If anything comes up along the way or you want to connect with others building with Runpod, join our Discord discord.com/invite/runpod
Join the Runpod Discord
Runpod points builders to its Discord for help along the way and connections with others building with Runpod.
Text in image
If anything comes up along the way or you want to connect with others building with Runpod, join our Discord discord.com/invite/runpod
Join the Runpod Discord
Runpod points builders to its Discord for help along the way and connections with others building with Runpod.
Text in image
If anything comes up along the way or you want to connect with others building with Runpod, join our Discord discord.com/invite/runpod
- 15:03
what makes sense the best? And just because it's the, the, the frontier model, maybe it doesn't make sense. Um, like I said, like, like I run a DeepSeek on a four H200s, and it does a really good job. Uh, Kimi is more for like long tasks that I'm trying to solve something like very, very detailed. Um, but like I said, the other thing is like making sure your like tooling tools like on there. That's a big thing. Um, context, like I, I'm not gonna be going through too much about like context, uh,
Join the Runpod Discord
Runpod points builders to its Discord for help along the way and connections with others building with Runpod.
Text in image
If anything comes up along the way or you want to connect with others building with Runpod, join our Discord discord.com/invite/runpod
Join the Runpod Discord
Runpod points builders to its Discord for help along the way and connections with others building with Runpod.
Text in image
If anything comes up along the way or you want to connect with others building with Runpod, join our Discord discord.com/invite/runpod
- 15:33
like wikis and all that, but the ability to have all that, uh, and like A- AI governance is the next step that we are gonna be going with Runpod. But just something to be, always be thinking about, especially if you're gonna be self-hos- um, hosting your, um, your LLM models. But yeah, is there any questions, like... And we'll, we'll also be at the booth, so like downstairs. But yeah, add me on LinkedIn, uh, if you wanna talk more. I'm always exploring every new model like every week, so uh,
Join the Runpod Discord
Runpod points builders to its Discord for help along the way and connections with others building with Runpod.
Text in image
If anything comes up along the way or you want to connect with others building with Runpod, join our Discord discord.com/invite/runpod
Questions at the Runpod booth
Vela points attendees to the Runpod booth downstairs for follow-up questions.
Text in image
We’ll be at the Runpod booth, if you have any questions
Connect and watch for the Benchpod template
Chris Vela invites viewers to connect on LinkedIn and scan the QR code for credits to try Runpod. He says the Benchpod template is planned for release at the end of the month.
Text in image
Add me on LinkedIn! And be on the lookout for the Benchpod Template Release runpod Chris Vela Forward Deployed Engineer https://github.com/[illegible] worker-vllm [ PASTE QR [illegible] - biggest size that gives it 30 seconds of screen time ] AI Engineer NEW YORK PRESENTED BY arize OCTOBER 13, 2026 EXPO SESSION McCarthy Room Interface text: Chrome File Edit View History Bookmarks Profiles Tab Window Help 72°F Tue Oct 13 3:36 PM
- 16:04
always have some like, uh, gotchas that I've been learning through the, through the time I've been doing all this work. And, uh, yeah, also scan the car- uh, QR code so you can get, uh, um, you'll get some credits and, um, you can test out Runpod. And like I said, the template will be out like at the end of the month, so uh, you can go play around with it.