AI Engineer Europe 2026
Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod
Read the talk
From a GPU model to a RunPod Serverless endpoint
Audry Hsu walks through deploying Qwen from a vLLM Hub listing, configuring worker capacity, and separating cold-start queue delay from inference execution.
From a talk by Audry Hsu
Before you start: Basic familiarity with containers, HTTP APIs, and language models will help you follow the deployment walkthrough.
Who manages the GPUs?
You have a private model or an open-source model from Hugging Face. How much infrastructure work should stand between that model and an application that uses it? RunPod’s division of responsibility is straightforward: developers bring their code and models; the platform supplies GPUs and manages the infrastructure around them.
Managing servers takes time away from building applications. Audry Hsu compares GPU infrastructure with the earlier move from maintaining on-premises servers to using AWS or Google Cloud: each layer of abstraction removes work developers previously had to handle themselves. GPU access adds another obstacle. At the time of the talk, she describes it as slow and opaque amid a global supply crunch, comparing demand to pandemic-era toilet-paper stockpiling. Her expectation of recovery rests on customers getting better at estimating their compute needs. The practical objective is to keep developers focused on the applications through which they create value.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A platform shaped by GPU users
Hsu traces RunPod’s beginnings to founders Zhen and Pardeep repurposing basement GPU rigs after unsuccessful crypto mining. In her account, they built the foundations of the platform in 2022, then offered free GPU access on Reddit in exchange for feedback. She describes the company as revenue-generating since those beginnings. The point of the story is the feedback loop: builders tried the infrastructure, their responses shaped it, and Reddit and Discord remain places where the company engages with users. Hsu promises engagement, not perfection.
At the time of the talk, Hsu reports more than 500,000 developers and more than 30 data centers worldwide, including locations in Europe and the EU. She also reports $120 million in annual recurring revenue, a company-announced milestone, rather than realized annual revenue. Her customer examples include AI cloud-native companies; the shared requirement is flexible, reliable GPU infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the compute environment
RunPod separates the environment that runs your workload from the catalog that helps you deploy it.
| Offering | Role | Mechanism |
|---|---|---|
| Pods | Sandbox development environment | Containers with allocated GPUs; bring Dockerfiles and code |
| Serverless | Bursty or batch workloads | Autoscaling workers that can spin down |
| Clusters | Heavy training | Multiple nodes with high-speed networking |
| Hub | Discover and deploy AI repositories | Preconfigured, vetted listings from RunPod and community contributors |
Hsu describes Serverless as avoiding idle compute costs by spinning workers down. For present-day budgeting, the current billing documentation is more precise: workers accrue compute charges from startup through full stop, including initialization and idle timeout; active workers stay running, and storage can also incur charges. That is a current billing clarification, not a reconstruction of the demo’s invoice. Hub, meanwhile, supplies repositories you can fork, watch, star, and deploy—it is the starting point for a deployment rather than another compute environment.
For real-time inference, Serverless adds controls over both capacity and readiness:
- Maximum workers: Bound how far the endpoint can scale out. Hsu also mentions spending caps.
- Always-on workers: Keep workers available with models already downloaded, so incoming requests need not wait for that preparation.
Autoscaling reduces the need to predict all compute demand in advance. Keeping workers ready addresses a different problem: the delay before a newly started worker can serve a request. Hsu presents this combination as a fast route to a production-ready API; the walkthrough that follows demonstrates deployment and a returned request, rather than a production-readiness assessment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Open the vLLM listing
The live walkthrough uses the console so the deployment steps are visible. Hsu also mentions CLI support and agent skills, but does not demonstrate those interfaces here. In Hub, she selects the vetted vLLM listing as the starting point for serving an LLM.
Opening the underlying GitHub repository reveals what the listing packages: setup instructions, a preconfigured Dockerfile, and defaults. Depending on the listing, environment variables let you customize its behavior. The template therefore removes setup work without hiding that a repository and container configuration sit underneath the deployment. After inspecting these pieces, Hsu clicks Deploy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Configure Qwen, then create the endpoint
The configuration sequence is small:
- Select a Qwen model to download from Hugging Face.
- Expand the advanced options and increase maximum model length for the serving context window.
- Leave the other settings at their defaults. Max LoRAs is another exposed option. Hsu explains that the engine configuration becomes flags passed to
vllm serve. - Create the Serverless endpoint.
The walkthrough does not specify an exact Qwen identifier or a numeric context length. Increasing the serving limit also does not independently extend the model’s supported context: the model must support the requested length, and the GPU must have sufficient memory.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Set worker capacity while initialization runs
Creating the endpoint starts worker initialization; it does not make a fresh model immediately ready to serve. While that preparation runs, Hsu inspects the deployment settings. The shown hardware preference is H100 GPUs, with A100 GPUs as backup. She describes the displayed price as a fraction of a cent per second and explains the cost in terms of workers running and handling requests. These are the demo’s displayed choices, not a claim that every endpoint uses that hardware or rate.
In the shown configuration, Hsu says maximum workers can be raised to 15. This sets the scale-out ceiling. The separate active workers setting keeps containers always on instead of allowing them to spin down. After discussing these controls, she saves the settings and turns to the endpoint’s request interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow a request from the queue to completion
The deployment provides an HTTP API endpoint to which an application or customer can send requests. Hsu uses the console’s Run action to submit several requests, including a question about how Big Ben got its name. The requests queue while she checks worker readiness.
The worker view distinguishes preparation from availability. Initializing workers are creating containers and downloading the model. Running workers have finished that preparation and can pick up queued requests; Hsu expects those workers to handle the work she just submitted. The telemetry view offers request counts, execution time, and delay time, although it is initially blank. Those separate measurements matter because waiting for capacity and executing inference are different parts of the request’s path.
A request then completes. Hsu reports about 41 seconds in the queue and about 1.5 seconds of execution time for one returned request. She attributes the longer initial wait to cold-start work, including downloading the model and initializing the first container, and expects subsequent requests to wait less. That expectation is not a measured latency distribution: this is one reported result from a newly initialized endpoint, without token counts or confirmed hardware for the request. Queue delay is distinct from model execution time; describing this result only as a 1.5-second response would omit most of the observed wait.
Hsu estimates that getting the endpoint deployed from the Hub listing took less than five minutes. The result establishes the scope of the demonstration: a preconfigured listing, a selected model, worker initialization, and a completed request. It is not a universal deployment-time guarantee or an evaluation of the generated answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a Hub listing to remote Python functions
Hsu closes by inviting questions and announcing a session at 4:00 on the RunPod Flash Python SDK. The planned workflow moves entirely into the terminal: define code locally, run it as a remote function on a GPU, and ultimately deploy it as a production-ready endpoint. That extends the workflow from deploying an existing Hub template to deploying your own functions. It is a preview, not an SDK demonstration in this recording; no substantive audience Q&A follows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Discover preconfigured AI repositories, review hardware requirements and create an endpoint.
Define remote GPU functions in Python and deploy them to Runpod Serverless.
Further reading
Worker template with Docker deployment instructions and model-serving configuration.
Reference for model selection, context length, LoRA adapters and worker concurrency.
- Runpod's founding storyArticle
Zhen Lu recounts the basement GPU experiments and Reddit feedback that shaped Runpod.
Updates since the talk
- Serverless billing and worker costsDocumentation
Explains compute charges during startup, execution and idle timeout, plus storage charges.
Read the complete timestamped transcript
- 0:00
[upbeat music] Audry.
- 0:15
Um, I am from RunPod. Um, this is an intro to RunPod. Can I just get a quick hands to see how many people have already heard of RunPod or maybe even used RunPod before?
- 0:27
Okay, newbies for everybody. Great. Um, so RunPod, we are a cloud AI infrastructure company. So we have the hardware, we have the GPUs, and we make it easy for developers to deploy, um, models.
- 0:43
And that can be your own private model, it can be an open source model from Hugging Face. Doesn't matter to us. You bring your code, and we'll bring the rest.
- 0:54
Um, just really quickly, what problems does RunPod solve? Like, why are we even here today? Um, infrastructure can be hard managing it. Um, [sighs]
- 1:05
I think about back in the day before we had AWS, um, Google Cloud, when everybody would have to have on-prem servers and manage those, maintain those. That is something that we don't want to have to do as developers.
- 1:18
Those are things that we happily, um, have moved away from and given off to DevOps, and now it's even, it's even more abstracted for us. GPU access is slow and opaque.
- 1:30
So, um, I don't know if you-- if anybody has tried to buy a GPU recently. We're in a global supply crunch. Um, it's a bit like in COVID when everybody went to the store and bought all the toilet paper because we didn't know how, how long they would need to be at home for.
- 1:48
We're a little bit in that right now. Um, but we expect the market will recover, um, as customers, companies, people figure out a little bit better-- uh, get a little bit better at, um, estimating what kind of compute they need.
- 2:01
Um, and then last, builder primary focus should be building. So again, um, we wanna build app... We as so-software developers, uh, we bring, bring the value through the applications that we build, um, not for managing the infrastructure.
- 2:19
And I think RunPod has a pretty unique story. These are, these are our founders, Zen and Pardeep. Um, so they had a couple of GPU rigs in their basement in 2022.
- 2:30
Um, failed crypto mining, and then so they were like, "What are we gonna do with our GPUs now?" Um, so they prototyped what is now the foundations of RunPod.
- 2:40
They posted on Reddit and said, "Hey, anyone wanna use these GPUs for free? Just give us feedback on it." And that is literally how our company has started, and we have been, um, revenue generating ever since.
- 2:56
Um, and the reason why I wanna tell this story is, um, not because it's, it's very, like, bootstrappy, but because, um, the origin, origin story of RunPod has always started with, uh, builders and getting feedback from the community, and that is still true today.
- 3:13
So I won't promise that we'll be perfect, but, um, we are definitely very engaged with, um, our users on, on Reddit. Um, on Reddit, on Discord. So, um, we're always trying to stay engaged with y'all.
- 3:26
Um, just at a glance, to give you an idea of RunPod, we have over five hundred thousand developers on our platforms, thirty-plus data centers across the world, including, um, Europe and the EU.
- 3:40
Um, and we've just passed a significant revenue milestone for us, a hundred and twenty million in annual recurring revenue.
- 3:50
Uh, these are just a few of our customers. Um, you might be surprised to see some of the AI cloud-native companies on here too, but, um, they come to us for the same reasons that most of our customers come to us.
- 4:03
It's, um, because they need flexible and reliable GPU infrastructure.
- 4:12
This is a really high over-- high-level overview of, um, different ways you can build on RunPod. So I would say our... at our core, um, Pods, it's our sandbox virtual environment.
- 4:26
We spin up a container for you, um, allocate GPUs to it, and we manage the rest. You just bring your, um, Docker files. You bring your code. Serverless, um, it's our auto-scaling product.
- 4:42
So when you're thinking more about, like, bursty workloads or batch workloads, um, Serverless is really great because, um, instead of being always on like a container is, Serverless, um, your workers spin down, and when they're idle, you don't pay for anything.
- 4:59
Clusters, um, if you're doing some heavy duty training, there's a place for you as well on RunPod. Um, multi-node clusters with high-speed networking. And then the Hub, which I'll, I'll, I'll switch to in a second.
- 5:12
Um, it's kind of like our central repository for AI repos. Um, these are already pre-configured, pre-vetted. Um, we have a couple of examples of listings by RunPod for popular models, but also our community con- um, contributes to them as well.
- 5:29
So they're just repos that you can fork, you can watch, and then, um, you can star and deploy on RunPod.
- 5:39
Um, so today we're gonna be talking mostly about Serverless. Um, so Serverless is best for real-time inference. I talked about the auto-scaling that comes with it. Um, why teams use it is mostly because they don't need to, um, preempt and figure out how much compute they need ahead of time.
- 6:01
Um- You can set, you can configure the number of max workers that you wanna scale up to. You can set limits for caps, for spending caps, and you can also configure workers that are always on, so they're, um, already have your models downloaded, and they can respond to requests, um, immediately.
- 6:19
For a lot of teams, serverless is the fastest way, um, if you want to start deploying a production-ready API.
- 6:30
And now I'm going to switch over and just show y'all really quick how easy it is to get started and deploy something.
- 6:45
Okay. Where are we? [clears throat] Okay, so right now I'm, um, gonna do everything via the console so that it's nice and pretty for you guys to see.
- 7:03
But we also have, um, CLI support. We have skills, um, to help work with RunPod, everything that's ready for your agent. So you don't have to read our documents, but since we're all humans here today, I'm gonna show you via the console.
- 7:19
Um, we'll start in the Hub, which is... If you're just trying to explore and see what's out there, what is something that you can get up and running right now, the Hub is a great place to start.
- 7:32
So like I mentioned, these are already vetted open source listings for, um, AI repos, and I am going to pick vllm. Um, and I'll just open the underlying
- 7:48
repository as well, so you c- guys can see what... It is literally just a GitHub, um, repo.
- 7:56
It tells you how to get set up for it. Um,
- 8:01
we can see there's already the Docker file here. It's already pre-configured for you. It's got some defaults for you. Um, depending on the listing, you can, uh, pass in different environmental variables,
- 8:15
um, to configure it how you wish. But I'm just gonna go ahead and click deploy, and
- 8:28
I have a model that I wanted. I've...
- 8:37
Let me see. I was gonna just pick Gwen.
- 8:44
Works well. This is gonna download it from Hugging Face, and just expand the advanced options, and look for the max model length. And I'm gonna bump this up for the context window and leave everything else as the defaults.
- 9:02
But there's settings for Max LoRAs. Um, all of these configuration options get passed as, um,
- 9:11
as flags to the, uh, vllm serve. And
- 9:19
I'm gonna spin it up as an endpoint here.
- 9:25
So this might take a minute since... or two, since this is the very... I just created it. We've gotta initialize my workers. Let's check out.
- 9:35
So the default configuration here is it's gonna deploy on some H100s, and A100s are the backup here. I have my pricing. This is l- a fraction of a cent per second.
- 9:48
Um, like I mentioned before, that this is only going to be charged for while the worker is actually running and handling a request. Max workers is where I can bump this up if I want to have
- 10:02
my workload scale up, up to 15 workers at a time. And I can set, [smacks lips] um, some active workers, ones that I want always to be on, that I don't want, um, the container to ever spin down.
- 10:16
And I can save that. Okay, so how do I in- how does one interact with the serverless endpoint? Um,
- 10:27
this is just an API, uh, HTTP endpoint right here. Um, we provisioned this endpoint for you. You can send requests to this. Your customers can send requests to this.
- 10:37
If I just hit run, and I'm gonna add a few. Let's...
- 10:45
What should, what should we ask the LLM today? Does anyone...
- 10:52
Okay. How did... I'm, I'm [REDACTED:origin], so how did Big Ben get its name? I don't know.
- 11:02
Um, okay, while these requests are queued, let me check on our workers.
- 11:08
Okay. We have a handful that are in- initializing. Um, this is the containers being created. That's the model being downloaded, um, getting ready. And the ones that are running, they've already finished.
- 11:21
These are probably gonna be the ones who are gonna pick up those requests that we just added. I've got telemetry about, um...
- 11:30
It's blank right now, but the number of requests, execution time, delay time. So you have observability into how your endpoints are operating.
- 11:41
And let's see. Okay, it's already done. I've got a request back, and it sat in the queue for about 41 seconds. Um, that's going to be a l- little bit longer than all of the subsequent requests because of some of the cold start time that I talked about, like downloading the model, um, initializing the first container.
- 12:02
But, um, execution time, only about one and a half seconds. So yeah, that was probably less than five minutes to get s- started and get something deployed, um, on serverless from a Hub listing.
- 12:22
Does anyone have any questions? This is, this is a very short and sweet intro. Um, we have another session later today at 4:00, um, and that one is gonna be focused on our Python Flash, um, SDK.
- 12:36
And that one is going to be completely via the terminal. Um, and I'm gonna sh- walk you through how I can spin up, uh, and deploy my code on...
- 12:47
my code as a remote, remote function onto a GPU, um, and, uh, deploy in the end, and make it, like, a production-ready endpoint here as well.
- 13:04
Okay. But that's all I got for today, so yeah. Thanks. Thanks for coming. [laughs] [audience applauds] [upbeat music]