Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

You have a private model or an open-source model from Hugging Face. How much infrastructure work should stand between that model and an application that uses it? RunPod’s division of responsibility is straightforward: developers bring their code and models; the platform supplies GPUs and manages the infrastructure around them.

Managing servers takes time away from building applications. Audry Hsu compares GPU infrastructure with the earlier move from maintaining on-premises servers to using AWS or Google Cloud: each layer of abstraction removes work developers previously had to handle themselves. GPU access adds another obstacle. At the time of the talk, she describes it as slow and opaque amid a global supply crunch, comparing demand to pandemic-era toilet-paper stockpiling. Her expectation of recovery rests on customers getting better at estimating their compute needs. The practical objective is to keep developers focused on the applications through which they create value.

Hsu traces RunPod’s beginnings to founders Zhen and Pardeep repurposing basement GPU rigs after unsuccessful crypto mining. In her account, they built the foundations of the platform in 2022, then offered free GPU access on Reddit in exchange for feedback. She describes the company as revenue-generating since those beginnings. The point of the story is the feedback loop: builders tried the infrastructure, their responses shaped it, and Reddit and Discord remain places where the company engages with users. Hsu promises engagement, not perfection.

At the time of the talk, Hsu reports more than 500,000 developers and more than 30 data centers worldwide, including locations in Europe and the EU. She also reports $120 million in annual recurring revenue, a company-announced milestone, rather than realized annual revenue. Her customer examples include AI cloud-native companies; the shared requirement is flexible, reliable GPU infrastructure.

RunPod separates the environment that runs your workload from the catalog that helps you deploy it.

Hsu describes Serverless as avoiding idle compute costs by spinning workers down. For present-day budgeting, the current billing documentation is more precise: workers accrue compute charges from startup through full stop, including initialization and idle timeout; active workers stay running, and storage can also incur charges. That is a current billing clarification, not a reconstruction of the demo’s invoice. Hub, meanwhile, supplies repositories you can fork, watch, star, and deploy—it is the starting point for a deployment rather than another compute environment.

For real-time inference, Serverless adds controls over both capacity and readiness:

• Maximum workers: Bound how far the endpoint can scale out. Hsu also mentions spending caps.
• Always-on workers: Keep workers available with models already downloaded, so incoming requests need not wait for that preparation.

Autoscaling reduces the need to predict all compute demand in advance. Keeping workers ready addresses a different problem: the delay before a newly started worker can serve a request. Hsu presents this combination as a fast route to a production-ready API; the walkthrough that follows demonstrates deployment and a returned request, rather than a production-readiness assessment.

The live walkthrough uses the console so the deployment steps are visible. Hsu also mentions CLI support and agent skills, but does not demonstrate those interfaces here. In Hub, she selects the vetted vLLM listing as the starting point for serving an LLM.

Opening the underlying GitHub repository reveals what the listing packages: setup instructions, a preconfigured Dockerfile, and defaults. Depending on the listing, environment variables let you customize its behavior. The template therefore removes setup work without hiding that a repository and container configuration sit underneath the deployment. After inspecting these pieces, Hsu clicks Deploy.

The configuration sequence is small:

1. Select a Qwen model to download from Hugging Face.
2. Expand the advanced options and increase maximum model length for the serving context window.
3. Leave the other settings at their defaults. Max LoRAs is another exposed option. Hsu explains that the engine configuration becomes flags passed to vllm serve.
4. Create the Serverless endpoint.

The walkthrough does not specify an exact Qwen identifier or a numeric context length. Increasing the serving limit also does not independently extend the model’s supported context: the model must support the requested length, and the GPU must have sufficient memory.

Creating the endpoint starts worker initialization; it does not make a fresh model immediately ready to serve. While that preparation runs, Hsu inspects the deployment settings. The shown hardware preference is H100 GPUs, with A100 GPUs as backup. She describes the displayed price as a fraction of a cent per second and explains the cost in terms of workers running and handling requests. These are the demo’s displayed choices, not a claim that every endpoint uses that hardware or rate.

In the shown configuration, Hsu says maximum workers can be raised to 15. This sets the scale-out ceiling. The separate active workers setting keeps containers always on instead of allowing them to spin down. After discussing these controls, she saves the settings and turns to the endpoint’s request interface.

The deployment provides an HTTP API endpoint to which an application or customer can send requests. Hsu uses the console’s Run action to submit several requests, including a question about how Big Ben got its name. The requests queue while she checks worker readiness.

The worker view distinguishes preparation from availability. Initializing workers are creating containers and downloading the model. Running workers have finished that preparation and can pick up queued requests; Hsu expects those workers to handle the work she just submitted. The telemetry view offers request counts, execution time, and delay time, although it is initially blank. Those separate measurements matter because waiting for capacity and executing inference are different parts of the request’s path.

A request then completes. Hsu reports about 41 seconds in the queue and about 1.5 seconds of execution time for one returned request. She attributes the longer initial wait to cold-start work, including downloading the model and initializing the first container, and expects subsequent requests to wait less. That expectation is not a measured latency distribution: this is one reported result from a newly initialized endpoint, without token counts or confirmed hardware for the request. Queue delay is distinct from model execution time; describing this result only as a 1.5-second response would omit most of the observed wait.

Hsu estimates that getting the endpoint deployed from the Hub listing took less than five minutes. The result establishes the scope of the demonstration: a preconfigured listing, a selected model, worker initialization, and a completed request. It is not a universal deployment-time guarantee or an evaluation of the generated answer.

Hsu closes by inviting questions and announcing a session at 4:00 on the RunPod Flash Python SDK. The planned workflow moves entirely into the terminal: define code locally, run it as a remote function on a GPU, and ultimately deploy it as a production-ready endpoint. That extends the workflow from deploying an existing Hub template to deploying your own functions. It is a preview, not an SDK demonstration in this recording; no substantive audience Q&A follows.