Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP

Ask a model to fix a failing test. The serving system can count the input tokens when the prompt arrives. The answer takes shape one token at a time, with each token becoming part of the context for the next. By the time the system knows the output length, it has already generated the answer. Admission control has to decide whether to start much earlier, while the GPU time this request will need is still uncertain.

Follow the request through a self-hosted deployment. The API gateway checks the key and counts calls. A router picks a replica, one running copy of the model. Its inference engine loads the weights onto the GPU and batches requests so several answers can advance together. Each active request also accumulates a key-value (KV) cache, the model state it keeps for the context it has already processed. A longer answer adds more state.

How much work the deployment can take depends on what the product promises. In interactive chat, people notice both a poor answer and a long wait for the next token. An asynchronous agent can tolerate delay, but a poor answer may mean rerunning an expensive task, so quality and throughput matter. Autocomplete needs to return many suggestions quickly and can use a less expensive model. Suppose a deployment can sustain eight concurrent requests while meeting its service-level objective (SLO). What happens when 40 arrive in two seconds?

Two incidents in Jain and Khatri's review show what can give way first. On November 8, 2023, heavy traffic at OpenAI exhausted memory on routing servers. Health checks began failing. The team added replicas, rolled them back, then cut incoming traffic and restored it gradually. An Anthropic model tier slowed down during a January 21, 2026 incident and began returning errors over roughly 90 minutes. The reported cause was insufficient compute for demand. OpenAI's follow-up emphasized shedding load gracefully. Anthropic's emphasized detecting the capacity limit earlier.

Before a request takes GPU memory, an admission layer can start it, defer it or refuse it. Starting it commits the engine to two phases of work. During prefill, the engine reads the prompt and builds its KV cache. During decode, it extends the answer and the cache one token at a time. That memory stays occupied as long as the request is running.

What does a short request inherit from the calls ahead of it? Consider A, which produces a short answer, followed by B, which produces a long one. When short request C arrives, B is still in memory and its cache can keep growing. C may need little work of its own and still miss its latency target. The decision about C depends partly on a commitment the engine made to B.

If active caches fill memory, the engine can evict work, swap a request out to wait, or return an error after spending resources on it. Any of those can cost the service its latency target. Teams that self-host for privacy or cost own those decisions along with the GPUs.

A person chatting with a model leaves gaps to read the replies. A coding agent can spend those same seconds issuing calls, retrying them and starting more work. In Jain's test-fix example, the agent might read about 14 files and run about 10 tests. Across the task, tool use and follow-up reasoning can generate 40–50 calls, some quick and some long-running. At the cluster, one user's task has become a burst of related requests with different costs.

Which component can decide whether there is room? The agent sees its own call and controls retries. The gateway knows the API key and its rate limit. The router knows which replicas are healthy. The engine sees its batch and cache and can decide what to run or evict. In this arrangement, the component with the best view of pressure discovers the cost after the request has entered.

Placement changes both visibility and maintenance cost. A proxy at ingress can inspect the incoming request and whatever metrics the engine exports. At the queue boundary, the scheduler can intervene as it chooses the next batch. During decode, it can pause a running request or reclaim its cache halfway through an answer. Those later interventions have more information, but changing the scheduler can leave the team maintaining an engine fork through upstream releases. Jain and Khatri choose ingress so the policy can run against a stock engine.

What reaches that proxy, and which decision sends work onward? The request supplies input length, tenant, agent session and, when available, a deadline. The engine supplies cache occupancy, the age of the oldest queued request, prompt-cache hits and load across replica pools. The proxy can now ask whether the deployment has room for this caller's work. Only an admission sends the request onward to acquire GPU memory.

These inputs suggest different actions. Defer a prompt that exceeds current memory headroom or the extra calls from a session with too much work in flight. Refuse a tenant that exceeds its share, or a request whose deadline would expire in the queue. Spare KV capacity favors admission. A long wait for the oldest queued request favors deferral. A cached prompt can go first because some of its processing is already paid for. When every pool is full, refuse the arrival. For any of these rules, the observation has to be timely. A one-minute average can wash out 40 calls in two seconds.

Admit. A 200 opens a stream. The engine has committed cache for the prompt, and the request decodes alongside its batch, one token per step. On the client, distinguish the wait for the first token from time spent streaming. Prefill alone can take seconds under load, so Khatri starts the streaming timeout at the first token. If the agent abandons the answer, close the connection. A local timeout that leaves it open can leave the engine generating tokens and holding memory. A stream can also break after the headers have arrived. The agent then has a partial answer and must decide whether it can continue or needs another attempt.

Defer. For a temporary burst, the proposed server returns 503 with Retry-After. It can estimate the delay from when running work is likely to free enough cache. A rising queue age with capacity due back soon is a reason to ask the client to wait. Under sustained saturation, repeatedly issuing short deferrals just reschedules the same demand.

When do ten deferred clients come back? If all ten retry immediately, they create another spike while the server is still busy. Across the original burst, one retry per call can turn 40 attempts into 80. Honoring Retry-After gives the engine time to make progress. Adding a small random delay, or jitter, spreads the returning clients out even when they all received the same suggested wait.

Reject. The reason determines what the agent can change.

• Caller over its share (429). The policy may have seen a quota violation or too many concurrent calls from this tenant or session. Reduce the agent's load by capping its in-flight calls.
• Provider overloaded (529 in the example). All replica pools are busy. Wait longer, switch to another model or provider, or park the task for later.

In either case, give retries a budget. Use exponential backoff with jitter and tell the user what happened once the budget runs out. Otherwise a refusal at ingress can immediately become another arrival.

A fixed allowance per API key helps tenants share the service, but it stays fixed as GPU load changes. A proxy that caps concurrent requests per replica gets closer to the hardware. Its blind spot is cost. A quick lookup and a long planning call each occupy one slot, even though their token counts, cache use and lifetimes differ. Once the queue grows, the client's timeout can end up making the refusal. A cost-aware policy estimates the incoming work in tokens and KV memory against the engine's current state.

Specifying a policy takes four choices. The unit says what gets admitted, such as a request, an agent session or a tenant. The signal is what the policy observes as the system gets busy. The rule converts that observation into a decision and takes time to run. The action determines what happens to the caller. These choices constrain each other. A deadline rule, for instance, needs the caller to supply a deadline.

What changes if the unit is a session? Imagine two agent tasks with 20 calls each and room for about half the calls. Split admissions evenly and each task gets roughly ten calls through. Both still depend on calls that failed to get in. All the capacity has been used, and neither task can finish. In the session-level alternative, A gets its full set of calls and finishes. B gets one clear deferral. A count of successful calls would miss the difference between zero completed tasks and one.

Choose the signal by the failure you need to detect. KV occupancy can reveal impending cache exhaustion. Pair occupancy with cache-hit rate when the cost is recomputation, since even a full cache may be doing valuable work by reusing prompt prefixes. Queue depth exposes accumulating waits. Deadline slack tells you how much delay a request can still tolerate. An estimate of output length helps plan for long answers. Cache pressure, for example, will miss overload on a deployment where the cache never comes close to filling.

The rule itself has to fit inside the time available for a decision. For the agent sending 40 calls in two seconds, a planner that takes five seconds can finish after the entire burst has arrived. Four research designs illustrate what that planning time buys and what cheaper rules can see.

• Single threshold. Compare KV occupancy with one limit for each incoming request. The example takes tens of microseconds per decision. It can respond quickly on each arrival, provided cache pressure actually rises under load.
• Dual gate. Require two signals, KV use and cache hits, to agree before acting. The example admits whole agent sessions and checks the signals in under 100 microseconds. It reported roughly four times SGLang's throughput in an offline test. That result leaves performance under live arrivals and retries open.
• Fleet optimizer. QLM plans across queues together to decide which requests to admit. It can account for work elsewhere in the fleet, at a reported cost of about five seconds per decision.
• Formal deadline bound. Calculate whether a request can still meet its time-to-first-token deadline in the worst case covered by the bound. The example makes a microsecond-scale decision and reported a drop in deadline misses from 99% to zero in its evaluation.

The papers used different models, engines and workloads, so their gains cannot establish a ranking. Jain compares them along two axes. Does the policy optimize deadlines or throughput? Does it admit individual requests or whole sessions? For a coding agent, Jain wants to admit sessions as live traffic arrives, using an engine he can deploy as shipped. The session-aware dual gate comes close because it can run without an engine fork. Its offline evaluation leaves that combination unproven.

Which rule would you try first on your own deployment? If requests carry deadlines, try a deadline bound. Otherwise, observe the deployment at peak load. If KV memory fills, try an occupancy threshold, or add a second gate when prefix reuse affects the cost of admission. If the cache has room while calls pile up, try a queue threshold. You can find out which branch applies before spending time tuning a threshold.

Start by measuring the load the replicas can sustain with admission control disabled. That establishes what 1× capacity means on this deployment. Specify the candidate policy's inputs, limits and refusal behavior. Then replay the same traffic trace with the policy enabled and disabled, sweeping offered load from 0.5× to 2× measured capacity. Below capacity a policy can appear to work simply because it has never had to refuse anything.

What score makes the cost of a refusal visible? Khatri's goodput under admission divides the number of requests meeting the SLO by the number of requests that arrived. For the streaming SLO here, the answer must start on time and avoid excessive gaps between tokens. Suppose the policy refuses 95 of 100 requests and serves the other five perfectly. Attainment among admitted requests is five out of five, or 100%. Goodput under admission counts all 100 in the denominator. Five percent of all arriving requests met the latency target.

The trace needs to preserve the behavior that creates pressure in the first place. Keep agent fan-out, retries and session membership. Uniform random arrivals discard the timing and dependencies the policy must handle. Set the deadline relative to latency measured without load so readers can tell how much slowdown the experiment allowed.

Khatri asks for seven disclosures alongside an admission result.

• Load. Give offered load as a multiple of measured capacity.
• Deadline. Express it relative to unloaded latency.
• Signal liveness. Show whether the metric the policy watches actually moved under stress.
• Variation. Give the repeat count and how much the runs differed.
• Parameters. Identify which values were tuned locally, borrowed or left at their defaults.
• Denominator. Say whether refusals counted toward goodput.
• Configuration. Record the model, engine and version, hardware, workload, SLO and baseline.

Jain and Khatri built AdmitPerf to compare policies on a common deployment. Its ingress library sits in front of an inference engine such as vLLM or SGLang and makes a decision for each arrival. Its benchmark harness exercises a policy against an engine on a real GPU. Both call the same decision function, so a benchmarked rule can go into the serving path without being reimplemented. Among the 16 papers they compared on evaluation setup, no two shared the same combination of baseline, engine, workload and SLO.

The harness has five operations. Bring up the GPU-backed engine, smoke-test that it serves requests, run each policy several times, compare the results and the size of the differences, then tear the infrastructure down to stop paying for it. Provisioning sits outside the benchmark. Loading the model takes minutes, and reprovisioning between candidates would change the deployment being used as the control.

How can the harness tell an empty cache from a missing metric? Each policy declares the signals it needs. A KV-threshold policy requires the fraction of KV cache in use. If the engine cannot supply it, the harness stops at startup. Without that check, a missing value could read as zero. The policy would admit every request and produce a plausible-looking benchmark of the admit-all baseline.

For a run that proceeds, the harness records each decision with the state the policy saw. The run directory contains four JSON artifacts covering the manifest, summary, decisions and outcomes. An engineer can inspect what triggered a refusal on a particular arrival as well as what changed over the whole run.

The GPU comparison used a half-billion-parameter model on an A10, with the engine capped at four simultaneous sequences. The arrival rate was 14 requests per second, with two repeats. With admission control disabled, p95 time to first token was about 2.5 seconds, with variation of roughly 700 milliseconds either way. That percentile describes the wait near the slow end of the observed requests.

What did a shorter queue buy? A policy allowing at most two waiting requests admitted about 63% of arrivals and shed about 37%. Its p95 time to first token fell to about one second. Goodput under admission stayed the same in this comparison. Admitted requests saw a shorter tail wait while the fraction of all arrivals served within the SLO held steady.

The engine metrics explain why queue depth worked here. KV occupancy peaked at about 0.5%. Four short active sequences left ample cache even as work accumulated outside the batch. A policy waiting for high KV occupancy would have kept admitting work. The waiting-request cap had an effect because the queue was where this deployment registered overload.

There is a laptop path for trying the mechanism. AdmitPerf includes a small fake engine that implements the three endpoints the harness uses and gets busy under load. A developer can exercise the policy there, then measure its behavior on the target GPU. AdmitPerf can also discover a policy packaged separately from its repository. The tool grew out of the LLM Inference at Scale handbook survey. Jain points readers to both projects on the AdmitPerf source-code slide.