# Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP

Harshul Jain, Audible · Parv Khatri, SAP

[Talk page](https://ai.engineer/talks/ntVpbyDTYaQ-your-self-hosted-llm-needs-say-no) · [Video](https://www.youtube.com/watch?v=ntVpbyDTYaQ)

## Your Self-Hosted LLM Needs to Say No

One request to fix a failing test can turn into dozens of model calls. Harshul Jain and Parv Khatri show how to decide which calls reach the GPU, how an agent should handle a refusal, and how to measure the fraction of arriving requests that get an answer on time.

### The prompt arrives before anyone knows the cost of the answer

[0:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=5s)

Ask a model to fix a failing test. The serving system can count the input tokens when the prompt arrives. The answer takes shape one token at a time, with each token becoming part of the context for the next. By the time the system knows the output length, it has already generated the answer. Admission control has to decide whether to start much earlier, while the GPU time this request will need is still uncertain.

Follow the request through a self-hosted deployment. The API gateway checks the key and counts calls. A router picks a replica, one running copy of the model. Its inference engine loads the weights onto the GPU and batches requests so several answers can advance together. Each active request also accumulates a key-value (KV) cache, the model state it keeps for the context it has already processed. A longer answer adds more state.

How much work the deployment can take depends on what the product promises. In interactive chat, people notice both a poor answer and a long wait for the next token. An asynchronous agent can tolerate delay, but a poor answer may mean rerunning an expensive task, so quality and throughput matter. Autocomplete needs to return many suggestions quickly and can use a less expensive model. Suppose a deployment can sustain eight concurrent requests while meeting its service-level objective (SLO). What happens when 40 arrive in two seconds?

Two incidents in Jain and Khatri's review show what can give way first. On November 8, 2023, heavy traffic at OpenAI exhausted memory on routing servers. Health checks began failing. The team added replicas, rolled them back, then cut incoming traffic and restored it gradually. An Anthropic model tier slowed down during a January 21, 2026 incident and began returning errors over roughly 90 minutes. The reported cause was insufficient compute for demand. OpenAI's follow-up emphasized shedding load gracefully. Anthropic's emphasized detecting the capacity limit earlier.

### What the engine commits when it starts a request

[8:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=529s)

Before a request takes GPU memory, an admission layer can start it, defer it or refuse it. Starting it commits the engine to two phases of work. During **prefill**, the engine reads the prompt and builds its KV cache. During **decode**, it extends the answer and the cache one token at a time. That memory stays occupied as long as the request is running.

What does a short request inherit from the calls ahead of it? Consider A, which produces a short answer, followed by B, which produces a long one. When short request C arrives, B is still in memory and its cache can keep growing. C may need little work of its own and still miss its latency target. The decision about C depends partly on a commitment the engine made to B.

If active caches fill memory, the engine can evict work, swap a request out to wait, or return an error after spending resources on it. Any of those can cost the service its latency target. Teams that self-host for privacy or cost own those decisions along with the GPUs.

A person chatting with a model leaves gaps to read the replies. A coding agent can spend those same seconds issuing calls, retrying them and starting more work. In Jain's test-fix example, the agent might read about 14 files and run about 10 tests. Across the task, tool use and follow-up reasoning can generate 40–50 calls, some quick and some long-running. At the cluster, one user's task has become a burst of related requests with different costs.

### Bring the request and the engine state together at ingress

[15:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=932s)

Which component can decide whether there is room? The agent sees its own call and controls retries. The gateway knows the API key and its rate limit. The router knows which replicas are healthy. The engine sees its batch and cache and can decide what to run or evict. In this arrangement, the component with the best view of pressure discovers the cost after the request has entered.

Placement changes both visibility and maintenance cost. A proxy at **ingress** can inspect the incoming request and whatever metrics the engine exports. At the **queue boundary**, the scheduler can intervene as it chooses the next batch. **During decode**, it can pause a running request or reclaim its cache halfway through an answer. Those later interventions have more information, but changing the scheduler can leave the team maintaining an engine fork through upstream releases. Jain and Khatri choose ingress so the policy can run against a stock engine.

What reaches that proxy, and which decision sends work onward? The request supplies input length, tenant, agent session and, when available, a deadline. The engine supplies cache occupancy, the age of the oldest queued request, prompt-cache hits and load across replica pools. The proxy can now ask whether the deployment has room for *this* caller's work. Only an admission sends the request onward to acquire GPU memory.

These inputs suggest different actions. Defer a prompt that exceeds current memory headroom or the extra calls from a session with too much work in flight. Refuse a tenant that exceeds its share, or a request whose deadline would expire in the queue. Spare KV capacity favors admission. A long wait for the oldest queued request favors deferral. A cached prompt can go first because some of its processing is already paid for. When every pool is full, refuse the arrival. For any of these rules, the observation has to be timely. A one-minute average can wash out 40 calls in two seconds.

Which path commits GPU memory?
Engine metrics travel back to ingress as information. Of the three decisions the proxy can make with that information, only admission sends work to the engine.
- Incoming request — Input length, tenant, session and deadline
- Engine metrics — KV occupancy, queue age, cache hits and replica load
- Ingress policy — Decides before the request acquires a KV cache
- Start on the engine — Allocate cache and join the running batch
- Try later — Give the client a delay while capacity returns
- Refuse this attempt — Return a reason the caller can act on
- Incoming request → Ingress policy (Request attributes)
- Engine metrics → Ingress policy (Exported state)
- Ingress policy → Start on the engine (Admit)
- Ingress policy → Try later (Defer)
- Ingress policy → Refuse this attempt (Reject)

### What the agent should do with each response

[22:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1328s)

**Admit.** A `200` opens a stream. The engine has committed cache for the prompt, and the request decodes alongside its batch, one token per step. On the client, distinguish the wait for the first token from time spent streaming. Prefill alone can take seconds under load, so Khatri starts the streaming timeout at the first token. If the agent abandons the answer, close the connection. A local timeout that leaves it open can leave the engine generating tokens and holding memory. A stream can also break after the headers have arrived. The agent then has a partial answer and must decide whether it can continue or needs another attempt.

**Defer.** For a temporary burst, the proposed server returns `503` with `Retry-After`. It can estimate the delay from when running work is likely to free enough cache. A rising queue age with capacity due back soon is a reason to ask the client to wait. Under sustained saturation, repeatedly issuing short deferrals just reschedules the same demand.

When do ten deferred clients come back? If all ten retry immediately, they create another spike while the server is still busy. Across the original burst, one retry per call can turn 40 attempts into 80. Honoring `Retry-After` gives the engine time to make progress. Adding a small random delay, or jitter, spreads the returning clients out even when they all received the same suggested wait.

**Reject.** The reason determines what the agent can change.

- **Caller over its share (`429`).** The policy may have seen a quota violation or too many concurrent calls from this tenant or session. Reduce the agent's load by capping its in-flight calls.
- **Provider overloaded (`529` in the example).** All replica pools are busy. Wait longer, switch to another model or provider, or park the task for later.

In either case, give retries a budget. Use exponential backoff with jitter and tell the user what happened once the budget runs out. Otherwise a refusal at ingress can immediately become another arrival.

### Twenty successful calls can still leave two tasks unfinished

[27:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1653s)

A fixed allowance per API key helps tenants share the service, but it stays fixed as GPU load changes. A proxy that caps concurrent requests per replica gets closer to the hardware. Its blind spot is cost. A quick lookup and a long planning call each occupy one slot, even though their token counts, cache use and lifetimes differ. Once the queue grows, the client's timeout can end up making the refusal. A cost-aware policy estimates the incoming work in tokens and KV memory against the engine's current state.

Specifying a policy takes four choices. The **unit** says what gets admitted, such as a request, an agent session or a tenant. The **signal** is what the policy observes as the system gets busy. The **rule** converts that observation into a decision and takes time to run. The **action** determines what happens to the caller. These choices constrain each other. A deadline rule, for instance, needs the caller to supply a deadline.

What changes if the unit is a session? Imagine two agent tasks with 20 calls each and room for about half the calls. Split admissions evenly and each task gets roughly ten calls through. Both still depend on calls that failed to get in. All the capacity has been used, and neither task can finish. In the session-level alternative, A gets its full set of calls and finishes. B gets one clear deferral. A count of successful calls would miss the difference between zero completed tasks and one.

Choose the signal by the failure you need to detect. KV occupancy can reveal impending cache exhaustion. Pair occupancy with cache-hit rate when the cost is recomputation, since even a full cache may be doing valuable work by reusing prompt prefixes. Queue depth exposes accumulating waits. Deadline slack tells you how much delay a request can still tolerate. An estimate of output length helps plan for long answers. Cache pressure, for example, will miss overload on a deployment where the cache never comes close to filling.

### Four ways to turn a signal into a decision

[34:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2041s)

The rule itself has to fit inside the time available for a decision. For the agent sending 40 calls in two seconds, a planner that takes five seconds can finish after the entire burst has arrived. Four research designs illustrate what that planning time buys and what cheaper rules can see.

- **Single threshold.** Compare KV occupancy with one limit for each incoming request. The example takes tens of microseconds per decision. It can respond quickly on each arrival, provided cache pressure actually rises under load.
- **Dual gate.** Require two signals, KV use and cache hits, to agree before acting. The example admits whole agent sessions and checks the signals in under 100 microseconds. It reported roughly four times SGLang's throughput in an offline test. That result leaves performance under live arrivals and retries open.
- **Fleet optimizer.** QLM plans across queues together to decide which requests to admit. It can account for work elsewhere in the fleet, at a reported cost of about five seconds per decision.
- **Formal deadline bound.** Calculate whether a request can still meet its time-to-first-token deadline in the worst case covered by the bound. The example makes a microsecond-scale decision and reported a drop in deadline misses from 99% to zero in its evaluation.

The papers used different models, engines and workloads, so their gains cannot establish a ranking. Jain compares them along two axes. Does the policy optimize deadlines or throughput? Does it admit individual requests or whole sessions? For a coding agent, Jain wants to admit sessions as live traffic arrives, using an engine he can deploy as shipped. The session-aware dual gate comes close because it can run without an engine fork. Its offline evaluation leaves that combination unproven.

Which rule would you try first on your own deployment? If requests carry deadlines, try a deadline bound. Otherwise, observe the deployment at peak load. If KV memory fills, try an occupancy threshold, or add a second gate when prefix reuse affects the cost of admission. If the cache has room while calls pile up, try a queue threshold. You can find out which branch applies before spending time tuning a threshold.

### Replay the burst and count every arrival

[38:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2315s)

Start by measuring the load the replicas can sustain with admission control disabled. That establishes what 1× capacity means on this deployment. Specify the candidate policy's inputs, limits and refusal behavior. Then replay the same traffic trace with the policy enabled and disabled, sweeping offered load from 0.5× to 2× measured capacity. Below capacity a policy can appear to work simply because it has never had to refuse anything.

What score makes the cost of a refusal visible? Khatri's **goodput under admission** divides the number of requests meeting the SLO by the number of requests that arrived. For the streaming SLO here, the answer must start on time and avoid excessive gaps between tokens. Suppose the policy refuses 95 of 100 requests and serves the other five perfectly. Attainment among admitted requests is five out of five, or 100%. Goodput under admission counts all 100 in the denominator. Five percent of all arriving requests met the latency target.

The trace needs to preserve the behavior that creates pressure in the first place. Keep agent fan-out, retries and session membership. Uniform random arrivals discard the timing and dependencies the policy must handle. Set the deadline relative to latency measured without load so readers can tell how much slowdown the experiment allowed.

Khatri asks for seven disclosures alongside an admission result.

- **Load.** Give offered load as a multiple of measured capacity.
- **Deadline.** Express it relative to unloaded latency.
- **Signal liveness.** Show whether the metric the policy watches actually moved under stress.
- **Variation.** Give the repeat count and how much the runs differed.
- **Parameters.** Identify which values were tuned locally, borrowed or left at their defaults.
- **Denominator.** Say whether refusals counted toward goodput.
- **Configuration.** Record the model, engine and version, hardware, workload, SLO and baseline.

### Benchmark the same decision function that handles live traffic

[43:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2631s)

Jain and Khatri built **AdmitPerf** to compare policies on a common deployment. Its ingress library sits in front of an inference engine such as vLLM or SGLang and makes a decision for each arrival. Its benchmark harness exercises a policy against an engine on a real GPU. Both call the same decision function, so a benchmarked rule can go into the serving path without being reimplemented. Among the 16 papers they compared on evaluation setup, no two shared the same combination of baseline, engine, workload and SLO.

The harness has five operations. Bring up the GPU-backed engine, smoke-test that it serves requests, run each policy several times, compare the results and the size of the differences, then tear the infrastructure down to stop paying for it. Provisioning sits outside the benchmark. Loading the model takes minutes, and reprovisioning between candidates would change the deployment being used as the control.

How can the harness tell an empty cache from a missing metric? Each policy declares the signals it needs. A KV-threshold policy requires the fraction of KV cache in use. If the engine cannot supply it, the harness stops at startup. Without that check, a missing value could read as zero. The policy would admit every request and produce a plausible-looking benchmark of the admit-all baseline.

For a run that proceeds, the harness records each decision with the state the policy saw. The run directory contains four JSON artifacts covering the manifest, summary, decisions and outcomes. An engineer can inspect what triggered a refusal on a particular arrival as well as what changed over the whole run.

### On the A10, the queue grew while the KV cache had room

[47:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2868s)

The GPU comparison used a half-billion-parameter model on an A10, with the engine capped at four simultaneous sequences. The arrival rate was 14 requests per second, with two repeats. With admission control disabled, p95 time to first token was about 2.5 seconds, with variation of roughly 700 milliseconds either way. That percentile describes the wait near the slow end of the observed requests.

What did a shorter queue buy? A policy allowing at most two waiting requests admitted about 63% of arrivals and shed about 37%. Its p95 time to first token fell to about one second. Goodput under admission stayed the same in this comparison. Admitted requests saw a shorter tail wait while the fraction of *all arrivals* served within the SLO held steady.

The engine metrics explain why queue depth worked here. KV occupancy peaked at about 0.5%. Four short active sequences left ample cache even as work accumulated outside the batch. A policy waiting for high KV occupancy would have kept admitting work. The waiting-request cap had an effect because the queue was where this deployment registered overload.

There is a laptop path for trying the mechanism. AdmitPerf includes a small fake engine that implements the three endpoints the harness uses and gets busy under load. A developer can exercise the policy there, then measure its behavior on the target GPU. AdmitPerf can also discover a policy packaged separately from its repository. The tool grew out of the *LLM Inference at Scale* handbook survey. Jain points readers to both projects on the [AdmitPerf source-code slide](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3084s).

## Key takeaways

- Make the admission decision before prefill commits GPU memory. A request that fits now can still put pressure on the batch as its answer and KV cache grow.
- Choose the admission unit around what must finish. Splitting capacity evenly across two dependent sets of calls can leave both agent tasks waiting.
- Give the agent an action it can take. `Retry-After` with jitter spreads returning traffic out. A caller-specific limit calls for lower concurrency. Pool-wide overload calls for a longer wait, another provider or parking the task.
- Check whether the signal moves under load, then score the policy across all arriving requests. On the A10, KV occupancy peaked at 0.5%. Capping the queue at two waiting requests cut p95 time to first token from about 2.5 seconds to about one second at the same reported goodput.

## Slides

### Saying No.

When a coding agent sends more requests than your self-hosted LLM can handle, admission control decides which requests get through.

[0:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=0s) [Title slide for “Saying No.” by Harshul Jain and Parv Khatri. The subtitle introduces admission control when a coding agent overwhelms a self-hosted LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a833c8fd7930b20ab68d449a0cf20190ccfdcd6357b48547f259463ea555bd5.webp)

Saying No.
Admission Control When Your Coding Agent Overwhelms Your Own LLM

Harshul Jain    Parv Khatri

AIE CODE SF 2026
45 MIN · 4 PARTS

01 Motivation
02 Admission control
03 Strategies
04 Benchmarking

### Four stops, one request’s journey

The orange card marks the starting question: why can the system admit only so much traffic? One request connects the next three stops: where the system decides whether to admit it, how it says no, and how to test whether the policy works.

[2:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=134s) [The orange Motivation card introduces the question of how much traffic infrastructure can admit. Three white cards follow for admission control, strategies and benchmarking.](https://assets.ai.engineer/sites/aiecode2025/sha256/dcca267ccbca07141816984a76f1aa9c01063d10158c35e3585a0ff2c16e0dcd.webp)

AGENDA
Four stops, one request’s journey

01 Motivation
Why infrastructure can only admit so much

02 Admission control
Where the decision lives and what it needs

03 Strategies
How a system says no

04 Benchmarking
How to test a policy

4 / 66

Interface text:
4 / 66  [illegible]

### Inference: a prompt goes in, an answer comes out

A user asks for help fixing a failing test. The model builds its reply one token at a time, using earlier tokens to choose the next, and streams back an explanation that the test expects a sorted list.

[2:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=140s) [A user asks for help fixing a failing test. The model generates “The,” “test” and “expects” in sequence. The streamed answer explains that the test expects a sorted list.](https://assets.ai.engineer/sites/aiecode2025/sha256/8661523d99329e84eeb60f53176c35440c285382546d12d7cbdcf5950d63130f.webp)

Inference: a prompt goes in, an answer comes out
STARTING POINT · LLM INFERENCE

1 · The prompt
someone asks
Fix this failing test

2 · The model
writes one token at a time
LLM
1  The
2  test
3  expects
4  next...
each word depends on the ones before

3 · The answer
streams back
The test expects a sorted list.
Fix line 42...

One request is one trip through the model. A product makes thousands of these trips a minute.
5 / 66

Interface text:
5 / 66  [illegible]

### Two public outages, two years apart

OpenAI cut incoming traffic after its routers hit memory limits. At Anthropic, Sonnet 4.5 slowed down before 5xx errors appeared. Both follow-ups point to deciding which requests to accept before they reach the engine.

[7:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=431s) [The OpenAI panel shows routers reaching a memory limit and names graceful load shedding as the follow-up. The Anthropic panel shows rising latency and server errors under insufficient compute and calls for earlier detection of capacity limits.](https://assets.ai.engineer/sites/aiecode2025/sha256/9926dabb263d3ce5a9014d13b2b212157f5bd59d092df07cc93ca18ee7194eed.webp)

TWO DAYS IT WENT WRONG
Two public outages, two years apart

OpenAI — Nov 8, 2023
router    router
memory limit
1  Heavy completion traffic all day
2  Routing nodes hit memory limits
3  Replicas added, then rolled back
4  Fix: cut inbound traffic, ramp slowly
Their follow-up: graceful load shedding.

Anthropic — Jan 21, 2026
latency
5xx
5xx
not enough compute
1  Sonnet 4.5 tier slowed down first
2  Errors followed, over about 90 minutes
3  Cause: not enough compute for demand
Their follow-up: detect capacity limits earlier.

Neither follow-up says a faster engine. Both ask for a decision earlier, before the engine.
More incidents in the appendix →
8 / 66

Interface text: [illegible]

### Both follow-ups point at the front door

Admission control sits between the client and the queue. Before a request takes GPU memory, it decides whether to start it now, queue it for later, or reject it. Here, a 429 tells the client to retry in 2 seconds.

[8:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=528s) [The pipeline highlights Admit between Client and Queue, followed by Route and Engine. Three panels show a request taking GPU memory immediately, waiting in a queue, or receiving a 429 response with a retry delay.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b069dfc6fbc366244cd54673f6fb1bcb3a8750eb36118e287dfd49f3f54a64d.webp)

WORKING BACKWARD · WHERE THE FIX LIVES
Both follow-ups point at the front door

Client → Admit → Queue → Route → Engine

Admit    start now
GPU memory
Into service, takes GPU memory now.

Hold    wait in queue
queue · served later
Accepted, served later.

Reject    not now
429
at capacity
retry in 2 s
A defined error: 429, 503 or 529.

This decision is admission control. Why does it have to happen before the engine?
9 / 66

### A yes holds memory for the request’s whole life

Prefill builds the KV cache. Each output token grows it until the last token frees the memory. Request B’s longer answer holds memory longer than A’s or C’s. At admission, the engine can’t tell how long that will be.

[10:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=612s) [Three request timelines show a wait, a blue prefill block and a yellow decode segment that grows as tokens are generated. Brackets mark memory held from prefill through the last token. Request B has the longest decode and holds memory longest.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9a6a14d840121491fa6d4123c6cc5671e24ef6d3dc0c6183ea82fe61b1a6581.webp)

REQUEST LIFECYCLE · PREFILL AND DECODE
A yes holds memory for the request’s whole life

Request A
short answer
prefill    decode
memory held    freed

Request B
long answer
prefill    decode
memory held    freed

Request C
short answer
prefill    decode
memory held    freed

How long will it run?
Unknown at the door, known only at the end

Waits: no memory used
Prefill: reads the prompt
Decode: one token at a time, cache grows

The KV cache grows until the last token. At the door, nobody knows how long that will be.
10 / 66

### Memory keeps growing after a request is admitted

As a request generates tokens, its KV cache grows. If GPU memory fills, the engine can drop the cache and rebuild it later, move it to CPU or disk, or cancel the request. Rejecting the request up front avoids wasting or delaying work.

[11:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=693s) [Stacks of KV-cache blocks approach a dashed GPU memory limit over time. Three panels describe evicting the cache and recomputing later, swapping it to CPU or disk, and preempting the request with an error.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba7b9db401783022ab333e247e8ffa0ff791a9356eb07eba5a72aa4aa702c5cf.webp)

WHY THE ENGINE CANNOT FIX IT ALONE
Memory keeps growing after a request is admitted

KV cache in GPU memory
illustrative
!   GPU memory full   !
time →

Evict
drop the cache, recompute later

Swap
move the cache to CPU or disk

Preempt
cancel the request, return an error

All three throw away or delay work already done. Saying no at the door avoids all three.
11 / 66

Interface text: 11/66   [illegible]

### More teams now host their own models

Teams self-host to keep code and prompts in-house, make costs predictable at volume and choose which model to run. They also take over a decision the API provider used to make: which requests their GPUs will serve.

[12:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=748s) [Three panels give reasons for self-hosting models. Privacy keeps code and prompts inside the network. Cost reflects owned or reserved GPUs. Control lets the team choose the model, version and serving setup.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc6fd767cabd2d69d090b546822017d6e6a8145b6da3ced4215b2b522d34f2ab.webp)

WHOSE PROBLEM THIS IS
More teams now host their own models

Privacy
code stays in-house
your code
Your code and prompts never leave your network.

Cost
predictable at volume
GPU
owned
You pay for GPUs you own or reserve, not per token.

Control
your choice of model
Models
Llama 3.3 70B
Qwen3 Coder
your fine-tune
You pick the model, the version and how it is served.

With a hosted API, the provider runs the front door. When you host it, that job is yours.
12 / 66

Interface text:
12 / 66
[illegible]

### Coding Agents changed what arrives

Serving stacks were tuned for people who pause to read between requests. A coding agent can make dozens of calls for one task, retry on timeout, and stall the whole plan while it waits for one slow call.

[13:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=797s) [A person at a laptop sends one request at a time and pauses to read. Beside it, a coding agent sends dozens of calls for one task and retries on timeout. A slow call stalls the agent’s plan.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a52a50f82577d453d771b57a67a9c7853f4e50adef41be9f6f26e4314c28aa4.webp)

MOTIVATION · WHY NOW
Coding Agents changed what arrives

A person chatting
the traffic we planned for
one at a time
· One request at a time
· Seconds of reading in between
· A slow answer is an annoyance

A coding agent
the traffic we got
+ retries
· Dozens of calls for one task
· No pause, retries on timeout
· A slow call stalls the whole plan

Serving stacks were tuned for the left panel. Agents send the right one. Here is one task.
13 / 66

Interface text:
13 / 66  [illegible]

### Forty requests, eight slots, three options

When forty requests compete for eight slots, queueing makes new arrivals wait. Evicting a running request discards its partial answer. Admission control can refuse a request early with a 429 and tell the caller to retry in two seconds.

[15:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=901s) [For forty requests and eight slots, three panels show the tradeoffs. Queueing delays request starts, eviction discards a partial answer, and early refusal sends the caller a 429 response with a two-second retry delay.](https://assets.ai.engineer/sites/aiecode2025/sha256/afee6e21af0d82b09969a70f26d563cb9fb1fbaf69dc6abccdf2e932e7ded5bb.webp)

PUTTING IT TOGETHER
Forty requests, eight slots, three options

Queue it
everyone waits
The line grows and every request starts later.

Evict it
drop running work
partial answer lost
Work already done is thrown away.

Refuse it
admission control
429
retry in 2 s
The caller gets a clear no, early, and can retry.

Most engines do the first two by default. This talk is about the third.
15 / 66

### Four stops, one request’s journey

A checkmark closes the section on capacity limits. The orange card marks the second stop, admission control. Where does the system decide whether to accept a request, and what information does it need?

[15:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=930s) [Four agenda cards run left to right. Fundamentals is checked, Admission control is highlighted in orange, and Strategies and Benchmarking follow.](https://assets.ai.engineer/sites/aiecode2025/sha256/70caa5eec7b35b0b9d9dcd7f2355bf021ec19c789558c9ec301a051241852d0f.webp)

AGENDA
Four stops, one request’s journey

✓ Fundamentals
Why infrastructure can only admit so much

02 Admission control
Where the decision lives and what it needs

03 Strategies
How a system says no

04 Benchmarking
How to test a policy

17 / 66

Interface text:
17/66  [illegible]

### Admission control

Where in the serving stack should a system decide whether to accept a request? This section asks what information the system needs to make that call and what response the client receives.

[15:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=941s) [Section 02 asks where the decision to say no lives. It covers where the decision sits, what it returns to the client, and what it needs to see. A presenter tile appears on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/de313455328fe647f5b98eb3535ac7049a9d2af428dd90c4ef4e90bd2ea249ed.webp)

02
Admission control
Where does the decision to say no live?

COVERS
Where the decision sits, what it returns to the client, and what it needs to see.

18 / 66

Interface text:
Parv Khatri

### Three moments to decide, only one is in front of the engine

At ingress, a gateway can refuse a request before the engine accepts it. The scheduler sees more as it picks the next batch or reclaims KV cache during decode. Changing either decision means maintaining an engine fork.

[17:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1039s) [Arrows connect three panels. At ingress, a proxy or gateway can refuse a request. At the queue boundary, the engine scheduler picks the next batch. During decode, it can preempt work or reclaim KV cache.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1ac10f3fac2c31bb650015bceb7af108fe31f8a50914c3ce6c2d2eb06a4fead.webp)

02 · ADMISSION CONTROL · WHEN TO DECIDE

Three moments to decide, only one is in front of the engine

Ingress
The admission layer: refuse before the request is accepted
READS
Request features and exposed engine metrics
NEEDS
A proxy or gateway. No engine change.

Queue boundary
Inside the engine: the scheduler picks the next batch
READS
Also the current batch state
NEEDS
Scheduler changes. A maintained fork.

During decode
Inside the engine: the scheduler preempts or reclaims KV
READS
Also decode progress
NEEDS
Scheduler changes. A maintained fork.

Later moments see more, but they are engine scheduling, not a separate layer. Ingress is the only one that needs no engine fork.

20 / 66

### The policy reads the request and the engine together

The gateway sees the request’s input length and any deadline. The engines see KV cache use and how long the oldest request has waited. The admission policy uses both views to admit, defer, or reject the request.

[19:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1175s) [Arrows point into a policy that can admit, defer, or reject. Request inputs are input length, client or tenant, agent session, and an optional deadline. Engine inputs are KV cache use, queue age, cache hit rate, and load per pool.](https://assets.ai.engineer/sites/aiecode2025/sha256/99f12c76d7dd76cd5dbbeb70c5b040c772be9c7807eff3b049750485160e4696.webp)

02 · ADMISSION CONTROL · WHAT IT MUST SEE
The policy reads the request and the engine together

From the request    gateway
• Input length
• Client or tenant
• Agent session
• Deadline, if it carries one

Policy
Admit
Defer
Reject

From the engines    engine
• KV cache use
• Queue age
• Cache hit rate
• Load per pool

The gateway has one view, the engine the other. Admission needs both.
22 / 66

Interface text: [illegible]

### Defer returns 503 with a Retry-After

During a burst, the engine estimates when a running batch will free enough KV cache for the request. Here the server returns 503 and tells the client to retry in two seconds. If capacity won’t return soon, the server rejects the request.

[23:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1439s) [An HTTP 503 Service Unavailable response specifies a two-second retry delay. The policy panel explains that a running batch is expected to free enough KV cache soon and that queue age is rising during a burst.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c3bf29c39f8c3e7f23b8ebf385d8bcd319287ee48a54cd78bd8ac2b9eaef736.webp)

02 · ADMISSION CONTROL · DEFER
Defer returns 503 with a Retry-After

The response
HTTP

HTTP/1.1 503 Service Unavailable
Retry-After: 2
{"error":"at_capacity"}

When to defer
the policy

CAPACITY RETURNS SOON
A running batch will free enough KV within τ.

A BURST, NOT A TREND
Queue age is rising but not saturated.

WHERE T COMES FROM
The engine: time until enough memory frees.

Defer is for load that will fit shortly. If capacity is not coming back, reject.
26 / 66

Interface text:
26 / 66  [illegible]

### Handling a defer: wait the time, plus jitter

Without Retry-After, all ten agents retry together. When the server asks them to wait two seconds, each agent adds a small random delay to that wait. The server then receives the ten retries spread across time.

[24:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1493s) [Client pseudocode waits for the Retry-After delay plus jitter before retrying. The diagram shows ten retries clustered together in the upper panel and spread across time in the lower panel.](https://assets.ai.engineer/sites/aiecode2025/sha256/8817144544783d4554adcb80850a130da334708a6a1ef87941a8e30b0a7b554c.webp)

Handling a defer: wait the time, plus jitter
02 · ADMISSION CONTROL · HANDLING A DEFER

The response
server and agent

HTTP/1.1 503 Service Unavailable
Retry-After: 2
# agent
sleep(retry_after + jitter())
retry(call)

Ten retries
what the server sees

Without Retry-After
all ten retry together

With Retry-After and jitter
spread across the window
Timeline: now, +2 s

Without a time, every agent picks its own delay, often zero. Defer turns a retry storm into a schedule you control.

27 / 66

### Reject returns 429 or 529

A 429 tells the caller to reduce its own load because its tenant or session is using more than its share. A 529 means every replica is full, so the provider is rejecting requests from all callers.

[25:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1538s) [The 429 card describes a client exceeding its share, with per-client load, concurrent calls, and quota as signals. The 529 card describes saturation across all replicas, with pool-level KV use and queue age as signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/13b03008795774acc4cbf7f0e6bf73dca0353f342cb9765c7bf9fe3096109c72.webp)

Reject returns 429 or 529
02 · ADMISSION CONTROL · REJECT

429    client over its share
Too many requests
MEANS
This tenant or session is using more than its share.
POLICY READS
Per-client load, concurrent calls, quota.

529    pool full
Provider overloaded
MEANS
Every replica is saturated. Nobody gets in.
POLICY READS
Pool-level KV use, queue age, no relief in sight.

A reject says why. The reason decides how the client should react.
28 / 66

### Strategies

The self-hosted LLM stack lacks a layer that checks capacity before accepting a request. Harshul Jain introduces admission control strategies for deciding when to turn a request away.

[27:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1650s) [Section 03 asks how a system can say no and on what basis. A recap identifies the need for a layer that checks available capacity. A presenter tile appears on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/eef1990d8eb2e0bfce54fdc56bba2eba020cfe59cb2d4ec21ccb728a39704506.webp)

03
Strategies
How can a system say no, and based on what?

SO FAR
No layer in today's stack decides whether there is capacity. That layer is ours to add.

9 min
30 / 66

Interface text:
harshul jain

### How to build a policy: four choices

First choose what to admit. A rule then uses a load signal to decide whether to admit the work, defer it with Retry-After, or reject it with 429. The unit constrains the signal: deadline slack only works for requests that carry a deadline.

[29:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1792s) [Four cards show how to choose the unit to admit, the load signal to watch, the rule that makes the decision, and the action the client sees. Actions include admitting, deferring with Retry-After, and rejecting with 429.](https://assets.ai.engineer/sites/aiecode2025/sha256/751ade166c858e7f555f26ce72af72bd6310121b7ed895f608dd1d89e91ae54e.webp)

How to build a policy: four choices
03 · STRATEGIES · BUILDING A POLICY

1 Unit
Request
Agent session
Tenant

WHAT GETS ADMITTED
One call, or the whole task

2 Signal
KV pressure
Queue depth
Deadline slack

WHAT IT WATCHES
Must move under load

3 Rule
Threshold
Dual gate
Optimizer
Formal bound

HOW IT DECIDES
Microseconds to seconds

4 Action
Admit
Defer with Retry-After
Reject with 429

WHAT THE CLIENT HEARS
A contract agents can follow

Each choice limits the next. Next: choice 1, what to admit.
32 / 66

Interface text:
32 / 66   [illegible]

### Admit single requests or whole sessions?

In this example, two agent tasks need 20 calls each. Admitting about half of each task’s calls leaves both waiting for the rest. Admitting whole sessions lets A finish and gives B one answer for all 20 calls: try again later.

[31:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1893s) [Orange squares mark admitted calls and gray squares mark deferred calls. In the per-request example, both sessions receive partial admission and neither finishes. With per-session admission, session A finishes and session B is deferred as a whole.](https://assets.ai.engineer/sites/aiecode2025/sha256/9330d4d3cc2452ee5529e702aad31d93674d5eb0de308fcdcab9aa6ae2ed97c9.webp)

Admit single requests or whole sessions?
03 · STRATEGIES · THE UNIT

Legend: Admitted, Deferred
ILLUSTRATIVE

Per request
About half of each task gets in. No task can finish.

Per session
Session A finishes. Session B gets one clear answer.

Session A · 20 calls
Session B · 20 calls

Coding agents send sessions. Next: choice 2, which signal to watch.
33 / 66

Interface text: 33 / 66 [illegible]

## Full transcript

[0:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=5s) Good morning everyone. My name is

[0:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=7s) Good morning everyone. My name is

[0:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=7s) Good morning everyone. My name is Hershel Jan and I have here with me

[0:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=10s) Hershel Jan and I have here with me

[0:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=10s) Hershel Jan and I have here with me Pearl. Today we are going to talk about

[0:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=13s) Pearl. Today we are going to talk about

[0:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=13s) Pearl. Today we are going to talk about saying no. It's called as admission

[0:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=16s) saying no. It's called as admission

[0:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=16s) saying no. It's called as admission control. How our system decides which

[0:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=20s) control. How our system decides which

[0:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=20s) control. How our system decides which request to get in when your coding

[0:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=22s) request to get in when your coding

[0:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=22s) request to get in when your coding agents overwhelm the LLMs you host

[0:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=25s) agents overwhelm the LLMs you host

[0:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=25s) agents overwhelm the LLMs you host yourself.

[0:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=27s) yourself.

[0:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=27s) yourself. Uh before we move on a quick intro about

[0:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=30s) Uh before we move on a quick intro about

[0:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=30s) Uh before we move on a quick intro about us.

[0:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=31s) us.

[0:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=32s) us. So I am a senior software engineer at

[0:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=34s) So I am a senior software engineer at

[0:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=34s) So I am a senior software engineer at Audible where I lead some of the MLAI

[0:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=37s) Audible where I lead some of the MLAI

[0:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=37s) Audible where I lead some of the MLAI infrastructure.

[0:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=38s) infrastructure.

[0:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=38s) infrastructure. Uh I am also writing an open-source

[0:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=40s) Uh I am also writing an open-source

[0:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=40s) Uh I am also writing an open-source handbook called as LLM inference at

[0:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=43s) handbook called as LLM inference at

[0:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=43s) handbook called as LLM inference at scale. It has received over 300 plus

[0:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=45s) scale. It has received over 300 plus

[0:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=45s) scale. It has received over 300 plus stars by now.

[0:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=47s) stars by now.

[0:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=47s) stars by now. I am also maintaining an open-source

[0:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=49s) I am also maintaining an open-source

[0:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=49s) I am also maintaining an open-source framework called agentship which helps

[0:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=51s) framework called agentship which helps

[0:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=51s) framework called agentship which helps you ship reliable and secure agentic

[0:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=55s) you ship reliable and secure agentic

[0:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=55s) you ship reliable and secure agentic harnesses in an ultra fast speed. Uh

[0:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=59s) harnesses in an ultra fast speed. Uh

[0:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=59s) harnesses in an ultra fast speed. Uh perv would you like to introduce

[1:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=60s) perv would you like to introduce

[1:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=60s) perv would you like to introduce yourself?

[1:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=61s) yourself?

[1:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=61s) yourself? Sure. Hi everyone. I'm Perv. I'm a

[1:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=63s) Sure. Hi everyone. I'm Perv. I'm a

[1:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=64s) Sure. Hi everyone. I'm Perv. I'm a senior software developer at SAP. I work

[1:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=66s) senior software developer at SAP. I work

[1:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=66s) senior software developer at SAP. I work on like cloud application and LLM

[1:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=69s) on like cloud application and LLM

[1:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=69s) on like cloud application and LLM infrastructure. I wrote an LLM fine-tune

[1:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=72s) infrastructure. I wrote an LLM fine-tune

[1:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=72s) infrastructure. I wrote an LLM fine-tune guide. I contribute to agentship and I'm

[1:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=75s) guide. I contribute to agentship and I'm

[1:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=75s) guide. I contribute to agentship and I'm like building a task tracker which is a

[1:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=78s) like building a task tracker which is a

[1:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=78s) like building a task tracker which is a local post task board for you and your

[1:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=80s) local post task board for you and your

[1:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=80s) local post task board for you and your agent.

[1:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=82s) agent.

[1:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=82s) agent. Uh apart from this if you would like to

[1:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=84s) Uh apart from this if you would like to

[1:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=84s) Uh apart from this if you would like to connect with us our LinkedIn QR codes

[1:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=87s) connect with us our LinkedIn QR codes

[1:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=87s) connect with us our LinkedIn QR codes are right there on this slide. Now back

[1:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=89s) are right there on this slide. Now back

[1:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=89s) are right there on this slide. Now back to you Hershel.

[1:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=92s) to you Hershel.

[1:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=92s) to you Hershel. Great. So

[1:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=95s) Great. So

[1:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=95s) Great. So here is the plan. We are going to start

[1:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=97s) here is the plan. We are going to start

[1:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=97s) here is the plan. We are going to start with understanding the motivation why

[1:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=101s) with understanding the motivation why

[1:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=101s) with understanding the motivation why any serving stack can admit so much of a

[1:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=105s) any serving stack can admit so much of a

[1:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=105s) any serving stack can admit so much of a traffic.

[1:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=106s) traffic.

[1:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=106s) traffic. Second, we try to dive into admission

[1:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=108s) Second, we try to dive into admission

[1:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=108s) Second, we try to dive into admission control where the decision to say no

[1:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=111s) control where the decision to say no

[1:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=111s) control where the decision to say no lives and what it needs to see. Third,

[1:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=115s) lives and what it needs to see. Third,

[1:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=115s) lives and what it needs to see. Third, we will go through some of the proven

[1:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=117s) we will go through some of the proven

[1:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=117s) we will go through some of the proven strategies, the different ways a system

[2:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=120s) strategies, the different ways a system

[2:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=120s) strategies, the different ways a system can actually make the admission

[2:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=121s) can actually make the admission

[2:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=121s) can actually make the admission decision. And fourth, we will try to

[2:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=125s) decision. And fourth, we will try to

[2:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=125s) decision. And fourth, we will try to dive into the benchmarking where you

[2:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=127s) dive into the benchmarking where you

[2:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=127s) dive into the benchmarking where you will learn how to prove that your policy

[2:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=131s) will learn how to prove that your policy

[2:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=131s) will learn how to prove that your policy for admission is working or it's not

[2:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=133s) for admission is working or it's not

[2:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=133s) for admission is working or it's not working.

[2:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=136s) working.

[2:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=136s) working. So let's get started with the

[2:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=138s) So let's get started with the

[2:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=138s) So let's get started with the motivation. Um

[2:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=141s) motivation. Um

[2:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=141s) motivation. Um and for this

[2:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=146s) everything today uh let's try to

[2:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=148s) everything today uh let's try to

[2:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=148s) everything today uh let's try to understand what is inference. Everything

[2:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=151s) understand what is inference. Everything

[2:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=151s) understand what is inference. Everything today happens during this inference

[2:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=154s) today happens during this inference

[2:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=154s) today happens during this inference itself. So inference is simply a moment

[2:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=157s) itself. So inference is simply a moment

[2:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=157s) itself. So inference is simply a moment when someone sends a prompt to the model

[2:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=160s) when someone sends a prompt to the model

[2:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=160s) when someone sends a prompt to the model and gets back an answer.

[2:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=163s) and gets back an answer.

[2:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=163s) and gets back an answer. So the step one is like a user prompt

[2:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=165s) So the step one is like a user prompt

[2:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=165s) So the step one is like a user prompt comes in. It's a sequence of words. Uh

[2:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=168s) comes in. It's a sequence of words. Uh

[2:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=168s) comes in. It's a sequence of words. Uh for example, it could be like failing a

[2:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=171s) for example, it could be like failing a

[2:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=171s) for example, it could be like failing a fix a failing test. The model which is

[2:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=175s) fix a failing test. The model which is

[2:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=175s) fix a failing test. The model which is running on your GPUs writes the answer

[2:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=177s) running on your GPUs writes the answer

[2:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=177s) running on your GPUs writes the answer one token at a time. A token is roughly

[3:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=180s) one token at a time. A token is roughly

[3:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=180s) one token at a time. A token is roughly like a word as I mentioned. Uh it writes

[3:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=183s) like a word as I mentioned. Uh it writes

[3:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=183s) like a word as I mentioned. Uh it writes the first word then uses that word to

[3:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=186s) the first word then uses that word to

[3:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=186s) the first word then uses that word to pick the second uh then it uses both to

[3:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=190s) pick the second uh then it uses both to

[3:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=190s) pick the second uh then it uses both to pick the third and it keeps on

[3:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=192s) pick the third and it keeps on

[3:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=192s) pick the third and it keeps on continuing like that and it cannot write

[3:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=196s) continuing like that and it cannot write

[3:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=196s) continuing like that and it cannot write all of them at once because every word

[3:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=198s) all of them at once because every word

[3:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=198s) all of them at once because every word depends on the previous one. So those

[3:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=202s) depends on the previous one. So those

[3:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=202s) depends on the previous one. So those once all the words are generated they

[3:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=204s) once all the words are generated they

[3:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=204s) once all the words are generated they stream back to the user uh who ask for

[3:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=208s) stream back to the user uh who ask for

[3:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=208s) stream back to the user uh who ask for the result and that volt prompt and the

[3:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=211s) the result and that volt prompt and the

[3:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=212s) the result and that volt prompt and the answer out is like that one request. Now

[3:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=214s) answer out is like that one request. Now

[3:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=214s) answer out is like that one request. Now I want you to hold on to one small

[3:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=217s) I want you to hold on to one small

[3:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=217s) I want you to hold on to one small detail like model has no idea how long

[3:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=221s) detail like model has no idea how long

[3:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=221s) detail like model has no idea how long this answer is going to be. It only

[3:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=224s) this answer is going to be. It only

[3:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=224s) this answer is going to be. It only finds out whether it writes when it only

[3:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=229s) finds out whether it writes when it only

[3:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=229s) finds out whether it writes when it only finds out when it writes the first last

[3:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=231s) finds out when it writes the first last

[3:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=231s) finds out when it writes the first last token.

[3:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=233s) token.

[3:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=233s) token. Keep that in mind. It will matter in the

[3:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=235s) Keep that in mind. It will matter in the

[3:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=235s) Keep that in mind. It will matter in the few minutes.

[4:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=241s) So a model never sits on the internet

[4:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=245s) So a model never sits on the internet

[4:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=245s) So a model never sits on the internet itself. Every request travels through a

[4:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=248s) itself. Every request travels through a

[4:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=248s) itself. Every request travels through a short pipeline. It starts in your app or

[4:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=252s) short pipeline. It starts in your app or

[4:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=252s) short pipeline. It starts in your app or your agent. Then it reaches your API

[4:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=255s) your agent. Then it reaches your API

[4:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=255s) your agent. Then it reaches your API gateway

[4:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=256s) gateway

[4:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=256s) gateway which checks your API key counts how

[4:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=258s) which checks your API key counts how

[4:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=258s) which checks your API key counts how many requests you are sending. Then a

[4:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=261s) many requests you are sending. Then a

[4:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=261s) many requests you are sending. Then a router picks a replica which which is

[4:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=263s) router picks a replica which which is

[4:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=263s) router picks a replica which which is just one running copy of the model and

[4:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=266s) just one running copy of the model and

[4:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=266s) just one running copy of the model and finally the requests land on the

[4:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=268s) finally the requests land on the

[4:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=268s) finally the requests land on the inference engine. The common some of the

[4:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=271s) inference engine. The common some of the

[4:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=271s) inference engine. The common some of the common inference engines are the VLMs,

[4:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=273s) common inference engines are the VLMs,

[4:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=274s) common inference engines are the VLMs, SGLAN, Tensor RT etc. engine is what

[4:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=279s) SGLAN, Tensor RT etc. engine is what

[4:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=279s) SGLAN, Tensor RT etc. engine is what actually hosts your model.

[4:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=282s) actually hosts your model.

[4:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=282s) actually hosts your model. It actually serves your model uh and it

[4:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=285s) It actually serves your model uh and it

[4:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=285s) It actually serves your model uh and it loads the model weights into your GPUs.

[4:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=288s) loads the model weights into your GPUs.

[4:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=288s) loads the model weights into your GPUs. It groups many requests into one batch

[4:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=290s) It groups many requests into one batch

[4:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=290s) It groups many requests into one batch so the GPU stays busy and for every

[4:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=293s) so the GPU stays busy and for every

[4:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=293s) so the GPU stays busy and for every request it keeps something called as a

[4:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=295s) request it keeps something called as a

[4:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=296s) request it keeps something called as a KV cache. In plain words, the KV cache

[4:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=298s) KV cache. In plain words, the KV cache

[4:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=298s) KV cache. In plain words, the KV cache is the memory the request builds up when

[5:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=302s) is the memory the request builds up when

[5:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=302s) is the memory the request builds up when it try when its answer is being written.

[5:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=305s) it try when its answer is being written.

[5:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=305s) it try when its answer is being written. Longer the answer, bigger this memory

[5:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=307s) Longer the answer, bigger this memory

[5:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=307s) Longer the answer, bigger this memory gets.

[5:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=308s) gets.

[5:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=308s) gets. Keep this picture in your head. Every

[5:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=310s) Keep this picture in your head. Every

[5:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=310s) Keep this picture in your head. Every box here makes some decision but one

[5:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=313s) box here makes some decision but one

[5:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=313s) box here makes some decision but one decision is missing and we will find it

[5:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=316s) decision is missing and we will find it

[5:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=316s) decision is missing and we will find it out in the few minutes.

[5:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=319s) out in the few minutes.

[5:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=319s) out in the few minutes. First, how much an engine can actually

[5:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=322s) First, how much an engine can actually

[5:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=322s) First, how much an engine can actually hold.

[5:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=326s) Every deployment has a ceiling and it

[5:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=328s) Every deployment has a ceiling and it

[5:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=328s) Every deployment has a ceiling and it comes from the three targets that pull

[5:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=330s) comes from the three targets that pull

[5:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=330s) comes from the three targets that pull against each other. Quality is how big

[5:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=333s) against each other. Quality is how big

[5:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=333s) against each other. Quality is how big your and precise your model is. Latency

[5:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=337s) your and precise your model is. Latency

[5:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=337s) your and precise your model is. Latency is how fast the first token arrives and

[5:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=340s) is how fast the first token arrives and

[5:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=340s) is how fast the first token arrives and every token after it. Throughput is how

[5:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=343s) every token after it. Throughput is how

[5:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=343s) every token after it. Throughput is how many requests your GPUs can serve at a

[5:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=345s) many requests your GPUs can serve at a

[5:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=345s) many requests your GPUs can serve at a same time. You know the old saying good,

[5:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=349s) same time. You know the old saying good,

[5:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=349s) same time. You know the old saying good, fast, cheap, but you have to pick two.

[5:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=351s) fast, cheap, but you have to pick two.

[5:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=351s) fast, cheap, but you have to pick two. It's the same here. You cannot pick all

[5:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=353s) It's the same here. You cannot pick all

[5:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=353s) It's the same here. You cannot pick all the three. You cannot prioritize all the

[5:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=355s) the three. You cannot prioritize all the

[5:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=355s) the three. You cannot prioritize all the three. So you can't So every product has

[5:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=359s) three. So you can't So every product has

[5:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=359s) three. So you can't So every product has to pick two. For an example like an

[6:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=361s) to pick two. For an example like an

[6:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=361s) to pick two. For an example like an interactive chat, you would want to go

[6:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=364s) interactive chat, you would want to go

[6:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=364s) interactive chat, you would want to go with the quality and the latency because

[6:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=366s) with the quality and the latency because

[6:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=366s) with the quality and the latency because users are more sensitive to that. It's

[6:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=369s) users are more sensitive to that. It's

[6:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=369s) users are more sensitive to that. It's quite obvious to the users if the

[6:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=371s) quite obvious to the users if the

[6:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=371s) quite obvious to the users if the quality is down or the latency is high.

[6:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=373s) quality is down or the latency is high.

[6:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=374s) quality is down or the latency is high. For something like an async agents, you

[6:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=376s) For something like an async agents, you

[6:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=376s) For something like an async agents, you would want to go with a higher

[6:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=377s) would want to go with a higher

[6:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=377s) would want to go with a higher throughput as well as quality. If

[6:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=380s) throughput as well as quality. If

[6:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=380s) throughput as well as quality. If quality is less, you will have to redo

[6:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=383s) quality is less, you will have to redo

[6:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=383s) quality is less, you will have to redo the task and redoing is very very

[6:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=385s) the task and redoing is very very

[6:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=385s) the task and redoing is very very expensive because these are the

[6:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=387s) expensive because these are the

[6:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=387s) expensive because these are the background and the asynchronous tasks.

[6:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=390s) background and the asynchronous tasks.

[6:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=390s) background and the asynchronous tasks. And the high volume real-time systems

[6:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=392s) And the high volume real-time systems

[6:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=392s) And the high volume real-time systems such as the autocompletes they would try

[6:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=395s) such as the autocompletes they would try

[6:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=395s) such as the autocompletes they would try to favor latency as well as and the

[6:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=397s) to favor latency as well as and the

[6:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=397s) to favor latency as well as and the throughput in favor of the quality

[6:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=399s) throughput in favor of the quality

[6:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=399s) throughput in favor of the quality because you don't need an expensive

[6:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=401s) because you don't need an expensive

[6:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=401s) because you don't need an expensive model for doing the autocompletes for

[6:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=403s) model for doing the autocompletes for

[6:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=403s) model for doing the autocompletes for you. But whichever corner you give up

[6:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=406s) you. But whichever corner you give up

[6:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=406s) you. But whichever corner you give up there is still a maximum load you can

[6:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=409s) there is still a maximum load you can

[6:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=409s) there is still a maximum load you can carry while hitting your SLO

[6:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=412s) carry while hitting your SLO

[6:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=412s) carry while hitting your SLO your service level objective. Let's say

[6:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=414s) your service level objective. Let's say

[6:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=414s) your service level objective. Let's say that on this deployment it's eight

[6:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=417s) that on this deployment it's eight

[6:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=417s) that on this deployment it's eight requests at once. In a normal minute,

[7:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=420s) requests at once. In a normal minute,

[7:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=420s) requests at once. In a normal minute, you never notice that limit. Now,

[7:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=423s) you never notice that limit. Now,

[7:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=423s) you never notice that limit. Now, picture the 40 requests arriving in the

[7:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=425s) picture the 40 requests arriving in the

[7:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=425s) picture the 40 requests arriving in the same two seconds.

[7:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=428s) It's such a high load.

[7:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=435s) And this is not like hypothetical. It

[7:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=437s) And this is not like hypothetical. It

[7:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=437s) And this is not like hypothetical. It has happened to like biggest providers

[7:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=439s) has happened to like biggest providers

[7:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=439s) has happened to like biggest providers in the world. Here are the two public

[7:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=441s) in the world. Here are the two public

[7:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=441s) in the world. Here are the two public postmortms two years apart.

[7:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=444s) postmortms two years apart.

[7:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=444s) postmortms two years apart. On November 8, 2023, OpenAI after a day

[7:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=447s) On November 8, 2023, OpenAI after a day

[7:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=447s) On November 8, 2023, OpenAI after a day of unusual heavy traffic, maybe not 40

[7:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=451s) of unusual heavy traffic, maybe not 40

[7:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=451s) of unusual heavy traffic, maybe not 40 requests, it would be some million

[7:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=453s) requests, it would be some million

[7:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=453s) requests, it would be some million requests, their routing servers ran out

[7:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=456s) requests, their routing servers ran out

[7:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=456s) requests, their routing servers ran out of the memory. They started failing

[7:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=457s) of the memory. They started failing

[7:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=457s) of the memory. They started failing their health checks. They added more

[7:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=460s) their health checks. They added more

[7:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=460s) their health checks. They added more replicas, then had to roll them back.

[7:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=462s) replicas, then had to roll them back.

[7:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=462s) replicas, then had to roll them back. The way out was to cut incoming traffic

[7:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=465s) The way out was to cut incoming traffic

[7:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=465s) The way out was to cut incoming traffic and ramp it up slowly. In January 21

[7:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=470s) and ramp it up slowly. In January 21

[7:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=470s) and ramp it up slowly. In January 21 2026 enthropic the sonnet 4.5 tire gear

[7:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=475s) 2026 enthropic the sonnet 4.5 tire gear

[7:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=475s) 2026 enthropic the sonnet 4.5 tire gear got slow first then it started returning

[7:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=479s) got slow first then it started returning

[7:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=479s) got slow first then it started returning errors over about 90 minutes. The cause

[8:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=482s) errors over about 90 minutes. The cause

[8:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=482s) errors over about 90 minutes. The cause they gave was simple not enough compute

[8:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=484s) they gave was simple not enough compute

[8:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=484s) they gave was simple not enough compute for the demand.

[8:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=487s) for the demand.

[8:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=487s) for the demand. And now let's look at what each team

[8:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=490s) And now let's look at what each team

[8:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=490s) And now let's look at what each team decided as their next set of the action

[8:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=492s) decided as their next set of the action

[8:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=492s) decided as their next set of the action item. Neither of them says we need a

[8:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=495s) item. Neither of them says we need a

[8:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=495s) item. Neither of them says we need a faster engine.

[8:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=497s) faster engine.

[8:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=497s) faster engine. Open AI's followup is graceful load

[8:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=500s) Open AI's followup is graceful load

[8:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=500s) Open AI's followup is graceful load shedding. Enthropics is detecting

[8:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=503s) shedding. Enthropics is detecting

[8:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=504s) shedding. Enthropics is detecting capacity limits earlier.

[8:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=506s) capacity limits earlier.

[8:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=506s) capacity limits earlier. It's the same idea in the different

[8:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=509s) It's the same idea in the different

[8:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=509s) It's the same idea in the different words which is decide on your request

[8:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=512s) words which is decide on your request

[8:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=512s) words which is decide on your request sooner

[8:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=514s) sooner

[8:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=514s) sooner to make sure which request gets in. And

[8:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=518s) to make sure which request gets in. And

[8:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=518s) to make sure which request gets in. And there are more incidents which we have

[8:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=521s) there are more incidents which we have

[8:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=521s) there are more incidents which we have linked in the appendix. If you are

[8:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=523s) linked in the appendix. If you are

[8:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=523s) linked in the appendix. If you are interested definitely go and read

[8:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=524s) interested definitely go and read

[8:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=524s) interested definitely go and read through them.

[8:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=529s) So

[8:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=532s) So

[8:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=532s) So if you walk backward from those

[8:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=534s) if you walk backward from those

[8:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=534s) if you walk backward from those follow-ups,

[8:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=535s) follow-ups,

[8:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=535s) follow-ups, they both point at the same place, the

[8:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=537s) they both point at the same place, the

[8:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=537s) they both point at the same place, the front door. Here is the pipeline again

[9:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=541s) front door. Here is the pipeline again

[9:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=541s) front door. Here is the pipeline again with the missing step filled in.

[9:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=545s) with the missing step filled in.

[9:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=545s) with the missing step filled in. We have a client, we have an admit

[9:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=548s) We have a client, we have an admit

[9:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=548s) We have a client, we have an admit layer, the Q router, and then the

[9:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=550s) layer, the Q router, and then the

[9:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=550s) layer, the Q router, and then the engine.

[9:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=552s) engine.

[9:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=552s) engine. It's never the client talking directly

[9:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=554s) It's never the client talking directly

[9:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=554s) It's never the client talking directly to your engine or client talking to your

[9:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=557s) to your engine or client talking to your

[9:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=557s) to your engine or client talking to your queue directly. There is an admission

[9:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=560s) queue directly. There is an admission

[9:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=560s) queue directly. There is an admission phase in between.

[9:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=562s) phase in between.

[9:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=562s) phase in between. Admission is a decision made before a

[9:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=565s) Admission is a decision made before a

[9:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=565s) Admission is a decision made before a request takes any GPU memory and it only

[9:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=568s) request takes any GPU memory and it only

[9:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=568s) request takes any GPU memory and it only has three possible answers.

[9:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=573s) Admission decides

[9:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=576s) Admission decides

[9:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=576s) Admission decides whether to start now or you want to

[9:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=581s) whether to start now or you want to

[9:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=582s) whether to start now or you want to hold the request. You want to accept it

[9:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=584s) hold the request. You want to accept it

[9:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=584s) hold the request. You want to accept it but make it wait. Third is the reject.

[9:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=587s) but make it wait. Third is the reject.

[9:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=587s) but make it wait. Third is the reject. That means not right now. It can throw

[9:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=590s) That means not right now. It can throw

[9:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=590s) That means not right now. It can throw different amount of errors here such as

[9:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=592s) different amount of errors here such as

[9:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=592s) different amount of errors here such as 429s and we will dive into this bit more

[9:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=596s) 429s and we will dive into this bit more

[9:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=596s) 429s and we will dive into this bit more deeper in the second part.

[10:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=600s) deeper in the second part.

[10:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=600s) deeper in the second part. this that is the decision that was

[10:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=601s) this that is the decision that was

[10:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=601s) this that is the decision that was missing from our pipeline.

[10:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=606s) So why does it have to happen before the

[10:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=609s) So why does it have to happen before the

[10:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=609s) So why does it have to happen before the engine

[10:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=611s) engine

[10:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=611s) engine and what would happen if you will

[10:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=613s) and what would happen if you will

[10:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=613s) and what would happen if you will actually accept the request what would

[10:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=616s) actually accept the request what would

[10:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=616s) actually accept the request what would be the cost of that yes

[10:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=619s) be the cost of that yes

[10:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=619s) be the cost of that yes so let's try to take a look at this like

[10:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=622s) so let's try to take a look at this like

[10:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=622s) so let's try to take a look at this like two phases of the inference like prefill

[10:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=624s) two phases of the inference like prefill

[10:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=624s) two phases of the inference like prefill and the decode. So first comes the

[10:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=626s) and the decode. So first comes the

[10:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=626s) and the decode. So first comes the prefill. That's the engine reading the W

[10:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=629s) prefill. That's the engine reading the W

[10:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=629s) prefill. That's the engine reading the W prompt, building the request KV cache.

[10:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=631s) prompt, building the request KV cache.

[10:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=631s) prompt, building the request KV cache. And then comes the decode. That's the

[10:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=633s) And then comes the decode. That's the

[10:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=633s) And then comes the decode. That's the engine writing the answer one token at a

[10:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=635s) engine writing the answer one token at a

[10:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=635s) engine writing the answer one token at a time. With every token, the cash grows a

[10:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=638s) time. With every token, the cash grows a

[10:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=638s) time. With every token, the cash grows a little more. All of that memory stays

[10:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=640s) little more. All of that memory stays

[10:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=640s) little more. All of that memory stays reserved until the request finishes.

[10:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=645s) reserved until the request finishes.

[10:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=645s) reserved until the request finishes. So a yes is not a moment. It's a

[10:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=648s) So a yes is not a moment. It's a

[10:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=648s) So a yes is not a moment. It's a commitment that lasts the request whole

[10:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=650s) commitment that lasts the request whole

[10:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=650s) commitment that lasts the request whole life. And here you can look at the

[10:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=653s) life. And here you can look at the

[10:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=653s) life. And here you can look at the example.

[10:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=655s) example.

[10:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=655s) example. There is a request A, there is a request

[10:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=657s) There is a request A, there is a request

[10:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=657s) There is a request A, there is a request B. Once you admit a request A which is a

[11:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=660s) B. Once you admit a request A which is a

[11:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=660s) B. Once you admit a request A which is a short answer request and then you admit

[11:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=662s) short answer request and then you admit

[11:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=662s) short answer request and then you admit a request B which is a long answer

[11:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=664s) a request B which is a long answer

[11:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=664s) a request B which is a long answer request. Now when you admit a request C

[11:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=666s) request. Now when you admit a request C

[11:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=666s) request. Now when you admit a request C which is again a short answer request,

[11:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=669s) which is again a short answer request,

[11:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=669s) which is again a short answer request, we don't know how longer it would take

[11:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=671s) we don't know how longer it would take

[11:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=671s) we don't know how longer it would take because there is a longer request which

[11:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=673s) because there is a longer request which

[11:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=673s) because there is a longer request which was already in the memory. Should we

[11:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=676s) was already in the memory. Should we

[11:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=676s) was already in the memory. Should we even take that request C or should we

[11:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=678s) even take that request C or should we

[11:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=678s) even take that request C or should we not even take the request C. So there is

[11:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=680s) not even take the request C. So there is

[11:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=680s) not even take the request C. So there is a cost associated of like admitting

[11:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=683s) a cost associated of like admitting

[11:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=683s) a cost associated of like admitting every request because admission could

[11:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=686s) every request because admission could

[11:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=686s) every request because admission could still result in missing out your SLOs's.

[11:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=694s) Another impact apart from the SLOs's is

[11:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=697s) Another impact apart from the SLOs's is

[11:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=697s) Another impact apart from the SLOs's is your memory.

[11:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=700s) your memory.

[11:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=700s) your memory. Here is what it does to your GPU. So

[11:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=703s) Here is what it does to your GPU. So

[11:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=703s) Here is what it does to your GPU. So every running request it keeps growing

[11:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=705s) every running request it keeps growing

[11:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=705s) every running request it keeps growing your cache as it writes. So your total

[11:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=707s) your cache as it writes. So your total

[11:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=707s) your cache as it writes. So your total memory keeps climbing. When the memory

[11:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=709s) memory keeps climbing. When the memory

[11:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=709s) memory keeps climbing. When the memory gets to the full engine has three ways

[11:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=711s) gets to the full engine has three ways

[11:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=711s) gets to the full engine has three ways to cut. Either it will aait

[11:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=714s) to cut. Either it will aait

[11:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=714s) to cut. Either it will aait or like throw away your request. Either

[11:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=717s) or like throw away your request. Either

[11:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=717s) or like throw away your request. Either it will put it to your waiting state

[11:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=719s) it will put it to your waiting state

[11:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=719s) it will put it to your waiting state that means it will swap or it will just

[12:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=723s) that means it will swap or it will just

[12:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=723s) that means it will swap or it will just say okay I won't be able to serve and I

[12:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=725s) say okay I won't be able to serve and I

[12:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=725s) say okay I won't be able to serve and I I will probably return you an error.

[12:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=729s) I will probably return you an error.

[12:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=729s) I will probably return you an error. Every one of those throws I mean every

[12:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=732s) Every one of those throws I mean every

[12:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=732s) Every one of those throws I mean every one of those causes a certain delay in

[12:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=736s) one of those causes a certain delay in

[12:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=736s) one of those causes a certain delay in processing of the request. and missing

[12:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=739s) processing of the request. and missing

[12:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=739s) processing of the request. and missing out your SLOs's again while also wasting

[12:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=741s) out your SLOs's again while also wasting

[12:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=741s) out your SLOs's again while also wasting your memory.

[12:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=750s) So why is it like our problem like we

[12:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=753s) So why is it like our problem like we

[12:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=753s) So why is it like our problem like we have con servers from open AI anthropic

[12:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=757s) have con servers from open AI anthropic

[12:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=757s) have con servers from open AI anthropic bold teams which are serving their own

[12:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=759s) bold teams which are serving their own

[12:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=759s) bold teams which are serving their own infrastructure

[12:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=761s) infrastructure

[12:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=761s) infrastructure for the inference.

[12:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=763s) for the inference.

[12:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=764s) for the inference. So it's a problem for us because

[12:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=768s) So it's a problem for us because

[12:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=768s) So it's a problem for us because there are teams who want privacy. They

[12:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=771s) there are teams who want privacy. They

[12:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=771s) there are teams who want privacy. They want to have their own models hosted on

[12:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=774s) want to have their own models hosted on

[12:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=774s) want to have their own models hosted on their own GPUs

[12:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=777s) their own GPUs

[12:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=777s) their own GPUs and they want it to be hosted on their

[12:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=779s) and they want it to be hosted on their

[12:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=779s) and they want it to be hosted on their own GPUs due to the cost perspectives

[13:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=782s) own GPUs due to the cost perspectives

[13:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=782s) own GPUs due to the cost perspectives but they have to also protect

[13:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=785s) but they have to also protect

[13:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=786s) but they have to also protect the burndown on those GPUs.

[13:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=788s) the burndown on those GPUs.

[13:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=788s) the burndown on those GPUs. So they need to effectively decide which

[13:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=791s) So they need to effectively decide which

[13:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=791s) So they need to effectively decide which request to serve across their inference

[13:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=794s) request to serve across their inference

[13:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=794s) request to serve across their inference ecosystem.

[13:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=802s) On top of

[13:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=804s) On top of

[13:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=804s) On top of and on top of that like the traffic

[13:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=806s) and on top of that like the traffic

[13:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=806s) and on top of that like the traffic arriving at the door has also changed.

[13:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=810s) arriving at the door has also changed.

[13:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=810s) arriving at the door has also changed. Earlier there used to be a person that

[13:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=812s) Earlier there used to be a person that

[13:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=812s) Earlier there used to be a person that was chatting. He would send one request

[13:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=814s) was chatting. He would send one request

[13:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=814s) was chatting. He would send one request at a time. There would be a seconds of

[13:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=816s) at a time. There would be a seconds of

[13:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=816s) at a time. There would be a seconds of reading in between and then there would

[13:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=818s) reading in between and then there would

[13:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=818s) reading in between and then there would be a slow answer. But now coding agents

[13:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=820s) be a slow answer. But now coding agents

[13:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=820s) be a slow answer. But now coding agents they have taken over. There are tons of

[13:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=822s) they have taken over. There are tons of

[13:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=822s) they have taken over. There are tons of calls that these coding agents they send

[13:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=825s) calls that these coding agents they send

[13:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=825s) calls that these coding agents they send along with the retries, pauses,

[13:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=829s) along with the retries, pauses,

[13:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=829s) along with the retries, pauses, um

[13:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=831s) um

[13:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=831s) um like the undo calls as well and it there

[13:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=834s) like the undo calls as well and it there

[13:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=834s) like the undo calls as well and it there is such a overwhelming traffic that your

[13:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=837s) is such a overwhelming traffic that your

[13:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=837s) is such a overwhelming traffic that your cluster has to handle and what that

[14:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=840s) cluster has to handle and what that

[14:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=840s) cluster has to handle and what that means is cluster has a responsibility to

[14:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=843s) means is cluster has a responsibility to

[14:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=843s) means is cluster has a responsibility to take a wise decision of which requests

[14:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=845s) take a wise decision of which requests

[14:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=845s) take a wise decision of which requests to accept and which uh requests to

[14:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=848s) to accept and which uh requests to

[14:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=848s) to accept and which uh requests to reject.

[14:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=855s) And this is exactly that visualization.

[14:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=857s) And this is exactly that visualization.

[14:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=857s) And this is exactly that visualization. You ask your agent that fix an o test.

[14:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=861s) You ask your agent that fix an o test.

[14:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=861s) You ask your agent that fix an o test. It runs all your it tries to run

[14:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=864s) It runs all your it tries to run

[14:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=864s) It runs all your it tries to run multiple tool calls that is reading of

[14:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=867s) multiple tool calls that is reading of

[14:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=867s) multiple tool calls that is reading of the files like around there could be

[14:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=869s) the files like around there could be

[14:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=869s) the files like around there could be like 14 files. There could be like

[14:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=872s) like 14 files. There could be like

[14:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=872s) like 14 files. There could be like around 10 tests that it has to run and

[14:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=875s) around 10 tests that it has to run and

[14:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=875s) around 10 tests that it has to run and it would combine together to make 40 50

[14:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=878s) it would combine together to make 40 50

[14:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=878s) it would combine together to make 40 50 calls for just one request and all these

[14:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=881s) calls for just one request and all these

[14:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=882s) calls for just one request and all these 40 requests they require like reaching

[14:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=884s) 40 requests they require like reaching

[14:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=884s) 40 requests they require like reaching out to the LLM getting back the

[14:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=886s) out to the LLM getting back the

[14:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=886s) out to the LLM getting back the response. Uh and in between some of

[14:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=890s) response. Uh and in between some of

[14:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=890s) response. Uh and in between some of these requests would be long running

[14:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=892s) these requests would be long running

[14:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=892s) these requests would be long running some of the requests would be short

[14:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=895s) some of the requests would be short

[14:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=895s) some of the requests would be short prompt request. So we need to decide

[14:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=897s) prompt request. So we need to decide

[14:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=897s) prompt request. So we need to decide effectively where which one to pick and

[15:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=900s) effectively where which one to pick and

[15:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=900s) effectively where which one to pick and which one not to pick.

[15:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=909s) Yeah. And that was the like reason why

[15:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=912s) Yeah. And that was the like reason why

[15:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=912s) Yeah. And that was the like reason why we need to worry about

[15:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=915s) we need to worry about

[15:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=915s) we need to worry about admission control. Why we need to worry

[15:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=918s) admission control. Why we need to worry

[15:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=918s) admission control. Why we need to worry about

[15:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=921s) about

[15:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=921s) about handling the traffic right way across

[15:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=923s) handling the traffic right way across

[15:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=923s) handling the traffic right way across our infancy ecosystem.

[15:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=932s) Okay. So this brings us to the admission

[15:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=936s) Okay. So this brings us to the admission

[15:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=936s) Okay. So this brings us to the admission control.

[15:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=937s) control.

[15:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=937s) control. Uh now moving on to the next slide. Uh

[15:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=942s) Uh now moving on to the next slide. Uh

[15:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=942s) Uh now moving on to the next slide. Uh that is slide number 18. So in this

[15:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=945s) that is slide number 18. So in this

[15:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=945s) that is slide number 18. So in this slide we will be covering three

[15:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=947s) slide we will be covering three

[15:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=947s) slide we will be covering three questions. Where does this decision

[15:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=949s) questions. Where does this decision

[15:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=950s) questions. Where does this decision from? which request to admit

[15:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=952s) from? which request to admit

[15:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=952s) from? which request to admit and which to reject sits back at your

[15:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=955s) and which to reject sits back at your

[15:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=955s) and which to reject sits back at your stack

[15:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=957s) stack

[15:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=957s) stack and the another question is what does it

[16:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=960s) and the another question is what does it

[16:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=960s) and the another question is what does it need to see to make a good call or

[16:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=963s) need to see to make a good call or

[16:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=963s) need to see to make a good call or judgment on the request and what does it

[16:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=967s) judgment on the request and what does it

[16:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=967s) judgment on the request and what does it returns back to the client. Moving on to

[16:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=970s) returns back to the client. Moving on to

[16:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=970s) returns back to the client. Moving on to the next slide. So here uh we see the

[16:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=975s) the next slide. So here uh we see the

[16:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=975s) the next slide. So here uh we see the pipeline again which each layer's jobs

[16:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=977s) pipeline again which each layer's jobs

[16:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=977s) pipeline again which each layer's jobs is written out. The agent sees its own

[16:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=981s) is written out. The agent sees its own

[16:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=981s) is written out. The agent sees its own call and decides when to retry.

[16:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=985s) call and decides when to retry.

[16:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=985s) call and decides when to retry. The gateway sees API key and request

[16:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=988s) The gateway sees API key and request

[16:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=988s) The gateway sees API key and request count and enforces a rate limit rate

[16:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=991s) count and enforces a rate limit rate

[16:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=991s) count and enforces a rate limit rate limiter for each and the router which

[16:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=995s) limiter for each and the router which

[16:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=995s) limiter for each and the router which sees replicas or healthy and which one

[16:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=997s) sees replicas or healthy and which one

[16:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=997s) sees replicas or healthy and which one to pick. Now the engine sees its KV

[16:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1001s) to pick. Now the engine sees its KV

[16:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1001s) to pick. Now the engine sees its KV cache and the current batch and decides

[16:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1004s) cache and the current batch and decides

[16:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1004s) cache and the current batch and decides the order of request to run and what to

[16:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1007s) the order of request to run and what to

[16:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1007s) the order of request to run and what to evict and print.

[16:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1010s) evict and print.

[16:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1010s) evict and print. So every layer decides something. Now

[16:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1014s) So every layer decides something. Now

[16:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1014s) So every layer decides something. Now ask each of them one question. Is the

[16:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1017s) ask each of them one question. Is the

[16:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1017s) ask each of them one question. Is the room for this request?

[17:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1020s) room for this request?

[17:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1020s) room for this request? The agent says not my job. The gateway

[17:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1023s) The agent says not my job. The gateway

[17:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1023s) The agent says not my job. The gateway says I cannot see it. And the router say

[17:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1027s) says I cannot see it. And the router say

[17:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1027s) says I cannot see it. And the router say not my job and the engine says I only

[17:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1029s) not my job and the engine says I only

[17:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1029s) not my job and the engine says I only find out when I have already said yes.

[17:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1033s) find out when I have already said yes.

[17:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1033s) find out when I have already said yes. The admission box is missing in in

[17:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1037s) The admission box is missing in in

[17:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1037s) The admission box is missing in in between.

[17:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1039s) between.

[17:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1040s) between. So now moving on to the next slide. Um

[17:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1043s) So now moving on to the next slide. Um

[17:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1043s) So now moving on to the next slide. Um so where to put this decision change

[17:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1048s) so where to put this decision change

[17:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1048s) so where to put this decision change what it can see and what it cost you to

[17:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1051s) what it can see and what it cost you to

[17:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1051s) what it can see and what it cost you to run. So there are three possible

[17:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1054s) run. So there are three possible

[17:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1054s) run. So there are three possible moments. The first is an invoice right

[17:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1058s) moments. The first is an invoice right

[17:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1058s) moments. The first is an invoice right in front of the engine. Here the policy

[17:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1061s) in front of the engine. Here the policy

[17:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1061s) in front of the engine. Here the policy sees the request itself plus whatever

[17:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1063s) sees the request itself plus whatever

[17:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1063s) sees the request itself plus whatever metrics the engine exposes. All you need

[17:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1067s) metrics the engine exposes. All you need

[17:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1067s) metrics the engine exposes. All you need is proxy or gateway. The engine stays

[17:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1069s) is proxy or gateway. The engine stays

[17:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1070s) is proxy or gateway. The engine stays exactly as it is. The second one is the

[17:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1073s) exactly as it is. The second one is the

[17:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1073s) exactly as it is. The second one is the Q boundary. Now the decision happens

[17:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1076s) Q boundary. Now the decision happens

[17:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1076s) Q boundary. Now the decision happens inside the engineuler

[17:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1078s) inside the engineuler

[17:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1078s) inside the engineuler as it picks the next batch to run.

[18:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1083s) as it picks the next batch to run.

[18:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1083s) as it picks the next batch to run. The third is during decoding when the

[18:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1085s) The third is during decoding when the

[18:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1086s) The third is during decoding when the scheduleuler can pause a request or take

[18:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1088s) scheduleuler can pause a request or take

[18:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1088s) scheduleuler can pause a request or take back its cache halfway through an

[18:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1091s) back its cache halfway through an

[18:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1091s) back its cache halfway through an answer. Those later movement can see

[18:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1094s) answer. Those later movement can see

[18:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1094s) answer. Those later movement can see more but they mean changing the

[18:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1096s) more but they mean changing the

[18:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1096s) more but they mean changing the engineuler.

[18:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1098s) engineuler.

[18:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1098s) engineuler. That means keeping your own modified

[18:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1100s) That means keeping your own modified

[18:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1100s) That means keeping your own modified copy of BLLM or SG lang and updating it

[18:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1104s) copy of BLLM or SG lang and updating it

[18:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1104s) copy of BLLM or SG lang and updating it every time original changes forever. So

[18:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1108s) every time original changes forever. So

[18:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1108s) every time original changes forever. So an ingress is the only movement that

[18:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1110s) an ingress is the only movement that

[18:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1110s) an ingress is the only movement that works with the engine exactly as it

[18:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1113s) works with the engine exactly as it

[18:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1113s) works with the engine exactly as it ships.

[18:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1114s) ships.

[18:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1114s) ships. So that's where we will stay for the

[18:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1116s) So that's where we will stay for the

[18:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1116s) So that's where we will stay for the rest of the talk. So now moving on to

[18:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1119s) rest of the talk. So now moving on to

[18:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1119s) rest of the talk. So now moving on to the next slide.

[18:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1121s) the next slide.

[18:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1122s) the next slide. Uh so what actually sits in ingress,

[18:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1126s) Uh so what actually sits in ingress,

[18:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1126s) Uh so what actually sits in ingress, right? Uh so you you you can think of it

[18:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1129s) right? Uh so you you you can think of it

[18:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1129s) right? Uh so you you you can think of it as a function. It can take in what it

[18:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1132s) as a function. It can take in what it

[18:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1132s) as a function. It can take in what it knows about the request and current

[18:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1134s) knows about the request and current

[18:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1134s) knows about the request and current state of the system and it returns one

[18:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1138s) state of the system and it returns one

[18:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1138s) state of the system and it returns one of the three answers

[19:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1141s) of the three answers

[19:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1141s) of the three answers like if we have to admit the request. So

[19:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1144s) like if we have to admit the request. So

[19:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1144s) like if we have to admit the request. So the admit is the request enters the

[19:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1147s) the admit is the request enters the

[19:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1147s) the admit is the request enters the engine now and memory is committed for

[19:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1149s) engine now and memory is committed for

[19:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1149s) engine now and memory is committed for this moment. If it's deferred with a

[19:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1153s) this moment. If it's deferred with a

[19:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1153s) this moment. If it's deferred with a time we can call toao hold on like uh

[19:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1156s) time we can call toao hold on like uh

[19:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1156s) time we can call toao hold on like uh hold on and come back after some time

[19:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1159s) hold on and come back after some time

[19:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1159s) hold on and come back after some time and there is another decision which is

[19:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1161s) and there is another decision which is

[19:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1162s) and there is another decision which is with uh reject with a code that no here

[19:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1166s) with uh reject with a code that no here

[19:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1166s) with uh reject with a code that no here is the reason why and until the policy

[19:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1169s) is the reason why and until the policy

[19:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1169s) is the reason why and until the policy says admit the request cost you nothing.

[19:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1172s) says admit the request cost you nothing.

[19:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1172s) says admit the request cost you nothing. It holds no GPU memory at all.

[19:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1177s) It holds no GPU memory at all.

[19:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1177s) It holds no GPU memory at all. Now moving on to the next slide.

[19:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1183s) uh to make the call well the policy

[19:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1186s) uh to make the call well the policy

[19:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1186s) uh to make the call well the policy needs two views at the same time from

[19:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1189s) needs two views at the same time from

[19:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1189s) needs two views at the same time from the request and from the engine. So from

[19:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1193s) the request and from the engine. So from

[19:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1193s) the request and from the engine. So from the request how long the input is, which

[19:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1196s) the request how long the input is, which

[19:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1196s) the request how long the input is, which client or tenant sends it, which agent

[19:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1199s) client or tenant sends it, which agent

[19:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1199s) client or tenant sends it, which agent session it belongs to, and a deadline.

[20:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1203s) session it belongs to, and a deadline.

[20:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1203s) session it belongs to, and a deadline. And from the engine point of view, how

[20:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1206s) And from the engine point of view, how

[20:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1206s) And from the engine point of view, how long uh KV cache is, how long the oldest

[20:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1210s) long uh KV cache is, how long the oldest

[20:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1210s) long uh KV cache is, how long the oldest request has been waiting in the queue,

[20:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1212s) request has been waiting in the queue,

[20:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1212s) request has been waiting in the queue, how often the prompt is already cached,

[20:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1215s) how often the prompt is already cached,

[20:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1215s) how often the prompt is already cached, and how how loaded each pool of replica

[20:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1219s) and how how loaded each pool of replica

[20:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1219s) and how how loaded each pool of replica is. Now look back at the ST. The gateway

[20:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1222s) is. Now look back at the ST. The gateway

[20:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1222s) is. Now look back at the ST. The gateway has left side and the engine has the

[20:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1226s) has left side and the engine has the

[20:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1226s) has left side and the engine has the right side. Nobody has the both.

[20:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1230s) right side. Nobody has the both.

[20:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1230s) right side. Nobody has the both. And that's really why admission has to

[20:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1233s) And that's really why admission has to

[20:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1233s) And that's really why admission has to be its own layer. Now moving on to the

[20:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1236s) be its own layer. Now moving on to the

[20:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1236s) be its own layer. Now moving on to the next slide.

[20:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1239s) next slide.

[20:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1239s) next slide. Here are those eight signals turned into

[20:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1241s) Here are those eight signals turned into

[20:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1241s) Here are those eight signals turned into simple rule. Each one is compared to

[20:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1244s) simple rule. Each one is compared to

[20:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1244s) simple rule. Each one is compared to threshold and when it crosses it pushes

[20:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1247s) threshold and when it crosses it pushes

[20:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1247s) threshold and when it crosses it pushes towards the outcome in the colored pale

[20:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1251s) towards the outcome in the colored pale

[20:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1251s) towards the outcome in the colored pale from the request side.

[20:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1253s) from the request side.

[20:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1253s) from the request side. If the input is too big to fit in the

[20:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1256s) If the input is too big to fit in the

[20:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1256s) If the input is too big to fit in the memory headroom just refer it. If a

[20:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1259s) memory headroom just refer it. If a

[20:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1259s) memory headroom just refer it. If a client or tenant is over

[21:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1262s) client or tenant is over

[21:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1262s) client or tenant is over it share reject it with 429.

[21:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1266s) it share reject it with 429.

[21:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1266s) it share reject it with 429. If one agent session has too many calls

[21:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1269s) If one agent session has too many calls

[21:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1269s) If one agent session has too many calls in flight defer with extras. If waiting

[21:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1273s) in flight defer with extras. If waiting

[21:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1273s) in flight defer with extras. If waiting would make a request miss its deadline

[21:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1275s) would make a request miss its deadline

[21:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1275s) would make a request miss its deadline anyway reject it from the engine side.

[21:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1279s) anyway reject it from the engine side.

[21:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1279s) anyway reject it from the engine side. Uh you can see at the bottom half of

[21:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1283s) Uh you can see at the bottom half of

[21:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1283s) Uh you can see at the bottom half of bottom half side. While KB cache is

[21:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1286s) bottom half side. While KB cache is

[21:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1286s) bottom half side. While KB cache is under high water mark admit it. When the

[21:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1291s) under high water mark admit it. When the

[21:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1291s) under high water mark admit it. When the oldest request has waited longer than

[21:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1293s) oldest request has waited longer than

[21:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1293s) oldest request has waited longer than your target defer it. If the prompt is

[21:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1296s) your target defer it. If the prompt is

[21:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1296s) your target defer it. If the prompt is already cached admit it first because

[21:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1299s) already cached admit it first because

[21:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1299s) already cached admit it first because it's cheap to serve. And when every pool

[21:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1301s) it's cheap to serve. And when every pool

[21:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1301s) it's cheap to serve. And when every pool is full, reject it with 529. One rule of

[21:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1305s) is full, reject it with 529. One rule of

[21:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1305s) is full, reject it with 529. One rule of thumb, pick the signal that changes in

[21:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1308s) thumb, pick the signal that changes in

[21:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1308s) thumb, pick the signal that changes in second, like a QH, not a one minute

[21:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1312s) second, like a QH, not a one minute

[21:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1312s) second, like a QH, not a one minute average. A coding agent sends 40 calls

[21:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1315s) average. A coding agent sends 40 calls

[21:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1315s) average. A coding agent sends 40 calls in 2 seconds.

[21:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1317s) in 2 seconds.

[21:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1317s) in 2 seconds. A minute average won't even notice. So

[22:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1320s) A minute average won't even notice. So

[22:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1320s) A minute average won't even notice. So that's what the policy reads. Now what

[22:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1323s) that's what the policy reads. Now what

[22:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1323s) that's what the policy reads. Now what does it say back? Now moving on to the

[22:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1326s) does it say back? Now moving on to the

[22:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1326s) does it say back? Now moving on to the next slide.

[22:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1328s) next slide.

[22:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1328s) next slide. We will take the three answers one at a

[22:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1331s) We will take the three answers one at a

[22:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1331s) We will take the three answers one at a time and look at each from both sides.

[22:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1335s) time and look at each from both sides.

[22:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1335s) time and look at each from both sides. What the server sends and what your

[22:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1337s) What the server sends and what your

[22:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1337s) What the server sends and what your agent should do with it. First, yes, the

[22:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1341s) agent should do with it. First, yes, the

[22:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1341s) agent should do with it. First, yes, the server sends back a 200 and the response

[22:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1344s) server sends back a 200 and the response

[22:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1344s) server sends back a 200 and the response starts streaming. At that moment, three

[22:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1346s) starts streaming. At that moment, three

[22:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1346s) starts streaming. At that moment, three things happens inside the engine. It

[22:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1349s) things happens inside the engine. It

[22:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1349s) things happens inside the engine. It commits KV cache from the whole prompt

[22:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1351s) commits KV cache from the whole prompt

[22:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1352s) commits KV cache from the whole prompt and the request joins the running batch

[22:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1354s) and the request joins the running batch

[22:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1354s) and the request joins the running batch and decode one token per step alongside

[22:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1357s) and decode one token per step alongside

[22:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1357s) and decode one token per step alongside everyone else

[22:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1360s) everyone else

[22:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1360s) everyone else and it holds that memory until it

[22:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1362s) and it holds that memory until it

[22:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1362s) and it holds that memory until it finishes or until the client

[22:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1364s) finishes or until the client

[22:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1364s) finishes or until the client disconnects.

[22:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1366s) disconnects.

[22:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1366s) disconnects. That's the commitment from part one just

[22:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1369s) That's the commitment from part one just

[22:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1369s) That's the commitment from part one just in the form of an HTTP response which is

[22:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1372s) in the form of an HTTP response which is

[22:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1372s) in the form of an HTTP response which is why the admission decision has to happen

[22:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1374s) why the admission decision has to happen

[22:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1374s) why the admission decision has to happen before 200 never after it.

[22:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1379s) before 200 never after it.

[22:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1379s) before 200 never after it. Now the client side if we are writing

[23:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1382s) Now the client side if we are writing

[23:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1382s) Now the client side if we are writing the agent here is how to handle 200.

[23:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1386s) the agent here is how to handle 200.

[23:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1386s) the agent here is how to handle 200. Watch the stream not the status code.

[23:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1389s) Watch the stream not the status code.

[23:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1390s) Watch the stream not the status code. First time is properly start your

[23:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1393s) First time is properly start your

[23:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1393s) First time is properly start your timeout clock when the first token

[23:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1395s) timeout clock when the first token

[23:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1395s) timeout clock when the first token arrives not when you send the request

[23:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1398s) arrives not when you send the request

[23:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1398s) arrives not when you send the request under the heavy load prefill alone can

[23:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1401s) under the heavy load prefill alone can

[23:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1401s) under the heavy load prefill alone can take few seconds. Second, if you decide

[23:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1404s) take few seconds. Second, if you decide

[23:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1404s) take few seconds. Second, if you decide to give up, actually close the

[23:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1406s) to give up, actually close the

[23:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1406s) to give up, actually close the connection. The server only frees the KV

[23:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1409s) connection. The server only frees the KV

[23:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1409s) connection. The server only frees the KV cache when the request finishes or the

[23:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1411s) cache when the request finishes or the

[23:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1411s) cache when the request finishes or the client disconnects. An agent that times

[23:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1414s) client disconnects. An agent that times

[23:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1414s) client disconnects. An agent that times out but leaves the connection open is

[23:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1416s) out but leaves the connection open is

[23:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1416s) out but leaves the connection open is still holding the GPU memory. And the

[23:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1419s) still holding the GPU memory. And the

[23:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1419s) still holding the GPU memory. And the third one, uh, a stream can fail halfway

[23:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1423s) third one, uh, a stream can fail halfway

[23:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1423s) third one, uh, a stream can fail halfway through, right? And that that's no

[23:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1426s) through, right? And that that's no

[23:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1426s) through, right? And that that's no status for that, right? and you're

[23:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1429s) status for that, right? and you're

[23:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1429s) status for that, right? and you're you're left with partial answer and you

[23:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1431s) you're left with partial answer and you

[23:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1431s) you're left with partial answer and you have to decide whether to retry or carry

[23:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1433s) have to decide whether to retry or carry

[23:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1433s) have to decide whether to retry or carry on. In short, 200 promise to start. It's

[23:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1438s) on. In short, 200 promise to start. It's

[23:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1438s) on. In short, 200 promise to start. It's not a promise to finish. Right?

[24:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1443s) not a promise to finish. Right?

[24:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1443s) not a promise to finish. Right? The second answer is dipper which

[24:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1445s) The second answer is dipper which

[24:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1445s) The second answer is dipper which basically means not yet. The server

[24:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1447s) basically means not yet. The server

[24:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1447s) basically means not yet. The server sends back 503 with retry header.

[24:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1451s) sends back 503 with retry header.

[24:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1451s) sends back 503 with retry header. The header tells the client how many

[24:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1453s) The header tells the client how many

[24:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1453s) The header tells the client how many seconds to wait. The policy picks that

[24:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1455s) seconds to wait. The policy picks that

[24:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1455s) seconds to wait. The policy picks that number from the engine. For example, how

[24:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1458s) number from the engine. For example, how

[24:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1458s) number from the engine. For example, how long until the running batch finishes

[24:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1459s) long until the running batch finishes

[24:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1460s) long until the running batch finishes and freeze the enough cash

[24:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1462s) and freeze the enough cash

[24:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1462s) and freeze the enough cash and you differ when capacity is coming

[24:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1465s) and you differ when capacity is coming

[24:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1465s) and you differ when capacity is coming back soon? It's a burst, not so not a

[24:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1467s) back soon? It's a burst, not so not a

[24:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1467s) back soon? It's a burst, not so not a trend, right? And for example, like if

[24:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1470s) trend, right? And for example, like if

[24:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1470s) trend, right? And for example, like if you think about it like Q age is rising

[24:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1474s) you think about it like Q age is rising

[24:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1474s) you think about it like Q age is rising but the system isn't saturated.

[24:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1479s) but the system isn't saturated.

[24:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1479s) but the system isn't saturated. If the capacity isn't coming back

[24:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1481s) If the capacity isn't coming back

[24:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1481s) If the capacity isn't coming back anytime soon, a dipper just kicks kicks

[24:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1484s) anytime soon, a dipper just kicks kicks

[24:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1484s) anytime soon, a dipper just kicks kicks the can down the road, right? And in

[24:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1488s) the can down the road, right? And in

[24:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1488s) the can down the road, right? And in that case, we can just reject it with

[24:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1490s) that case, we can just reject it with

[24:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1490s) that case, we can just reject it with the honest answer.

[24:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1493s) the honest answer.

[24:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1493s) the honest answer. Moving on to the next slide. On the

[24:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1495s) Moving on to the next slide. On the

[24:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1495s) Moving on to the next slide. On the agent side, handling a defer is simple.

[24:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1499s) agent side, handling a defer is simple.

[24:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1499s) agent side, handling a defer is simple. Sleep for retry after time and a little

[25:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1501s) Sleep for retry after time and a little

[25:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1502s) Sleep for retry after time and a little random jitter, then retry again. The

[25:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1505s) random jitter, then retry again. The

[25:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1505s) random jitter, then retry again. The jitter just means a small random extra

[25:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1507s) jitter just means a small random extra

[25:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1507s) jitter just means a small random extra delay. So client don't like don't call

[25:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1511s) delay. So client don't like don't call

[25:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1511s) delay. So client don't like don't call or like don't at all come back at the

[25:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1514s) or like don't at all come back at the

[25:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1514s) or like don't at all come back at the exact same moment. They will have to

[25:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1516s) exact same moment. They will have to

[25:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1516s) exact same moment. They will have to wait.

[25:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1519s) wait.

[25:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1519s) wait. Look what the server sees from 10

[25:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1521s) Look what the server sees from 10

[25:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1521s) Look what the server sees from 10 retries without retry after every agent

[25:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1525s) retries without retry after every agent

[25:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1525s) retries without retry after every agent picks its own delay. And that's usually

[25:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1529s) picks its own delay. And that's usually

[25:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1529s) picks its own delay. And that's usually that's usually zero. So they all come

[25:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1532s) that's usually zero. So they all come

[25:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1532s) that's usually zero. So they all come back at once. That's how 40 calls turns

[25:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1535s) back at once. That's how 40 calls turns

[25:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1535s) back at once. That's how 40 calls turns into 80.

[25:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1538s) into 80.

[25:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1538s) into 80. With retry after and jitter the retry

[25:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1541s) With retry after and jitter the retry

[25:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1541s) With retry after and jitter the retry spreads out across the window serves

[25:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1543s) spreads out across the window serves

[25:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1543s) spreads out across the window serves asked for.

[25:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1550s) Okay. So if if we see the uh reject

[25:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1555s) Okay. So if if we see the uh reject

[25:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1555s) Okay. So if if we see the uh reject reject returns 429 or 529 the third

[25:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1559s) reject returns 429 or 529 the third

[25:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1559s) reject returns 429 or 529 the third answer is no.

[26:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1561s) answer is no.

[26:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1561s) answer is no. Uh and there there are two different

[26:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1563s) Uh and there there are two different

[26:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1563s) Uh and there there are two different codes for it because there are two

[26:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1565s) codes for it because there are two

[26:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1565s) codes for it because there are two different reasons. A 429s means too many

[26:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1568s) different reasons. A 429s means too many

[26:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1568s) different reasons. A 429s means too many requests.

[26:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1570s) requests.

[26:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1570s) requests. This particular client or agent session

[26:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1573s) This particular client or agent session

[26:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1573s) This particular client or agent session is using more than its share. The policy

[26:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1576s) is using more than its share. The policy

[26:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1576s) is using more than its share. The policy saw something like per client load, the

[26:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1578s) saw something like per client load, the

[26:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1578s) saw something like per client load, the number of concurrent calls or a kota.

[26:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1581s) number of concurrent calls or a kota.

[26:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1581s) number of concurrent calls or a kota. The problem is the caller and then the

[26:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1584s) The problem is the caller and then the

[26:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1584s) The problem is the caller and then the 529 means the provider is overloaded.

[26:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1588s) 529 means the provider is overloaded.

[26:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1588s) 529 means the provider is overloaded. Enthropic uses this code. Every replica

[26:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1591s) Enthropic uses this code. Every replica

[26:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1591s) Enthropic uses this code. Every replica is sold. So nobody is getting in right

[26:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1594s) is sold. So nobody is getting in right

[26:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1594s) is sold. So nobody is getting in right now.

[26:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1596s) now.

[26:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1596s) now. The problem is pool not the caller.

[26:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1600s) The problem is pool not the caller.

[26:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1600s) The problem is pool not the caller. Now uh handling a reject. So how should

[26:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1604s) Now uh handling a reject. So how should

[26:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1604s) Now uh handling a reject. So how should agent react on 429? Reduce your own

[26:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1608s) agent react on 429? Reduce your own

[26:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1608s) agent react on 429? Reduce your own load. Cap how many calls that agent has

[26:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1611s) load. Cap how many calls that agent has

[26:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1611s) load. Cap how many calls that agent has in flight at once.

[26:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1613s) in flight at once.

[26:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1614s) in flight at once. And on 529 retrying harder only makes

[26:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1617s) And on 529 retrying harder only makes

[26:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1617s) And on 529 retrying harder only makes this worse. Waiting longer switching to

[27:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1620s) this worse. Waiting longer switching to

[27:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1620s) this worse. Waiting longer switching to different model provider or park the

[27:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1622s) different model provider or park the

[27:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1622s) different model provider or park the task for later. and one either and on

[27:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1627s) task for later. and one either and on

[27:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1627s) task for later. and one either and on either one keep a retry budget. Use

[27:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1629s) either one keep a retry budget. Use

[27:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1629s) either one keep a retry budget. Use expansion exponential back off with

[27:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1632s) expansion exponential back off with

[27:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1632s) expansion exponential back off with jitter which means waiting longer after

[27:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1635s) jitter which means waiting longer after

[27:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1635s) jitter which means waiting longer after each failed try and after few tries tell

[27:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1639s) each failed try and after few tries tell

[27:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1639s) each failed try and after few tries tell the user what's going on because a

[27:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1641s) the user what's going on because a

[27:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1641s) the user what's going on because a reject that gets retrieded at full speed

[27:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1643s) reject that gets retrieded at full speed

[27:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1644s) reject that gets retrieded at full speed is how you end up in one of the postmodm

[27:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1646s) is how you end up in one of the postmodm

[27:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1646s) is how you end up in one of the postmodm from the talk at we described at the

[27:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1649s) from the talk at we described at the

[27:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1649s) from the talk at we described at the starting.

[27:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1653s) Now over to you Hershel

[27:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1660s) three.

[27:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1661s) three.

[27:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1661s) three. Thank you P. Uh so this is the module

[27:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1663s) Thank you P. Uh so this is the module

[27:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1664s) Thank you P. Uh so this is the module three where we will try to learn about

[27:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1666s) three where we will try to learn about

[27:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1666s) three where we will try to learn about the different strategies uh that have

[27:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1668s) the different strategies uh that have

[27:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1668s) the different strategies uh that have been rolled out for the admission

[27:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1670s) been rolled out for the admission

[27:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1670s) been rolled out for the admission control. So far we have seen that no

[27:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1672s) control. So far we have seen that no

[27:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1672s) control. So far we have seen that no layer in the today's stack asks where uh

[27:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1675s) layer in the today's stack asks where uh

[27:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1675s) layer in the today's stack asks where uh whether there is a room and that layer

[27:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1678s) whether there is a room and that layer

[27:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1678s) whether there is a room and that layer is ours to add and now we give it a

[28:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1681s) is ours to add and now we give it a

[28:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1681s) is ours to add and now we give it a brain. So we will look at why counting

[28:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1683s) brain. So we will look at why counting

[28:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1683s) brain. So we will look at why counting requests isn't enough. Breaking a policy

[28:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1686s) requests isn't enough. Breaking a policy

[28:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1686s) requests isn't enough. Breaking a policy down into four different choices. See

[28:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1688s) down into four different choices. See

[28:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1688s) down into four different choices. See what four different research paper chose

[28:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1692s) what four different research paper chose

[28:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1692s) what four different research paper chose and finish with a decision tree that you

[28:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1695s) and finish with a decision tree that you

[28:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1695s) and finish with a decision tree that you can use onto your own ecosystem.

[28:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1700s) So let's start with what most team

[28:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1702s) So let's start with what most team

[28:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1702s) So let's start with what most team already have which is counting. Yes.

[28:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1706s) already have which is counting. Yes.

[28:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1706s) already have which is counting. Yes. Counting the number of the requests.

[28:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1710s) Counting the number of the requests.

[28:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1710s) Counting the number of the requests. A simple idea is to rate limit at the

[28:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1712s) A simple idea is to rate limit at the

[28:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1712s) A simple idea is to rate limit at the gateway which gives API key a fixed

[28:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1715s) gateway which gives API key a fixed

[28:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1715s) gateway which gives API key a fixed allowance set in advance. It's great for

[28:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1718s) allowance set in advance. It's great for

[28:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1718s) allowance set in advance. It's great for keeping the clients fair, keeping your

[28:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1720s) keeping the clients fair, keeping your

[28:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1720s) keeping the clients fair, keeping your tenants fair with each other,

[28:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1723s) tenants fair with each other,

[28:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1723s) tenants fair with each other, but it does nothing to protect your GPUs

[28:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1727s) but it does nothing to protect your GPUs

[28:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1727s) but it does nothing to protect your GPUs because it doesn't change with the load

[28:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1730s) because it doesn't change with the load

[28:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1730s) because it doesn't change with the load that is happening onto your GPUs or your

[28:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1732s) that is happening onto your GPUs or your

[28:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1732s) that is happening onto your GPUs or your cluster. A fixed cap at the proxy gets

[28:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1737s) cluster. A fixed cap at the proxy gets

[28:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1737s) cluster. A fixed cap at the proxy gets closer. It counts how many requests are

[28:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1739s) closer. It counts how many requests are

[29:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1740s) closer. It counts how many requests are in flight on each replica. The problem

[29:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1743s) in flight on each replica. The problem

[29:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1743s) in flight on each replica. The problem is that the quick crep and a long

[29:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1745s) is that the quick crep and a long

[29:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1745s) is that the quick crep and a long planning to call

[29:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1748s) planning to call

[29:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1748s) planning to call count exactly the same while the real

[29:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1750s) count exactly the same while the real

[29:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1750s) count exactly the same while the real capacity of the GPU changes with every

[29:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1752s) capacity of the GPU changes with every

[29:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1752s) capacity of the GPU changes with every batch. So in practice it mostly cues and

[29:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1757s) batch. So in practice it mostly cues and

[29:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1757s) batch. So in practice it mostly cues and clients timeout ends up doing the

[29:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1760s) clients timeout ends up doing the

[29:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1760s) clients timeout ends up doing the refusing.

[29:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1762s) refusing.

[29:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1762s) refusing. Neither one knows whether our 40 calls

[29:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1765s) Neither one knows whether our 40 calls

[29:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1765s) Neither one knows whether our 40 calls are cheap or expensive.

[29:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1767s) are cheap or expensive.

[29:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1767s) are cheap or expensive. The cost of air policy does though and

[29:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1771s) The cost of air policy does though and

[29:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1771s) The cost of air policy does though and it estimates each request will actually

[29:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1775s) it estimates each request will actually

[29:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1775s) it estimates each request will actually cost what each request will cost in the

[29:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1779s) cost what each request will cost in the

[29:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1779s) cost what each request will cost in the KV memory and the tokens using the live

[29:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1782s) KV memory and the tokens using the live

[29:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1782s) KV memory and the tokens using the live state from your engine. And it says no

[29:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1785s) state from your engine. And it says no

[29:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1785s) state from your engine. And it says no before the request actually takes up the

[29:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1787s) before the request actually takes up the

[29:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1787s) before the request actually takes up the memory. So the short version is count is

[29:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1790s) memory. So the short version is count is

[29:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1790s) memory. So the short version is count is not a cost.

[29:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1795s) How do you build your own policy? So

[29:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1797s) How do you build your own policy? So

[29:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1797s) How do you build your own policy? So there are four choices here. The first

[29:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1799s) there are four choices here. The first

[30:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1800s) there are four choices here. The first is the unit

[30:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1802s) is the unit

[30:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1802s) is the unit which tells you what you are actually

[30:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1803s) which tells you what you are actually

[30:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1804s) which tells you what you are actually admitting. Are you admitting a single

[30:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1806s) admitting. Are you admitting a single

[30:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1806s) admitting. Are you admitting a single request? Are you admitting an agent

[30:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1809s) request? Are you admitting an agent

[30:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1809s) request? Are you admitting an agent session or are you admitting a tenant?

[30:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1813s) session or are you admitting a tenant?

[30:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1813s) session or are you admitting a tenant? The two second is the signal

[30:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1817s) The two second is the signal

[30:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1817s) The two second is the signal which means what is that the policy is

[30:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1820s) which means what is that the policy is

[30:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1820s) which means what is that the policy is watching for taking an admission

[30:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1822s) watching for taking an admission

[30:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1822s) watching for taking an admission decision. This could be as simple as

[30:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1824s) decision. This could be as simple as

[30:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1824s) decision. This could be as simple as your memory pressure which is your KB

[30:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1827s) your memory pressure which is your KB

[30:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1827s) your memory pressure which is your KB pressure. It could be your Q depth. It

[30:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1829s) pressure. It could be your Q depth. It

[30:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1829s) pressure. It could be your Q depth. It could be your deadline slack. That means

[30:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1832s) could be your deadline slack. That means

[30:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1832s) could be your deadline slack. That means whether your request is able to meet

[30:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1834s) whether your request is able to meet

[30:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1834s) whether your request is able to meet your SLOs's or not.

[30:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1837s) your SLOs's or not.

[30:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1837s) your SLOs's or not. And whatever it is, it has to actually

[30:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1839s) And whatever it is, it has to actually

[30:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1839s) And whatever it is, it has to actually move when the system gets busy. So three

[30:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1843s) move when the system gets busy. So three

[30:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1843s) move when the system gets busy. So three is the rule which means how does the

[30:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1846s) is the rule which means how does the

[30:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1846s) is the rule which means how does the signal gets turned into a decision.

[30:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1850s) signal gets turned into a decision.

[30:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1850s) signal gets turned into a decision. So rule is used for that and that can be

[30:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1854s) So rule is used for that and that can be

[30:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1854s) So rule is used for that and that can be anything from a simple threshold that

[30:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1856s) anything from a simple threshold that

[30:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1856s) anything from a simple threshold that takes microscond to a full optimizer

[30:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1858s) takes microscond to a full optimizer

[30:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1858s) takes microscond to a full optimizer that takes seconds.

[31:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1861s) that takes seconds.

[31:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1861s) that takes seconds. Fourth is the action. Once you have the

[31:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1863s) Fourth is the action. Once you have the

[31:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1864s) Fourth is the action. Once you have the rule that gives you a decision what do

[31:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1866s) rule that gives you a decision what do

[31:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1866s) rule that gives you a decision what do you do with it?

[31:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1869s) you do with it?

[31:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1869s) you do with it? What does your how does your client gets

[31:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1871s) What does your how does your client gets

[31:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1871s) What does your how does your client gets impacted through those actions?

[31:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1874s) impacted through those actions?

[31:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1874s) impacted through those actions? That's the admit defer and the reject

[31:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1877s) That's the admit defer and the reject

[31:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1877s) That's the admit defer and the reject that per just talked about in the module

[31:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1880s) that per just talked about in the module

[31:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1880s) that per just talked about in the module two. Each choice limits the next one.

[31:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1884s) two. Each choice limits the next one.

[31:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1884s) two. Each choice limits the next one. For example, a deadline signal only

[31:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1886s) For example, a deadline signal only

[31:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1886s) For example, a deadline signal only works if your request actually carry the

[31:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1888s) works if your request actually carry the

[31:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1888s) works if your request actually carry the deadlines. So let's go through the first

[31:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1891s) deadlines. So let's go through the first

[31:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1891s) deadlines. So let's go through the first three.

[31:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1897s) Choice one. Do you admit a single

[31:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1899s) Choice one. Do you admit a single

[31:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1899s) Choice one. Do you admit a single request or do you admit a poll sessions?

[31:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1904s) request or do you admit a poll sessions?

[31:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1904s) request or do you admit a poll sessions? Say you have two agent task 20 calls

[31:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1906s) Say you have two agent task 20 calls

[31:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1906s) Say you have two agent task 20 calls each room for almost half of them.

[31:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1910s) each room for almost half of them.

[31:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1910s) each room for almost half of them. You can uh point you can see the top

[31:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1914s) You can uh point you can see the top

[31:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1914s) You can uh point you can see the top row. So if you admit per request about

[31:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1917s) row. So if you admit per request about

[31:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1917s) row. So if you admit per request about half of each task gets in but neither

[32:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1920s) half of each task gets in but neither

[32:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1920s) half of each task gets in but neither task can finish because each one is

[32:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1923s) task can finish because each one is

[32:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1923s) task can finish because each one is stuck waiting on the calls that didn't

[32:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1925s) stuck waiting on the calls that didn't

[32:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1925s) stuck waiting on the calls that didn't make it. You have used all the capacity

[32:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1928s) make it. You have used all the capacity

[32:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1928s) make it. You have used all the capacity and completed nothing. If you admit per

[32:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1932s) and completed nothing. If you admit per

[32:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1932s) and completed nothing. If you admit per session, session A gets all of its call

[32:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1935s) session, session A gets all of its call

[32:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1935s) session, session A gets all of its call and finishes. Session B gets one clear

[32:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1937s) and finishes. Session B gets one clear

[32:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1937s) and finishes. Session B gets one clear answer. Come back later. And this

[32:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1940s) answer. Come back later. And this

[32:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1940s) answer. Come back later. And this picture is illustrative, but the logic

[32:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1942s) picture is illustrative, but the logic

[32:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1942s) picture is illustrative, but the logic holds.

[32:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1944s) holds.

[32:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1944s) holds. Our 40 calls are one session. Coding

[32:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1947s) Our 40 calls are one session. Coding

[32:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1947s) Our 40 calls are one session. Coding agents send sessions and yet most

[32:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1950s) agents send sessions and yet most

[32:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1950s) agents send sessions and yet most published policies admit only the single

[32:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1953s) published policies admit only the single

[32:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1953s) published policies admit only the single requests.

[32:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1958s) So which metric do you watch for what

[32:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1960s) So which metric do you watch for what

[32:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1960s) So which metric do you watch for what particular problem?

[32:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1964s) This is the

[32:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1966s) This is the

[32:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1966s) This is the question that we are trying to answer.

[32:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1969s) question that we are trying to answer.

[32:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1969s) question that we are trying to answer. It depends on the problem that you are

[32:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1971s) It depends on the problem that you are

[32:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1971s) It depends on the problem that you are trying to prevent. If you are if you

[32:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1974s) trying to prevent. If you are if you

[32:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1974s) trying to prevent. If you are if you want to keep the cash from overflowing,

[32:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1977s) want to keep the cash from overflowing,

[32:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1977s) want to keep the cash from overflowing, you definitely have to watch your KV

[32:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1979s) you definitely have to watch your KV

[32:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1979s) you definitely have to watch your KV pressure.

[33:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1982s) pressure.

[33:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1982s) pressure. If you want to avoid your wasted

[33:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1984s) If you want to avoid your wasted

[33:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1984s) If you want to avoid your wasted recomputee, you have to watch out

[33:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1987s) recomputee, you have to watch out

[33:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1987s) recomputee, you have to watch out the KV used together with the cash hit

[33:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1989s) the KV used together with the cash hit

[33:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1990s) the KV used together with the cash hit rate. A full cache is fine if it's being

[33:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1992s) rate. A full cache is fine if it's being

[33:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1992s) rate. A full cache is fine if it's being reused.

[33:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1995s) reused.

[33:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1995s) reused. If you want to keep your weights to be

[33:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1998s) If you want to keep your weights to be

[33:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1998s) If you want to keep your weights to be shorter, you have to watch for your

[33:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2000s) shorter, you have to watch for your

[33:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2000s) shorter, you have to watch for your queue depth and you have to take actions

[33:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2002s) queue depth and you have to take actions

[33:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2002s) queue depth and you have to take actions based on that. If you want to meet your

[33:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2005s) based on that. If you want to meet your

[33:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2005s) based on that. If you want to meet your latency targets, you need to watch your

[33:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2007s) latency targets, you need to watch your

[33:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2007s) latency targets, you need to watch your deadline slack. Take actions based on

[33:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2009s) deadline slack. Take actions based on

[33:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2010s) deadline slack. Take actions based on that. If you need to plan for your long

[33:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2012s) that. If you need to plan for your long

[33:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2012s) that. If you need to plan for your long answers, you need to predict for the

[33:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2014s) answers, you need to predict for the

[33:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2014s) answers, you need to predict for the output length.

[33:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2017s) output length.

[33:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2017s) output length. And look at the last column

[33:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2020s) And look at the last column

[33:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2020s) And look at the last column fails when it tells you when each signal

[33:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2024s) fails when it tells you when each signal

[33:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2024s) fails when it tells you when each signal stops working. Look at the first row. KV

[33:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2028s) stops working. Look at the first row. KV

[33:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2028s) stops working. Look at the first row. KV pressure fails when the cache never

[33:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2030s) pressure fails when the cache never

[33:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2030s) pressure fails when the cache never fills up.

[33:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2032s) fills up.

[33:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2032s) fills up. Remember that row? It comes back to bite

[33:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2034s) Remember that row? It comes back to bite

[33:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2034s) Remember that row? It comes back to bite someone in the part four.

[34:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2041s) So what are the four ways to take that

[34:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2043s) So what are the four ways to take that

[34:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2043s) So what are the four ways to take that decision?

[34:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2047s) That is how do you convert your signal

[34:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2049s) That is how do you convert your signal

[34:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2049s) That is how do you convert your signal into a decision from the cheapest to the

[34:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2052s) into a decision from the cheapest to the

[34:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2052s) into a decision from the cheapest to the most expensive. Here are the ways a

[34:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2056s) most expensive. Here are the ways a

[34:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2056s) most expensive. Here are the ways a threshold just compares one number to

[34:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2058s) threshold just compares one number to

[34:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2058s) threshold just compares one number to the one limit. Example is a policy named

[34:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2063s) the one limit. Example is a policy named

[34:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2063s) the one limit. Example is a policy named clued weight that was uh created

[34:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2067s) clued weight that was uh created

[34:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2067s) clued weight that was uh created somewhere in 2026.

[34:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2069s) somewhere in 2026.

[34:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2069s) somewhere in 2026. It takes tens of microsconds per

[34:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2071s) It takes tens of microsconds per

[34:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2071s) It takes tens of microsconds per decision.

[34:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2073s) decision.

[34:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2074s) decision. Second is a dual gate which needs two

[34:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2077s) Second is a dual gate which needs two

[34:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2077s) Second is a dual gate which needs two signals to agree before it acts. Example

[34:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2080s) signals to agree before it acts. Example

[34:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2080s) signals to agree before it acts. Example is a paper called concour

[34:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2083s) is a paper called concour

[34:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2083s) is a paper called concour uh which checks like KVUs and the cash

[34:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2085s) uh which checks like KVUs and the cash

[34:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2085s) uh which checks like KVUs and the cash hits under 100 microsconds.

[34:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2089s) hits under 100 microsconds.

[34:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2089s) hits under 100 microsconds. Third is an optimizer which solves like

[34:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2091s) Third is an optimizer which solves like

[34:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2091s) Third is an optimizer which solves like one big planning problem across all the

[34:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2093s) one big planning problem across all the

[34:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2094s) one big planning problem across all the cues at once. Example is a QLM.

[34:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2097s) cues at once. Example is a QLM.

[34:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2097s) cues at once. Example is a QLM. It's thorough but it takes about 5

[34:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2099s) It's thorough but it takes about 5

[34:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2099s) It's thorough but it takes about 5 seconds per decision.

[35:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2102s) seconds per decision.

[35:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2102s) seconds per decision. And the fourth is the formal bound uh

[35:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2105s) And the fourth is the formal bound uh

[35:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2105s) And the fourth is the formal bound uh which does the math on the worst case.

[35:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2108s) which does the math on the worst case.

[35:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2108s) which does the math on the worst case. Can this request still meets its

[35:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2110s) Can this request still meets its

[35:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2110s) Can this request still meets its deadline?

[35:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2112s) deadline?

[35:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2112s) deadline? The example is the Kronos on which uses

[35:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2116s) The example is the Kronos on which uses

[35:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2116s) The example is the Kronos on which uses the like time to first token and it uh

[35:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2119s) the like time to first token and it uh

[35:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2119s) the like time to first token and it uh provides you like a microcond level

[35:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2121s) provides you like a microcond level

[35:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2121s) provides you like a microcond level latency.

[35:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2123s) latency.

[35:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2123s) latency. And now hold that cost column up against

[35:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2126s) And now hold that cost column up against

[35:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2126s) And now hold that cost column up against our agent. It sends 40 calls in 2

[35:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2128s) our agent. It sends 40 calls in 2

[35:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2128s) our agent. It sends 40 calls in 2 seconds. A 5-second decision shows up

[35:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2131s) seconds. A 5-second decision shows up

[35:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2131s) seconds. A 5-second decision shows up after every party is over.

[35:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2138s) So what the four papers did

[35:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2142s) So what the four papers did

[35:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2142s) So what the four papers did as the complete policy fluid weight

[35:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2144s) as the complete policy fluid weight

[35:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2144s) as the complete policy fluid weight admits single request uses the threshold

[35:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2146s) admits single request uses the threshold

[35:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2146s) admits single request uses the threshold on your KV use and reports 10 to 25%

[35:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2150s) on your KV use and reports 10 to 25%

[35:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2150s) on your KV use and reports 10 to 25% improvement. Conqueror is the one that

[35:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2153s) improvement. Conqueror is the one that

[35:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2153s) improvement. Conqueror is the one that admits your VO agents. It uses your dual

[35:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2156s) admits your VO agents. It uses your dual

[35:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2156s) admits your VO agents. It uses your dual gate and reports four to 4x

[36:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2160s) gate and reports four to 4x

[36:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2160s) gate and reports four to 4x uh the throughput of the SG lang but it

[36:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2162s) uh the throughput of the SG lang but it

[36:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2162s) uh the throughput of the SG lang but it was tested offline.

[36:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2165s) was tested offline.

[36:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2165s) was tested offline. Uh QLM admits the request plans across

[36:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2169s) Uh QLM admits the request plans across

[36:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2169s) Uh QLM admits the request plans across the W feed and reports 40 to 90% gains

[36:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2173s) the W feed and reports 40 to 90% gains

[36:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2173s) the W feed and reports 40 to 90% gains at about 5 seconds per decision. Kronos

[36:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2176s) at about 5 seconds per decision. Kronos

[36:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2176s) at about 5 seconds per decision. Kronos admits requests against their deadline

[36:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2178s) admits requests against their deadline

[36:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2178s) admits requests against their deadline slack with a formal bound and takes

[36:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2181s) slack with a formal bound and takes

[36:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2181s) slack with a formal bound and takes deadline misses from 99% down to zero.

[36:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2185s) deadline misses from 99% down to zero.

[36:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2185s) deadline misses from 99% down to zero. Definitely very amazing accomplishments.

[36:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2189s) Definitely very amazing accomplishments.

[36:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2189s) Definitely very amazing accomplishments. So each purper used a different model

[36:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2192s) So each purper used a different model

[36:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2192s) So each purper used a different model engine and a workload. So please don't

[36:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2194s) engine and a workload. So please don't

[36:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2194s) engine and a workload. So please don't read this as a leaderboard.

[36:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2197s) read this as a leaderboard.

[36:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2197s) read this as a leaderboard. The question that matters for this room

[36:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2199s) The question that matters for this room

[36:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2199s) The question that matters for this room is does any of them fit a coding agent

[36:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2205s) and the answer is no. We placed every

[36:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2209s) and the answer is no. We placed every

[36:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2209s) and the answer is no. We placed every paper on the two axises. One axis is

[36:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2212s) paper on the two axises. One axis is

[36:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2212s) paper on the two axises. One axis is what it optimizes deadlines or the

[36:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2215s) what it optimizes deadlines or the

[36:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2215s) what it optimizes deadlines or the throughputs. Other is what it admits

[36:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2217s) throughputs. Other is what it admits

[36:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2217s) throughputs. Other is what it admits single request or the VO agent sessions.

[37:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2221s) single request or the VO agent sessions.

[37:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2221s) single request or the VO agent sessions. A coding agent needs a policy that

[37:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2223s) A coding agent needs a policy that

[37:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2223s) A coding agent needs a policy that admits bold sessions, works online with

[37:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2225s) admits bold sessions, works online with

[37:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2226s) admits bold sessions, works online with live traffic and runs engine you haven't

[37:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2228s) live traffic and runs engine you haven't

[37:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2228s) live traffic and runs engine you haven't fought

[37:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2231s) fought

[37:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2231s) fought and nothing sits there cleanly. Conc

[37:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2233s) and nothing sits there cleanly. Conc

[37:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2233s) and nothing sits there cleanly. Conc doesn't need a for but it was only

[37:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2236s) doesn't need a for but it was only

[37:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2236s) doesn't need a for but it was only tested offline.

[37:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2238s) tested offline.

[37:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2238s) tested offline. So in practice you will pick the closest

[37:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2240s) So in practice you will pick the closest

[37:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2240s) So in practice you will pick the closest match and build just build your own. And

[37:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2243s) match and build just build your own. And

[37:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2243s) match and build just build your own. And here is how you can choose which polic

[37:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2247s) here is how you can choose which polic

[37:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2247s) here is how you can choose which polic like which kind of a policy you want to

[37:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2249s) like which kind of a policy you want to

[37:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2249s) like which kind of a policy you want to try it out.

[37:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2251s) try it out.

[37:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2251s) try it out. So here is that decision tree.

[37:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2254s) So here is that decision tree.

[37:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2254s) So here is that decision tree. First

[37:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2256s) First

[37:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2256s) First the question that you have to answer is

[37:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2258s) the question that you have to answer is

[37:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2258s) the question that you have to answer is does your request carry any deadlines?

[37:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2260s) does your request carry any deadlines?

[37:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2260s) does your request carry any deadlines? If yes, set up a some sort of a formal

[37:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2264s) If yes, set up a some sort of a formal

[37:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2264s) If yes, set up a some sort of a formal bound on that deadline. This will take

[37:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2266s) bound on that deadline. This will take

[37:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2266s) bound on that deadline. This will take some microsconds level of latency.

[37:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2270s) some microsconds level of latency.

[37:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2270s) some microsconds level of latency. If there is no deadline requirement, the

[37:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2273s) If there is no deadline requirement, the

[37:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2273s) If there is no deadline requirement, the second question that you can ask is does

[37:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2276s) second question that you can ask is does

[37:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2276s) second question that you can ask is does the KV cache fill at the peak load? If

[38:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2280s) the KV cache fill at the peak load? If

[38:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2280s) the KV cache fill at the peak load? If your KV cache fills at the peak load,

[38:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2285s) your KV cache fills at the peak load,

[38:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2285s) your KV cache fills at the peak load, then

[38:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2287s) then

[38:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2287s) then you would want something like a way to

[38:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2291s) you would want something like a way to

[38:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2291s) you would want something like a way to detect the cache prefixes used. So you

[38:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2294s) detect the cache prefixes used. So you

[38:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2294s) detect the cache prefixes used. So you can use some sort of a dual gate

[38:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2295s) can use some sort of a dual gate

[38:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2295s) can use some sort of a dual gate mechanism or a KV threshold mechanism.

[38:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2299s) mechanism or a KV threshold mechanism.

[38:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2299s) mechanism or a KV threshold mechanism. Um and then if your KV cache gets filled

[38:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2304s) Um and then if your KV cache gets filled

[38:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2304s) Um and then if your KV cache gets filled at the peak load, it does not gets

[38:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2307s) at the peak load, it does not gets

[38:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2307s) at the peak load, it does not gets filled at the peak load, then you what

[38:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2308s) filled at the peak load, then you what

[38:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2308s) filled at the peak load, then you what you could do is you can set up a Q level

[38:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2310s) you could do is you can set up a Q level

[38:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2310s) you could do is you can set up a Q level of threshold on it.

[38:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2315s) And that gets us to the benchmarking

[38:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2318s) And that gets us to the benchmarking

[38:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2318s) And that gets us to the benchmarking like once you have a policy, how do you

[38:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2321s) like once you have a policy, how do you

[38:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2321s) like once you have a policy, how do you benchmark?

[38:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2323s) benchmark?

[38:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2323s) benchmark? So I will be handing it over to pop.

[38:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2330s) Thank you. So now moving on to part four

[38:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2333s) Thank you. So now moving on to part four

[38:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2333s) Thank you. So now moving on to part four which is benchmarking. As we just saw no

[38:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2337s) which is benchmarking. As we just saw no

[38:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2337s) which is benchmarking. As we just saw no publish policy fit a coding agent

[38:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2339s) publish policy fit a coding agent

[39:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2340s) publish policy fit a coding agent cleanly. So you are going to adopt one

[39:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2342s) cleanly. So you are going to adopt one

[39:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2342s) cleanly. So you are going to adopt one or you you will have to build your own

[39:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2345s) or you you will have to build your own

[39:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2345s) or you you will have to build your own and then you have to prove if it's

[39:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2347s) and then you have to prove if it's

[39:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2347s) and then you have to prove if it's actually working or not. So to now

[39:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2351s) actually working or not. So to now

[39:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2351s) actually working or not. So to now moving on to the next slide

[39:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2354s) moving on to the next slide

[39:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2354s) moving on to the next slide you would have to test your policies

[39:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2356s) you would have to test your policies

[39:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2356s) you would have to test your policies right so for that you need four things

[39:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2359s) right so for that you need four things

[39:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2359s) right so for that you need four things first is your cluster your policy and

[39:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2363s) first is your cluster your policy and

[39:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2363s) first is your cluster your policy and trace and score. So let's dig deeper

[39:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2366s) trace and score. So let's dig deeper

[39:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2366s) trace and score. So let's dig deeper into each. So your cluster of the first

[39:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2369s) into each. So your cluster of the first

[39:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2369s) into each. So your cluster of the first measure it it is capacity with no

[39:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2373s) measure it it is capacity with no

[39:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2373s) measure it it is capacity with no admission control at all. The load your

[39:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2375s) admission control at all. The load your

[39:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2375s) admission control at all. The load your replica can sustain on their own. Every

[39:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2378s) replica can sustain on their own. Every

[39:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2378s) replica can sustain on their own. Every load level at after this is described as

[39:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2382s) load level at after this is described as

[39:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2382s) load level at after this is described as multiple of your number like multiple of

[39:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2385s) multiple of your number like multiple of

[39:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2385s) multiple of your number like multiple of that number and uh about the policy u

[39:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2389s) that number and uh about the policy u

[39:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2389s) that number and uh about the policy u which signals it reads what limit it

[39:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2392s) which signals it reads what limit it

[39:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2392s) which signals it reads what limit it uses and what it does when it says no.

[39:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2396s) uses and what it does when it says no.

[39:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2396s) uses and what it does when it says no. The third one is trace which is a record

[39:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2398s) The third one is trace which is a record

[39:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2398s) The third one is trace which is a record of real traffic that you replay using

[40:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2401s) of real traffic that you replay using

[40:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2402s) of real traffic that you replay using code coding agent traffic ramping from

[40:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2404s) code coding agent traffic ramping from

[40:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2404s) code coding agent traffic ramping from half your capacity up to twice your

[40:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2407s) half your capacity up to twice your

[40:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2408s) half your capacity up to twice your capacity.

[40:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2409s) capacity.

[40:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2410s) capacity. Four is a score. Then they run the same

[40:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2413s) Four is a score. Then they run the same

[40:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2413s) Four is a score. Then they run the same trace twice. Once with the policy and

[40:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2416s) trace twice. Once with the policy and

[40:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2416s) trace twice. Once with the policy and once without and then you can compare

[40:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2419s) once without and then you can compare

[40:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2419s) once without and then you can compare your result. The score is where the most

[40:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2422s) your result. The score is where the most

[40:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2422s) your result. The score is where the most people go wrong. So let's look look at

[40:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2425s) people go wrong. So let's look look at

[40:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2425s) people go wrong. So let's look look at that. Now moving on to the next slide

[40:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2428s) that. Now moving on to the next slide

[40:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2428s) that. Now moving on to the next slide which is uh how how you score. The score

[40:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2432s) which is uh how how you score. The score

[40:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2432s) which is uh how how you score. The score we recommend is called goodput under

[40:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2435s) we recommend is called goodput under

[40:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2435s) we recommend is called goodput under admission. So you can see the formula on

[40:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2438s) admission. So you can see the formula on

[40:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2438s) admission. So you can see the formula on the screen. The top of the fraction is

[40:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2441s) the screen. The top of the fraction is

[40:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2441s) the screen. The top of the fraction is the request that met their SLO their

[40:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2444s) the request that met their SLO their

[40:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2444s) the request that met their SLO their latency target. That means the answer

[40:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2447s) latency target. That means the answer

[40:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2447s) latency target. That means the answer started on time and it never stalled for

[40:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2450s) started on time and it never stalled for

[40:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2450s) started on time and it never stalled for too long between tokens.

[40:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2453s) too long between tokens.

[40:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2453s) too long between tokens. The bottom of the fraction is every

[40:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2455s) The bottom of the fraction is every

[40:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2456s) The bottom of the fraction is every request that arrived including the one

[40:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2459s) request that arrived including the one

[40:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2459s) request that arrived including the one which got rejected. Why does the bottom

[41:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2462s) which got rejected. Why does the bottom

[41:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2462s) which got rejected. Why does the bottom matter so much?

[41:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2465s) matter so much?

[41:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2465s) matter so much? So try this. Uh refuse 95% of your

[41:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2469s) So try this. Uh refuse 95% of your

[41:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2469s) So try this. Uh refuse 95% of your traffic and serve the remaining 5%

[41:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2471s) traffic and serve the remaining 5%

[41:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2471s) traffic and serve the remaining 5% perfectly. If you only count what you

[41:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2475s) perfectly. If you only count what you

[41:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2475s) perfectly. If you only count what you admitted, your SLO attainment is 100%.

[41:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2479s) admitted, your SLO attainment is 100%.

[41:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2479s) admitted, your SLO attainment is 100%. Looks amazing, right? Just ship it.

[41:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2482s) Looks amazing, right? Just ship it.

[41:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2482s) Looks amazing, right? Just ship it. But no like good put under admission

[41:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2485s) But no like good put under admission

[41:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2485s) But no like good put under admission says 5%. Shedding looks three when you

[41:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2489s) says 5%. Shedding looks three when you

[41:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2489s) says 5%. Shedding looks three when you only account the request you kept. It

[41:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2493s) only account the request you kept. It

[41:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2493s) only account the request you kept. It isn't.

[41:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2495s) isn't.

[41:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2495s) isn't. So moving on to slide 42.

[41:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2499s) The second thing to get right is the

[41:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2502s) The second thing to get right is the

[41:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2502s) The second thing to get right is the workload. A benchmark only tell you

[41:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2504s) workload. A benchmark only tell you

[41:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2504s) workload. A benchmark only tell you something if it looks like your real

[41:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2507s) something if it looks like your real

[41:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2507s) something if it looks like your real traffic.

[41:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2509s) traffic.

[41:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2509s) traffic. So replay agent traces with their fan

[41:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2512s) So replay agent traces with their fan

[41:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2512s) So replay agent traces with their fan out retries session not uniform random

[41:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2516s) out retries session not uniform random

[41:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2516s) out retries session not uniform random arrival.

[41:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2517s) arrival.

[41:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2517s) arrival. Describe load as multiple of the

[42:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2520s) Describe load as multiple of the

[42:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2520s) Describe load as multiple of the capacity you measured and sweep from

[42:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2523s) capacity you measured and sweep from

[42:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2523s) capacity you measured and sweep from half of it to double it.

[42:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2528s) half of it to double it.

[42:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2528s) half of it to double it. Set your guideline as multiple of your

[42:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2531s) Set your guideline as multiple of your

[42:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2531s) Set your guideline as multiple of your latency with no load. And remember a

[42:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2534s) latency with no load. And remember a

[42:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2534s) latency with no load. And remember a benchmark that never goes past capacity

[42:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2537s) benchmark that never goes past capacity

[42:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2537s) benchmark that never goes past capacity never actually tests the policy because

[42:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2539s) never actually tests the policy because

[42:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2539s) never actually tests the policy because the policy never has to say no. So you

[42:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2542s) the policy never has to say no. So you

[42:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2542s) the policy never has to say no. So you have to make sure that you reach at that

[42:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2545s) have to make sure that you reach at that

[42:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2545s) have to make sure that you reach at that point.

[42:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2547s) point.

[42:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2547s) point. Now

[42:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2549s) Now

[42:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2549s) Now move moving on to the seven uh seven

[42:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2552s) move moving on to the seven uh seven

[42:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2552s) move moving on to the seven uh seven things every admission result should

[42:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2554s) things every admission result should

[42:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2554s) things every admission result should disclose.

[42:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2556s) disclose.

[42:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2556s) disclose. Signal liess is one of the seven things

[42:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2559s) Signal liess is one of the seven things

[42:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2559s) Signal liess is one of the seven things every admission result should disclose.

[42:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2562s) every admission result should disclose.

[42:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2562s) every admission result should disclose. Here is the full full list that you can

[42:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2565s) Here is the full full list that you can

[42:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2565s) Here is the full full list that you can see on your screen.

[42:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2567s) see on your screen.

[42:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2567s) see on your screen. The load as a multiple of measured

[42:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2569s) The load as a multiple of measured

[42:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2570s) The load as a multiple of measured capacity

[42:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2571s) capacity

[42:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2571s) capacity deadline sets relative to latency with

[42:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2574s) deadline sets relative to latency with

[42:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2574s) deadline sets relative to latency with no load.

[42:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2576s) no load.

[42:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2576s) no load. Signal liveless which is no paper

[42:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2579s) Signal liveless which is no paper

[42:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2579s) Signal liveless which is no paper report.

[43:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2581s) report.

[43:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2581s) report. How many times you repeated the run and

[43:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2584s) How many times you repeated the run and

[43:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2584s) How many times you repeated the run and how much the result varied not one lucky

[43:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2587s) how much the result varied not one lucky

[43:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2587s) how much the result varied not one lucky run. Where each parameters comes from,

[43:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2590s) run. Where each parameters comes from,

[43:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2590s) run. Where each parameters comes from, turned on this setup, borrowed from

[43:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2592s) turned on this setup, borrowed from

[43:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2592s) turned on this setup, borrowed from somewhere else or left at the default.

[43:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2596s) somewhere else or left at the default.

[43:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2596s) somewhere else or left at the default. Whether the rejected request count in a

[43:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2599s) Whether the rejected request count in a

[43:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2599s) Whether the rejected request count in a good put denominator and the full

[43:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2602s) good put denominator and the full

[43:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2602s) good put denominator and the full configuration

[43:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2605s) and the full configuration is model,

[43:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2608s) and the full configuration is model,

[43:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2608s) and the full configuration is model, engine, version, hardware, workload,

[43:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2611s) engine, version, hardware, workload,

[43:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2611s) engine, version, hardware, workload, SLOs's and baseline. None of this ask

[43:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2615s) SLOs's and baseline. None of this ask

[43:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2615s) SLOs's and baseline. None of this ask anyone to change how they evaluate. It

[43:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2618s) anyone to change how they evaluate. It

[43:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2618s) anyone to change how they evaluate. It just ask them to say what they did. Use

[43:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2621s) just ask them to say what they did. Use

[43:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2621s) just ask them to say what they did. Use it as a checklist when you read on on a

[43:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2624s) it as a checklist when you read on on a

[43:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2624s) it as a checklist when you read on on a paper and when your benchmark and when

[43:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2627s) paper and when your benchmark and when

[43:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2627s) paper and when your benchmark and when you benchmark your own policy. Doing all

[43:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2631s) you benchmark your own policy. Doing all

[43:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2631s) you benchmark your own policy. Doing all seven by hand is painful. So we built a

[43:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2634s) seven by hand is painful. So we built a

[43:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2634s) seven by hand is painful. So we built a tool that records all of this uh as as a

[43:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2638s) tool that records all of this uh as as a

[43:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2638s) tool that records all of this uh as as a default.

[44:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2640s) default.

[44:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2640s) default. So uh the tool is admin perf and it's it

[44:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2645s) So uh the tool is admin perf and it's it

[44:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2645s) So uh the tool is admin perf and it's it it's two things in one repository. So in

[44:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2649s) it's two things in one repository. So in

[44:09](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2649s) it's two things in one repository. So in production if you see it's a library you

[44:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2653s) production if you see it's a library you

[44:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2653s) production if you see it's a library you put in front of VLLM or SG lang it

[44:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2656s) put in front of VLLM or SG lang it

[44:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2656s) put in front of VLLM or SG lang it decides admit defer or reject for every

[44:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2660s) decides admit defer or reject for every

[44:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2660s) decides admit defer or reject for every request that's ingress layer from part

[44:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2663s) request that's ingress layer from part

[44:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2663s) request that's ingress layer from part two.

[44:26](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2666s) So if you also see we on the right side

[44:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2671s) So if you also see we on the right side

[44:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2671s) So if you also see we on the right side uh it's uh in in a benchmark it's a

[44:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2674s) uh it's uh in in a benchmark it's a

[44:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2674s) uh it's uh in in a benchmark it's a harness that runs any policy against a

[44:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2676s) harness that runs any policy against a

[44:36](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2676s) harness that runs any policy against a real engine on a real GPU and reports

[44:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2680s) real engine on a real GPU and reports

[44:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2680s) real engine on a real GPU and reports what the policy cost

[44:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2683s) what the policy cost

[44:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2683s) what the policy cost both sides calls the same function

[44:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2685s) both sides calls the same function

[44:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2685s) both sides calls the same function policy decide. So the policy you

[44:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2689s) policy decide. So the policy you

[44:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2689s) policy decide. So the policy you benchmark is exactly the policy you

[44:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2691s) benchmark is exactly the policy you

[44:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2691s) benchmark is exactly the policy you deploy.

[44:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2693s) deploy.

[44:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2693s) deploy. We build it because across those 16

[44:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2696s) We build it because across those 16

[44:56](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2696s) We build it because across those 16 paper no two share the same baseline

[45:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2700s) paper no two share the same baseline

[45:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2700s) paper no two share the same baseline engine workload or SLO admitf makes a

[45:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2703s) engine workload or SLO admitf makes a

[45:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2704s) engine workload or SLO admitf makes a head-to-head comparison possible.

[45:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2707s) head-to-head comparison possible.

[45:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2707s) head-to-head comparison possible. Now moving on to the next slide.

[45:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2711s) Now moving on to the next slide.

[45:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2711s) Now moving on to the next slide. So using its pive command. So you can

[45:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2715s) So using its pive command. So you can

[45:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2715s) So using its pive command. So you can see on the screen we have five commands.

[45:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2718s) see on the screen we have five commands.

[45:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2718s) see on the screen we have five commands. So infra up starts a real VLM on a real

[45:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2721s) So infra up starts a real VLM on a real

[45:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2722s) So infra up starts a real VLM on a real GPU. Infra smoke checks that it's

[45:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2725s) GPU. Infra smoke checks that it's

[45:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2725s) GPU. Infra smoke checks that it's actually serving. Then we have bench

[45:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2727s) actually serving. Then we have bench

[45:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2727s) actually serving. Then we have bench runs every policy several times each.

[45:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2731s) runs every policy several times each.

[45:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2731s) runs every policy several times each. Then we have bench computer tell you

[45:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2734s) Then we have bench computer tell you

[45:34](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2734s) Then we have bench computer tell you which policy won and how by how much.

[45:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2737s) which policy won and how by how much.

[45:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2737s) which policy won and how by how much. And then we have infraown which which

[45:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2740s) And then we have infraown which which

[45:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2740s) And then we have infraown which which basically means it should shuts all all

[45:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2743s) basically means it should shuts all all

[45:43](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2743s) basically means it should shuts all all off so you stop paying. And that's my

[45:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2746s) off so you stop paying. And that's my

[45:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2746s) off so you stop paying. And that's my favorite one.

[45:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2748s) favorite one.

[45:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2748s) favorite one. Uh and so now provisioning and

[45:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2750s) Uh and so now provisioning and

[45:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2750s) Uh and so now provisioning and benchmarking are separate commands on

[45:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2752s) benchmarking are separate commands on

[45:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2752s) benchmarking are separate commands on purpose. Loading a model takes minutes

[45:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2755s) purpose. Loading a model takes minutes

[45:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2755s) purpose. Loading a model takes minutes and every policy has to face exactly

[45:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2758s) and every policy has to face exactly

[45:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2758s) and every policy has to face exactly same deployment.

[45:59](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2759s) same deployment.

[46:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2760s) same deployment. If you are reprovisioned between policy,

[46:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2762s) If you are reprovisioned between policy,

[46:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2762s) If you are reprovisioned between policy, you would change the very thing you are

[46:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2765s) you would change the very thing you are

[46:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2765s) you would change the very thing you are trying to hold a constant.

[46:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2768s) trying to hold a constant.

[46:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2768s) trying to hold a constant. The deployment is constant. The policy

[46:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2770s) The deployment is constant. The policy

[46:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2770s) The deployment is constant. The policy is variable.

[46:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2775s) So now uh here a policy must declare the

[46:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2779s) So now uh here a policy must declare the

[46:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2779s) So now uh here a policy must declare the signal it reads. This is how admit perf

[46:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2782s) signal it reads. This is how admit perf

[46:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2782s) signal it reads. This is how admit perf deals with the trap I just showed you.

[46:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2785s) deals with the trap I just showed you.

[46:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2785s) deals with the trap I just showed you. Every policy has to declare the signal

[46:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2787s) Every policy has to declare the signal

[46:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2787s) Every policy has to declare the signal it needs. This KV cache threshold policy

[46:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2791s) it needs. This KV cache threshold policy

[46:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2791s) it needs. This KV cache threshold policy says I require KV used correction which

[46:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2795s) says I require KV used correction which

[46:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2795s) says I require KV used correction which is the share of the KV cache in use.

[46:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2800s) is the share of the KV cache in use.

[46:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2800s) is the share of the KV cache in use. If the engine cannot report that signal,

[46:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2802s) If the engine cannot report that signal,

[46:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2802s) If the engine cannot report that signal, the run stops at the startup with an

[46:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2805s) the run stops at the startup with an

[46:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2805s) the run stops at the startup with an error. You can actually read without the

[46:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2808s) error. You can actually read without the

[46:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2808s) error. You can actually read without the check a missing signal reads as a zero

[46:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2811s) check a missing signal reads as a zero

[46:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2811s) check a missing signal reads as a zero and KV cache threshold that always sees

[46:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2814s) and KV cache threshold that always sees

[46:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2814s) and KV cache threshold that always sees zero admits everything. It's quietly

[46:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2817s) zero admits everything. It's quietly

[46:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2817s) zero admits everything. It's quietly turned into the no admit baseline while

[47:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2821s) turned into the no admit baseline while

[47:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2821s) turned into the no admit baseline while your results looks completely normal. So

[47:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2824s) your results looks completely normal. So

[47:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2824s) your results looks completely normal. So policy that cannot see its signal should

[47:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2827s) policy that cannot see its signal should

[47:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2827s) policy that cannot see its signal should fail loudly and not just fail silently.

[47:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2834s) Um so here

[47:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2837s) Um so here

[47:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2837s) Um so here what we did is every decision is

[47:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2840s) what we did is every decision is

[47:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2840s) what we did is every decision is recorded with the state it saw. Every

[47:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2842s) recorded with the state it saw. Every

[47:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2842s) recorded with the state it saw. Every run leaves behind a folder you can open

[47:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2845s) run leaves behind a folder you can open

[47:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2845s) run leaves behind a folder you can open inspect. There would be four files as

[47:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2847s) inspect. There would be four files as

[47:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2847s) inspect. There would be four files as you can see on the screen. Those are

[47:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2850s) you can see on the screen. Those are

[47:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2850s) you can see on the screen. Those are manifest JSON summary JSON decision JSON

[47:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2853s) manifest JSON summary JSON decision JSON

[47:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2853s) manifest JSON summary JSON decision JSON and uh out outcome JSON. So

[47:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2859s) and uh out outcome JSON. So

[47:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2859s) and uh out outcome JSON. So what each does

[47:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2861s) what each does

[47:41](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2861s) what each does is

[47:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2862s) is

[47:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2862s) is uh you can take a look deeply once you

[47:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2866s) uh you can take a look deeply once you

[47:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2866s) uh you can take a look deeply once you run the results. Now we we will be

[47:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2868s) run the results. Now we we will be

[47:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2868s) run the results. Now we we will be looking at the shading cut. So shade

[47:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2871s) looking at the shading cut. So shade

[47:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2871s) looking at the shading cut. So shade shading 37% cut tail latency. So here

[47:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2875s) shading 37% cut tail latency. So here

[47:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2875s) shading 37% cut tail latency. So here are the four results how they will look

[47:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2877s) are the four results how they will look

[47:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2877s) are the four results how they will look like on a real GPU. The setup small

[48:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2880s) like on a real GPU. The setup small

[48:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2880s) like on a real GPU. The setup small model we used quen 2.5 uh and it's a

[48:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2884s) model we used quen 2.5 uh and it's a

[48:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2884s) model we used quen 2.5 uh and it's a half billion

[48:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2886s) half billion

[48:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2886s) half billion model on a A10 GPU. The engine was kept

[48:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2890s) model on a A10 GPU. The engine was kept

[48:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2890s) model on a A10 GPU. The engine was kept out at four sequ sequences at a time.

[48:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2894s) out at four sequ sequences at a time.

[48:14](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2894s) out at four sequ sequences at a time. Eight request at 14 per second two

[48:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2897s) Eight request at 14 per second two

[48:17](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2897s) Eight request at 14 per second two repeats.

[48:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2900s) repeats.

[48:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2900s) repeats. If you look at the first row with no

[48:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2903s) If you look at the first row with no

[48:23](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2903s) If you look at the first row with no admission control, the P95 time to first

[48:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2907s) admission control, the P95 time to first

[48:27](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2907s) admission control, the P95 time to first token was about two two and two and a

[48:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2909s) token was about two two and two and a

[48:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2909s) token was about two two and two and a half around 2 and a half seconds and P

[48:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2912s) half around 2 and a half seconds and P

[48:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2912s) half around 2 and a half seconds and P 45 sorry P95 means 95% of the requests

[48:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2917s) 45 sorry P95 means 95% of the requests

[48:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2917s) 45 sorry P95 means 95% of the requests were faster than that and it swung by

[48:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2920s) were faster than that and it swung by

[48:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2920s) were faster than that and it swung by about 700 milliseconds either way.

[48:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2924s) about 700 milliseconds either way.

[48:44](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2924s) about 700 milliseconds either way. Now the last row if you see with a queue

[48:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2927s) Now the last row if you see with a queue

[48:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2927s) Now the last row if you see with a queue up policy that allows at most two

[48:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2930s) up policy that allows at most two

[48:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2930s) up policy that allows at most two request waiting 60 63% of requests were

[48:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2934s) request waiting 60 63% of requests were

[48:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2934s) request waiting 60 63% of requests were admitted and the P95 dropped to about

[48:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2938s) admitted and the P95 dropped to about

[48:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2938s) admitted and the P95 dropped to about around 1 second that's 2.5 times better

[49:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2943s) around 1 second that's 2.5 times better

[49:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2943s) around 1 second that's 2.5 times better with the exact the good same put

[49:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2947s) with the exact the good same put

[49:07](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2947s) with the exact the good same put now if we look at the signal matter more

[49:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2950s) now if we look at the signal matter more

[49:10](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2950s) now if we look at the signal matter more than the threshold

[49:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2952s) than the threshold

[49:12](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2952s) than the threshold but this This is the result I love most.

[49:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2955s) but this This is the result I love most.

[49:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2955s) but this This is the result I love most. Same run. KV cache usage peaked at half

[49:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2958s) Same run. KV cache usage peaked at half

[49:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2958s) Same run. KV cache usage peaked at half a percent. The queue got 24 requests

[49:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2961s) a percent. The queue got 24 requests

[49:21](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2961s) a percent. The queue got 24 requests deep on a small model. KV cache is huge

[49:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2965s) deep on a small model. KV cache is huge

[49:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2965s) deep on a small model. KV cache is huge compared to the four short sequences. So

[49:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2968s) compared to the four short sequences. So

[49:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2968s) compared to the four short sequences. So it never comes close to filling. All of

[49:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2971s) it never comes close to filling. All of

[49:31](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2971s) it never comes close to filling. All of the pressure showed up in the queue.

[49:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2973s) the pressure showed up in the queue.

[49:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2973s) the pressure showed up in the queue. Instead, a KV cache pressure policy

[49:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2975s) Instead, a KV cache pressure policy

[49:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2975s) Instead, a KV cache pressure policy would have botched a flat line and

[49:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2978s) would have botched a flat line and

[49:38](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2978s) would have botched a flat line and admitted everything. that that's the row

[49:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2982s) admitted everything. that that's the row

[49:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2982s) admitted everything. that that's the row of choice from two. So

[49:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2988s) now here and you you can try it today.

[49:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2990s) now here and you you can try it today.

[49:50](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2990s) now here and you you can try it today. You don't need a GPU. There is a small

[49:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2993s) You don't need a GPU. There is a small

[49:53](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2993s) You don't need a GPU. There is a small fake engine that speaks the three

[49:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2995s) fake engine that speaks the three

[49:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2995s) fake engine that speaks the three endpoints admit puff uses and it gets

[49:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2998s) endpoints admit puff uses and it gets

[49:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2998s) endpoints admit puff uses and it gets busy under load just like a real one. So

[50:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3002s) busy under load just like a real one. So

[50:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3002s) busy under load just like a real one. So you can test the whole setup on your

[50:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3003s) you can test the whole setup on your

[50:04](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3004s) you can test the whole setup on your laptop. If your right policy is your own

[50:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3006s) laptop. If your right policy is your own

[50:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3006s) laptop. If your right policy is your own package,

[50:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3008s) package,

[50:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3008s) package, admit perf discovers it automatically.

[50:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3011s) admit perf discovers it automatically.

[50:11](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3011s) admit perf discovers it automatically. No change to the repo and contribute

[50:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3013s) No change to the repo and contribute

[50:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3013s) No change to the repo and contribute contributions are very welcome. New

[50:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3016s) contributions are very welcome. New

[50:16](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3016s) contributions are very welcome. New policies, workload, engine adapter or

[50:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3019s) policies, workload, engine adapter or

[50:19](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3019s) policies, workload, engine adapter or matrices.

[50:25](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3025s) Okay.

[50:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3029s) So if you want to remember like one

[50:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3032s) So if you want to remember like one

[50:32](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3032s) So if you want to remember like one slide just make it this one that

[50:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3037s) slide just make it this one that

[50:37](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3037s) slide just make it this one that when you are accepting the request in

[50:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3040s) when you are accepting the request in

[50:40](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3040s) when you are accepting the request in your inference clusters

[50:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3042s) your inference clusters

[50:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3042s) your inference clusters you need to have a mechanism to say no

[50:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3045s) you need to have a mechanism to say no

[50:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3045s) you need to have a mechanism to say no early something that works at the

[50:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3047s) early something that works at the

[50:47](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3047s) early something that works at the ingress something before saying yes to

[50:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3051s) ingress something before saying yes to

[50:51](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3051s) ingress something before saying yes to the request before that request goes

[50:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3054s) the request before that request goes

[50:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3054s) the request before that request goes into the GPU before that request takes

[50:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3057s) into the GPU before that request takes

[50:57](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3057s) into the GPU before that request takes up your memory.

[51:00](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3060s) and per added clearly. There are a

[51:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3063s) and per added clearly. There are a

[51:03](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3063s) and per added clearly. There are a couple of error codes that your

[51:05](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3065s) couple of error codes that your

[51:06](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3066s) couple of error codes that your admission policies return. You need to

[51:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3068s) admission policies return. You need to

[51:08](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3068s) admission policies return. You need to handle those requests and those errors.

[51:15](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3075s) And with that we would like to come to

[51:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3078s) And with that we would like to come to

[51:18](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3078s) And with that we would like to come to the closure. We would like to thank you

[51:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3080s) the closure. We would like to thank you

[51:20](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3080s) the closure. We would like to thank you all for your patience, for your time.

[51:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3084s) all for your patience, for your time.

[51:24](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3084s) all for your patience, for your time. And here are like two of our projects.

[51:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3088s) And here are like two of our projects.

[51:28](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3088s) And here are like two of our projects. It all started as part of the LLM

[51:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3090s) It all started as part of the LLM

[51:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3090s) It all started as part of the LLM inference at scale handbook. We and when

[51:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3093s) inference at scale handbook. We and when

[51:33](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3093s) inference at scale handbook. We and when we were doing the survey, we found the

[51:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3095s) we were doing the survey, we found the

[51:35](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3095s) we were doing the survey, we found the gaps and we prepared admit puff. So here

[51:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3099s) gaps and we prepared admit puff. So here

[51:39](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3099s) gaps and we prepared admit puff. So here is the source code for it. Feel free to

[51:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3102s) is the source code for it. Feel free to

[51:42](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3102s) is the source code for it. Feel free to go to it and like experiment with it.

[51:45](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3105s) go to it and like experiment with it.

[51:46](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3106s) go to it and like experiment with it. Raise the issues, contribute to it. idea

[51:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3109s) Raise the issues, contribute to it. idea

[51:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3109s) Raise the issues, contribute to it. idea is to

[51:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3112s) is to

[51:52](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3112s) is to build your admission policies, benchmark

[51:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3114s) build your admission policies, benchmark

[51:54](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3114s) build your admission policies, benchmark them and be confident that it uh helps

[51:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3118s) them and be confident that it uh helps

[51:58](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3118s) them and be confident that it uh helps you keep your cluster stable. Yeah.

[52:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3121s) you keep your cluster stable. Yeah.

[52:01](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3121s) you keep your cluster stable. Yeah. Thank you everyone.

[52:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3122s) Thank you everyone.

[52:02](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=3122s) Thank you everyone. Thank you everyone.