# Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Harshul Jain, Audible · Tanmay Sah

[Talk page](https://ai.engineer/talks/y2W4FNAuPEA-deep-dive-on-llm-inference-at-scale) · [Video](https://www.youtube.com/watch?v=y2W4FNAuPEA)

## Deep Dive on LLM Inference at Scale

Harshul Jain and Tanmay Sah build inference capacity from attention and GPU memory, then explain how quantization, caching, batching, and serving engines change the costs of producing tokens.

### The model fits. The requests still might not.

[0:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=13s)

Loading **Mistral 7B** onto a GPU consumes roughly 15 GB before requests add their own memory requirements. Longer prompts then increase memory use and delay the first generated token. Send five requests to a plain sequential implementation, and they wait behind one another. Harshul Jain, a senior software engineer at Audible, and Tanmay Sah, an AI researcher and quantitative modeler, begin their workshop with these three symptoms because each points to a different part of inference. [8:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=498s)

Inference is recurring work: every user, session, and generated token consumes resources. The workshop frames the response as reducing token usage or making the service use its hardware more efficiently. Its beginner-to-intermediate progression follows the second route, first explaining the bottlenecks and then separating model optimization from serving optimization.

The accompanying project, LLM Inference at Scale, collects slides, a benchmark report, and Jupyter notebooks; its workshop README points to the exercises. The opening demonstration walks through previously run results—Jain does not trust conference Wi-Fi—and establishes three measurements:

- **Memory:** How much additional storage does a longer context consume?
- **Time to first token (TTFT):** How long does the user wait before output begins?
- **Throughput:** How many tokens or requests can the service produce per second?

### Every token adds keys and values across the model

[12:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=736s)

The inference pipeline converts text into tokens, converts tokens into embeddings, and passes those embeddings through transformer layers. The Mistral example has 32 layers. Each generated token extends the sequence used to generate the next one. Within those layers, attention works alongside normalization and feed-forward computation; attention is the part to inspect when explaining growing key-and-value storage. [12:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=752s)

Attention projects each token into query, key, and value vectors. Queries and keys determine attention scores against preceding tokens; values supply the information combined through those scores. Ten tokens require ten sets of vectors, and a thousand require a thousand. This happens across layers and KV heads, so storing the vectors costs much more memory than storing the prompt text.

How much does one token cost? The workshop uses 128 dimensions per head, 32 layers, eight KV heads, and a 16-bit representation:

\[
\text{KV bytes per token}
=2\times128\times32\times8\times2
=131{,}072.
\]

The first factor of two stores both K and V; the final factor uses two bytes per element. That is approximately 131 KB per token. Eight KV heads, rather than 32, already reflect a memory-saving architectural choice: grouped query attention, explained later. [14:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=884s)

Follow one request through the capacity change. At roughly 4,000 tokens, its KV storage occupies about half a GB. Increase the context to roughly 16,000 tokens, and the requirement becomes about 2.1 GB. Alternatively, keep the 4,000-token context and serve 80 users concurrently: aggregate KV storage reaches roughly 42 GB. A 24 GB GPU cannot hold that cache, even before loading weights and accounting for overhead. Context length and concurrency multiply the same per-token cost.

GPU memory therefore divides into relatively fixed model weights, working overhead, and the remaining space for KV vectors. For a fixed GPU, model, and representation, adding users leaves less memory for each user’s context. The memory chart illustrates this directly: a target of 160 concurrent users requires shorter contexts than a smaller batch can support.

### The initial wait and the gaps between tokens have different causes

[20:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1233s)

**Prefill** processes the prompt, constructs its key and value vectors, and performs attention computation across the input sequence. Longer prompts require more matrix-heavy work, explaining the increase in TTFT. **Decode** then generates output sequentially, computing attention for each new token against the preceding sequence. The gaps between output tokens introduce the workshop’s fourth metric: **inter-token latency**. [20:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1233s)

The GPU memory hierarchy explains why decode can be slow despite ample arithmetic capacity. In the workshop’s simplified picture, high-bandwidth memory, or HBM, holds large amounts of data. Much smaller shared memory supports faster local transfers. Computation moves chunks from HBM into shared memory, performs arithmetic, and writes results back. Generating tokens one after another repeatedly demands this movement.

The roofline model describes the distinction through **arithmetic intensity**, the number of floating-point operations performed per byte transferred. Prefill processes many input tokens together and does substantial arithmetic with the data it loads. Decode reads weights and preceding KV vectors to calculate one new token, doing less arithmetic per byte. Under this model, prefill tends toward a compute limit and decode toward a memory-bandwidth limit. [25:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1532s)

Where does the user wait, and what carries context between steps? The diagram separates prefill from repeated decode. It also looks ahead to the serving section’s KV cache: retaining vectors avoids rebuilding them, while decode still has to read the retained context.

The notebook results match that distinction. Prefill time grows visibly with input length. After excluding cold start, decode time stays nearer an average, but it is not constant: longer context means retrieving more preceding KV data. A comparatively flat decode curve does not remove the context-length cost.

Prompt processing and repeated generation
Prefill creates prompt state. The later serving optimization retains that state in a KV cache; each decode step reads preceding context and extends the sequence.
- Input tokens — Longer prompts supply more initial work.
- Prefill — Construct K/V and perform prompt attention.
- Retained KV state — Cache preceding vectors instead of rebuilding them.
- Decode step — Read preceding K/V and calculate the new token.
- Generated token — Extend the sequence before the next step.
- Input tokens → Prefill (Initial processing contributes to TTFT)
- Prefill → Retained KV state (Retain computed vectors)
- Retained KV state → Decode step (Read preceding context)
- Decode step → Generated token (Produce next token)
- Generated token → Decode step (Repeat; step timing sets inter-token latency)

### A batch can fit in memory and still miss its latency target

[28:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1711s)

A first estimate of concurrency divides memory available for KV storage by storage per user:

\[
\text{concurrent users}
\lesssim
\frac{\text{available KV memory}}
{\text{context tokens per user}\times\text{KV bytes per token}}.
\]

This estimates a memory ceiling. A service also has a latency objective: larger batches can increase TTFT and inter-token latency, preventing it from using every theoretically available slot. [28:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1711s)

The workshop presents a tradeoff triangle of quality, latency, and throughput. Context contributes to quality when it contains information the task needs; shortening it to fit more users can remove that information. Different applications choose different priorities:

- **Premium chat:** Preserve useful context and fast responses, accepting fewer concurrent users per GPU and a higher cost.
- **Asynchronous agents:** Preserve task quality and process many concurrent long-running tasks, accepting more latency where the workflow permits it.

The capacity calculator combines GPU VRAM, bandwidth, FLOPS, and hourly cost. Its procedure fixes the most important requirement first—latency for interactive chat or minimum batch size for asynchronous work—then preserves the necessary context and compares feasible hardware. The chat example selects a 10-millisecond latency target as an input; the calculator does not demonstrate that every candidate achieves it. [34:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2086s)

A GPU with a higher hourly price may nevertheless produce tokens more cheaply if its useful throughput rises enough. Cost per million tokens therefore matters more than rental price alone. That comparison must use achievable concurrency under the application’s context and latency requirements; throughput the service cannot use is a poor basis for its cost estimate.

### Compression changes the representation—and introduces assumptions

[37:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2276s)

Sah introduces two deliberately fake algorithms. The “ostrich algorithm” ignores an inconvenient problem long enough to proceed with an assumption. The “World Cup algorithm” breaks a large problem into smaller pieces and carries useful results forward. These jokes distinguish two recurring optimization moves: assuming compression preserves useful behavior, and arranging work into manageable parts. [38:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2306s)

Weight quantization attacks fixed model memory. Sah’s **GPT-OSS** illustration uses a 120-billion-parameter model, approximately 240 GB of BF16 weights, and one 80 GB H100. Reducing the representation to FP8 gives approximately 120 GB, still too large; MXFP4 is described as bringing it to around 65 GB. These approximate footprints illustrate the speaker’s capacity decision. Compression matters here because it crosses the threshold from several GPUs to one. [39:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2366s)

For Mistral 7B, the alternatives include `int8`, `int4`, and `NF4`. Preserving quality is the ostrich assumption that must eventually face a relevant benchmark. **Post-training quantization** compresses an already trained model; **quantization-aware training** incorporates quantization into training or fine-tuning. Fewer bits establish a storage benefit, while task quality still needs evaluation.

Attention variants change how much separate K/V information the architecture maintains. Sah illustrates the heads by splitting 4,096 columns into 32 blocks of 128 columns. The important comparison is the degree of sharing:

- **Multi-head attention (MHA):** Maintains separate heads, with work that can run in parallel.
- **Multi-query attention (MQA):** Shares a single KV representation across queries.
- **Grouped query attention (GQA):** Shares KV representations within groups of queries, retaining more separation than MQA while storing less than MHA.

**Multi-head latent attention (MLA)** compresses key and value information into a latent representation. Position complicates the design: rotary position embeddings, or RoPE, encode token position, so the compressed representation must also accommodate position-related key information. Sparse attention changes a different part of the calculation by selecting important preceding tokens instead of attending to the whole preceding sequence.

### FlashAttention reduces traffic; quantization frees room

[48:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2880s)

**FlashAttention** returns to the memory hierarchy. The expensive pattern repeatedly loads attention data, calculates intermediate results, and writes those results to HBM. Tiling divides the matrices into pieces that can be processed in fast on-chip memory. Online softmax carries summary information forward across tiles, avoiding repeated movement of full intermediate matrices through HBM. This changes how attention computation moves data. [48:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2880s)

The attention scorecard compares storage, throughput, and quality. Reducing 32 KV heads to eight reduces that storage component fourfold—the architectural choice already present in the Mistral memory calculation. GQA is presented as retaining quality close to MHA in many uses, while MQA’s more aggressive sharing can cost quality. Those judgments depend on the task.

The scorecard also briefly includes sliding-window attention, linear attention’s summarization approach, and **Mamba** state-space models. These alternatives change which preceding information is consulted or how it is retained. Sah also raises diffusion-based generation as a possible future departure from sequential token generation; the workshop’s bottleneck model is built around the autoregressive architecture it actually demonstrates.

The quantization notebook gives the earlier capacity example another lever. FP16 Mistral weights occupy about 15 GB; the `int8` result is about 7.5 GB, and the `int4` footprint is described as about 4.5 GB. Reducing the weight allocation leaves more of the same GPU for KV storage. That space can support longer contexts or additional concurrent requests, provided latency remains acceptable. [55:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3355s)

Memory savings do not establish a speedup. The notebook’s throughput plot uses theoretical values rather than throughput tests, and Jain notes that some benchmarks they studied showed lower throughput with `int8`. A smaller representation can improve capacity while its execution path still produces fewer tokens per second.

The attention demonstration ends with a consequential correction: Jain revises the earlier MLA savings figure from 56× to 14× relative to MHA because the calculation omitted the number of layers. The corrected figure applies to the workshop’s comparison; MLA’s storage formula varies with the architecture. Every layer’s retained state belongs in the memory calculation. [59:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3560s)

### Retain computed vectors, then manage their storage and scheduling

[1:00:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3614s)

Serving optimization starts with computation repeated inside one request. A simple implementation rebuilds preceding tokens’ keys and values even though earlier steps already produced them. A **KV cache** retains those vectors for later decode steps. This trades memory for saved computation, making the earlier growing-memory problem the cost of an important optimization. [1:00:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3636s)

Paged attention addresses allocation waste. Jain’s deliberately small example reserves 2 KB for a request that needs only 1 KB, wasting half its reservation. Requiring contiguous storage also limits where a growing request can fit. Operating-system paging supplies the alternative: keep a contiguous logical view while mapping it to physical memory blocks at different addresses, allocating blocks as tokens arrive.

How can ordered token state occupy separate memory regions? The diagram separates the request’s logical view from its physical placement. The sequence retains its order, while the allocator can use separate blocks and add storage when the request grows. This removes the requirement to find one large continuous region.

The remaining serving mechanisms address different kinds of waste:

- **Continuous batching:** A fixed batch waits for all its requests to finish before accepting replacements. Continuous batching admits new work as capacity opens, reducing unused slots while longer requests continue.
- **Prefix caching:** A KV cache saves repeated computation within one request; prefix caching reuses computation for the same leading tokens across different requests.
- **KV quantization:** Lower-precision cached keys and values reduce storage per token, leaving room for more context or concurrent requests.

These mechanisms work on allocation, timing, repeated prompt work, and representation size respectively. Jain recommends using **vLLM**, which packages the serving optimizations, rather than implementing each from scratch. The next benchmark tests what changes when those capabilities replace the plain baseline.

Logical sequence, separate physical blocks
Paging preserves an ordered view of KV state without requiring contiguous physical storage. New tokens trigger additional block allocation.
- Logical KV sequence — The request sees token state in order.
- Address mapping — Resolve logical storage to physical addresses.
- Physical memory blocks — Store KV data in separate regions.
- New tokens — Require additional storage as generation continues.
- Logical KV sequence → Address mapping (Preserve logical order)
- Address mapping → Physical memory blocks (Map to separate physical addresses)
- New tokens → Physical memory blocks (Allocate blocks when needed)

### More aggregate tokens can coexist with a longer initial wait

[1:06:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3979s)

The benchmark holds Mistral 7B and its set of input questions fixed on an H100, comparing a plain Hugging Face baseline with vLLM configurations. The baseline produces about 51 tokens per second. Default vLLM includes paged attention, continuous batching, and KV caching; the workshop reports nearly 15× higher throughput, alongside higher TTFT and lower inter-token latency. Aggregate output and an individual request’s initial wait are separate measurements. [1:07:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4039s)

Adding prefix caching increases throughput further and reduces TTFT while leaving inter-token latency approximately unchanged. That fits its purpose: reuse removes prompt-processing work. Adding KV quantization leaves throughput and latency roughly similar in this benchmark while reducing KV usage. Saved computation and saved storage produce different observable changes.

**Speculative decoding** attacks sequential output generation. A smaller model drafts perhaps four or five tokens; the larger target model becomes the referee that decides how many to accept. The loop repeats. Accepting several drafts advances the output farther per verification cycle, while poor agreement between draft and target reduces the progress obtained from the extra drafting work. [1:12:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4322s)

Sah is skeptical of ordinary speculative decoding based on his personal tests and identifies draft-target alignment as the problem. The workshop benchmark also reports approximately similar results after adding it. That limits the conclusion to those tests; Sah suggests domains with more predictable output patterns, such as coding, as possible candidates.

The variants are “different flavors of same kind of soda,” in Sah’s phrasing:

- **Self-speculative decoding:** An auxiliary head within the target model performs drafting work.
- **EAGLE:** A smaller predictor uses features from a main-model layer to predict features, changing the relationship between drafting and the target model. This is Sah’s preferred approach.
- **Medusa:** Parallel token predictions propose output beyond a single next token.

### Shared agent context changes the engine comparison

[1:14:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4469s)

Prefix caching becomes especially useful when agents repeatedly send the same instructions and shared context. Sah contrasts a simple whole-prompt hash lookup with a radix tree. A changed word defeats a whole-prompt hash match. A prefix tree represents common beginnings; a radix tree compresses paths without branches. **SGLang** uses this structure for prefix caching. Reuse follows shared token prefixes, rather than the semantic similarity of differently worded requests. [1:14:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4469s)

**TensorRT-LLM** introduces another engine choice. TensorRT is an SDK; TensorRT-LLM is an inference engine comparable in role to vLLM and SGLang. Its NVIDIA-specific optimization reaches into layers and hardware execution, making the target hardware part of the engine decision. [1:16:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4597s)

The workshop’s vLLM–SGLang comparison uses ShareGPT questions on an H100. The presenters report no statistical difference for that standard workload, with similar request rates, TTFT, and latency. They then change the workload to repeated agentic branching. [1:18:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4686s)

The agent example first requests a proposal for solving traffic congestion in a city. The model produces a proposal; a second turn asks it to review that proposal and rate it from one to ten. Repeating those turns retains common instructions and context while introducing changing output. Prefix caching can preserve work for the shared beginning and compute the material that diverges.

For this branching setup, Sah reports SGLang performing three to four times better. The multiplier is setup-dependent and is not tied to a single named metric in the spoken comparison. It provides a reason to benchmark repeated-prefix workflows separately from ordinary API traffic. Jain recommends vLLM as the default for standard serving and trying SGLang when agent workloads are not performing well enough.

The comparison closes with a Clarifai benchmark that also includes TensorRT-LLM. Its value is another set of workload and hardware conditions to compare, rather than a universal engine ranking. Hardware-focused optimization and repeated-context reuse address different opportunities; the application determines which opportunity matters.

### The next layer is engineering the cache itself

[1:22:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4936s)

The engine landscape extends into session routing and serving multiple models; NVIDIA Dynamo is mentioned in connection with agentic session routing. The deployment sequence remains practical: establish a baseline, choose a model that satisfies the task, compress its weights where appropriate, and use a serving engine for caching and scheduling. Mistral 7B is the workshop’s teaching model, rather than Jain’s recommended production choice.

KV cache engineering is the next layer of study: eviction strategies, cache compression, and hybrid memory. Retaining vectors saves computation, but creates new questions about which state to keep and how much memory to spend on it. These topics extend the workshop’s original capacity calculation into policies for managing scarce memory. [1:24:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5080s)

Distributed inference adds another set of problems beyond these single-GPU foundations. Jain treats it as material for a separate hands-on workshop. The useful closing question is whether a technique solves a problem the application actually has: begin with the bottleneck, then evaluate the proposed change through the same memory, latency, quality, and throughput requirements.

## Key takeaways

- KV storage multiplies across tokens, layers, KV heads, precision, and concurrent requests. The workshop’s roughly 4,000-token context across 80 users requires about 42 GB of KV storage alone.
- Prefill performs prompt-wide computation before output begins. Decode repeatedly moves weights and preceding context for each new token, making bandwidth and inter-token latency central.
- Memory capacity sets an upper bound on concurrency; latency objectives determine how much of that capacity the application can use.
- Quantization frees storage, but task quality and execution speed need separate evaluation. The workshop corrects its MLA comparison to 14× savings after restoring the missing layer multiplier.
- Paging reduces allocation waste, continuous batching fills available slots, prefix caching saves shared prompt work, and KV quantization reduces retained-state size.
- Standard traffic and repeated agent prefixes can produce different engine comparisons. Benchmark the context reuse your application creates, then study eviction, compression, and distributed serving as its bottlenecks demand.

## Resources

- [The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents](https://arxiv.org/abs/2603.19328) — Sah and coauthors study how runtime verification affects multi-step agent completion and compute cost. Useful for connecting the workshop’s asynchronous-agent serving discussion to the additional work introduced by verification and recovery.

## Related talks

- [Introduction to LLM serving with SGLang](https://ai.engineer/talks/Ahtaha9fEM0-introduction-llm-serving-sglang) — Continue exploring the serving engine discussed for shared prefixes and agentic branching.
- [From model weights to API endpoint with TensorRT-LLM](https://ai.engineer/talks/Lko9lTGD_9U-from-model-weights-api-endpoint-tensorrt-llm) — Follow the NVIDIA-focused engine choice toward an API-serving deployment.
- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://ai.engineer/talks/YXowceUKYJI-kv-cache-aware-routing-p-d-disaggregation) — Extend the prefill, decode, and cache foundations into routing and distributed serving.

## Slides

### LLM Inference at Scale — With First Principles

The two-hour workshop moves from the memory equation to production engines, with live demos reproducible from the repository.

[0:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=19s) [Workshop title slide introducing LLM inference from the memory equation to production engines.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c46a8db5dc64a90846efe30d9330a8adbfe98acf5bc4c8831f3e277884f7b25.webp)

LLM Inference at Scale -
With first principles

AIE WORKSHOP 2026 · 2 HOURS · LIVE DEMOS

From the memory equation to production engines

Every number computed live. Every demo reproducible from the repo.

scan → repo
Github.com/harshuljain13/
llm-inference-at-scale

Interface text:
PowerPoint  Edit  View  Window  Help
97%  Mon Jun 29 12:13 PM

### Meet Your Speakers

Harshul Jain builds ML/AI platforms and writes an LLM inference handbook; Tanmay Sah researches agent verifiers and world models.

[0:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=41s) [Speaker profiles for Harshul Jain and Tanmay Sah, with portraits and brief biographies.](https://assets.ai.engineer/sites/aiecode2025/sha256/c15bd6e4d37ca462ab1afb45eb9214b27815291e3d58e3b10cd56c47ee424b43.webp)

YOUR SPEAKERS
Meet your Speakers

Harshul Jain
Sr. Software Engineer, Audible Inc
Building ML/AI platforms, LLM Inference Handbook

Tanmay Sah, PHD
Senior Quantitative Modeler
AI researcher building agent verifiers and world models.

Github.com/harshuljain13/llm-inference-at-scale · every demo runs from this repo

Interface text:
PowerPoint  Edit  View  Window  Help
97%
Mon Jun 29 12:13 PM

### How Deep Are You in LLM Inference?

The speakers poll the room, from newcomers to people deploying and tuning inference engines in production. The workshop targets beginner and intermediate levels.

[1:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=79s) [Audience poll with four experience levels: curious, API user, hands-on, and in production.](https://assets.ai.engineer/sites/aiecode2025/sha256/e26d9dbdb1622e02fc8b885dc1f66ddd229480d99b0627349de9ab001a59ba08.webp)

How deep are you in LLM inference?

AUDIENCE POLL · SHOW OF HANDS

01
Curious
New to it — heard the term, never deployed a model.

02
API user
Call LLM APIs — curious what runs under the hood.

03
Hands-on
Run models locally — batch, measure latency, tinker.

04
In production
Own the GPU bill — deploy and tune inference engines.

Wherever you land — today is built to move you one level up.

Interface text:
PowerPoint  Edit  View  Window  Help
97%  Mon Jun 29 12:14 PM

### Agenda — The Arc

The workshop moves from inference pain points and their foundations through model and serving optimizations to serving engines. Multimodal and distributed inference are left in the repo to explore.

[1:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=113s) [Agenda moving from the problem statement and foundations through model and serving optimizations to serving engines.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf1398b74c2245cf2ac595ba6dd95a6b1311498a8e583026af2d166265400b18.webp)

WHAT WE COVER
Agenda - The arc

00
Problem statement
Pain Points

01
The Foundations
Metrics and internals

02
Model Optimizations
Attention mechanisms + Quantization

03
Serving optimizations
Optimizations on serving

04
Serving Engines
SGLang · Tensor RT
Nvidia Dynamo

Multimodal · Distributed Inference — left in the repo to explore.

Interface text:
PowerPoint  Edit  View  Window  Help
97%
Mon Jun 29 12:14 PM

### Problem Statement

The opening section introduces the pain points of inference, starting with what LLM inference means.

[2:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=158s) [Section 00 divider titled Problem Statement, subtitled Pain Points of Inference.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9afaa5deace7872f1209f30fb058137f03a13945f0f50ea5301921042e81acf.webp)

00
Problem Statement
Pain Points of Inference

10 min · Demo A

Interface text:
PowerPoint  Edit  View  Window  Help
97%
Mon Jun 29 12:15 PM

### Every Interaction with AI Is an Inference Call

Messages, autocomplete and tool calls all run inference. The slide estimates a $23B inference-as-a-service market in 2026 and $72K per year for a 25-person team on Opus.

[2:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=163s) [Slide presents claims about AI coding agents, AI products, the inference market, and team costs.](https://assets.ai.engineer/sites/aiecode2025/sha256/f33311eced2e9e6d1063d99fb68e9be6b30bbeaf83e3711d648d532f1ef414f2.webp)

WHAT IS LLM INFERENCE?
Every interaction with AI is an inference call

52% — AI coding agents
of new code is AI-authored · 1 in 7 PRs involve an agent · 84–91% of devs use AI tools

∞ — AI products
ChatGPT · Claude · Gemini · Copilot — every message, autocomplete, and tool call is inference

$23B — The market
inference-as-a-service in 2026 (from $18.6B in ’25) · cost / 1M tokens fell 1,200× — volume grew faster

$72K — The real cost
per year for a 25-person team on Opus · one 50-turn coding session ≈ 1M tokens

Raise PRs, chat with AI, or ship agent features — you're doing LLM inference, and it's expensive at scale.

Interface text:
PowerPoint  Edit  View  Window  Help
97%  Mon Jun 29 12:15 PM

### Why LLM inference matters

SemiAnalysis puts the profit drain from replacing Google search queries with LLMs at $36B. Business Insider describes companies budgeting tokens as compute becomes scarce.

[3:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=193s) [Three article summaries describe search inference costs and corporate token budgets.](https://assets.ai.engineer/sites/aiecode2025/sha256/13c2eaf02f51de9f0a901e5cf0ddff8d95ff432437b864d5d40105823e0e6633.webp)

WHY LLM INFERENCE MATTERS

From $100B search shockwaves to the corporate ‘Token Hunger Games’

SEMIANALYSIS
“The inference cost of search disruption”
› Replacing Google search queries with LLMs is a $36B profit drain.
› At scale, inference cost far exceeds training cost.
› Query cost must drop below 0.5¢ to keep search profitable.

BUSINESS INSIDER
“It’s time to put AI on a diet”
› The ‘all-you-can-eat’ era is over — companies count token calories.
› Massive sticker shock from coding agents
› C-suites force teams to audit and budget token spend.

BUSINESS INSIDER
“AI Token Hunger Games are coming”
› Engineers compete for compute budgets as caps land.
› Finite compute gives way to strict token allocations.
› Token limits are the new office power struggle.

The cost of a token is now a board-level number — and it starts with inference.

Interface text:
PowerPoint  Edit  View  Window  Help
97%  Mon Jun 29 12:15 PM

### Inference now costs more than training

The slide compares GPT-3’s $4.6M one-time training cost with more than $4.6M per week for inference, which scales with users, tokens and sessions. The speaker calls this an old OpenAI statistic.

[4:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=244s) [Bars compare a stated one-time GPT-3 training cost with recurring weekly inference cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/28d50fecadc4400ab7a090976751941883e0b2d8631189855ab505798999b91d.webp)

THE COST PROBLEM · OPEN AI
Inference now costs more than training

TRAINING · GPT-3 · one-time
$4.6M

INFERENCE · serving · every second, growing
> $4.6M / week
▸ and rising

Training is a one-time capital cost. Inference is an operating cost that scales with every user, every token, every session.

Interface text:
PowerPoint  Edit  View  Window  Help
97%  Mon Jun 29 12:16 PM

### The field moves quickly

From vLLM’s PagedAttention in 2023 to Nvidia Dynamo and M* in 2026, the timeline shows how quickly inference changes. The course builds foundations for evaluating what ships next.

[4:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=277s) [Timeline highlights inference developments from 2023 through 2026.](https://assets.ai.engineer/sites/aiecode2025/sha256/e26c89a8668f5b74fb58e7f2b3b75d9f1119730b6608fec182e28c5adf474540.webp)

WHY NOW
The field moves quickly

2023
vLLM
PagedAttention ships

2024
SGLang
Radix Attention

2025
DeepSeek-V3
MLA goes mainstream

2026
Nvidia Dynamo · M*
Distributed KV, new attention

You will not memorize these. You will learn the foundations that let you evaluate whatever ships next.

Interface text:
PowerPoint  Edit  View  Window  Help
98%  Mon Jun 29 12:17 PM

### Feel the pain

A five-minute GPU demo loads Mistral-7B to expose growing memory use, slow time to first token that scales linearly with input text, and a one-user-at-a-time limitation.

[5:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=319s) [A GPU demo introduction lists memory growth, time to first token, and single-user limitations.](https://assets.ai.engineer/sites/aiecode2025/sha256/47e9c0972a7ee8452b8f0fd125ea80946bd19fb807245cbe326d1ffbbe9796cb.webp)

Feel the pain

LIVE DEMO A · 5 MINS · GPU

demos/demo_a_feel_the_pain.ipynb

Load Mistral-7B on a real GPU. Measure what the API never shows you.

› Memory compounds with static weights + new context + users
› Slow TTFT and scales linearly to input text
› One user at a time limitation

Everything that follows explains these three numbers.

Interface text:
PowerPoint  Edit  View  Window  Help
98%
Mon Jun 29 12:17 PM

### Scan and pull up the repo

The repo collects LLM inference resources in one place, with 8 runnable demo notebooks and 55+ modules across 12 chapters. Scan the QR code or open it on GitHub to follow along.

[5:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=328s) [Workshop repository slide shows a QR code and usage statistics.](https://assets.ai.engineer/sites/aiecode2025/sha256/72d1eb411965da3b8fdc25fb2d008a5454f2349d6799d18758ff4f1295b221bf.webp)

FOLLOW ALONG
Scan and pull up the repo

Github.com/harshuljain13/llm-inference-at-scale

8 runnable demo notebooks · 55+ modules across 12 chapters · every slide number reproduced live.

300+
clones

500+
unique visitors

2500+
views

90+ stars
stars increasing

Powered By Molab - https://molab.marimo.io/notebooks

Interface text:
PowerPoint  Edit  View  Window  Help
Mon Jun 29 12:18 PM

### Workshop README

The workshop README links to the slides and provides clone instructions and Molab links for the demo notebooks.

[6:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=384s) [Workshop README with quick-start instructions and demo notebook links.](https://assets.ai.engineer/sites/aiecode2025/sha256/a34e13664c1269163ee489228b25751f7550082bcd27c77b9c4810ece695fdf8.webp)

LLM Inference at Scale: Workshop

AI Engineering World's Fair 2026 | June 29 | 2 hours

By Harshul Jain | Slides

Quick Start (Do This Before the Workshop)

1. Clone the repo

git clone https://github.com/harshuljain13/llm-inference-at-scale.git
cd llm-inference-at-scale/workshop/demos

Demo Notebooks with Workshop links

Demo | What it proves | Run on
demo_a_feel_the_pain.ipynb | Memory, TTFT, queue, KV growth | https://molab.marimo.io/notebooks/[illegible]
demo_b_memory_equation.ipynb | Architecture, weights, KV derivation | https://molab.marimo.io/notebooks/[illegible]
[partially visible] Prefill fast,

Interface text:
github.com/harshuljain13/llm-inference-at-scale/tree/master/workshop
master | llm-inference-at-scale / workshop / | Top
README.md | fix the molab url in readme | yesterday
README.md
Apps | AIE World Fair 2026 | talks | transformer videos | voice-agents-videos | papers | Feast | Research-Exp | Recommender systems | CFPs | LLM-Inference | Learning | vizuara-voiceAI | MLFeatureStores | job | Distributed Training | Blogs | Agents | All Bookmarks
New Chrome available | Ask Gemini

### Workshop materials

The repository README links to a folder with the workshop slides and a benchmark report, which attendees can download.

[6:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=388s) [A Drive folder containing a presentation file and a PDF.](https://assets.ai.engineer/sites/aiecode2025/sha256/07ae9760974dd112c71cf92ed7264f9e34a1c58cb738561b5cb4b233b628fead.webp)

Name
LLM_Inference_at_S...
vllm_sglang_bench...
PDF

Interface text:
Drive
Search in Drive
New
Home
My Drive
Shared with me
Recent
Starred
Spam
Trash
Storage
drive.google.com/drive/u/2/folders/[illegible]
AIE World Fair 2026
Ask Gemini
New Chrome available
Apps
 talks
transformer videos
voice-agents-videos
papers
Feast
Research-Exp
Recommender systems
CFPs
LLM-Inference
Learning
vizuara-voiceAI
MLFeatureStores
job
Distributed Training
Blogs
Agents
All Bookmarks

### Workshop demo notebooks

The workshop maps each notebook to a topic and runtime, from KV growth to quantization. The presenters preconfigured the notebooks and assets for experimentation, starting with demo A.

[6:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=401s) [A table of demo notebooks, their topics, and links.](https://assets.ai.engineer/sites/aiecode2025/sha256/027f35494da9df4194f1f2901a7d06daf834c35b03865d13560b842742260ac4.webp)

git clone https://github.com/harshuljain13/llm-inference-at-scale.git
cd llm-inference-at-scale/workshop/demos

Demo Notebooks with Workshop links

Demo | What it proves | Run on
demo_a_feel_the_pain.ipynb | Memory, TTFT, queue, KV growth | https://molab.marimo.io/notebooks/[illegible]
demo_b_memory_equation.ipynb | Architecture, weights, KV derivation | https://molab.marimo.io/notebooks/[illegible]
demo_c_prefill_vs_decode.ipynb | Prefill fast, decode slow, roofline | https://molab.marimo.io/notebooks/[illegible]
demo_d_capacity_calculator.ipynb | Max users, GPU selection (widgets) | https://colab.research.google.com/github/harshuljain13/llm-inference-at-scale/blob/master/workshop/demos/demo_d_capacity_calculator.ipynb
demo_e_attention_comparison.ipynb | GQA/MLA comparison | https://molab.marimo.io/notebooks/[illegible]
demo_e_quantization_batching.ipynb | INT4/INT8 quantization | https://molab.marimo.io/notebooks/[illegible]
demo_f_vllm.ipynb | vLLM: PagedAttention, prefix, KV quant, speculative | Lightning.ai (offline)

Interface text:
github.com/harshuljain13/llm-inference-at-scale/tree/master/workshop
master
llm-inference-at-scale / workshop /
Top
AIE World Fair 2026
Ask Gemini
New Chrome available
Apps · talks · transformer videos · voice-agents-videos · papers · Feast · Research-Exp · Recommender systems · CFPs · LLM-Inference · Learning · vizuara-voiceAI · MLFeatureStores · job · Distributed Training · Blogs · Agents · All Bookmarks

### Molab notebook platform

Molab’s homepage offers cloud-hosted Python and SQL notebooks, with tutorials and searchable examples. The presenter opens it ahead of the workshop’s first demo.

[7:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=440s) [Molab homepage with tutorials and example notebooks.](https://assets.ai.engineer/sites/aiecode2025/sha256/852cd57488ce7295a7793ca678b34986030076536b2a9419622fcb2f34a6b7eb.webp)

Welcome to molab 🍃

molab lets you create, run, and share cloud-hosted marimo notebooks — rapidly experiment with Python and SQL in marimo, the next-generation programming environment designed for data and stored as Python.

+ New notebook

Tutorials
Welcome to marimo. A quick tour of cells, reactivity, and the editor.
Plotting. Render interactive charts with matplotlib, altair, and plotly.
Interactive UI. Build sliders, dropdowns, and forms that react to user input.
Markdown. Write rich text, math, and embed Python values inline.
SQL. Query DataFrames and databases with native SQL cells.

Examples
Search notebooks...
All · algorithm · analysis · cad · community · dashboard · finance · geo · jam · llm · math · matplotlib · ml · research · sql · stem · widget

Example preview: Investment value over time
Example preview labels: PC10, PC9, PC8, PC7, PC6
[Other example preview text illegible]

Interface text:
marimo
Sign In
molab.marimo.io/notebooks
Ask Gemini
New Chrome available
Apps · AIE World Fair 2026 · talks · transformer videos · voice-agents-videos · papers · Feast · Research-Exp · Recommender systems · CFPs · LLM-Inference · Learning · vizuara-voiceAI · MLFeatureStores · job · Distributed Training · Blogs · Agents · All Bookmarks

### Model memory demo

The notebook sets up Mistral-7B in FP16 on a 102.0 GB Blackwell GPU, then prepares to measure its memory use against the ~14.5 GB theoretical estimate.

[7:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=456s) [A notebook showing model configuration, GPU information, and a memory-consumption exercise.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb53bc866e41e6dc18592077a14442e4f74c2c67c88e924639025731089f1ecf.webp)

Problem 1: Memory Consumption

A 7B parameter model in FP16 should theoretically need ~14.5 GB. Let's load it and measure the actual GPU memory consumed.

import torch
import time
import matplotlib.pyplot as plt
from matplotlib.patches import Patch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Config
MODEL_ID = 'mistralai/Mistral-7B-v0.1'  # non-gated, no auth needed
DEVICE = 'cuda'
DTYPE = torch.float16

# GPU info
gpu_name = torch.cuda.get_device_name(0)
gpu_mem_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f'GPU: {gpu_name} ({gpu_mem_gb:.1f} GB)')
print(f'Precision: FP16')
print(f'Model: {MODEL_ID}')

WARNING: The directory '/home/marimo/.cache/pip' or its parent directory is not owned or is not writabl[illegible]
[notice] A new release of pip is available: 25.3 -> 26.1.2
[notice] To update, run: python -m pip install --upgrade pip
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (102.0 GB)
Precision: FP16
Model: mistralai/Mistral-7B-v0.1

# Measure GPU memory before and after loading
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
mem_before = torch.cuda.memory_allocated() / 1e9

print('Loading model... (30-60 seconds)')
model = AutoModelForCausalLM.from_pretrained(

Interface text:
marimo
Examples / demo_a_feel_the_pain.ipynb
Preview
Share
Fork
Sign In
Read-only preview. Run it now without saving, or fork to edit in your workspace.
molab.marimo.io/notebooks/[illegible]
made with marimo

### Model memory consumption

The notebook sets up a before-and-after GPU memory measurement for a 7B FP16 Mistral model, estimated at ~14.5 GB. The speaker walks through previously run results on a 102 GB Blackwell GPU.

[7:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=475s) [The memory-consumption exercise with code to load a model and measure GPU memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/45b089967267e19c3ee1bca9249727d088b3a67062f87c02809a989561e427c2.webp)

Problem 1: Memory Consumption

A 7B parameter model in FP16 should theoretically need ~14.5 GB. Let's load it and measure the actual GPU memory consumed.

# GPU info
gpu_name = torch.cuda.get_device_name(0)
gpu_mem_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f'GPU: {gpu_name} ({gpu_mem_gb:.1f} GB)')
print(f'Precision: FP16')
print(f'Model: {MODEL_ID}')

WARNING: The directory '/home/marimo/.cache/pip' or its parent directory is not owned or is not writabl[illegible]
[notice] A new release of pip is available: 25.3 -> 26.1.2
[notice] To update, run: python -m pip install --upgrade pip
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (102.0 GB)
Precision: FP16
Model: mistralai/Mistral-7B-v0.1

# Measure GPU memory before and after loading
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
mem_before = torch.cuda.memory_allocated() / 1e9

print('Loading model... (30-60 seconds)')
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=DTYPE, device_map='auto', token=False)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=False)

torch.cuda.synchronize()
model_mem_gb = torch.cuda.memory_allocated() / 1e9 - mem_before

print(f'\n--- MODEL LOADED ---')
print(f'GPU memory used by model: {model_mem_gb:.2f} GB')
print(f'This is JUST the weights. No users served yet.')

Interface text:
marimo
Examples / demo_a_feel_the_pain.ipynb
Preview · Share · Fork · Sign In
Read-only preview. Run it now without saving, or fork to edit in your workspace.
made with marimo

[8:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=490s) [Notebook introduces GPU memory measurement for a 7B FP16 model and shows model-loading code.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9e7ab65b981ae4ed85e8b118467e6de10561fe2b6e7640187d3958219611808.webp)

Problem 1: Memory Consumption

A 7B parameter model in FP16 should theoretically need ~14.5 GB. Let's load it and measure the actual GPU memory consumed.

# GPU info
gpu_name = torch.cuda.get_device_name(0)
gpu_mem_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
print(f'GPU: {gpu_name} ({gpu_mem_gb:.1f} GB)')
print(f'Precision: FP16')
print(f'Model: {MODEL_ID}')

WARNING: The directory '/home/marimo/.cache/pip' or its parent directory is not owned or is not writabl[illegible]
[notice] A new release of pip is available: 25.3 -> 26.1.2
[notice] To update, run: python -m pip install --upgrade pip
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (102.0 GB)
Precision: FP16
Model: mistralai/Mistral-7B-v0.1

# Measure GPU memory before and after loading
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
mem_before = torch.cuda.memory_allocated() / 1e9

print('Loading model... (30-60 seconds)')
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=DTYPE, device_map='auto', token=False)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=False)

torch.cuda.synchronize()
model_mem_gb = torch.cuda.memory_allocated() / 1e9 - mem_before

print(f'\n--- MODEL LOADED ---')
print(f'GPU memory used by model: {model_mem_gb:.2f} GB')
print(f'This is JUST the weights. No users served yet.')
print(f'On a {gpu_mem_gb:.0f} GB GPU, you have {gpu_mem_gb - model_mem_gb:.1f} GB left[illegible]

Interface text:
marimo
Examples / demo_a_feel_the_pain.ipynb
Preview · Share · Fork · Sign In
Read-only preview. Run it now without saving, or fork to edit in your workspace.
made with marimo

### Capacity Calculator Summary

The calculator divides available GPU memory by KV cache memory per user to estimate capacity. Model precision, context length, the SLO and GPU choice also shape throughput and cost.

[53:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3205s) [Notebook summary shows the capacity equation and factors affecting inference cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/bfb4ad4ac9c1c01acb53642bbccf943079b5c751247333ca2ac5437b90ef4efa.webp)

GPU | $/hr | ITL | tok/s | $/M tok | VRAM | Status
MI300X (192GB) | $10.00 | 5ms | 1321 | $2.10 | 28/192GB | ✓
H100 80GB | $8.10 | 8ms | 835 | $2.70 | 28/80GB | ✓
H200 (141GB) | $12.50 | 6ms | 1196 | $2.90 | 28/141GB | ✓
[illegible] | $1.01 | 47ms | 150 | — | 28/24GB | won't fit
[illegible] | $0.81 | 94ms | 75 | — | 28/24GB | won't fit
[illegible] | $3.67 | 18ms | 387 | — | 28/40GB | ITL 18ms > SLO
A100 80GB | $5.12 | 14ms | 508 | — | 28/80GB | ITL 14ms > SLO
L40S (48GB) | $2.40 | 33ms | 215 | — | 28/48GB | ITL 33ms > SLO

Summary

The capacity equation:
Max users = (GPU_VRAM - model_weights - overhead) / (KV_per_token x context_length)

What determines your cost:
• Model precision (FP16 vs INT4) sets weight memory
• Context length sets KV per user
• SLO caps batch size, which caps throughput
• GPU choice (VRAM + bandwidth) determines $/M tokens

Next: How to push these limits with optimizations (GQA, prefix caching, engines).

Interface text:
demo_d_capacity_calculator.ipynb
Save in GitHub to keep changes
File  Edit  View  Insert  Runtime  Tools  Help
Commands  + Code  + Text  Run all  Copy to Drive
To exit full screen, press and hold esc
Double-click (or enter) to edit
Variables  Terminal  12:47 PM  Python 3

### Weight Quantization Experiment

The notebook sets up a comparison of Mistral-7B at three precisions, measuring GPU memory, inference throughput and sample perplexity.

[53:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3236s) [Notebook introduces measurements of GPU memory, throughput and quality at three model precisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/46dc2934b8636b9770e3bdbcf70df112b5b5a5abed390c45c1898cc89faecf9b.webp)

and paste into the cells as pre-computed results.

# Install dependencies
import subprocess, sys
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '-q', '--no-deps',
    'openai', 'matplotlib>=3.9', 'narwhals', 'scikit-learn>=1.5', 'bitsandbytes',
    'accelerate'])

WARNING: The directory '/home/marimo/.cache/pip' or its parent directory is not owned or is not writ[illegible]

Experiment 1: Weight Quantization

We load the same model (Mistral-7B) at three precisions and measure:

• GPU memory consumed
• Inference throughput (tokens/sec)
• Quality (perplexity on a sample, to show it barely changes)

Theory: [illegible]

Interface text:
marimo
Your notebooks
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
4.02s
molab.marimo.io/notebooks/[illegible]
New Chrome available
Ask Gemini

### FP16 Baseline Setup

The notebook sets up Mistral-7B with 64 output tokens for throughput measurement. Its FP16 baseline estimates ~14.5 GB for 7.24B parameters at two bytes each.

[54:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3241s) [Notebook shows model parameters and describes the FP16 baseline memory estimate.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ef2b10c4825a25f3cf7b294c668a8289d6ea0b73b41694cbd361903e94135a2.webp)

import torch
import time
import gc
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# === PARAMETERS ===
MODEL_ID = 'mistralai/Mistral-7B-v0.1'  # Non-gated model
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
N_OUTPUT_TOKENS = 64  # Tokens to generate for throughput measurement
PROMPT = 'Explain the concept of gradient descent in machine learning:'

# Store results for each precision
quant_results = {}  # {precision: {memory_gb, throughput_tok_s, load_time_s}}

print(f'Device: {DEVICE}')
if DEVICE == 'cuda':
    print(f'GPU: {torch.cuda.get_device_name(0)}')
    print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')

Load 1: FP16 (Baseline)

Standard half-precision. 7.24B params x 2 bytes = ~14.5 GB.

Interface text:
marimo
Your notebook
[illegible] CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
6.17s
AIE World Fair 2026
talks
transformer videos
voice-agents-videos
papers
Feast
Research-Exp
Recommender systems
CFPs
LLM-Inference
Learning
vizuara-voiceAI
MLFeatureStores
job
Distributed Training
Blogs
Agents
All Bookmarks
Ask Gemini
New Chrome available

### FP16 Baseline Measurement

The notebook loads Mistral-7B in FP16, measures memory and throughput, then frees the model. Its baseline estimate is about 14.5 GB for 7.24B parameters.

[54:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3251s) [Notebook displays GPU and memory information above code for measuring model memory and throughput.](https://assets.ai.engineer/sites/aiecode2025/sha256/63c59d6f313f110ba29a39fa7109dc83b155bbfb58ddf08f045793efb7ece7bb.webp)

print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')

Device: cuda
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
VRAM: 102.0 GB

Load 1: FP16 (Baseline)

Standard half-precision. 7.24B params x 2 bytes = ~14.5 GB.

def measure_model(model, tokenizer, label):
    """Measure memory usage and throughput for a loaded model."""
    # Memory measurement
    if DEVICE == 'cuda':
        torch.cuda.synchronize()
        mem_gb = torch.cuda.memory_allocated() / 1e9
    else:
        mem_gb = 0.0

    # Tokenize prompt
    [illegible]

Interface text:
marimo
Your notebooks
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace

[54:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3276s) [Notebook code loads Mistral-7B in FP16, measures performance, and frees memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/9be27bbc7e7fa7d097a22f624d779a671e86516ed8f178016d38166125b32bb5.webp)

[partially cropped] print(f'  Throughput: {throughput:.1f} tok/s')
return {'memory_gb': mem_gb, 'throughput_tok_s': throughput}

# --- FP16 ---
print('Loading Mistral-7B in FP16...')
t0_load_fp16 = time.perf_counter()
model_fp16 = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float16, device_map='auto', token=False
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, token=False)
load_time_fp16 = time.perf_counter() - t0_load_fp16

print(f'  Load time: {load_time_fp16:.1f}s')
quant_results['FP16'] = measure_model(model_fp16, tokenizer, 'FP16')
quant_results['FP16']['load_time_s'] = load_time_fp16

# Free memory
del model_fp16
gc.collect()
if DEVICE == 'cuda':
    torch.cuda.empty_cache()

Interface text:
marimo
Your notebooks
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace

### FP16 results and INT8 setup

The FP16 run uses 14.48 GB and generates 69.2 tokens/s. The notebook then sets up INT8 quantization, expecting roughly 7.2 GB by storing each parameter in one byte.

[55:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3311s) [Notebook reports FP16 load time, memory and throughput above an introduction to INT8 quantization.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5ea61ca71d7164f82b19c61df08a520fd3ce5e30573fa316157e6491b105cd9.webp)

Loading weights: 100% | 291/291 [00:07<00:00, 38.55it/s]
Load time: 30.1s
[transformers] Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
[transformers] Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Memory: 14.48 GB
Throughput: 69.2 tok/s

Load 2: INT8 (2x compression)

8-bit quantization via bitsandbytes. Each parameter stored in 1 byte instead of 2. Expected: ~7.2 GB memory, minimal quality loss.

# --- INT8 ---
print('Loading Mistral-7B in INT8...')
quant_config_8bit = BitsAndBytesConfig(load_in_8bit=True)  # 8-bit quantization config

t0_load_int8 = time.perf_counter()
model_int8 = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, quantization_config=quant_config_8bit, device_map='auto', token=False
)

Interface text:
marimo
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
Ask Gemini
New Chrome available

### Quantization comparison

INT4 uses less weight memory than FP16 and INT8, while INT8 has the lowest displayed throughput. The speaker describes the figures as theoretical and says no throughput test is being done here.

[57:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3427s) [Table and bar charts compare memory, throughput, and estimated maximum users for FP16, INT8, and INT4.](https://assets.ai.engineer/sites/aiecode2025/sha256/f8860ecebb8ae5d2f9f30968e1d67b71150947a24e983aac8e22d49f60886961.webp)

print(f'INT4 uses {memories[0]/memories[2]:.1f}x less memory than FP16.')
print(f'On an A100-80, that means {int((80-memories[2]-8)/0.268)} users instead of {int((80-memories[0]-8)/0.268)}.')

Precision  Memory (GB)  Throughput  Max Users (80GB)
FP16       14.48        69.2 tok/s   214
INT8        7.54        13.7 tok/s   240
INT4        4.52        44.4 tok/s   251

Weight Memory (lower = more users)
Y-axis: GPU Memory (GB)
Y-axis ticks: 0, 2, 4, 6, 8, 10, 12, 14
FP16: 14.5
INT8: 7.5
INT4: 4.5

Throughput (higher = faster)
Y-axis: Tokens/second
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70
FP16: 69
INT8: 14
INT4: 44

--- KEY INSIGHT ---

Interface text:
marimo
Your notebook
32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
Ask Gemini
New Chrome available
Apps · AIE World Fair 2026 · talks · transformer videos · voice-agents-videos · papers · Feast · Research-Exp · Recommender systems · CFPs · LLM-Inference · Learning · vizuara-voiceAI · MLFeatureStores · job · Distributed Training · Blogs · Agents · All Bookmarks

### Optimization summary

The notebook estimates that INT4 supports 251 users versus 214 with FP16 on an A100-80. It says the three optimizations can be stacked for a 10–20x improvement.

[57:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3450s) [Summary shows a quantization row and notes that optimizations can be combined.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a28f36945ab8e1b9d7a5ebf69868ddc0a848956a842c41022d2d0c1a2661f30.webp)

FP16  INT8  INT4  FP16  INT8  INT4

--- KEY INSIGHT ---
INT4 uses 3.2x less memory than FP16.
On an A100-80, that means 251 users instead of 214.

Summary: Three Optimizations

Optimization | What It Does | Speedup | Tradeoff
Quantization | Fewer bits per weight | 2-4x more users | Minor quality loss at INT4

Key insight: These are orthogonal. You can stack all three:
• INT4 weights (4x less memory)

In practice, you get 10-20x improvement. The engines (Part 5) implement all of these.

Interface text:
marimo
Your notebooks
CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
Double-click (or enter) to edit
molab.marimo.io/notebooks/[illegible]
AIE World Fair 2026
Ask Gemini
New Chrome available
Apps  talks  transformer videos  voice-agents-videos  papers  Feast  Research-Exp  Recommender systems  CFPs  LLM-Inference  Learning  vizuara-voiceAI  MLFeatureStores  job  Distributed Training  Blogs  Agents  All Bookmarks

### Attention mechanisms compared

The lab introduces a GPU benchmark of MHA, GQA, MLA and FlashAttention, using Mistral-7B to measure each mechanism’s memory use and latency.

[57:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3454s) [Notebook introduces a lab comparing attention mechanisms using Mistral-7B.](https://assets.ai.engineer/sites/aiecode2025/sha256/77181875a79efb6608e0c2bfb609342d675bc1723ffceda7c4bd996ec3353cdb.webp)

import marimo as mo

Open in Colab
Open in Molab

Attention Mechanisms Compared

This lab benchmarks MHA vs GQA vs MLA vs FlashAttention on a real GPU. Load Mistral-7B and measure actual memory and latency for each mechanism.

# Import
import subprocess, sys

Interface text:
marimo
Your notebook
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
molab.marimo.io/notebooks/[illegible]
New Chrome available

### Attention lab setup

The setup cell installs transformers and accelerate for model loading, then attempts an optional FlashAttention install that silently ignores failure.

[57:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3459s) [Notebook code installs model-loading dependencies and optionally installs FlashAttention.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b033c82facea961c849bc7e7560ff726acc8ee3551619191d829d119ce728eb.webp)

# Import
import subprocess, sys

# Install dependencies (run once)
def pip_install(*packages, extra_args=None):
    """Install packages via pip subprocess."""
    # Compute cmd
    cmd = [sys.executable, "-m", "pip", "install", "-q"] + list(packages)
    # Conditional check
    if extra_args:
        cmd += extra_args
    # Run shell command
    subprocess.check_call(cmd)

pip_install("transformers", "accelerate")  # HuggingFace model loading
# flash-attn: optional, install silently ignores failure
import os
# Set environment variable
os.environ.setdefault("CUDA_HOME", "/usr/local/cuda")
# Run shell command
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "flash-attn", "--no-build-isolation"], capture_output=True)

# Import

Interface text:
marimo
Your notebook
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
New Chrome available

### KV cache memory per token

The notebook compares Mistral-7B’s per-token KV cache calculations: MHA uses 32 heads, GQA-8 uses 8 KV heads, and MLA counts 576 values per layer.

[57:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3470s) [Experiment lists per-token memory calculations for MHA, GQA-8, and MLA.](https://assets.ai.engineer/sites/aiecode2025/sha256/c3409dd6fd652438474be2bf64b6703d930a842e9a1664c1c9feb3bbc1e8a92c.webp)

Experiment 1: KV Cache Memory per Token

For Mistral-7B with different attention variants, calculate bytes per token:
• MHA: 32 heads x 128 dim x 2 (K+V) x 32 layers x 2 bytes
• GQA-8: 8 KV heads x 128 dim x 2 x 32 x 2
• MLA: 512 + 64 = 576 values x 32 layers x 2 bytes

[illegible] print("No GPU detected. Run on Molab/Colab for real measurements.")  # GPU model name
gpu_vram_gb = 80  # Set gpu_vram_gb
gpu_bw_gbs = 2039  # total VRAM in GB  # Compute gpu_bw_gbs  # A100 bandwidth in GB/s (default)  # Print result to stdout  # Set gpu_vram_gb  # assume A100 for calculations  # Set gpu_bw_gbs

# Mistral-7B architecture constants
N_LAYERS_m = 32          # transformer layers
N_QUERY_HEADS_m = 32     # query attention heads
N_KV_HEADS_GQA = 8       # KV heads with GQA (4:1 ratio)
HEAD_DIM_m = 128         # dimension per head
[illegible]

Interface text:
marimo
molab.marimo.io/notebooks/[illegible]
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
New Chrome available

### GPU resource configuration

After the notebook reports no GPU detected, the presenter checks its configured resources: 4 CPU cores, 32 GiB of memory, and an RTX Pro 6000 (Blackwell).

[58:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3491s) [Notebook resource dialog shows four CPU cores, 32 GiB of memory, and an RTX Pro 6000 GPU.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea6be0992b20e2902e814839098cdb7a43d4a669a7a19eecd9fd2031f05ed02a.webp)

Configure Resources

CPU Cores: 4 cores
Memory: 32 GiB
GPUs: RTX Pro 6000 (Blackwell)
Save and restart

Attention Mechanisms Compared

This lab benchmarks MHA vs GQA vs MLA vs FlashAttention on a real GPU. Load Mistral-7B and measure actual

Open in Molab

Interface text:
marimo
Your noteb[illegible]
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
Restart
New Chrome available
Apps · AIE World Fair 2026 · talks · transformer videos · voice-agents-videos · papers · Feast · Research-Exp · Recommender systems · CFPs · LLM-Inference · Learning · vizuara-voiceAI · MLFeatureStores · job · Distributed Training · Blogs · Agents · All Bookmarks

### Attention mechanisms notebook

The notebook reconnects to an existing session and introduces a lab comparing MHA, GQA, MLA and FlashAttention on a GPU using Mistral-7B.

[58:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3496s) [Notebook introduction beneath a reconnected-session notice.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5ee94d03059364d38bdc89cf8c02c9daf514d7ee833a5baeeb50932c72352ab.webp)

Reconnected
You have reconnected to an existing session.
Restart

import marimo as mo

Open in Colab
Open in Molab

Attention Mechanisms Compared

This lab benchmarks MHA vs GQA vs MLA vs FlashAttention on a real GPU. Load Mistral-7B and measure actual

Interface text:
marimo
Your notebook
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
New Chrome available
Ask Gemini
Apps · AIE World Fair 2026 · talks · transformer videos · voice-agents-videos · papers · Feast · Research-Exp · Recommender systems · CFPs · LLM-Inference · Learning · vizuara-voiceAI · MLFeatureStores · job · Distributed Training · Blogs · Agents · All Bookmarks
molab.marimo.io/notebooks/[illegible]

### KV cache memory per token

The notebook compares KV cache formulas for Mistral-7B: MHA uses 32 heads, GQA-8 uses 8 KV heads, and MLA uses 576 values per layer. The runtime reports CPU and no detected GPU.

[58:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3510s) [Notebook displays a CPU detection result and formulas comparing KV cache memory for MHA, GQA-8, and MLA.](https://assets.ai.engineer/sites/aiecode2025/sha256/8afdcf8dfdeecd20aa4def58cf18db5f496a4f9d4a012c6c3468a0c5fd072c5a.webp)

Experiment 1: KV Cache Memory per Token

For Mistral-7B with different attention variants, calculate bytes per token:
• MHA: 32 heads x 128 dim x 2 (K+V) x 32 layers x 2 bytes
• GQA-8: 8 KV heads x 128 dim x 2 x 32 x 2
• MLA: 512 + 64 = 576 values x 32 layers x 2 bytes

Visible code:
    gpu_bw_gbs = 2039  # Get GPU specifications for reference in calculations
    print(f'GPU: {gpu_name}, VRAM: {gpu_vram_gb:.1f} GB')
else:  # Set gpu_name
    print('No GPU detected. Run on Molab/Colab for real measurements.')  # GPU model name
    gpu_vram_gb = 80  # Set gpu_vram_gb
    gpu_bw_gbs = 2039  # total VRAM in GB  # Compute gpu_bw_gbs  # A100 bandwidth in GB/s (default)  # Print result to stdout  # Set gpu_vram_gb  # assume A100 for calculations  # Set gpu_bw_gbs

Device: cpu
No GPU detected. Run on Molab/Colab for real measurements.

Interface text:
marimo
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
New Chrome available

### Attention Mechanisms Compared

The notebook reports CPU and compares KV cache savings across attention mechanisms. The speaker corrects MLA’s displayed 56× saving to 14× versus MHA: the calculation omitted the layer count.

[59:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3542s) [Notebook device-detection code reports CPU and says no GPU was detected.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac3207577fe0a9e47ec6ea10669eb6bde15756400a937d9ed5cce154c2d90472.webp)

import torch, gc, time, numpy as np, matplotlib.pyplot as plt
os.environ['HF_HUB_DISABLE_TELEMETRY'] = '1'
# Environment setup for HuggingFace
device_type = 'cuda' if torch.cuda.is_available() else 'cpu'  # suppress telemetry
print(f'Device: {device_type}')
# Device setup: prefer GPU, fall back to CPU
if device_type == 'cuda':
    # Print result to stdout
    gpu_props = torch.cuda.get_device_properties(0)
    gpu_name = gpu_props.name
    # Conditional check
    gpu_vram_gb = gpu_props.total_memory / 1000000000.0
    gpu_bw_gbs = 2039  # Get GPU specifications for reference in calculations
    print(f'GPU: {gpu_name}, VRAM: {gpu_vram_gb:.1f} GB')
else:  # Set gpu_name
    print('No GPU detected. Run on Molab/Colab for real measurements.')  # GPU model name
    gpu_vram_gb = 80  # Set gpu_vram_gb
    gpu_bw_gbs = 2039  # total VRAM in GB  # Compute gpu_bw_gbs  # A100 bandwidth in GB/s (default)  # Print result to stdout  # Set gpu_vram_gb  # assume A100 for calculations  # Set gpu_bw_gbs

Device: cpu
No GPU detected. Run on Molab/Colab for real measurements.

Interface text:
marimo
Your notebook
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
molab.marimo.io/notebooks/[illegible]

[1:00:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3600s) [Notebook compares MHA, GQA-8 and MLA; the displayed 56× MLA saving is corrected by the speaker to 14×.](https://assets.ai.engineer/sites/aiecode2025/sha256/6aecdac9fcb9efb9cdf5839e76129c30afb4c22c1ea9a9ad375efeac76660ba2.webp)

[illegible]
inference engines.")

Key Takeaways

Mechanism | KV bytes/tok | Savings vs MHA | Production use
MHA | ~524 KB | 1x | GPT-2, OPT, BLOOM
GQA-8 | ~131 KB | 4x | Mistral, Llama 3, Gemma
MLA | ~1.2 KB | 56x | DeepSeek-V2/V3

FlashAttention is orthogonal -- it reduces HBM traffic for the attention kernel (6x fewer reads) without changing the KV cache size. Combine with GQA for best results.

See the comparison module markdown for full analysis.

Interface text:
marimo
Your notebook
4 CPU · 32 GiB · RTX Pro 6000
Share
Duplicate
Harshul Jain's Workspace
PYTHON
MARKDOWN
SQL
GENERATE WITH AI
molab.marimo.io/notebooks/[illegible]

### KV cache, serving & engines

After model optimization with GQA and INT4, the talk turns to serving, pairing concepts with demos from PagedAttention through RadixAttention.

[1:00:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3611s) [Section 03 introduces KV cache, serving and inference engines.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab7cb26501811ffe2618903cd15535ab599e3cf59df9375e2cbf42de3a847d66.webp)

03

KV cache, serving & engines

Model is smaller — GQA + INT4. Now optimize serving. Each concept, then the demo that proves it: PagedAttention through RadixAttention.

### Cache the past, compute once

The KV cache stores key and value vectors so each decode step can reuse them instead of recomputing past tokens. That trades compute for memory: about 131 KB per token for Mistral.

[1:00:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3634s) [Slide contrasts recomputing previous tokens with storing and reusing keys and values, trading compute for memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/e06bbf32e2c03a1ac548aa7c58a2ee4a269af2cdc477a0f4ad0383daf7d275a8.webp)

THE KV CACHE · WHY IT EXISTS

Cache the past, compute once

WITHOUT CACHE
O(n²)
Recompute every token, every step
Each new token re-derives the keys and values of all previous tokens. Work grows quadratically with context.

→

WITH CACHE
O(n)
Store the past, compute only the new
Keys and values are written once and reused. Each step computes a single token — work grows linearly.

The cache trades compute for memory — about 131 KB per token for Mistral. That growing pile is what the four levers manage.

## Full transcript

[0:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=13s) Uh so good afternoon everyone. Um my

[0:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=16s) name is Hershel Jan and he is Tanisha.

[0:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=20s) Uh and we would like to welcome you all

[0:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=22s) in this two hours workshop on the LLM

[0:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=25s) inference. Uh so the goal of this

[0:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=28s) workshop is to understand this domain

[0:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=32s) from the first principles uh dive deeper

[0:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=35s) into it and like understand what's going

[0:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=38s) on throughout the industry.

[0:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=41s) Uh a bit of background about us. So I am

[0:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=45s) a senior software engineer at Audible.

[0:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=48s) uh have been building MLA data platforms

[0:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=51s) for the past five years and on the sides

[0:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=54s) I have been writing this opensource

[0:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=56s) handbook on LLM inference

[0:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=59s) and Tanme he is the senior quantitative

[1:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=63s) modeler at XAN cup bank corporation he

[1:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=66s) recently completed his PhD and he has

[1:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=69s) been actively doing research in the

[1:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=72s) agent verifiers and the world models

[1:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=78s) Uh so a quick show of hands here. Uh Vu

[1:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=82s) here is like brand new to the LLM

[1:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=84s) inference.

[1:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=87s) Okay, great. And Vu here has like

[1:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=89s) deployed these models in production.

[1:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=93s) They have been tuning it. They have been

[1:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=95s) serving the production traffic.

[1:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=98s) Okay, great.

[1:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=101s) So this workshop is targeted towards the

[1:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=104s) beginner and the intermediate level. U

[1:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=108s) and all of the slides and exercises they

[1:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=110s) are in the repo. I will share that soon.

[1:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=115s) Here is the quick agenda for the

[1:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=117s) workshop. We will start with the problem

[2:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=120s) statement. We will try to understand few

[2:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=122s) of the pain points around LLM inference.

[2:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=126s) uh then we understand what causes those

[2:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=128s) pain points and build our foundations

[2:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=130s) from there.

[2:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=132s) Then we will dive into like two kind of

[2:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=135s) the optimizations that we do like the

[2:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=137s) model optimizations and the serving

[2:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=139s) optimizations.

[2:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=141s) uh and then we start learning about

[2:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=143s) different serving engines that are

[2:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=145s) available to deploy our LLM inference

[2:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=149s) solutions in production and we will

[2:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=151s) showcase some benchmarks and the

[2:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=153s) decision chart on like which engine to

[2:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=156s) use.

[2:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=160s) Cool. So to understand the pain points

[2:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=163s) first we need to know what is like LLM

[2:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=166s) inference. So, and probably a lot of us

[2:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=169s) already know this. Um, but yeah,

[2:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=172s) anything that you ask your AI to do like

[2:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=175s) whether it be generate a video, audio,

[2:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=178s) analyze any text, uh, analyze your

[3:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=181s) medical reports or like your tax bills,

[3:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=184s) all of that is like an LLM inference.

[3:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=188s) And this market is like approximately

[3:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=191s) $23 billion today.

[3:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=195s) uh semi analysis recently shared that if

[3:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=198s) you want to model like a Google search

[3:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=200s) queries with LLMs, you need like a

[3:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=203s) profit drain of like $36 billion

[3:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=207s) and query cost has to be less than 0.5

[3:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=211s) cents to keep your search business

[3:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=213s) profitable.

[3:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=215s) On the other hand, the business insider

[3:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=218s) mentioned like your AI has to be put on

[3:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=220s) diet

[3:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=222s) and everyone has to start auditing and

[3:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=224s) budgeting their token usage and all of

[3:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=227s) this is happening. Why? Because your

[3:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=230s) hardware is limited, compute is

[3:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=232s) expensive, your inference is expensive

[3:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=235s) and with the growing need of like more

[3:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=238s) and more AI usage, this inference cost

[4:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=240s) is rising more and more.

[4:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=243s) So

[4:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=245s) this stat, it's an old stat from the

[4:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=248s) open AI, but it's still it's still true.

[4:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=251s) So if you look at the like training cost

[4:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=254s) of the GPT3, it was like around $4.6

[4:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=257s) million. It was a one-time cost. But if

[4:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=260s) you see the inference cost that has been

[4:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=263s) like uh it's a recurring cost because

[4:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=266s) it's a operating cost that scales with

[4:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=268s) every user that comes in that every

[4:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=270s) token that comes in every session that

[4:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=272s) uh is being initiated on the like AI

[4:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=279s) and there are only two ways to basically

[4:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=282s) counter this. Uh one way is you reduce

[4:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=285s) your token usage.

[4:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=287s) um alternative is you should try to

[4:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=290s) optimize your inference solutions as a

[4:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=293s) inference service provider for your

[4:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=296s) customers and for yourself. And so this

[4:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=299s) um we have been seeing like lot and lot

[5:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=302s) of like new solutions coming out every

[5:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=304s) then and now. Um and so the idea would

[5:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=308s) be like okay we will try to build those

[5:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=310s) foundations that will help us understand

[5:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=313s) and evaluate like whatever ships next.

[5:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=317s) Um so yeah to get started like we will

[5:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=321s) do a quick demo like it's a short demo

[5:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=324s) of like what are the different pain

[5:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=325s) points around inference

[5:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=328s) u and so this is the repo uh I mean you

[5:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=333s) can pull it or you can also open it on

[5:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=335s) the GitHub

[5:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=337s) uh it's called LLM inference at scale a

[5:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=339s) bit of background here like four months

[5:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=342s) back when I didn't knew anything on the

[5:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=343s) LLM inference um I started learning it I

[5:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=347s) saw like lot of resources were

[5:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=348s) scattered. So we started putting it uh

[5:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=351s) like all together in one place uh so

[5:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=353s) that it could benefit people.

[5:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=356s) Um yes. So let me actually get out of

[6:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=360s) this slideshow mode and probably

[6:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=364s) go into

[6:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=368s) I will go to this extended mode.

[6:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=373s) Um

[6:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=376s) okay great.

[6:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=379s) Uh yeah so in this uh repository if you

[6:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=383s) see a readme file there is like a link

[6:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=386s) to the slides. Uh so this it will be

[6:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=389s) like this folder where you have like a

[6:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=391s) pptx and there is like a benchmark

[6:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=393s) report in there.

[6:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=396s) uh you can always like download it and

[6:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=399s) then for the demo purposes uh we have

[6:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=402s) couple of Jupyter notebooks. Uh we have

[6:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=404s) like collaborated with Moab who are the

[6:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=407s) like Google collab alternative

[6:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=410s) and what they basically provide you is

[6:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=412s) like a free RTX 6000 GPU. It's a 100 GB

[6:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=416s) V RAM GPU.

[6:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=418s) So

[6:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=419s) and we have like already set up these

[7:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=421s) notebooks so that it becomes easy to

[7:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=424s) like experiment with and like all of the

[7:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=427s) assets and everything are preset for

[7:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=429s) you. Uh

[7:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=432s) so uh we will start with like a simple

[7:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=435s) demo a

[7:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=438s) probably

[7:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=441s) let me just see

[7:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=449s) Okay.

[7:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=451s) Um yeah. So

[7:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=455s) when it comes to the inference, you need

[7:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=457s) to do an inference on a certain model,

[7:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=460s) right? Uh for the workshop purposes, we

[7:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=462s) are using a simple ML 7B model. Uh it's

[7:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=466s) a small model of around 15GB in size. So

[7:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=469s) we we are going to like load that into

[7:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=471s) the GPU. So,

[7:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=475s) and we would look like some of the GPU

[7:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=477s) stats as well. So, we see like okay, we

[7:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=479s) are working on the 6,000 Blackwell. Uh,

[8:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=482s) and you might be thinking I'm not

[8:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=483s) running the cells because I don't trust

[8:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=485s) the Wi-Fi at conferences.

[8:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=488s) So, yeah. So, I would probably be just

[8:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=491s) going over the results uh that we kind

[8:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=493s) of ran previously.

[8:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=498s) Uh, yeah. So, we have like a GPU which

[8:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=500s) is like 102GB.

[8:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=503s) Uh now the first thing that comes to my

[8:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=506s) mind is like what's my memory

[8:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=507s) consumption looks like when I do the LLM

[8:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=509s) inference. So I load this model

[8:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=513s) uh and I see like okay I have like a 15

[8:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=516s) GB here. So I have roughly like 87.5GB.

[8:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=520s) And now when I do the like inference

[8:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=523s) here

[8:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=525s) uh what I notice is like the more the

[8:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=528s) number of inputs I pass more is the

[8:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=531s) memory that I need.

[8:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=533s) uh and it's increasing slowly but it's

[8:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=535s) still increasing. [snorts] So imagine

[8:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=537s) like if you have a context length of

[8:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=539s) like around 4,000 or 16,000 or 32,000

[9:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=543s) uh tokens. Uh so this memory could like

[9:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=546s) really grow big and it you could

[9:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=548s) actually get like all of those out of

[9:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=550s) memory issues.

[9:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=553s) Uh so definitely this is like your

[9:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=555s) problem one like your memory increasing

[9:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=557s) with the increase in tokens. So in form

[9:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=559s) of like a simple visualization it looks

[9:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=561s) like this.

[9:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=565s) The second problem that you would see is

[9:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=567s) like

[9:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=568s) the time to your first token it's very

[9:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=571s) very slow. Uh we measure it by a metric

[9:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=574s) called TTFT. It's a short short form of

[9:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=577s) it. uh and when you try to like measure

[9:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=581s) the TTFT uh with the like input size you

[9:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=586s) would see like longer the context

[9:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=589s) you would see like this uh TTFT being

[9:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=592s) slow. So now there are two problems.

[9:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=594s) Your memory increases with the token

[9:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=596s) size. Your TTFT increases with the token

[9:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=599s) size.

[10:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=600s) Uh sorry not the token size, the context

[10:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=602s) size.

[10:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=606s) Uh and then the third is the like

[10:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=608s) throughput. The throughput is like how

[10:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=611s) many tokens can you serve per second and

[10:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=614s) then how many users can you serve per

[10:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=616s) second. So if you take a very very

[10:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=618s) vanilla implementation on your local

[10:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=620s) system

[10:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=622s) uh it would be like very sequential. So

[10:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=624s) if you send like five requests all those

[10:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=627s) five requests would be catered like

[10:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=629s) sequentially rather than parallelly.

[10:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=632s) Uh and so like your request basically

[10:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=636s) takes more time to complete if you have

[10:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=638s) like multiple users.

[10:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=641s) So these are the like three problems.

[10:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=643s) There is a fourth one. I haven't

[10:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=644s) described it here. probably we will

[10:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=646s) build that intuition as we move forward.

[10:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=650s) Uh but let's remember like these are the

[10:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=652s) three problems the memory TTFT and the

[10:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=655s) throughput.

[10:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=658s) Cool. Uh I will go back to the slides.

[11:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=667s) Okay, perfect.

[11:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=675s) Uh so it should be this.

[11:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=680s) >> Uh is it visible?

[11:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=695s) Oh, you you need the

[11:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=698s) Oh, okay. Workshop. Okay.

[11:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=707s) Yeah. So, within that repository, if you

[11:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=710s) see a workshop folder, you see that

[11:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=713s) readme and then the readme has all the

[11:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=715s) links, the slides and the demos.

[12:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=724s) Uh, does that work?

[12:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=729s) Okay, perfect.

[12:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=736s) Okay. uh so let's start working through

[12:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=738s) the foundations like let's start

[12:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=740s) understanding

[12:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=742s) uh what are the reasons behind those

[12:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=744s) pain points and for that like we have to

[12:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=747s) look at this inference pipeline

[12:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=749s) um

[12:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=752s) so we get like an input text uh that

[12:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=755s) text could have like any number of words

[12:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=759s) you convert those into the tokens so for

[12:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=762s) simplicity you can assume one word equal

[12:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=765s) to one token

[12:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=767s) uh then you kind of convert them into

[12:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=769s) like the embeddings and then you send it

[12:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=771s) to the like transformers

[12:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=774s) uh like there are 32 layers of

[12:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=776s) transformers but that's specific to the

[12:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=777s) Mistl 7B different models have different

[13:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=780s) kind number of layers uh and then you

[13:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=783s) generate a new token and that token

[13:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=785s) basically goes back to the input then

[13:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=787s) you generate another token and that

[13:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=788s) keeps on going now in this entire

[13:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=791s) pipeline you would see like 95% of your

[13:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=795s) compute is like taken by these

[13:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=797s) transformer layers. So it's worth

[13:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=799s) looking at like what goes within this

[13:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=801s) transformer layer.

[13:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=805s) Within this transformer layer you would

[13:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=806s) have like more layers. You have like a

[13:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=809s) normalization layer, you have an

[13:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=811s) attention layer, you have a feed forward

[13:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=814s) layer and all.

[13:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=815s) And attention layer is the one I think

[13:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=818s) that has been very very famous.

[13:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=820s) Attention is all you need paper. I think

[13:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=822s) that's very well known. So attention is

[13:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=825s) the most compute intensive layer and we

[13:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=829s) need to understand what goes within that

[13:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=831s) attention layer.

[13:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=834s) So what does attention do? Attention uh

[13:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=838s) so if you have an input text it needs to

[14:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=840s) find the attention scores of every token

[14:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=843s) with respect to all of the previous

[14:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=844s) tokens.

[14:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=846s) And to do that what it needs to do is

[14:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=849s) like it needs to project every token

[14:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=851s) into like a key query and the value

[14:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=853s) space. So in like in a simpler terms

[14:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=858s) just u understand this like if you have

[14:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=862s) 10 tokens then it needs like the 10

[14:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=864s) different query key and the value

[14:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=866s) vectors. If there are 100 tokens you

[14:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=869s) would need 100 key and the value

[14:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=870s) vectors. If there are thousand tokens

[14:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=872s) you would need thousand key value

[14:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=874s) vectors. And so like your number of the

[14:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=876s) key and the value vectors they increase

[14:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=878s) as you increase the input size.

[14:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=884s) Uh and if you calculate the like KV size

[14:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=888s) per token

[14:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=890s) uh for a mist 7B it comes out to be 131

[14:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=893s) KV. Uh this is because like

[14:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=897s) uh you have two vectors K and V. You

[15:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=900s) have to multiply the size uh one vector

[15:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=903s) is like 128 dimensions. you have to

[15:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=905s) multi multiply it by 32 transformer

[15:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=908s) layers and then you have to multiply it

[15:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=910s) by the KV heads for ML 7B it's gate KV

[15:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=915s) heads it's not like 32 because it uses a

[15:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=918s) different kind of an attention mechanism

[15:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=921s) u which we will talk about for sure but

[15:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=925s) yeah so the KV size per token is like

[15:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=929s) your 131 KB now imagine if you have 4K

[15:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=933s) context uh So that size becomes like

[15:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=935s) half a GB. Uh if you do like 16k context

[15:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=940s) that size becomes 2.1 GB. Uh now

[15:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=944s) multiplay by the users like assume you

[15:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=946s) can serve multiple users together

[15:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=950s) at the same time within that GPU you

[15:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=953s) could have like 42GB with a 4K context

[15:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=956s) and 80 users. Um and if your GPU is only

[16:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=960s) like let's say 24GB, you are already

[16:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=963s) running out of the memory. So you cannot

[16:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=965s) serve that many users with that many

[16:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=967s) context.

[16:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=972s) To visualize this, look at a GPU memory.

[16:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=975s) So the GPU memory has like a model

[16:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=977s) weights which are pretty fixed. These

[16:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=980s) are pre-trained weights. Uh there is

[16:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=983s) like an overhead that is also fixed.

[16:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=985s) that also like that changes but it does

[16:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=987s) not change that much u overall you can

[16:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=990s) assume it's fixed and then there is like

[16:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=993s) a leftover memory so this leftover

[16:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=996s) memory is what being used by your KB me

[16:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=999s) like key and the value vectors so

[16:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1004s) assume like you have a one user you can

[16:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1008s) only serve that many key and the value

[16:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1011s) vectors or that many tokens which can

[16:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1014s) like fit in this entire 80GB uh like

[16:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1018s) memory that is left.

[17:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1022s) Uh

[17:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1024s) so we can show this with a simple demo

[17:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1027s) too. Uh

[17:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1033s) okay, let me

[17:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1038s) Okay, great.

[17:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1043s) Okay, great.

[17:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1045s) Uh, let me see if I can actually run

[17:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1049s) this.

[17:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1051s) What

[17:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1056s) probably?

[17:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1058s) Where the heck is this happening?

[17:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1069s) Okay, great.

[17:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1072s) Yeah. So you would see like the GPU is

[17:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1075s) attached.

[18:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1087s) So here we are just trying to confirm

[18:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1089s) the like memory based on the maths and

[18:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1091s) based on the intuition that we have

[18:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1093s) built. So the model memory is like let's

[18:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1096s) say if you have a 7 billion parameters

[18:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1098s) you are doing a 16 bit precision your

[18:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1101s) total memory comes out to be 14.6 GB

[18:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1103s) with you can basically verify that with

[18:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1106s) the maths. So if you do all that math

[18:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1110s) that comes out to be the 14.6GB.

[18:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1113s) Uh now comes the KVK and the KV size. So

[18:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1118s) this KV size is like your 131 KB per

[18:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1120s) token.

[18:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1122s) Uh and if you do that maths and you try

[18:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1125s) to like visualize this.

[18:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1129s) [sighs]

[18:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1132s) Oh, sure.

[19:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1140s) Wait.

[19:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1143s) Okay. And then let's just visualize

[19:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1147s) this.

[19:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1149s) Okay. Great.

[19:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1151s) Uh yeah so this is the like a memory

[19:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1154s) chart. So if you see like as your

[19:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1157s) context increases your memory keeps

[19:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1159s) increasing.

[19:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1161s) Then another thing to realize is like as

[19:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1164s) your users increase

[19:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1166s) uh then also your memory increases. So

[19:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1169s) if you want to serve like 160 users

[19:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1173s) uh on a GPU you can support like uh you

[19:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1177s) can only support like a lesser context

[19:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1179s) length. So there is always a tradeoff

[19:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1182s) between what context length you can

[19:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1184s) serve versus how much cost you can save

[19:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1188s) by like putting your multiple

[19:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1191s) uh users or the concurrent users into

[19:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1193s) like a single GPU. So you have to always

[19:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1196s) take that tradeoff and and we will go

[19:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1198s) through that like uh in couple of more

[20:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1201s) slides.

[20:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1206s) >> Uh can you repeat please?

[20:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1214s) Uh I'm sorry I'm cannot hear you.

[20:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1224s) >> Yeah.

[20:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1233s) Cool.

[20:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1235s) Uh okay. Great. So let me pull back.

[20:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1241s) So that was like memory. Uh we need to

[20:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1244s) understand uh why we had like uh slower

[20:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1248s) time to first token

[20:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1251s) uh when we increase the context length.

[20:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1254s) So for that like we need to understand

[20:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1256s) the two phases of inference and those

[20:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1259s) phases are like the prefill and the

[21:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1260s) decode phase. I think you would all seen

[21:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1263s) like a lot of articles but we just

[21:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1265s) wanted to explain it. Uh so when you

[21:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1269s) send like a lot of uh like when you send

[21:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1271s) these input tokens

[21:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1274s) what you want to do is uh you want to

[21:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1276s) build those key and the value vectors

[21:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1278s) that I mentioned for all the tokens.

[21:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1281s) Then you want to compute the attention

[21:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1283s) scores of every token with respect to

[21:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1286s) the previous token. All this operation

[21:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1288s) that you do it's a very very metricsh

[21:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1291s) heavy uh it's a very very compute heavy

[21:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1294s) operation and we all know like the GPUs

[21:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1297s) they are like very well suited for a

[21:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1300s) heavy compute workload so we call like a

[21:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1304s) prefill to be like a compute bound uh

[21:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1307s) and it does take some time to complete

[21:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1310s) so whatever time that this phase takes

[21:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1312s) to complete that's your time to the

[21:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1315s) first token

[21:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1317s) So if if you have like a more input

[21:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1319s) tokens, you have to generate more key

[22:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1322s) value vectors. You have to do lot more

[22:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1324s) attention math and because of that your

[22:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1328s) TTFT becomes more more slower.

[22:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1332s) Where is if once you generate one token

[22:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1335s) you need to keep doing this to generate

[22:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1337s) another tokens sequentially one after

[22:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1340s) another. But in that process every time

[22:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1343s) you have to build the key and the value

[22:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1346s) vectors of all the previous tokens

[22:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1349s) which is same as prefill like you were

[22:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1351s) building key value vectors there also

[22:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1353s) here also but in decode phase you are

[22:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1356s) only computing the attention math for

[22:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1358s) the new token

[22:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1361s) and that is why it's a very less it's

[22:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1364s) lesser compute oriented

[22:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1366s) and it's also called as memory bound. We

[22:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1368s) will see it shortly why it's called as

[22:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1370s) memory bound.

[22:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1374s) So in a classic timeline you would see

[22:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1378s) prefill and decode phase like this. So

[23:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1381s) time taken by prefill that's your time

[23:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1383s) to first token and then your time taken

[23:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1386s) by every decode step that's your

[23:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1390s) uh basically your inter token latency.

[23:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1393s) So that's like the fourth metric

[23:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1396s) uh that you need to worry about like

[23:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1399s) what's the time being taken by your

[23:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1401s) decode step.

[23:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1405s) Okay, cool. Uh

[23:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1408s) now why why does the like decode step or

[23:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1412s) why does decode takes time and why it's

[23:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1416s) being called as like a memory bound

[23:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1418s) operation? Let's try to understand that.

[23:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1420s) uh to understand that we need to look at

[23:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1424s) how the metrics map basically works on

[23:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1426s) the GPU on a high level. So GPU has two

[23:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1429s) kind of memories. You have a high

[23:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1431s) bandwidth memory. You have a shared

[23:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1434s) memory.

[23:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1436s) So the high bandwidth memory is a larger

[23:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1438s) size but a lower me like lower

[24:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1440s) bandwidth.

[24:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1442s) By lower bandwidth I mean like you can

[24:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1444s) transfer data out of it at a lower rate

[24:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1447s) compared to the shared memory. So the

[24:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1449s) shared memory is smaller in size but it

[24:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1452s) has a very very high bandwidth. Uh that

[24:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1455s) means you can transfer data in and out

[24:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1457s) of it with a very first thing. So when a

[24:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1460s) when you have to do a metric math so you

[24:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1463s) have to pick the data in chunks from the

[24:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1466s) high bandwidth memory you have to put it

[24:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1468s) into the shared memory.

[24:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1471s) Do that math write back the result into

[24:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1473s) the high bandwidth memory.

[24:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1477s) uh for the prefill phase when you have

[24:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1482s) to do this you have to do this matrix

[24:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1485s) math only once but for the decode phase

[24:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1488s) you have to do this metric math uh like

[24:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1493s) uh again and again because you're

[24:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1495s) generating each and every token

[24:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1497s) sequentially

[24:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1499s) and so like you it doesn't matter like

[25:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1502s) how fast is your decode

[25:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1505s) because now you can transfer your data

[25:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1508s) out of the high bandwidth memory into

[25:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1511s) the S shared memory at a certain speed

[25:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1514s) because you are limited by the high

[25:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1516s) bandwidth memory bandwidth speed

[25:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1519s) and so that governs your like token

[25:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1522s) sealing like at what rate can you

[25:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1524s) actually generate tokens out of the

[25:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1527s) decode step.

[25:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1532s) If you look at this in the roof line

[25:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1535s) plot

[25:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1537s) uh so there is a left section which is

[25:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1541s) called to be a memory bound.

[25:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1543s) Mathematically it's governed by the

[25:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1545s) arithmetic intensity.

[25:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1547s) Arithmetic intensity is the number of

[25:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1550s) flip-flop operations that you perform

[25:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1552s) per bite of data being transferred. So

[25:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1555s) for the decode step

[25:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1558s) uh decode step since you are

[26:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1561s) transferring lot of data like the key

[26:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1564s) and the value vectors of the all the

[26:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1566s) previous tokens the model weights but

[26:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1568s) you are doing the like less computation

[26:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1570s) because you're computing attention math

[26:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1572s) for only one token. Uh so it's

[26:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1576s) arithmetic intensity is very low but for

[26:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1579s) a prefill phase you are transferring the

[26:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1581s) data once but then like you are doing

[26:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1585s) this heavy computation and so it's

[26:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1588s) arithmetic intensity is very high. So

[26:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1590s) now you know like in terms of

[26:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1592s) mathematics like why the computer like

[26:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1596s) why the arithmetic intensity of prefill

[26:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1598s) is very high compared to your decor.

[26:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1603s) Uh [snorts]

[26:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1604s) okay

[26:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1606s) so this is like another small small

[26:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1609s) demo. Uh

[26:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1612s) every time I have to Okay.

[26:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1617s) Okay, great.

[27:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1620s) I hope this is already running. So yeah,

[27:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1623s) again we are loading the model.

[27:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1627s) Now this is the like the prefill cost.

[27:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1630s) So what we are basically doing is uh we

[27:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1632s) are getting the like um the input uh

[27:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1635s) text and then we are trying to generate

[27:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1638s) this um the prefill step the amount of

[27:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1642s) time it takes. We see like as we

[27:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1644s) increase the like size of the input

[27:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1646s) tokens this prefill is increasing. So

[27:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1649s) you and this is the reason why your TDF

[27:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1652s) increases

[27:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1654s) and then like your decode time. So the

[27:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1657s) decode time is like on average it stays

[27:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1660s) about the same. Uh and so if it is like

[27:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1664s) assuming like you ignore the like cold

[27:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1666s) start your decode time is like

[27:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1668s) approximately around the average line.

[27:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1670s) it it it it is still impacted by like u

[27:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1676s) the input size. It's not like it's a

[27:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1678s) constant uh time and it is because it

[28:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1682s) still needs to pull the key and the

[28:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1683s) value vectors from the memory for all

[28:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1685s) the previous tokens. So there is still

[28:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1688s) like uh that uh basically small increase

[28:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1692s) in time that you would see with the

[28:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1694s) decode step. And then this is the like

[28:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1697s) classic uh roof line plot.

[28:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1701s) Uh okay.

[28:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1707s) Presentation.

[28:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1711s) Okay. Five. Okay. Great.

[28:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1715s) Okay. So

[28:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1718s) now now let's try to understand like uh

[28:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1721s) the throughput dimension. You want to

[28:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1724s) understand how many users you can

[28:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1726s) actually serve and I think we saw like a

[28:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1729s) diagram of the GPU memory where we saw

[28:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1732s) okay there is some memory that is free

[28:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1734s) for the key and the value vectors to

[28:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1735s) grow.

[28:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1737s) So assume like you have just a single

[29:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1740s) user

[29:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1742s) uh

[29:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1744s) what's the total KV size that you have

[29:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1747s) you can basically support it's defined

[29:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1749s) by your context limit.

[29:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1752s) uh the max users that you can support is

[29:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1755s) like whatever is your GPU uh

[29:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1758s) availability like whatever is the memory

[29:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1760s) that is available in the GPU you divide

[29:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1762s) it by the key and the value size per

[29:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1764s) user uh and when you do that like it

[29:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1768s) comes out to be like your the concurrent

[29:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1770s) users.

[29:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1773s) Now assume like your GPU is fixed, your

[29:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1778s) model is fixed.

[29:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1780s) Uh

[29:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1781s) so

[29:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1783s) your KV size per token is fixed. There

[29:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1786s) are only two dimensions that are left

[29:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1788s) here which is context and your

[29:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1791s) concurrent users.

[29:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1793s) If you want to serve more concurrent

[29:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1795s) users, you have to reduce the context

[29:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1797s) length. If you reduce the context

[29:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1799s) length, you could impact your quality.

[30:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1802s) Uh so these are the two dimensions right

[30:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1805s) now that we are trading off.

[30:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1809s) Then if we but can you actually serve

[30:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1812s) the like max number of concurrent users?

[30:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1817s) Uh in an ideal world probably not

[30:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1820s) because

[30:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1822s) every business has like a latency SLO

[30:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1825s) that we have to meet.

[30:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1828s) So

[30:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1830s) if you remember like in the decode step

[30:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1832s) I said the time for the decode still

[30:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1835s) increases if you have more inputs.

[30:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1838s) It also increases if you have more

[30:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1839s) users.

[30:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1841s) So ultimately

[30:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1844s) uh your inter token latency also gets

[30:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1847s) impacted

[30:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1849s) if you have like a higher batch size and

[30:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1851s) your TTF also gets impacted. So now

[30:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1854s) there is a third dimension you have to

[30:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1856s) worry about which is like your latency.

[30:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1858s) So the three dimensions that you have is

[31:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1861s) like a quality latency and the

[31:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1863s) throughput. So it comes out to be like

[31:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1865s) this trade-off triangle where you have

[31:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1868s) to choose between the two. So for a

[31:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1872s) premium chat application

[31:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1875s) you would want to prioritize definitely

[31:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1877s) the quality and you want to prioritize

[31:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1879s) the like the latency. You would not want

[31:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1882s) your users to wait infinitely for the

[31:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1885s) like or like not infinitely but probably

[31:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1888s) for the larger latency.

[31:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1891s) You can always sacrifice the number of

[31:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1893s) users you can support on the GPU and

[31:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1895s) probably take that costed being more

[31:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1898s) customers

[31:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1900s) in form of like and and like if you

[31:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1903s) consider like an agent uh sorry the

[31:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1906s) async agent workload you would want to

[31:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1909s) like prioritize definitely quality and

[31:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1910s) the throughput

[31:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1912s) uh because these are the longunning

[31:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1915s) tasks

[31:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1916s) uh and you would want to like serve as

[31:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1919s) many as concurrent tasks. fast as

[32:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1921s) possible but with a very very higher

[32:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1923s) quality.

[32:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1927s) And often like we think like okay if the

[32:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1931s) GPU is like a very expensive GPU

[32:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1935s) uh that might not be a good fit for us.

[32:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1939s) Uh but it turns out that could actually

[32:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1941s) serve you the lowest cost per million uh

[32:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1944s) tokens.

[32:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1947s) Uh but you really have to trust your

[32:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1950s) kind of calculations on the max users

[32:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1953s) that you want and like uh you really

[32:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1956s) have to make those estimations uh

[32:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1959s) correctly.

[32:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1963s) Uh

[32:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1969s) so we do have like uh

[32:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1975s) let me just

[32:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1979s) Where is this?

[33:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1982s) Okay, great. So, for the capacity

[33:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1985s) calculator, uh there is like a link to

[33:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1988s) the collab because I was facing certain

[33:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1991s) issues with molab. I had to migrate out

[33:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1993s) the wall widget library and I didn't

[33:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1996s) have time. So, being lazy, I just picked

[33:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=1999s) collab there. Uh apologies to Moab.

[33:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2005s) Uh

[33:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2007s) so my VR is connected.

[33:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2015s) Okay.

[33:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2026s) Wi-Fi probably.

[33:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2031s) Okay. Great.

[34:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2042s) So what we have done over here is we

[34:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2044s) have like shaded some like the GPUs with

[34:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2047s) their V RAMs, bandwidths, the flip-flops

[34:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2050s) and the cost per hours. Um

[34:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2054s) then we kind of like built this simple

[34:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2057s) uh like uh capacity calculator. This is

[34:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2060s) just a KV visualizer uh where you kind

[34:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2063s) of like when you increase the number of

[34:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2065s) tokens uh you see like your KV size it

[34:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2068s) increases and when you increase the

[34:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2071s) number of users your size is like

[34:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2073s) increasing at a much faster rate

[34:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2077s) and then

[34:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2080s) in this capacity calculator uh

[34:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2086s) let it run.

[34:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2089s) So we have like a model which we which

[34:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2092s) is like a 7 billion parameter model that

[34:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2095s) we selected.

[34:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2098s) We set the like precision to be FP16. Uh

[35:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2101s) now we decide the way we basically go by

[35:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2104s) the GPU decision is you have to decide

[35:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2108s) what's your like you have to fix one

[35:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2111s) dimension first which you care about the

[35:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2113s) most.

[35:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2115s) for premium chat I mentioned like

[35:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2117s) latency is definitely the one

[35:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2121s) uh and then like for the async workloads

[35:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2123s) the batch the minimum batch size that

[35:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2125s) you want to serve for from like a single

[35:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2128s) GPU that is the second dimension so you

[35:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2131s) want to fix these first so I will go

[35:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2134s) about like in a premium chat application

[35:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2138s) uh

[35:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2139s) so I can go ahead with like 10

[35:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2141s) milliseconds latency a minimum batch

[35:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2145s) size I don't care like I can so I'm okay

[35:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2147s) with like probably two

[35:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2150s) uh

[35:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2153s) okay so probably with the seven

[35:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2156s) concurrent users on a single GPU and

[35:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2159s) then like my context limit is very

[36:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2161s) important to me because I want to focus

[36:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2163s) on the quality as well

[36:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2166s) uh and so like I do see like some of the

[36:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2169s) GPUs so the H18GB

[36:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2172s) it's like a $8 per hour but like am I

[36:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2178s) 300x is it? Yeah. So it's like around

[36:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2181s) $10 per hour but if you do all that

[36:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2186s) throughput math that we shared in the

[36:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2188s) mathematics before you could find like

[36:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2191s) your cost per million dollar tokens that

[36:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2194s) could be very very that could be like

[36:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2196s) lesser. So

[36:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2199s) you need to do such calculations by

[36:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2201s) fixing those dimensions and you need to

[36:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2204s) decide your GPU to like reduce your kind

[36:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2207s) of inference cost. This is at least the

[36:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2210s) first step that you can take towards

[36:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2211s) optimizing the inference.

[36:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2216s) Okay, cool.

[37:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2221s) So the next slide. So let me

[37:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2228s) Okay, great.

[37:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2230s) And so like now the next thing is about

[37:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2233s) the model optimization. So we are now

[37:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2235s) basically have built that foundation

[37:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2237s) where we understood some of the pain

[37:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2239s) points, reason behind those pain points,

[37:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2241s) why those were happening

[37:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2243s) um how we could like address that GPU

[37:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2248s) capacity thing. We need to understand

[37:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2250s) what can we do like what can we further

[37:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2252s) do about it. So it it is about the model

[37:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2256s) optimization and I think I would like to

[37:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2258s) invite Tan I he can talk more about

[37:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2260s) these model optimizations provided he

[37:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2263s) has worked uh on this like during his

[37:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2266s) research times

[37:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2268s) okay I can control

[37:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2273s) yeah here okay hi everyone uh mic check

[37:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2278s) am I audible at last yeah okay so hi I'm

[38:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2282s) Tesha I work as a senior quant modeler

[38:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2285s) and also I am an AI researcher. My work

[38:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2288s) focuses on a agent verification and

[38:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2290s) right now building world models. So for

[38:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2293s) this one model optimization

[38:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2296s) before we start model optimization so I

[38:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2299s) created a research template so that it

[38:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2301s) will be easy for us to understand all

[38:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2303s) these complex things. I so our template

[38:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2307s) is simple. First we will identify the

[38:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2309s) problem. Second step we will solve the

[38:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2312s) problem using two algorithms. These are

[38:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2314s) just fake algorithms. So first algorithm

[38:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2316s) is called ostrich algorithm. Whenever we

[38:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2320s) see uh just like ostrich whenever we see

[38:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2322s) a problem ostrich put their head into

[38:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2325s) the sand. So same thing we will do

[38:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2327s) whenever we face a problem we will just

[38:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2330s) ignore it. So this is an important

[38:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2332s) algorithm we should follow. Second one

[38:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2335s) is created it is called world cup

[38:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2337s) algorithm. For example, we don't know

[39:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2341s) who will win this FIFA World Cup. So,

[39:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2343s) what organizers did, they uh break the

[39:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2348s) 48 teams into 12 groups, uh then round

[39:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2352s) 32. So, round 32 right now is currently

[39:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2354s) going on. Uh then round 16, then

[39:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2357s) quarterfinals, uh then semi-finals and

[39:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2360s) finals. So what they are doing is that

[39:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2364s) uh they are breaking it into a smaller

[39:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2366s) problems and the useful results are

[39:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2369s) moving forward. So same analogy or same

[39:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2372s) algorithm we will use uh to understand

[39:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2375s) this model optimization all those

[39:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2377s) things. So yeah let's start. So I have

[39:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2383s) one H100 GPU.

[39:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2386s) I have to use this open-source model

[39:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2389s) what is called GPTOSS

[39:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2391s) 120 billion parameter model. So right

[39:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2395s) now I think it's so they have trained it

[39:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2397s) on BF float 16 and weight is 240 GB.

[40:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2402s) What should I do?

[40:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2404s) This is the problem we have. So first

[40:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2407s) thing what we have to deal do is that

[40:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2410s) 240 GB and 80 uh GB H100.

[40:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2416s) So and I have to fit only in one GPU or

[40:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2419s) not in multiple GPU. So what can we do?

[40:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2422s) I think simple step is that just

[40:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2425s) compress it. But how should we compress

[40:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2429s) it? Uh that's the another challenge. So

[40:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2431s) if we compress BF BF float 16 to FP8 uh

[40:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2436s) then it will be around 120 GB but our

[40:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2439s) GPU H100 is still 80 GB. So what I think

[40:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2443s) they did is that they compressed it into

[40:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2445s) further MX uh MX FP4 and I think size is

[40:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2451s) around 65 GB. So this is something we

[40:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2455s) can do uh compress but question so and

[41:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2460s) we will use over this ostrich algorithm

[41:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2463s) we are assuming that uh there is no loss

[41:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2465s) in compressing a bigger model into a a

[41:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2469s) smaller size. Second thing

[41:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2472s) in uh in this one okay yeah so in this

[41:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2477s) one in this slide we have used this

[41:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2478s) mistral 7B so 7 billion parameters so

[41:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2482s) it's a small model 7 billion parameters

[41:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2484s) so uh so if you multiply it by two bytes

[41:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2487s) so it so weight of it's around is 14

[41:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2491s) 14.5 GB which can easily fit into H100

[41:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2495s) or even a a40 so

[41:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2499s) so Next uh what we can do is that like

[41:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2503s) mistral 7B instead of compressing it a

[41:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2506s) floating point 16 we can apply different

[41:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2509s) techniques like int 8 or int4 or nf4. So

[41:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2513s) basically we have to just use ostrich

[41:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2515s) algorithm and just believe that uh there

[41:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2518s) is no quality loss kind of things but

[42:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2521s) somehow we also have to mathematically

[42:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2523s) prove that by doing some kind of test

[42:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2525s) testing on some external benchmark that

[42:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2527s) whether it is working or not. So the and

[42:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2532s) this comes under post training

[42:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2534s) quantization kind of thing. One can also

[42:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2536s) do uh this one uh during finetuning one

[42:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2540s) can also do this kind of quantization.

[42:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2542s) This comes under a quant training kind

[42:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2544s) of thing. So uh let's move to our next

[42:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2548s) problem.

[42:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2551s) So

[42:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2554s) we have this huge matrices just just

[42:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2558s) imagine imagine uh 1,000 by 1,000 uh

[42:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2563s) dimension matrix A and another matrix

[42:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2567s) matrix um 1,000 by 1,000. So if we

[42:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2571s) multiply uh if we multiply by this two

[42:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2574s) matrices so number of operations will be

[42:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2577s) 1,000 raised to the power q and this is

[43:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2581s) kind of a problem in terms of uh uh in

[43:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2585s) terms of computing. So we wondered our

[43:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2587s) matrix multiplication should be fast and

[43:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2590s) it should save memory. So what should we

[43:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2594s) do? We have a giant matrix. Okay, let's

[43:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2598s) take this one. Uh, Mr. 4096 by 4096.

[43:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2602s) What should we do uh to

[43:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2605s) solve our problem of speeding up the

[43:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2608s) things and saving the memory 4096 by

[43:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2611s) 4096.

[43:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2613s) So first thing is that we will use just

[43:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2615s) our world cup algorithm. We can decide a

[43:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2618s) random number just break the block

[43:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2621s) vertically. It does not matter what you

[43:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2623s) are choosing it. So you have so let's

[43:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2627s) say uh we have 4096 uh columns we will

[43:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2631s) break it uh we will break it into a

[43:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2634s) group of 128 column each. So 128, 128, 128

[44:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2640s) 128 uh vertical vert uh vertically so we

[44:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2644s) will get a 30 we will get this 32 blocks

[44:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2647s) if we divide this 4096

[44:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2649s) then

[44:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2651s) what will happen by doing this thing? So

[44:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2653s) if we just divide this one vertical

[44:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2656s) vertically then we can use a multiple

[44:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2659s) GPU to speed up the process. So this

[44:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2662s) kind of thing is called multi head

[44:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2665s) attention.

[44:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2667s) So what else can we do? We have a big

[44:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2670s) matrix like

[44:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2673s) uh as I have mentioned that ostrich

[44:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2675s) algorithm. So our main problem is

[44:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2679s) sizing. So what we what we can do is

[44:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2682s) that instead of having all those 32 uh

[44:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2685s) 32 vertical blocks we will throw away uh

[44:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2688s) 31 blocks and we will assume that one

[44:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2691s) block is sufficient enough that all the

[44:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2694s) queries uh can handle those blocks. Our

[44:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2698s) loss will be almost negligible and we

[45:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2702s) come up with this algorithm uh which uh

[45:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2704s) and this algorithm is called a

[45:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2706s) multiquery attention. So as we can see

[45:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2710s) right now we are at two spectrum. One is

[45:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2712s) multi head attention where we split it

[45:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2716s) into 32 blocks and use different uh

[45:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2720s) different uh GPUs or do some parallel

[45:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2722s) processing and at the same time we are

[45:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2725s) just throwing 31 blocks and uh we are

[45:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2728s) calling this is as a multi-query

[45:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2731s) attention. So

[45:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2734s) uh so at both extreme we should be come

[45:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2736s) up with a middle ground like something

[45:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2739s) we can say that

[45:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2741s) instead of throwing all the 31 uh maybe

[45:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2744s) we can group we can group we can group

[45:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2746s) some of the blocks together so that uh

[45:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2750s) uh and we can assume that uh similar

[45:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2753s) blocks will attend to a um similar kind

[45:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2757s) of uh queries. So this kind of technique

[46:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2760s) comes under grouped query attention

[46:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2762s) which is very popular right now. Uh even

[46:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2764s) in uh even in mistral or in other models

[46:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2768s) this grouped query attention works. So

[46:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2771s) right now we have understand that we

[46:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2774s) have a big matrix uh we can divide it

[46:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2776s) the way we want and doing some

[46:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2778s) mathematical calculation prove that loss

[46:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2781s) is almost negligible kind of thing. So

[46:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2783s) what else we can do?

[46:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2786s) So after that uh after this grouped

[46:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2789s) query attention

[46:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2792s) uh

[46:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2794s) see

[46:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2796s) uh we have a big matrix uh one is one is

[46:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2800s) key and one is value.

[46:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2802s) Let's compress that matrix into a latent

[46:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2806s) vector and then come up with some

[46:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2809s) algorithm to uh reconstruct from latent

[46:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2812s) vector uh to our original matrix. So

[46:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2815s) this kind of a strategy comes under this

[46:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2818s) one um multi head latent latent

[47:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2821s) attention but again it has some problems

[47:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2824s) with rope because rope is position

[47:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2826s) dependent and uh and it is position

[47:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2828s) independent kind of thing. So yeah one

[47:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2831s) needs to also include some uh index for

[47:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2833s) keys also so that one can map it. But

[47:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2837s) again main problem is that why why we

[47:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2841s) are why we are multiplying all those big

[47:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2844s) matrices. So because that's how this

[47:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2847s) attention mechanism works that

[47:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2850s) each token will pay attention to every

[47:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2853s) token. So how about let's don't pay

[47:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2856s) attention to all the previous token only

[47:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2858s) pay attention to the important tokens uh

[47:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2861s) which is important for us. So this is a

[47:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2864s) kind of uh this kind of field is uh

[47:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2867s) evolving. So this comes under sparse uh

[47:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2869s) deepseek sparse attention. So

[47:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2873s) uh yeah and yeah yeah yeah so okay next

[48:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2880s) yeah so next one is flash attention. So

[48:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2884s) uh so in flash attention so main pro so

[48:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2887s) main problem is that uh

[48:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2891s) uh so so currently so so currently not

[48:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2894s) currently so right now almost everyone

[48:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2896s) uses flash attention but way in 2022 or

[48:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2899s) 2023 uh so that's how it works that's

[48:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2903s) how it works is that uh so

[48:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2908s) uh this

[48:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2909s) Q K query and A and key matrices they

[48:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2913s) were in HBM. Uh it loads uh it uh first

[48:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2918s) uh it loads into uh this one uh tensor

[48:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2921s) core and it do some uh it do some

[48:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2924s) calculation and then it will uh write it

[48:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2926s) back to uh HBM and then uh this process

[48:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2929s) goes on multiple times. So in flash

[48:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2932s) attention uh what they did is that

[48:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2936s) uh is that instead of multiplying the

[48:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2938s) whole matrices so they just divided it

[49:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2941s) into like our world cup algorithm

[49:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2943s) divided the bigger matrices into a small

[49:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2946s) tile and only put those small tiles uh

[49:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2948s) into a SBM so that uh it can process

[49:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2952s) multiplication fast and just uh keep uh

[49:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2956s) keeping track of this some three

[49:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2957s) variables so that they can calculate

[49:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2959s) this online softmax.

[49:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2963s) Yeah.

[49:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2965s) Next one. So, yeah. So, so this is just

[49:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2968s) mathematics. So, if we have a multi head

[49:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2971s) attention if it is 524

[49:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2974s) uh KV uh then it depends upon how much

[49:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2978s) how much grouping we want and so if

[49:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2982s) instead of 32 KV head we only want to

[49:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2986s) use uh 8 KV heads. So uh so so we can

[49:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2990s) get a compression of 4x times and this

[49:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2993s) multi head latent attention this formula

[49:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2996s) depends on the model to model how many

[49:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=2999s) layers your model have. So in the

[50:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3001s) original deepseek paper uh I think they

[50:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3003s) have some 128 dimension

[50:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3007s) 128 d 12 I don't remember the exact

[50:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3010s) dimension but according to that uh they

[50:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3013s) have used uh this one latent vector in

[50:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3016s) which they have used 512 as a dimension

[50:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3019s) and some 64

[50:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3021s) for for rope index. So and then they

[50:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3025s) show that it is a 50x 56x

[50:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3029s) uh more compressed than multi head

[50:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3032s) attention.

[50:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3037s) Okay.

[50:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3039s) Yeah. So uh so yeah so this is uh so

[50:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3042s) this is the uh this is the trade-off uh

[50:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3045s) trade-off diagram. So here I think we

[50:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3046s) have not talked about this linear

[50:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3048s) attention or mamba. So main problem is

[50:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3052s) just all this m Matrix multiplication.

[50:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3055s) Right now everyone is using attention.

[50:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3057s) Suppose in future

[51:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3060s) uh if we don't want to use attention or

[51:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3063s) rather than generating tokens

[51:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3065s) sequentially just use maybe diffusion

[51:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3067s) models where we can generate everything

[51:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3069s) simultaneously. So all these algorithms

[51:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3072s) will change also. But here I think they

[51:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3075s) have two more. One is linear attention

[51:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3077s) and one is mamba. So according to uh

[51:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3080s) this slide so

[51:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3083s) if we are not compressing anything so

[51:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3085s) MHA is just we are parallelizing the

[51:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3088s) process so there is no quality loss so

[51:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3091s) it's a good and then this uh grouped

[51:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3094s) query attention which is I think almost

[51:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3096s) uh every model is using uh just GQA and

[51:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3100s) DSA kind of thing or yeah

[51:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3105s) I think same thing we are providing in

[51:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3106s) the attention mechanism scorecard So uh

[51:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3110s) so I think uh this one mha quality is

[51:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3113s) good throughput is uh throughput is okay

[51:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3116s) and for grouped query attention it

[51:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3119s) depends upon your use case also though

[52:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3122s) yeah though

[52:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3124s) quality is almost similar to uh multi

[52:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3127s) head attention but use case also matters

[52:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3129s) a lot yeah multi-query attention is just

[52:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3132s) one extreme we are

[52:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3135s) I don't know why but we are just

[52:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3137s) assuming that we only need one block and

[52:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3140s) all the queries will attend to that

[52:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3142s) smaller smaller block. So, so quality is

[52:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3146s) not that great for M for MQA and this

[52:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3150s) multi head latent attention. So yeah if

[52:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3154s) you have tried some this deep seat

[52:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3156s) models so I think uh they are doing

[52:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3158s) great job yeah in in quality wise

[52:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3162s) besides that sliding window so all these

[52:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3165s) are sub techniques which

[52:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3168s) yeah yeah all these are some techniques

[52:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3170s) like I just slide the windows all those

[52:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3173s) things and instead of yeah instead of

[52:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3176s) multiplying everything so linear

[52:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3179s) attention is just saying that sum

[53:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3180s) summarize everything first uh and then

[53:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3183s) look up into it and then mamba this is

[53:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3186s) just a state space model. Yeah,

[53:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3192s) I can cover that. Okay.

[53:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3195s) Uh cool. Uh thank you T.

[53:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3198s) So for the model like optimizations we

[53:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3202s) also have like the two notebooks

[53:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3205s) here.

[53:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3208s) So there will be

[53:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3212s) I have to go to this.

[53:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3222s) Okay. Uh so for the quantization uh like

[53:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3227s) the demo

[53:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3229s) uh this is is this already run? No. Let

[53:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3233s) me just run this.

[54:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3243s) Okay. So we are loading the model which

[54:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3245s) is like uh ML 7B.

[54:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3252s) Uh so

[54:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3257s) this one is like with the FP16 baseline.

[54:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3269s) Wait.

[54:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3271s) Uh, did it run?

[54:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3275s) Okay. So, it's uh two millisecond run.

[54:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3280s) Did this run? Okay. So, yeah, this time

[54:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3283s) it's fetching that model with the FP16

[54:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3287s) precision.

[54:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3297s) the Wi-Fi.

[55:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3303s) It's going to take time.

[55:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3307s) Okay.

[55:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3311s) >> Yeah, it because it's downloading the

[55:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3313s) weights from the hugging face.

[55:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3317s) >> Huh.

[55:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3320s) Yeah. So, MOLAB is like running online.

[55:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3323s) >> Yes.

[55:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3326s) because it needs to make the network

[55:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3327s) call through to the hugging phase and

[55:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3329s) like it fetching

[55:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3332s) I don't know like but it's taking time

[55:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3334s) to download probably

[55:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3352s) Okay.

[55:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3355s) So good. Okay. So here we see like the

[55:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3359s) memory size is like 15 GB around

[56:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3362s) approximately with the FP16 precision.

[56:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3365s) We are trying to do the 2x compression

[56:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3368s) as Tmet talked about with the int8.

[56:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3373s) Let's download. Okay. So we do see like

[56:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3376s) your memory size is now like 7.5 GB.

[56:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3381s) What that means is now you have a more s

[56:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3383s) more memory for your KV to basically

[56:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3386s) grow. That means you can either serve

[56:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3388s) higher context limit or you can serve

[56:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3390s) the higher concurrent users there.

[56:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3396s) If you do the like in your basic you are

[56:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3399s) doing the 4x compression so that with

[56:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3402s) the 4x compression it would be more

[56:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3405s) lower. It would be I think around

[56:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3408s) 3 to 4 GB.

[56:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3411s) Yeah. 4.5 GB

[56:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3415s) and Yep. So this is Wait.

[57:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3422s) So this is just a basic plot

[57:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3425s) of like

[57:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3427s) so these are the like theoretical

[57:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3429s) numbers. uh we are not doing the like

[57:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3431s) any throughput test here but uh usually

[57:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3433s) you would see like your memory increases

[57:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3435s) so pro you would also have like a bit of

[57:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3438s) higher uh throughput.

[57:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3441s) Uh from some of the benchmarks that we

[57:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3443s) studied we saw like the intate uh

[57:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3446s) compression it does have like a lower

[57:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3448s) throughput.

[57:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3452s) Okay. And then there is like a demo on

[57:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3456s) the like the attention mechanisms.

[57:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3462s) So for the attention okay I have to run

[57:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3467s) this.

[57:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3478s) Uh okay so it has run. Oh, wait. Why

[58:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3484s) does it say no GPU detected?

[58:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3489s) It should say the GPU should be

[58:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3491s) detected.

[58:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3498s) Oh, okay.

[58:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3517s) Wait,

[58:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3530s) but this is surprising.

[58:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3537s) Yeah, I guess it's not like able to

[58:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3539s) detect the GPU for some reason.

[59:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3544s) Uh we do have like a GPU here.

[59:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3551s) Uh okay, never mind.

[59:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3554s) Yeah. Yeah. So, but the like basic idea

[59:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3556s) here was more like

[59:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3560s) as you try to move towards like

[59:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3562s) compressing the computation like by

[59:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3565s) using different attention mechanisms

[59:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3567s) like moving from the multi head to the

[59:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3570s) grouped query attention and then to the

[59:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3573s) MLA you would start seeing some

[59:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3575s) optimizations.

[59:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3578s) Um I think yesterday night we were doing

[59:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3581s) some benchmarking. Uh I wanted to

[59:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3584s) correct this part. Uh so it wasn't like

[59:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3587s) 50 56x it was 14x. Uh basically the demo

[59:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3592s) had a mistake of like a computation uh

[59:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3595s) where it did not multiply the number of

[59:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3598s) layers.

[1:00:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3601s) Uh yeah so apologies for that. Uh so

[1:00:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3604s) this MLA is like a 14x savings work in

[1:00:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3608s) comparison to like your multi head

[1:00:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3610s) attention.

[1:00:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3614s) Uh so now that we have understanding of

[1:00:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3618s) the pain points, the foundations, the

[1:00:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3621s) one side of the optimizations which is

[1:00:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3623s) the model optimizations,

[1:00:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3625s) we want to talk about what can you do on

[1:00:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3629s) the like the serving side.

[1:00:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3633s) So

[1:00:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3636s) the first thing is we saw like when you

[1:00:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3638s) perform like a simple decode step you

[1:00:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3642s) are pulling it you are basically pulling

[1:00:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3644s) the model weights and then you are

[1:00:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3645s) recomputing the key and the value

[1:00:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3647s) vectors for all the previous tokens even

[1:00:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3651s) though you already computed the those

[1:00:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3653s) vectors for the tokens.

[1:00:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3656s) So there is definitely like a lot of

[1:00:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3659s) compute wastage.

[1:01:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3661s) Uh and if you kind of analyze the time

[1:01:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3664s) complexity of it, it would come out to

[1:01:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3666s) be O of N². Uh and the way to resolve

[1:01:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3670s) that is like a classic trade-off against

[1:01:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3671s) the memory. You can maintain a memory of

[1:01:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3675s) those vectors against the tokens and you

[1:01:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3678s) can reference that memory. So that

[1:01:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3680s) memory was called as like KV cache.

[1:01:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3684s) uh and the like the flow looks something

[1:01:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3686s) like this

[1:01:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3690s) and then based on this KV cache there

[1:01:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3692s) were like four optimizations that were

[1:01:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3694s) really possible.

[1:01:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3696s) Um

[1:01:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3698s) the first one is about the page

[1:01:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3701s) detention. So what's the different

[1:01:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3704s) what's the problem today? So when you

[1:01:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3706s) send like multiple requests as the input

[1:01:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3708s) to the GPU

[1:01:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3710s) these requests are in a batch

[1:01:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3714s) uh every request is allocated like a

[1:01:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3717s) continuous memory storage let's say of

[1:02:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3721s) I'm just taking an example like let's

[1:02:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3723s) set uh 2 KB

[1:02:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3726s) however like your request needed only

[1:02:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3728s) let's say

[1:02:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3730s) uh 1 KB so there is like u 50% of that

[1:02:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3736s) memory fragmentation.

[1:02:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3739s) Uh and this fragmentation basically

[1:02:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3742s) leads to the memory wastage. That means

[1:02:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3746s) there was a space in the memory where

[1:02:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3748s) you could have served more requests but

[1:02:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3751s) you could not because you were looking

[1:02:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3753s) for that contigious block of the memory.

[1:02:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3756s) So an inspiration to was being taken

[1:02:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3759s) from like how the OS works like you

[1:02:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3762s) maintain a logical memory and you

[1:02:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3764s) basically have a physical memory.

[1:02:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3767s) So in the logical memory it would still

[1:02:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3771s) feel like

[1:02:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3773s) uh that the KV vector for the like every

[1:02:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3777s) token is like a contiguous

[1:03:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3780s) but it will be mapping to a different

[1:03:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3782s) physical address.

[1:03:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3786s) So that really helped like saving a lot

[1:03:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3790s) of memory. Uh and it was only possible

[1:03:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3794s) because you they considered like memory

[1:03:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3796s) as a set of blocks and you would be

[1:03:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3799s) dynamically allocating those blocks as

[1:03:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3801s) the request need as the like new tokens

[1:03:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3805s) comes in and they need that kind of

[1:03:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3807s) memory.

[1:03:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3810s) The another lever is like when you are

[1:03:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3814s) sending multiple requests

[1:03:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3817s) in the batch

[1:03:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3819s) GPU is like taking those requests

[1:03:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3824s) but it does not accepts the new batch

[1:03:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3826s) unless all the requests in that batch

[1:03:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3828s) gets completed. So the diagram looks

[1:03:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3831s) more like a page retention but here it

[1:03:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3833s) is more about like when is GPU available

[1:03:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3837s) to take the next batch. So there is a

[1:04:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3841s) time period where GPU is like sitting

[1:04:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3843s) really idle

[1:04:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3845s) and you want to like resolve for that

[1:04:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3849s) and for that like the idea was like okay

[1:04:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3851s) let's do that continuous batching.

[1:04:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3857s) So the continuous batching also really

[1:04:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3859s) helped with like throughput because now

[1:04:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3861s) you can ship more requests pretty

[1:04:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3864s) quickly. Keep making sure like GPU

[1:04:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3866s) always uh get is always like occupied

[1:04:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3870s) and it's not like uh sitting idle. So

[1:04:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3873s) you are saving on that compute.

[1:04:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3876s) The third is the like prefix caching. So

[1:04:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3879s) you remember like the KV cache helped

[1:04:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3881s) you save the computation for a single

[1:04:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3884s) request across the tokens.

[1:04:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3887s) But what if like you have the same

[1:04:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3890s) tokens across multiple requests? How do

[1:04:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3893s) you basically save against that? So the

[1:04:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3896s) prefix caching uh which was introduced

[1:04:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3899s) by VLM

[1:05:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3903s) exactly counters that

[1:05:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3906s) and then the third is like we talked

[1:05:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3909s) about the fourth actually. So we talked

[1:05:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3912s) about quantizing the model

[1:05:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3915s) but you could also you can also like

[1:05:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3918s) quantize the KV weights.

[1:05:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3921s) So that means now you you need like a

[1:05:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3924s) lesser space for your key and the value

[1:05:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3927s) vectors. That means you can serve more

[1:05:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3929s) key and the value vectors in the memory.

[1:05:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3932s) And that means like you can serve more

[1:05:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3934s) tokens. That means you can serve more

[1:05:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3936s) context context limit. And that means

[1:05:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3938s) like you can serve more model quality

[1:05:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3945s) and all of this is like uh already

[1:05:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3949s) present in the VLM.

[1:05:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3951s) You don't really need to reinvent that

[1:05:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3954s) wheel

[1:05:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3956s) uh and you can like deploy this VLM in

[1:05:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3959s) production and you could see that

[1:06:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3962s) basically growth.

[1:06:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3965s) So next we have like a benchmark that we

[1:06:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3968s) did. So this benchmark was

[1:06:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3972s) let me see if I have that

[1:06:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3977s) here

[1:06:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3979s) the demos.

[1:06:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3983s) So doing this benchmark takes like

[1:06:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3986s) around 1 hour because you have to

[1:06:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3988s) continuously stop and like restart the

[1:06:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3990s) VLM servers and you have to load the

[1:06:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3993s) models and all. So it does take a lot of

[1:06:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3996s) time in doing the testing but I can like

[1:06:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=3999s) really tell you here what we are doing.

[1:06:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4002s) So we have kept the model as same like

[1:06:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4005s) the Mistful 7B.

[1:06:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4008s) Uh and then we have like the set of

[1:06:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4011s) input questions that we are sending. Uh

[1:06:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4015s) consider them as the prompts. Then we

[1:06:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4017s) have couple of helper functions here

[1:06:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4019s) like checking the server is up or not.

[1:07:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4022s) The server is the VLM server.

[1:07:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4025s) Then there are helper functions to get

[1:07:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4028s) the VLM metrics.

[1:07:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4030s) uh and I will talk about like what those

[1:07:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4032s) metrics are. Uh then there are like lot

[1:07:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4035s) of the benchmarks and all and then you

[1:07:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4038s) have to measure uh the KV usage and all.

[1:07:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4043s) So these are the like helper functions.

[1:07:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4045s) So the baseline is very simple like we

[1:07:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4047s) have a hugging phase baseline.

[1:07:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4050s) Uh this is the raw like sending the text

[1:07:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4053s) to the LLM getting back the response. We

[1:07:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4057s) see some results here. We saw like

[1:07:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4060s) hugging phase has a throughput of like

[1:07:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4062s) around 51 tokens per second. Time to

[1:07:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4064s) first token was like 54 and then the

[1:07:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4066s) inter token latency was 19.

[1:07:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4070s) Uh this bas uh this was all run on the

[1:07:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4072s) h100.

[1:07:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4075s) Uh and then we start like a very default

[1:07:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4079s) VLM server. So by default VLM provides

[1:08:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4082s) you the page detention, continuous

[1:08:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4084s) batching

[1:08:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4086s) and the KV caching.

[1:08:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4088s) So three things are present by default

[1:08:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4092s) and when you try to compare those

[1:08:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4095s) benchmarks you see your throughput is

[1:08:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4098s) like almost 15x you are able to serve

[1:08:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4102s) more tokens per second

[1:08:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4105s) then

[1:08:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4106s) your time to the first token

[1:08:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4110s) uh that also rises

[1:08:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4114s) and then your v the inter token latency

[1:08:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4117s) kind goes down and then your KV versus

[1:08:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4121s) users and the versus context rate

[1:08:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4122s) increases for sure.

[1:08:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4127s) Now when you apply the prefix caching to

[1:08:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4130s) it

[1:08:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4132s) so with the prefix caching you see like

[1:08:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4134s) your throughput increases

[1:08:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4137s) more your TDF decreases your inter token

[1:09:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4141s) latency is approximately same uh and

[1:09:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4144s) then your KV cache usage versus the

[1:09:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4147s) users it's kind of going down

[1:09:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4150s) the vers context it's not going down

[1:09:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4152s) it's approximately same I think this is

[1:09:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4155s) also approximately same it's like not

[1:09:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4158s) that uh big of a deal

[1:09:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4161s) when you apply the like KV quantization

[1:09:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4164s) on top of it.

[1:09:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4167s) So it becomes like so so you see like

[1:09:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4171s) your throughput is like almost similar.

[1:09:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4174s) Your time to first token is similar.

[1:09:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4176s) Your token latency is similar but then

[1:09:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4180s) your KV usage actually goes down. And

[1:09:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4182s) this is because like you have quantized

[1:09:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4185s) your key value space.

[1:09:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4188s) Uh and then there is a concept of

[1:09:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4191s) speculative decoding that TME will talk

[1:09:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4194s) about. Uh

[1:09:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4196s) so when you try to benchmark those so

[1:10:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4200s) you also see like there is a uh like a

[1:10:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4203s) bit of like the less KV usage there

[1:10:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4206s) although like the results are

[1:10:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4207s) approximately same.

[1:10:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4215s) So yeah, I mean overall like these are

[1:10:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4219s) the like the metrics across probably I

[1:10:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4222s) should

[1:10:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4225s) zoom out. Okay, it's not zoom out. It's

[1:10:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4227s) not working.

[1:10:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4230s) Great. So yeah, this is the like VLM

[1:10:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4234s) benchmarks. Um it's your production

[1:10:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4236s) default by the way. uh we will also

[1:10:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4239s) share that decision tree uh when we try

[1:10:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4243s) to talk about like the other engines.

[1:10:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4249s) So

[1:10:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4251s) yeah, so we should talk about like what

[1:10:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4254s) are some of the other inference

[1:10:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4256s) optimizations we can do on top of it and

[1:10:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4259s) what were some of the other solutions

[1:11:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4261s) that came out.

[1:11:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4265s) Uh so I would like to again invite

[1:11:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4267s) Tanme. He's going to talk about like

[1:11:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4270s) some of these optimizations.

[1:11:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4278s) Oh, sorry. Uh I'm so sorry. Uh I didn't

[1:11:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4281s) enable the slides.

[1:11:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4285s) Uh what was the Okay, great.

[1:11:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4290s) Perfect.

[1:11:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4290s) >> Which one?

[1:11:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4291s) >> The speculative.

[1:11:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4292s) >> Yeah. Thank you, Hersel. Yeah.

[1:11:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4296s) So,

[1:11:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4297s) so all these are like speculative

[1:11:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4299s) decoding all these are the uh so so what

[1:11:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4303s) we say uh different flavors of same kind

[1:11:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4306s) of soda. So this uh this technique comes

[1:11:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4310s) under decoding accelerator. So first one

[1:11:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4313s) so we are only talking about this

[1:11:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4315s) speculative decoding but there are other

[1:11:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4318s) variants like self speculative eagle

[1:12:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4321s) medusa

[1:12:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4322s) I only like I think uh this one eagle

[1:12:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4325s) algorithm

[1:12:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4327s) personally I don't think

[1:12:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4330s) speculative decoding works because main

[1:12:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4332s) problem is alignment okay so let's start

[1:12:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4335s) with what is uh speculative decoding

[1:12:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4338s) main problem is that in transformer

[1:12:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4340s) architecture All these tokens are

[1:12:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4342s) generated sequentially one by one by

[1:12:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4346s) one. How about just use a smaller model

[1:12:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4350s) and let a smaller model to generate

[1:12:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4354s) maybe let's say four or five tokens and

[1:12:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4358s) this teacher model or we can say

[1:12:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4360s) according to our world cup algorithm we

[1:12:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4362s) can say referee. So referee will decide

[1:12:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4365s) how many uh tokens it accept and this

[1:12:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4369s) loop keeps on going on and our

[1:12:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4373s) assumption is that there are certain

[1:12:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4376s) domain where this kind of things will

[1:12:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4379s) work like maybe in decode maybe in

[1:13:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4381s) coding or where almost there is no

[1:13:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4384s) creativity uh each uh code or syntax is

[1:13:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4388s) almost similar. So maybe it can help it.

[1:13:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4391s) But uh based on personal testing, I

[1:13:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4394s) didn't find this speculative decoding

[1:13:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4397s) useful at all. But other techniques like

[1:13:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4401s) uh self speculative decoding where

[1:13:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4404s) teacher model also have one head

[1:13:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4407s) auxiliary head and it will do same

[1:13:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4409s) similar kind of things what this base

[1:13:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4412s) model or small model is doing it. But

[1:13:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4416s) then this eagle came Eagle 1 2 3 I don't

[1:13:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4419s) know how many version versions are but

[1:13:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4422s) it is just saying that instead of

[1:13:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4425s) creating instead of generating tokens uh

[1:13:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4427s) let's uh train a small model inside

[1:13:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4430s) train a small model and just take a

[1:13:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4433s) features from one of its uh one of main

[1:13:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4437s) models layer so that instead of

[1:13:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4439s) generating token uh it will generate uh

[1:14:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4441s) this features so so uh so eagle is uh

[1:14:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4445s) Eagle is better compared to this other

[1:14:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4449s) kind of technologies and then another

[1:14:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4451s) one is Medusa which is just saying that

[1:14:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4454s) just generate all the tokens parallelly.

[1:14:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4458s) Uh okay. So here so here in this slide

[1:14:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4463s) >> yeah the next slide.

[1:14:25](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4465s) >> Okay.

[1:14:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4469s) Okay. Yeah. Okay. Now we come to uh now

[1:14:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4473s) we will come to this one prefix caching.

[1:14:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4475s) So I don't know whether people are using

[1:14:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4477s) this one static prefix caching or not

[1:14:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4479s) but thing is that main problem with

[1:14:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4482s) prefix caching is that sometimes we type

[1:14:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4486s) and make a small kind of mistake and

[1:14:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4488s) this standard static prefix caching is

[1:14:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4491s) basically it takes a prompt do some

[1:14:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4493s) hashing and then next time when user

[1:14:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4495s) asks similar kind of question it will

[1:14:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4497s) try to match the hash. So if hash is uh

[1:15:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4501s) if hash is equal then then it will

[1:15:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4504s) instead of recomputing all those K and B

[1:15:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4506s) it will just uh take it from from the

[1:15:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4509s) storage but you know that sometimes we

[1:15:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4511s) make a mistake or maybe we can just

[1:15:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4513s) change a word or letter something like

[1:15:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4515s) that then we have a very higher uh cache

[1:15:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4519s) uh hit cache miss hit rate so that's why

[1:15:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4523s) uh this one uh radics tree so radics

[1:15:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4526s) tree is becoming very popular and also

[1:15:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4529s) also because of agent. So I think almost

[1:15:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4532s) everyone is doing agent and most of the

[1:15:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4534s) computation is going during TT during

[1:15:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4537s) test time inference kind of thing where

[1:15:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4539s) we keep on asking same kind of questions

[1:15:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4542s) and prompt for example you are an expert

[1:15:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4545s) software engineer multiply by 200 times.

[1:15:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4549s) This kind of loop keeps on going inside

[1:15:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4552s) this uh agentic agentic kind of things

[1:15:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4555s) where it is al necessary to keep uh or

[1:15:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4559s) store similar kind of things in a radics

[1:16:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4563s) tree. So radics tree is just so so radic

[1:16:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4567s) tree is just advanced version of this

[1:16:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4569s) prefix tree where where we will just

[1:16:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4572s) where we will just collapse a node if it

[1:16:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4574s) does not have a does not have any branch

[1:16:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4578s) and for this kind of work where keep on

[1:16:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4583s) repeating same thing this uh red x tree

[1:16:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4588s) helps a lot and st lang uh use this kind

[1:16:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4592s) of algorithm

[1:16:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4593s) for prefix caching.

[1:16:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4597s) Okay. Yeah. Then there is another thing.

[1:16:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4599s) One is tensor RT LLM. This is very

[1:16:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4602s) confusing. When I first started, I was I

[1:16:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4606s) was just confused. What is tensor RTLM?

[1:16:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4609s) So yeah. So tensor RT is just a uh it's

[1:16:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4613s) just a standard uh SDK kind of thing.

[1:16:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4616s) Tensor RTLM is just an inference engine

[1:16:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4619s) just like VLM, SG lang. But problem is

[1:17:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4623s) that it is related to Nvidia. They

[1:17:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4627s) optimized each and every layer and every

[1:17:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4630s) problem as I mentioned in our world cup

[1:17:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4633s) algorithm. They just break everything

[1:17:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4635s) and optimized everything at hardware

[1:17:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4638s) level also. So uh yeah. So okay next.

[1:17:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4647s) Yeah. So for this workshop we also uh

[1:17:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4650s) did some benchmarking like which is best

[1:17:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4654s) uh so our setup was something similar

[1:17:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4657s) was so so we did two kind of testing.

[1:17:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4659s) First one is without uh without agentic

[1:17:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4663s) testing where we just so

[1:17:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4666s) we use this shared GPT uh this one data

[1:17:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4668s) set and uh just ask those questions uh

[1:17:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4673s) using VLM and SG lang.

[1:17:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4677s) Okay.

[1:18:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4686s) Yeah. Okay.

[1:18:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4689s) And let me just zoom it up. Okay, great.

[1:18:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4694s) Okay. Yeah. So, yeah, for this workshop,

[1:18:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4696s) we used H100 and of our first testing

[1:18:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4700s) was that uh we just uh we just asked uh

[1:18:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4704s) we take questions from shared GPT and

[1:18:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4706s) put it into VLM, SG lang and we found

[1:18:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4710s) that actually there's no statistical

[1:18:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4712s) difference between which one is better.

[1:18:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4715s) So both have almost similar kind. So

[1:18:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4718s) both are fulfilling similar kind of

[1:18:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4720s) request per second uh TTFT and latency.

[1:18:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4724s) So but only difference we have seen

[1:18:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4727s) during agentic uh agentic branching. So

[1:18:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4732s) uh what we did was that we asked that

[1:18:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4734s) similar kind of question that you are

[1:18:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4736s) the best this one software engineer in

[1:18:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4738s) the world just solve the problem of

[1:19:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4742s) traffic congestion in this city kind of

[1:19:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4744s) thing. Then we put this into LLM. LLM

[1:19:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4749s) generates some output. Then we did

[1:19:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4751s) another uh round two also. So once this

[1:19:15](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4755s) LLM generates this output, then in round

[1:19:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4759s) two we have specially mentioned that uh

[1:19:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4763s) provide uh review the proposal and give

[1:19:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4766s) ratings from 1 to 10. So this uh two

[1:19:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4770s) turns we did uh and this loop keeps on

[1:19:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4774s) uh repeating it. Uh what we found is

[1:19:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4776s) that for this kind of

[1:19:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4780s) uh workflow where everything is standard

[1:19:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4783s) all those prompts and context

[1:19:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4785s) engineering comes into the picture. If

[1:19:47](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4787s) we do proper this agentic branching then

[1:19:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4790s) I think uh this HG lang is three to four

[1:19:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4793s) times better. But again this depends

[1:19:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4796s) upon the different setup maybe uh if you

[1:19:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4799s) do it uh you may get different results.

[1:20:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4803s) Okay. Yeah. So I think uh did we

[1:20:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4807s) uploaded it on GitHub? Okay.

[1:20:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4810s) >> Yeah. So the PDF is like also in the

[1:20:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4814s) drive. Uh it's the same link as the

[1:20:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4817s) slides.

[1:20:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4819s) So a quick summary here.

[1:20:23](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4823s) So on a standard API workload throughput

[1:20:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4826s) you would see like a VLM and the SG lang

[1:20:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4830s) would behave same. So if you don't have

[1:20:33](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4833s) if you have like a standard workload

[1:20:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4835s) definitely go with VLM. It's the

[1:20:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4837s) production default anyways. But what

[1:20:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4839s) Tanme was also saying is when you try to

[1:20:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4843s) like make it like agentic workloads that

[1:20:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4846s) is where like your SG link really shines

[1:20:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4851s) uh and uh it kind of like provides you

[1:20:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4853s) all the benefits.

[1:20:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4857s) So yeah keep like VLM as a default but

[1:21:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4860s) if you have agentic workloads probably

[1:21:03](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4863s) try to move as the towards the SG lang.

[1:21:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4865s) if you're not happy with DB LLM. Uh but

[1:21:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4872s) uh okay. Uh let me

[1:21:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4881s) Okay.

[1:21:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4886s) And then like there is like the like a

[1:21:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4889s) comparison that is done at the 120

[1:21:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4892s) billion like for the GPTO OSS 120

[1:21:35](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4895s) billion. Um this is a benchmark that was

[1:21:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4899s) prepared by clarify. So there is like a

[1:21:44](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4904s) blog link here. Oh nice.

[1:21:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4908s) Okay. Yeah. So they did the similar

[1:21:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4912s) benchmark and they included like a

[1:21:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4914s) tensor RT LLM in it.

[1:21:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4918s) Definitely you can always go through

[1:22:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4920s) these benchmarks and try to understand

[1:22:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4922s) which basically suits your use case. As

[1:22:06](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4926s) we mentioned like Tensor RT they try to

[1:22:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4928s) optimize the hardware side as well

[1:22:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4930s) having the peak hardware performance.

[1:22:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4937s) Wait uh this is

[1:22:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4942s) okay.

[1:22:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4946s) Yeah.

[1:22:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4947s) And then like in terms of when you want

[1:22:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4950s) to dep pick like your engines

[1:22:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4954s) once you figure out like between VLM, SG

[1:22:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4957s) lang tenserati so that there are some

[1:22:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4959s) new engines that are popping up Nvidia

[1:22:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4962s) Dynamo for sure. Uh so they are also for

[1:22:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4966s) the agentic uh session routing.

[1:22:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4969s) Uh hugging phase is always there. It's a

[1:22:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4972s) simple no server. Then there is like an

[1:22:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4975s) MSAR

[1:22:57](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4977s) engine that was recently proposed by

[1:22:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4979s) Stanford. They are for like the

[1:23:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4982s) multimodel.

[1:23:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4985s) Uh so definitely you could explore those

[1:23:08](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4988s) and when you try to basically just to

[1:23:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4991s) like give a quick summary uh you we

[1:23:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4994s) start with like a baseline

[1:23:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=4997s) we try to find what model could fit our

[1:23:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5000s) use cases.

[1:23:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5002s) Um,

[1:23:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5004s) so you could pick like uh Deep Seek, you

[1:23:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5007s) could pick like don't pick like a

[1:23:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5010s) Mistral 7B. I mean, it's not good. Uh,

[1:23:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5014s) but yeah, so you pick your model and you

[1:23:38](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5018s) want to like have a smaller memory and

[1:23:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5021s) you want to try to fit that bigger model

[1:23:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5023s) into smaller memory so that you could

[1:23:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5025s) save cost on the GPU cost. So you can do

[1:23:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5029s) like all those com quantization

[1:23:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5032s) then you can apply all those serving

[1:23:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5034s) optimizations by using the right serving

[1:23:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5036s) engine under the hood. So that can

[1:23:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5039s) really provide you that

[1:24:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5042s) throughut that you really want.

[1:24:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5047s) And

[1:24:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5050s) now something that you can do uh after

[1:24:13](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5053s) going back home pro because we cannot

[1:24:16](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5056s) like actually go over all the material

[1:24:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5058s) here uh is definitely reading about some

[1:24:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5062s) of the source informations like

[1:24:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5064s) different attention mechanisms different

[1:24:26](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5066s) like these engines like try to just read

[1:24:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5069s) the different benchmarks which are

[1:24:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5071s) present online as well

[1:24:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5074s) and then there are a lot of like

[1:24:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5077s) in-depth guides or the next phases of it

[1:24:40](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5080s) which is like learning about some KV

[1:24:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5083s) eviction strategies. So world is moving

[1:24:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5086s) towards having a separate KV cache

[1:24:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5088s) engineering domain. So you want to

[1:24:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5091s) understand what's going on in there. So

[1:24:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5093s) KV cache KV eviction cache compressions

[1:24:56](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5096s) hybrid memories. So there are like lot

[1:24:59](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5099s) of solutions that are happening around

[1:25:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5100s) there. So always try to stick to those

[1:25:04](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5104s) foundations or like the fundamentals or

[1:25:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5107s) the first principles and try to see like

[1:25:10](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5110s) which solution basically solves what

[1:25:12](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5112s) problem and whether you actually need

[1:25:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5114s) that problem to be solved for your use

[1:25:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5117s) case

[1:25:18](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5118s) and then there is like distributed LLM

[1:25:21](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5121s) inference which is like a different

[1:25:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5124s) painoint altogether. Uh you would

[1:25:27](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5127s) probably need like a two-hour workshop

[1:25:29](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5129s) there as well.

[1:25:31](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5131s) uh to like go over like all the

[1:25:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5134s) internals do all the hands-on.

[1:25:41](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5141s) Yes. And this is something we are trying

[1:25:43](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5143s) to propose for the AI engineer New York

[1:25:45](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5145s) session uh which is to like dive deeper

[1:25:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5148s) into the advanced sections of the LLM

[1:25:50](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5150s) inference. So this workshop was more for

[1:25:53](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5153s) the like beginner and the intermediate

[1:25:55](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5155s) level. Um so in this form we do have

[1:25:58](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5158s) like a feedback as well plus also the

[1:26:01](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5161s) interest. Um if uh you think like we

[1:26:05](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5165s) need certain improvements on certain

[1:26:07](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5167s) sections definitely give that feedback

[1:26:09](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5169s) as well and if you want to see this

[1:26:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5171s) workshop in like New York uh fair you I

[1:26:17](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5177s) mean definitely feel free to enroll your

[1:26:19](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5179s) interest.

[1:26:22](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5182s) Uhhuh.

[1:26:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5184s) How is it possible?

[1:26:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5188s) Well,

[1:26:34](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5194s) let me just check.

[1:26:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5206s) >> Huh?

[1:26:48](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5208s) >> URL works, right? Not the QR code. Okay.

[1:26:52](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5212s) Probably I forgot to link those two

[1:26:54](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5214s) together.

[1:27:00](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5220s) Z

[1:27:02](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5222s) G A five.

[1:27:11](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5231s) Okay, cool. Yes. So if you can give that

[1:27:14](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5234s) feedback let me just okay

[1:27:20](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5240s) that will be fine um and yeah I think we

[1:27:24](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5244s) would like to wrap this workshop then

[1:27:28](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5248s) I'm sure like lot of you would be having

[1:27:30](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5250s) a lot of questions so we can take all

[1:27:32](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5252s) those like offline uh we can meet uh and

[1:27:36](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5256s) we can uh like talk about those

[1:27:37](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5257s) questions.

[1:27:39](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5259s) >> Yeah sure. Uh thank you everyone. Thanks

[1:27:42](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5262s) for joining. Uh I think it was really

[1:27:46](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5266s) meaningful and all of you like came

[1:27:49](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5269s) here. Uh thanks a lot.

[1:27:51](https://www.youtube.com/watch?v=y2W4FNAuPEA&t=5271s) >> Yeah. Thanks.
