# The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Byung-Gon (Gon) Chun, FriendliAI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/Hvb2LfMH58c-frontier-ai-inference-cloud-agents) · [Video](https://www.youtube.com/watch?v=Hvb2LfMH58c)

## The Frontier AI Inference Cloud for Agents

Byung-Gon (Gon) Chun explains how FriendliAI serves agents by reusing growing context, preserving cache locality, and scheduling model calls around the time needed to finish a task.

### One tower defense game, two different bills

[0:12](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=12s)

Byung-Gon (Gon) Chun, founder and CEO of FriendliAI, introduces agentic inference from the serving side. FriendliAI grew out of a research team at Seoul National University; Chun credits that team with inventing continuous batching and describes its **Orca** work as an influence on **vLLM**. The question now is how inference infrastructure should change when agents become its customers. [0:41](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=41s)

Chun sees two developments coming together: agents spreading through software, operations, and knowledge work, and open-weight models becoming capable enough to make those workflows economical. A coding example gives that claim a concrete test. The same agent receives the same assignment—build a tower defense game—with an open-weight **GLM 5.2** model served by FriendliAI and an Anthropic **Opus 4.8** model. Chun describes both results as clearly usable, without claiming that the games are identical. GLM 5.2 and Opus 4.8 are the model labels Chun reports for these demonstrated runs. Exact checkpoints and evaluation conditions are not supplied, so this is a speaker-reported demonstration rather than an independently reproduced benchmark. [2:02](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=122s)

The bills differ substantially: about $1.50 for the Opus 4.8 run and 27 cents for the GLM 5.2 run on FriendliAI, a reported 5.6-fold cost difference. For this assignment, each result clears the practical quality threshold, making the cheaper run useful. That is the promise of open weights in an agent workflow. But the model bill is only part of the story: the serving system also determines how quickly and reliably the agent reaches its finished result.

### The latency clock runs until the task finishes

[3:16](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=196s)

Chat gives an inference service a simple unit of work: a person asks a question, the model answers, and the person reads the response. Request latency measures that wait. An agent’s assignment can contain many model calls and tool calls, continuing autonomously between them. Its user is waiting for completed work, so the main performance measure becomes **end-to-end task latency**. A fast response helps insofar as it advances the task. [3:46](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=226s)

The task typically runs through a plan–act–observe loop. Planning invokes the LLM; acting may invoke a tool; observing appends the tool’s result to the context before the next planning call. This creates gaps between inference calls while tools execute. The agent can also create sub-agents that run in parallel. Requests therefore arrive from an ongoing program whose next move depends on earlier results, rather than from a fixed sequence of independent questions.

### Every observation adds context—and repeats old work

[5:01](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=301s)

FriendliAI’s internal coding-agent runs show long inputs that grow as the task progresses. Each observation joins the next model input, so consecutive steps share a large beginning, or **prefix**, followed by newly appended material. Processing the entire input again means spending compute on context the service has already processed. In Chun’s words, it is work “we already did.” [5:31](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=331s)

A long-horizon research task makes the accumulation easier to understand: an agent is asked to explain a decoding framework in vLLM. The work has multiple stages, each containing sub-agents that make inference calls and tool calls. Tasks of this kind can involve tens or hundreds of inference steps over minutes or hours, while shared context continues through the work. Repeatedly processing that history adds cost and waiting time throughout the chain.

Three properties complicate serving: context grows, tool execution interrupts the sequence of model calls, and the number of calls depends on the input. A fixed request-rate assumption cannot capture the whole workload. The inference service needs to help an evolving task finish, including preserving useful state between calls when the agent is busy elsewhere.

### Compute the prefix once, then keep it available

[7:03](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=423s)

FriendliAI organizes its serving stack around four pillars: prefix caching, key-value cache management, cache-aware routing, and agent-aware optimization. These sit above model-level work such as sparse attention for long context, error reduction, and resilient serving. The shared objective is to reduce the time to finished work by recognizing the structure of agent workloads. [7:27](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=447s)

**Prefix caching** stores the key-value, or KV, state computed for the shared beginning of an input. On a later step, the service reuses that state and processes only the new suffix. Prefill is the input-processing work that would otherwise happen again before generation begins. Reading cached state is cheaper than recomputing that prefill, improving time to first token and reducing repeated compute. A longer task offers more opportunities to reuse the initial work. [8:08](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=488s)

The saving depends on cached state fitting somewhere and moving efficiently when needed. FriendliAI describes complementary mechanisms:

- **GPU memory management:** Pack more active context into GPU memory and reduce the KV memory footprint, allowing more useful state to remain available.
- **Hierarchical caching:** Use GPU memory, host memory, and disks to extend cache capacity beyond GPU limits.
- **Reuse across replicas:** Make a prefix usable across serving replicas, rather than restricting its value to one instance.

These mechanisms address capacity and availability; the talk does not develop the specific memory-reduction technique or policies for moving state between tiers.

### Route task A back to its cached prefix

[9:26](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=566s)

At cluster scale, a cache can exist and still go unused. A load balancer that spreads requests evenly may send a later agent call to a destination without its prefix. That destination performs cold prefill, repeating the computation that caching was meant to save. A cache-aware router considers where the required state already resides. [9:26](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=566s)

Follow task A in the talk’s routing example. Its first request goes through path one, where the prefix state is computed and cached. The agent then acts, observes the result, and calls the model again with the shared prefix plus appended context. Sending the second request through path one preserves locality: the service reuses the prefix and processes the suffix. Sending it somewhere without that state requires processing the prefix again. The destination changes the work needed to answer the agent’s next call.

What does routing the second request back to path one save? The diagram separates the agent’s progression from the location of its cached state. The warm route brings the new request and existing KV state together; the cold route repeats prefix computation. Locality still has to share the decision with load balancing, because repeatedly favoring one cache-rich path can make it a hotspot.

Task A: warm reuse or cold prefill
The second request can reuse prefix state on path one. A destination without that state must recompute it; the router must also consider load.
- Task A: first request — Processes the prefix through path one.
- Path one holds prefix KV — Computed prefix state is available for reuse.
- Task A: next request — Shared prefix plus context appended after tool execution.
- Warm cache hit — Reuse prefix KV and process the new suffix.
- Cold prefill — A destination without the prefix repeats its computation.
- Task A: first request → Path one holds prefix KV (Compute and cache)
- Task A: first request → Task A: next request (Act, observe, call again)
- Task A: next request → Warm cache hit (Route back to path one)
- Path one holds prefix KV → Warm cache hit (Supply prefix state)
- Task A: next request → Cold prefill (Route somewhere without the prefix)

### Schedule a call as part of a longer program

[10:07](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=607s)

Agent-aware optimization extends this reasoning beyond routing. Scheduling every LLM call independently discards information about the longer program that produced it. Agent-level context could change which work proceeds now, what input gets prepared next, and which cached state remains available. [10:07](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=607s)

Chun identifies three possible decisions:

- **Preemption:** Interrupt the right work when knowledge of the larger agent task suggests a better scheduling choice.
- **Speculative prefill:** Prepare context for a likely next step before its model call arrives.
- **Cache eviction:** Use agent-level context to decide which state to discard.

These are presented as the next frontier, with examples rather than a complete scheduling policy. Their benefit may appear in a later call, which is why optimizing the whole task changes how the service judges a decision.

### Measure the finished game, then choose how to deploy

[10:57](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=657s)

The closing demonstration holds the model constant and changes the inference provider. **Kilo Code** uses the same GLM 5.2 model to create a simple mobile game through FriendliAI’s API and another provider’s API. Chun reports that FriendliAI finishes the task end to end 2× faster and attributes the result to its agent-focused cloud design. This measures the combined serving stack on that task; it does not isolate how much each caching, routing, or scheduling mechanism contributes. [10:57](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=657s)

The proposed production stack starts with an agent the builder already uses, adds a capable open-weight model, and serves it through infrastructure designed for task performance. Chun names MiniMax and Kimi among the model options and identifies Kilo and LG as production customers. Quality, speed, reliability, and cost belong in the same decision: lower token costs become valuable when the combined system completes useful work.

A Kilo customer testimonial supplies a separate comparison. Its split test of GLM5 usage against other third-party providers and direct model-lab access describes FriendliAI as consistently seven times faster, with a significantly lower error rate, and as a core component of Kilo’s stack. The testimonial does not specify the timing metric or quantify the error-rate difference. Its multiplier therefore belongs to that customer split test, separate from the GLM 5.2 model and 2× result in the mobile-game demonstration. [12:45](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=765s)

The same serving stack is offered through three deployment choices:

- **Model API:** A serverless API for calling frontier open-weight models, presented as the quickest way to begin.
- **Dedicated endpoints:** An isolated deployment with guaranteed service-level agreements for production workloads.
- **BYOG—bring your own GPU:** Friendli inference running on the customer’s own infrastructure.

The choice concerns how to consume the stack: through a shared API service, an isolated deployment, or infrastructure the customer supplies. [13:20](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=800s)

Chun closes by inviting builders to try frontier open-weight models in an existing agent stack. The practical question is whether the combination completes the builder’s tasks at an acceptable quality, cost, and speed. The tower defense comparison demonstrates a lower model bill; the mobile-game comparison demonstrates faster completion with the model held constant. Together they explain why economical agents require attention to both model choice and the inference system carrying the loop.

## Key takeaways

- End-to-end task latency captures the agent’s actual wait: model calls alternate with tool execution, context grows, and the number of calls depends on the input.
- Prefix caching reuses computed KV state and processes the new suffix. Memory management, hierarchical caching, and reuse across replicas keep that state available.
- Cache-aware routing preserves locality while balancing load. Evenly distributing requests can send an agent away from its cached prefix and repeat prefill.
- Agent-level context can inform preemption, speculative prefill, and cache eviction, allowing a decision now to improve a later step.
- Keep the comparisons separate: $1.50 versus 27 cents concerns the tower defense model comparison; 2× faster completion concerns the same-model mobile-game demonstration; seven times faster concerns Kilo’s customer split test.

## Related talks

- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://ai.engineer/talks/YXowceUKYJI-kv-cache-aware-routing-p-d-disaggregation) — Continues the serving-infrastructure topic with cache-aware routing and prefill/decode topology on Kubernetes.
- [Context Engineering for Complex Codebases: Research, Plan, Implement](https://ai.engineer/talks/rmvDxxNubIg-context-engineering-for-complex-codebases) — Pairs the serving-side explanation of repeated context with the application-side problem of organizing context for coding tasks.

## Slides

### Frontier AI Inference Cloud for Agents

FriendliAI CEO Byung-Gon Chun introduces his talk on agentic inference: what changed, why it matters, and how FriendliAI rebuilt its inference cloud for agents.

[0:21](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=21s) [Title slide for Byung-Gon Chun’s talk on frontier AI inference for agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f529cf279c2a32c4ed45b6bdd8249ccfeacc0bf1557d10c0cb43df2761871ae.webp)

AI ENGINEER WORLD'S FAIR 2026

Frontier AI Inference Cloud for Agents

Byung-Gon Chun (Gon)
Founder and CEO, FriendliAI

© FriendliAI Corp. All Rights Reserved.

Interface text: CC

### Who we are

FriendliAI grew out of a Seoul National University research team. Chun says the team invented continuous batching and its Orca work inspired vLLM; the company now has teams in San Francisco and Seoul.

[0:39](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=39s) [FriendliAI describes its inference service, research roots, continuous batching work, and San Francisco and Seoul locations.](https://assets.ai.engineer/sites/aiecode2025/sha256/ecb580f1594f558babede5253051d6a3c593a4150265b263b9bbecc293fcf86f.webp)

Who we are
The Frontier AI Inference Cloud for Agents

FriendliAI runs frontier AI inference for agents at scale, faster, cheaper, and more reliably.

We are the team that invented continuous batching, the inference optimization now standard across the industry; our Orca work inspired vLLM.

Research roots
San Francisco + Seoul

© FriendliAI Corp. All Rights Reserved.

Interface text: CC

### 2026 is the year of agents in production

Chun links agents moving into production across coding, operations, research and knowledge work with open-weight models that he says can power frontier-quality agents at much lower token costs.

[1:23](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=83s) [The slide links growth in production agents with the capability and lower cost of open-weight models.](https://assets.ai.engineer/sites/aiecode2025/sha256/0cda0de6b8080c93e801888e635d69038fb94650d44f15707b00edd62558b2cf.webp)

2026 is the year of agents in production

Agents are going exponential

Coding, operations, research, and knowledge work are moving from demos to production workflows.

Open-weight models reach the frontier and make agents economical

Open-weight models can power frontier-quality agents with much lower token costs.

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Open-weight models deliver frontier intelligence

The same coding-agent prompt produced two different tower-defense games. GLM-5.2 on FriendliAI and Anthropic’s Opus 4.8 both completed the task, which Chun describes as usable.

[2:01](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=121s) [Two tower-defense game screenshots compare outputs from FriendliAI GLM-5.2 and Anthropic Opus-4.8 using the same prompt and coding agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b3a8d376c3bee18f99cf6ff4d1d954ee54b8cf956be862045ccdde74bcb86c2.webp)

Open-weight models deliver frontier intelligence

FriendliAI/GLM-5.2

TOWER DEFENSE
GOLD 220
LIVES 20
WAVE 1/12
ENEMIES 8
Cannon 50g
Frost 70g
Sniper 100g
Pause
Speed 1x
Wave Active
[illegible]

Anthropic/Opus-4.8

TOWER DEFENSE
GOLD 80
LIVES 20
WAVE 0/12
ENEMIES 0
SPAWN
Pause
2x
TOWER SHOP
Cannon — Solid single-target damage — 50g
Frost — Slows enemies in range — 65g
Sniper — [illegible] — 85g
Pick a tower, then click a green tile to build. Keys 1/2/3 select, Esc cancels.
ENEMIES
Normal — balanced
Fast — low HP, quick
Tank — heavy, slow
Boss — every 5th wave
Wave 1 in 6s
Start Next Wave Early

Same prompt, same coding agent: both complete the workflow

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Frontier-quality agents at a fraction of the cost

For the same tower-defense coding task, GLM-5.2 on FriendliAI cost $0.269 versus $1.523 for Opus 4.8: about 5.6× cheaper, with fewer tokens used.

[2:45](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=165s) [A table compares token counts and costs for Opus 4.8 and GLM-5.2, with a stated 5.6-fold cost difference for the same agent task.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee9c751d8dbc7b269d4041fe1bc7d10f730c4d6fb49f565924cb29e1b87d8e73.webp)

Frontier-quality agents at a fraction of the cost

Model | Provider | Tokens | Cost
claude-opus-4-8 | Anthropic | 407,296 | $1.523
zai-org/GLM-5.2 | FriendliAI | 259,332 | $0.269

5.6× cheaper for the same agent task

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### From chat requests to agent tasks

Chat measures latency for one response. An agent task may run many model and tool calls autonomously, so its latency is the time to finish the whole task.

[3:26](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=206s) [Side-by-side panels contrast single chat requests and response latency with agent tasks involving multiple model and tool calls and whole-task latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8fcc8ec3869cc405b0ca1c43b3b2d9531428734a0a57f9564506bd4144ec266.webp)

From chat requests to agent tasks

Past – Chat

Request

A human asks, the model replies, the human reads.

Short input, short output

Latency = time to one response

Today – Agents

Task

Each task is many model calls plus tool calls, run autonomously

Latency = time to finish the whole task

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### An agent runs a session of tasks

An agent completes a task by repeatedly planning with an LLM, calling a tool, and adding the result to its context. This loop alternates model inference with tool execution.

[4:14](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=254s) [An agent repeats a plan, action, and observe loop until a task is complete.](https://assets.ai.engineer/sites/aiecode2025/sha256/b432e0d8a167693b047e572f5671575975a268ea92f63221fb559f49425086a3.webp)

An agent runs a session of tasks

• A session → many tasks
• A task → many inferences + tool calls
• Plan → Action → Observe

Repeat until the task is done

Plan
decide the next step
LLM

Action
call a tool
Tool

Observe
append result to context
Context

Repeat

Inference
Tool execution

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Context accumulates, prefixes repeat

In internal GLM-5.2 coding-agent runs, prompts stretch much farther than completions. Observations accumulate in context, and consecutive agent steps usually share a large prefix.

[4:59](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=299s) [Prompt and completion length distributions illustrate long inputs and repeated shared prefixes in coding agent runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/6620aede18f40a036c37c170d4019fc005f8973eda42ef622855c7521b02eeb6.webp)

Context accumulates, prefixes repeat

• Much longer inputs
• Context accumulates during a task
• A large shared prefix is reused

Prompt length distribution
Y-axis: Probability
Y-axis ticks: 0.00, 0.02, 0.04, 0.06, 0.08, 0.10, 0.12
X-axis: Length (tokens)
X-axis ticks: 0, 100k, 200k, 300k, 400k, 500k, 600k, 700k

Completion length distribution
Y-axis: Probability
Y-axis ticks: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7
X-axis: Length (tokens)
X-axis ticks: 0, 5k, 10k, 15k, 20k, 25k, 30k

Internal coding agent runs with GLM-5.2

© FriendliAI Corp. All Rights Reserved.

Interface text: CC

### One task can run hundreds of steps

The trace branches through search ×5, extract ×17 and verify ×75 before synthesis. A task can involve tens to hundreds of inferences over minutes or hours while context grows.

[5:45](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=345s) [A task trace shows search, extraction, verification, and synthesis stages, with many inference steps and growing context.](https://assets.ai.engineer/sites/aiecode2025/sha256/52fe7de5d0bd4358ea7c975db5600683b40d637ba62e22f84885c900dea67dce.webp)

One task can run hundreds of steps

Claude Code + GLM-5.2 on FriendliAI

Example: “Explain the speculative decoding framework in vLLM”

MAIN ×1
SCOPE ×1
SEARCH ×5
EXTRACT ×17
VERIFY ×75
SYNTHESIZE ×1

tens–hundreds
inferences per task

minutes–hours
wall-clock per task

growing
context

© FriendliAI Corp. All Rights Reserved.

Interface text: CC

### Agentic inference is a different problem

Agent workloads accumulate context and interleave tool work with model calls. Because call counts depend on the input, Chun argues that latency should be measured across the whole task.

[6:33](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=393s) [Four characteristics: growing context, interleaved non-model work, input-dependent call counts, and end-to-end task latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/36918dab1f9f55d1801c1ec34274c647fc5d6bc0424883f030d90cd9ee0ea9a9.webp)

Agentic inference is a different problem

01 Context accumulates and grows
02 Non-model work is interleaved
03 The number of calls is input-dependent
04 The real metric is end-to-end task latency

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### FriendliAI

FriendliAI says it rebuilt its inference cloud for agentic workloads, aiming to reduce end-to-end task latency rather than optimize a single request.

[7:01](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=421s) [FriendliAI describes its inference cloud for agents and its goal of optimizing for the task.](https://assets.ai.engineer/sites/aiecode2025/sha256/251a860caa4f64ca8311531e5605fadb3e8472682cf5879a1055a74c917b7991.webp)

FriendliAI

• We rebuilt the frontier inference cloud for agents around these challenges.
• Optimize for the task, not just the request.

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Key engineering

FriendliAI organizes its agentic inference stack around prefix caching, KV cache management, cache-aware routing, and agent-aware optimization, supported by model-level optimization and serving techniques.

[7:25](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=445s) [Four engineering areas are prefix caching, KV cache management, cache-aware routing, and agent-aware optimization.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c022aaf7e3890f3558da5b568d13f209e83b35ad4930174cb2d4c0a91c8bbd9.webp)

Key engineering

01 Prefix caching
02 Key Value (KV) cache management
03 Cache-aware routing
04 Agent-aware optimization
(the next frontier)

+ everything else: model-level optimization, techniques to reduce errors, fast kernels, resilient serving, and more

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Prefix caching

Agent steps reuse a cached shared prefix and process only each new suffix, avoiding repeated prefill. The speaker says this reduces compute and improves time to first token, with greater value on longer tasks.

[8:06](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=486s) [A shared prefix cache serves multiple steps, each processing a new suffix.](https://assets.ai.engineer/sites/aiecode2025/sha256/f74caba5259c19337681813765f320868fabf24116db042c7b7707166b4fe2b3.webp)

Prefix caching

• Compute the shared prefix once
• Skip redundant prefill
• The longer the task, the bigger the win

Shared prefix KV
cached once

Step 1: + new suffix
Step 2: + new suffix
Step 3: + new suffix
Step N: + new suffix

Reuse ≪ recompute.
Each step pays only for the green suffix.

© FriendliAI Corp. All Rights Reserved.

Interface text: CC

### Key Value (KV) cache management

Frugal memory management packs more context into GPU memory; quantization reduces the footprint, while hierarchical and distributed caching extend capacity and share prefixes across replicas.

[8:40](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=520s) [Four techniques: frugal memory management, quantization, hierarchical caching, and distributed caching.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c2b829f50917876f32386cdebf30762029adf0993ab35bdb8254138eabbfc2b.webp)

Key Value (KV) cache management

Frugal memory management
KV cache quantization
Hierarchical KV caching
Distributed KV caching

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Cache-aware routing in distributed clusters

A cache-aware router sends requests toward GPU pods with matching cached prefixes while balancing load. In the example, both Task A requests take the same path to preserve cache locality.

[9:24](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=564s) [A router directs task requests to GPU pods using prefix cache hits and load.](https://assets.ai.engineer/sites/aiecode2025/sha256/200273b1db2e54b18afb03696e67ed775c84afdd1ac3d3ee6b938fdc73012df7.webp)

Cache-aware routing in distributed clusters

• Maximize KV cache hits
• Balance load
• Win at cluster scale

Task A req
Task B req
Task A req
Task C req

Cache-aware router
prefix hit + load

GPU pod 1
holds prefix A

GPU pod 2
holds prefix B

GPU pod 3
holds prefix C

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Agent-aware optimization

The optimizer uses context from an entire agent program to schedule work, prefill likely next steps, and choose cache evictions. The goal is lower end-to-end task latency.

[10:06](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=606s) [Agent program context informs scheduling, caching, and routing across GPUs.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e3350bd41ed389981444d796cdab5598baf6e7be774134a35c6067cc8f16d94.webp)

Agent-aware optimization

Optimize with agent-level context, not isolated calls

e.g., preemptive scheduling, speculative prefill, and agent-aware cache eviction

Agent program
c1 → tool → c2 → tool → c3

Scheduling, caching, and routing
based on program-level context

GPUs
GPUs

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Model Crush Saga demonstration

Two game boards show Model Crush Saga. The presenter says FriendliAI completed the same GLM-5.2 and Kilo Code task end to end 2x faster than another inference provider.

[11:12](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=672s) [Two side-by-side Model Crush Saga game boards with different scores.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d87ff265062d290035dd6f753324062d3348f0fdfc5173b63ab3d4974184874.webp)

Model Crush Saga
SCORE
510
Inference Engine

Model Crush Saga
SCORE
600
Inference Engine

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Run GLM-5.2 on Kilo Code with FriendliAI

The demo uses GLM-5.2 with Kilo Code and FriendliAI to create a simple mobile game.

[11:22](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=682s) [Title slide: Run GLM-5.2 on Kilo Code with FriendliAI.](https://assets.ai.engineer/sites/aiecode2025/sha256/345a98752e2c172155bf15e3e68b538fe6b6561d037cfeda84dde8598d6daca2.webp)

Run GLM-5.2
on Kilo Code
with FriendliAI

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### A winning production agent stack

The proposed stack pairs an agent and an open-weight frontier model with FriendliAI inference to deliver a faster, lower-cost production agent.

[11:25](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=685s) [Diagram combines an agent, an open-weight frontier model, and FriendliAI inference into a production agent stack.](https://assets.ai.engineer/sites/aiecode2025/sha256/97413023d2876aca623291a51f66dcd0eecaac4985bbbcc60f04faff63176430.webp)

A winning production agent stack

AGENT
Your agent
Kilo, OpenCode, Claude Code, Cursor, Pi, OpenClaw, Hermes Agent, and more

MODEL
Open-weight frontier model
GLM-5.2, MiniMax M3, Kimi K2.7, and more

INFERENCE
FriendliAI: Frontier AI inference cloud for agents

= A faster, lower-cost production agent

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### FriendliAI: Trusted in production

Chun says FriendliAI powers production teams ranging from AI startups to global enterprises, including Kilo and LG.

[12:07](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=727s) [Section slide titled FriendliAI: Trusted in production.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b68b5ea28b3581fc7cd6153c4cdc6deeb1d19600bd7c7fc73e747cf93e7e03b.webp)

FriendliAI
Trusted in production

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Testimonial from Kilo

Kilo reports that in a GLM-5 split test, FriendliAI was consistently 7 times faster and had a significantly lower error rate than other third-party providers and direct usage from Z.ai.

[12:43](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=763s) [Kilo testimonial claims FriendliAI was seven times faster and had a significantly lower error rate in a GLM-5 split test.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcce3b9108a6498e7d5caf40d46652eb581f3c0970ccfe07a6e181a1161f0183.webp)

Testimonial from Kilo

“Over the past year, Kilo Code has tested several different inference providers hosting a range of both open and closed models. In a split test of GLM-5 usage, they were especially impressed by FriendliAI’s results. Compared against both other third-party providers and direct usage (from the model lab, Z.ai), FriendliAI was consistently 7 times faster and had a significantly lower error rate. FriendliAI is now a core component of the Kilo stack, enabling high-performance access to the latest open-weight models.”

© FriendliAI Corp. All Rights Reserved.

### Deploy your way

FriendliAI offers a serverless Model API, isolated dedicated endpoints with guaranteed SLAs, and BYOG to run Friendli inference on your own infrastructure.

[13:22](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=802s) [Three deployment options: Model API, Dedicated Endpoints, and Bring Your Own GPU.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ec06b0cf9e868cee138e3d8eac87b2fc9ee2d8fd5126e5aab24682a7dd97f5c.webp)

Deploy your way

Model API
Dedicated Endpoints
BYOG — Bring Your Own GPU

© FriendliAI Corp. All Rights Reserved.

Interface text:
CC

### Thank you

Chun invites agent builders to try frontier open-weight models on FriendliAI. The closing slide provides a QR code and the friendli.ai address.

[14:22](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=862s) [Closing slide invites viewers to try frontier open-weight models on FriendliAI, with a QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dbd26f5c81c511d2cb00712175ea9d534b1b170475028175791132a549897bf.webp)

Thank you!

The Frontier AI Inference Cloud for Agents
Try out frontier open-weight models on FriendliAI today!

friendli.ai

© FriendliAI Corp. All Rights Reserved.

Interface text: CC

### AI Engineer World's Fair

[14:52](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=892s) [AI Engineer World's Fair title card on a black background, with the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c300c28f520392bc3f5b9c84c2c49cb235b4d0b0e51e3a22a8cf60f5fe244aa.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:12](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=12s) Let's, uh, get started. Uh, hi, everyone. Thank you for c-coming. Uh, this is the l-late afternoon in the last day, um, so I really appre-appreciate it. I'm Gon, founder and CEO of FriendliAI. Today, I wanna talk about agentic inference. So I'll first walk through what changed, uh, why it matters, and how we rebuilt the inference cloud for agents.

[0:41](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=41s) Before we go deeper, let me briefly introduce FriendliAI. FriendliAI is the frontier AI inference cloud for agents. So we run inference for agents at scale, uh, faster, cheaper, and more reliably. We are born from a research team at Seoul National University, and those research roots still define us. We are the team that invented continuous batching, the inference optimization that is now standard across the industry,

[1:11](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=71s) and our Orca work inspired vLLM, a widely used open source framework. Today, we operate globally, headquartered in San Francisco, with a team in Seoul to scale frontier inference.

[1:25](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=85s) As you know, 2026 is the year agents go into massive production, and it's driven by two trends coming together. First, agents are going exponential. AI an-agents are driving explosive adoption across software, operations, and knowledge work. Second, open-weight models have reached the frontier and make agents economical. They now rival closed frontier models in capability, which

[1:55](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=115s) means you can run frontier quality agents on open models with much lower token cost.

[2:02](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=122s) Let me make the open-weight model part concrete. Open-weight models are now strong enough for these types of real agentic workflows. Here, we gave the exact same task, building a tower defense game with a coding agent, to two models. On the left is GLM 5.2, an open-weight model running on FriendliAI. On the right is Anthropic's Opus 4.8. The important point is not that the outputs are identical.

[2:32](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=152s) The point is that both complete the task at a level that is clearly usable. For many agentic workflows, open-weight models have crossed the quality threshold.

[2:46](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=166s) But the economics are very different. For the same te-task, Opus 4.8 costs about $1.50, GLM 5.2 on FriendliAI costs 27 cents, about 5.6 times cheaper. So this is the promise I mentioned earlier. Open-weight models give you frontier quality agents at a fraction of the cost. But model cost is only one part of the story. To make

[3:16](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=196s) agent actually fast and reliable, the inference stack itself has to change. So let's look at what actually happens inside an agentic workload. So first, let's look at changes in the workload. In the past, the dominant usage was chat. The basic unit was a request. A person asks a question, the model answers, and the person reads it. Latency meant how fast did I get one response.

[3:46](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=226s) Agents are different. The basic unit is a task. A task may involve many model calls, many tool calls, and they may run autonomously for a while. So the user does not really care about the latency of one individual request. The user cares about when the whole task is completed. That means we have to optimize for tasks, not just individual requests.

[4:16](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=256s) Let's look at agentic workloads more closely. An agent really runs a session made up of tasks. Each task typically runs in a loop. First it plans, which usually means an LLM call. Then it acts, maybe by calling a tool. Then it observes the result and adds the, that back into the context, and it repeats this until the task is done. So we are constantly alternating between LLM inference

[4:45](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=285s) and one or more non-LLM tool executions. So there is a gap between LLM calls. An agent can also create sub-agents and run them in parallel.

[5:01](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=301s) Agent inputs also look very different from chat. The graph here shows the prompt and completion length dis-distributions of our internal coding agent runs with GLM 5.2, which we use day to day. They are much longer. They grow as the task progresses since every observation gets appended back into the context. There is an important pattern here. Consecutive agent steps usually share a

[5:31](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=331s) huge prefix. If we recompute the same prefix every time, we are burning a lot of compute on work we already did. So this is one of the biggest opportunities in agentic inference.

[5:46](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=346s) So how token hungry are agents? Now let's look at a long-horizon task example like deep research. We ran Explain the Spec decoding framework in vLLM using code code with GLM 5.2 on FriendliAI. There are multiple stages, and each stage is composed of sub-agents which run multiple inferences and tool calls. So it might run tens or even hundreds of inference steps,

[6:17](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=377s) sometimes over minutes or hours, and the shared context keeps going the whole time. For the user, what matters is not the latency of a single token or one call, what matters is when is my task completed.

[6:35](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=395s) So agentic inference is not just chat with more requests. It's a different problem. The context grows over time. Tool work is interleaved between model calls. The number of model calls depends on the input, so you can't really plan around a fixed request rate plan. And the real metric is end-to-end task latency, not a single request latency.

[7:03](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=423s) This is where FriendliAI comes in. We rebuilt the frontier inference cloud specifically for agentic workloads around the challenges I just walked through. And we set one goal: optimize end-to-end task latency, uh, the task, not just the request. So how do we do that? Let me show you the key engineering behind it.

[7:27](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=447s) Here's the engineering map for how we think about it. We built the stack layer by layer around agentic workloads. There are four big pillars I'm gonna cover today: prefix caching, key value, in short, KV, cache management, cache-aware routing, agent-aware optimization. And of course, underneath, we need model layer optimization like sparse attention for long context, techniques to

[7:56](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=476s) reduce errors, fast corners, resilient serving, and more. In this talk, I'm gonna focus on the four pillars.

[8:08](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=488s) Let's start with prefix caching. Since agent steps share a large prefix, we compute key value for the prefix once and cache it. Then on later steps, we reuse the cache key value and only process the new suffix. Reading from cache is much cheaper than recomputing prefill, so this improves time to first token and reduces compute on every step. And the longer the task runs in agents, the more valuable this

[8:38](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=518s) becomes.

[8:41](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=521s) But caching only works if the k- KV cache actually fits and can move around efficiently. So we need strong KV cache management. We use frugal memory management to pack more active context onto each GPU memory. We use KV contiguation to reduce the memory footprint. We u- we use hierarchical caching across GPU memory, host memory, and disks, so

[9:11](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=551s) we can go beyond GPU limits. And we also use disputed caching, so one prefix can be served across replicas, not just inside one instance.

[9:26](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=566s) At global cluster scale, routing becomes really important. A naive load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality. A cache-aware router at a global scale does something smarter. It sends a request to the path that already has the right prefix cached, turning a cold prefill into a warm cache hit. At the same time, it still has to balance load, so one path doesn't become a

[9:56](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=596s) hotspot. In this example, the two requests of task A go to the same path one for cache locality.

[10:07](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=607s) The next piece is agent-aware optimization, and this is the next frontier of agentic inference. Today, most systems schedule each LLM call as if it were independent. They don't really understand that this call is part of a l- longer agent program. But if the optimizer knows the agent-level context, it can make better decisions. For example, preempting the right work,

[10:37](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=637s) speculatively prefilling context for a likely next step, or making a better cache eviction decision based on agent-level context. So the goal is to reduce end-to-end task latency, not just make one call look fast.

[10:57](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=657s) When we put all of this together, this is the payoff. We are using the same model, GLM5.2, with Kilo Code to create a simple mobile game. We ran the same task with model APIs of FriendliAI and another well-known inference provider. As you can see, FriendliAI completes the same task end to end 2x faster thanks to our agentic-centric cloud design.

[11:27](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=687s) So what does this unlock in practice? A stronger production agent stack. Take an agent you already like. Now plug in open-weight frontier models like GLM5.2, MiniMax, and Kimi served on FriendliAI. The model gives you frontier quality capability and better economics. FriendliAI gives you the speed, reliability, and end-to-end task performance needed in production.

[11:57](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=717s) That combination, quality, speed, reliability, and cost, is what makes agents actually useful and economical in production

[12:09](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=729s) FriendliAI is currently powering teams in production from AI native startups to global enterprises. I'd like to highlight a couple here. Kilo is a hugely popular agentic AI coding tool serving millions of users.

[12:28](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=748s) LG is a global enterprise whose businesses range from electronics to healthcare to energy. Very different companies, but they all need the same thing: fast, reliable, cost-effective agentic inference.

[12:45](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=765s) This testimonial from our client Kilo says it all. Over the past year, Kilo Code has tested several inference providers hosting both open and closed models. In a split test of GLM5 usage compared against other third-party providers and direct usage from the model lab g.ai, FriendliAI was consistently seven times faster with a significantly lower error rate. Today, FriendliAI is a core

[13:15](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=795s) component of the Kilo stack.

[13:20](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=800s) And you can consume this however fits your stack. Model API is the fastest way to start. Call Frontier open-weight models through our serverless API, dedicated endpoints give you your own isolated deployment with guaranteed SLAs for production workloads. And BYOG, bring your own GPU, lets you run Friendli inference on your own infrastructure. Same stack, three ways to deploy.

[13:49](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=829s) To wrap up, there are three things to remember. First, Frontier open-weight models make production agents economically scalable. Second, agents are not just chat with more calls. Agentic inference requires optimizing end-to-end task latency with the challenges I mentioned. Third, FriendliAI is built as an inference cloud for that world. Fast, reliable, cost-effective agentic

[14:19](https://www.youtube.com/watch?v=Hvb2LfMH58c&t=859s) inference. Thank you for attending my session. If you're building agents, give, uh, Frontier open-weight models a try on FriendliAI today. You can get started at friendli.ai in minutes. And, uh, thank you. I'll be around after the session. Um, thank you.
