The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Byung-Gon (Gon) Chun, founder and CEO of FriendliAI, introduces agentic inference from the serving side. FriendliAI grew out of a research team at Seoul National University; Chun credits that team with inventing continuous batching and describes its Orca work as an influence on vLLM. The question now is how inference infrastructure should change when agents become its customers. 0:41

Chun sees two developments coming together: agents spreading through software, operations, and knowledge work, and open-weight models becoming capable enough to make those workflows economical. A coding example gives that claim a concrete test. The same agent receives the same assignment—build a tower defense game—with an open-weight GLM 5.2 model served by FriendliAI and an Anthropic Opus 4.8 model. Chun describes both results as clearly usable, without claiming that the games are identical. GLM 5.2 and Opus 4.8 are the model labels Chun reports for these demonstrated runs. Exact checkpoints and evaluation conditions are not supplied, so this is a speaker-reported demonstration rather than an independently reproduced benchmark. 2:02

The bills differ substantially: about $1.50 for the Opus 4.8 run and 27 cents for the GLM 5.2 run on FriendliAI, a reported 5.6-fold cost difference. For this assignment, each result clears the practical quality threshold, making the cheaper run useful. That is the promise of open weights in an agent workflow. But the model bill is only part of the story: the serving system also determines how quickly and reliably the agent reaches its finished result.

Chat gives an inference service a simple unit of work: a person asks a question, the model answers, and the person reads the response. Request latency measures that wait. An agent’s assignment can contain many model calls and tool calls, continuing autonomously between them. Its user is waiting for completed work, so the main performance measure becomes end-to-end task latency. A fast response helps insofar as it advances the task. 3:46

The task typically runs through a plan–act–observe loop. Planning invokes the LLM; acting may invoke a tool; observing appends the tool’s result to the context before the next planning call. This creates gaps between inference calls while tools execute. The agent can also create sub-agents that run in parallel. Requests therefore arrive from an ongoing program whose next move depends on earlier results, rather than from a fixed sequence of independent questions.

FriendliAI’s internal coding-agent runs show long inputs that grow as the task progresses. Each observation joins the next model input, so consecutive steps share a large beginning, or prefix, followed by newly appended material. Processing the entire input again means spending compute on context the service has already processed. In Chun’s words, it is work “we already did.” 5:31

A long-horizon research task makes the accumulation easier to understand: an agent is asked to explain a decoding framework in vLLM. The work has multiple stages, each containing sub-agents that make inference calls and tool calls. Tasks of this kind can involve tens or hundreds of inference steps over minutes or hours, while shared context continues through the work. Repeatedly processing that history adds cost and waiting time throughout the chain.

Three properties complicate serving: context grows, tool execution interrupts the sequence of model calls, and the number of calls depends on the input. A fixed request-rate assumption cannot capture the whole workload. The inference service needs to help an evolving task finish, including preserving useful state between calls when the agent is busy elsewhere.

FriendliAI organizes its serving stack around four pillars: prefix caching, key-value cache management, cache-aware routing, and agent-aware optimization. These sit above model-level work such as sparse attention for long context, error reduction, and resilient serving. The shared objective is to reduce the time to finished work by recognizing the structure of agent workloads. 7:27

Prefix caching stores the key-value, or KV, state computed for the shared beginning of an input. On a later step, the service reuses that state and processes only the new suffix. Prefill is the input-processing work that would otherwise happen again before generation begins. Reading cached state is cheaper than recomputing that prefill, improving time to first token and reducing repeated compute. A longer task offers more opportunities to reuse the initial work. 8:08

The saving depends on cached state fitting somewhere and moving efficiently when needed. FriendliAI describes complementary mechanisms:

• GPU memory management: Pack more active context into GPU memory and reduce the KV memory footprint, allowing more useful state to remain available.
• Hierarchical caching: Use GPU memory, host memory, and disks to extend cache capacity beyond GPU limits.
• Reuse across replicas: Make a prefix usable across serving replicas, rather than restricting its value to one instance.

These mechanisms address capacity and availability; the talk does not develop the specific memory-reduction technique or policies for moving state between tiers.

At cluster scale, a cache can exist and still go unused. A load balancer that spreads requests evenly may send a later agent call to a destination without its prefix. That destination performs cold prefill, repeating the computation that caching was meant to save. A cache-aware router considers where the required state already resides. 9:26

Follow task A in the talk’s routing example. Its first request goes through path one, where the prefix state is computed and cached. The agent then acts, observes the result, and calls the model again with the shared prefix plus appended context. Sending the second request through path one preserves locality: the service reuses the prefix and processes the suffix. Sending it somewhere without that state requires processing the prefix again. The destination changes the work needed to answer the agent’s next call.

What does routing the second request back to path one save? The diagram separates the agent’s progression from the location of its cached state. The warm route brings the new request and existing KV state together; the cold route repeats prefix computation. Locality still has to share the decision with load balancing, because repeatedly favoring one cache-rich path can make it a hotspot.

Agent-aware optimization extends this reasoning beyond routing. Scheduling every LLM call independently discards information about the longer program that produced it. Agent-level context could change which work proceeds now, what input gets prepared next, and which cached state remains available. 10:07

Chun identifies three possible decisions:

• Preemption: Interrupt the right work when knowledge of the larger agent task suggests a better scheduling choice.
• Speculative prefill: Prepare context for a likely next step before its model call arrives.
• Cache eviction: Use agent-level context to decide which state to discard.

These are presented as the next frontier, with examples rather than a complete scheduling policy. Their benefit may appear in a later call, which is why optimizing the whole task changes how the service judges a decision.

The closing demonstration holds the model constant and changes the inference provider. Kilo Code uses the same GLM 5.2 model to create a simple mobile game through FriendliAI’s API and another provider’s API. Chun reports that FriendliAI finishes the task end to end 2× faster and attributes the result to its agent-focused cloud design. This measures the combined serving stack on that task; it does not isolate how much each caching, routing, or scheduling mechanism contributes. 10:57

The proposed production stack starts with an agent the builder already uses, adds a capable open-weight model, and serves it through infrastructure designed for task performance. Chun names MiniMax and Kimi among the model options and identifies Kilo and LG as production customers. Quality, speed, reliability, and cost belong in the same decision: lower token costs become valuable when the combined system completes useful work.

A Kilo customer testimonial supplies a separate comparison. Its split test of GLM5 usage against other third-party providers and direct model-lab access describes FriendliAI as consistently seven times faster, with a significantly lower error rate, and as a core component of Kilo’s stack. The testimonial does not specify the timing metric or quantify the error-rate difference. Its multiplier therefore belongs to that customer split test, separate from the GLM 5.2 model and 2× result in the mobile-game demonstration. 12:45

The same serving stack is offered through three deployment choices:

• Model API: A serverless API for calling frontier open-weight models, presented as the quickest way to begin.
• Dedicated endpoints: An isolated deployment with guaranteed service-level agreements for production workloads.
• BYOG—bring your own GPU: Friendli inference running on the customer’s own infrastructure.

The choice concerns how to consume the stack: through a shared API service, an isolated deployment, or infrastructure the customer supplies. 13:20

Chun closes by inviting builders to try frontier open-weight models in an existing agent stack. The practical question is whether the combination completes the builder’s tasks at an acceptable quality, cost, and speed. The tower defense comparison demonstrates a lower model bill; the mobile-game comparison demonstrates faster completion with the model held constant. Together they explain why economical agents require attention to both model choice and the inference system carrying the loop.