FriendliAI

Chun introduces the economics through a coding agent building a tower defense game. He reports usable results from models labeled GLM 5.2 on FriendliAI and Anthropic Opus 4.8, with bills of $0.27 and about $1.50 respectively—a roughly 5.6-fold difference. These are the recording’s reported model labels and demonstrated runs; exact checkpoints and evaluation conditions are not supplied. 2:02–3:18

The serving problem extends beyond token prices. An agent plans with a model, acts through tools, appends observations to its context, and repeats; subagents can also run in parallel. Tool execution creates gaps between inference calls, and the number of calls depends on the task’s progress. Chun therefore makes end-to-end task latency the central metric: the user waits for completed work across the whole loop. His research-task example illustrates how multiple stages and subagents can produce tens or hundreds of inference steps over minutes or hours. 3:46–7:00

Consecutive agent steps often share a substantial prefix because earlier observations remain in subsequent prompts. Prefix caching computes that prefix’s attention key-value, or KV, state once, then reuses it while processing only the new suffix. Avoiding repeated prefill—the processing of input context before generation—reduces compute and improves time to first token. 5:01 8:08

Reuse also requires capacity and access to the stored state. Chun describes efficient GPU memory management, reducing the KV footprint, hierarchical caching across GPU memory, host memory, and disk, and making prefixes usable across replicas. These mechanisms keep more context available beyond a single GPU or serving instance. The recording does not explain the specific memory-reduction algorithm or the policies governing movement between tiers, so those implementation details remain unspecified. 8:41–9:22

At cluster scale, distributing calls evenly can send a later request somewhere without its cached prefix, forcing cold prefill. Chun’s task-A example routes successive requests through the same path so the next call can reuse existing KV state and process the appended context. Cache-aware routing must weigh that locality against load: repeatedly choosing a cache-rich destination can create a hotspot. 9:26–10:04

Agent-aware optimization adds information about the longer program surrounding each call. Chun identifies preemption, speculative prefill for a likely next step, and cache eviction informed by agent context as possible decisions. Speculative prefill prepares input context; it is distinct from speculative decoding, which drafts and verifies output tokens. These scheduling examples are presented as an optimization frontier, without a complete policy or proof that every example is deployed. Their intended benefit is faster completion across later steps of the task. 10:07–10:55

The closing mobile-game demonstration holds GLM 5.2 constant while changing inference providers. Chun reports that Kilo Code completes the task through FriendliAI’s API 2× faster than through another provider’s API. That result concerns the combined serving stack on one task and does not isolate the contribution of caching, routing, or scheduling. A separate Kilo testimonial concerns GLM-5 usage and reports sevenfold faster performance with a lower error rate; the recording does not specify its timing metric or quantify the error reduction. Neither multiplier is the tower-defense demonstration’s 5.6-fold model-cost difference. 10:57 12:45

Chun closes with three ways to consume the inference stack: a serverless Model API, dedicated endpoints described as isolated deployments with guaranteed service-level agreements, and BYOG, which runs Friendli inference on customer-supplied infrastructure. The supplied product sources further distinguish OpenAI-compatible Model APIs, dedicated GPU capacity with GPU-time billing and reserved options, and BYOG operated by FriendliAI within the customer’s cloud environment. The practical evaluation remains whether the chosen model and serving arrangement complete useful tasks at acceptable quality, speed, reliability, and cost. 13:20–14:38