[
  {
    "startMs": 12960,
    "text": "Let's, uh, get started. Uh, hi, everyone. Thank you for c-coming. Uh, this is the l-late afternoon in the last day, um, so I really appre-appreciate it. I'm Gon, founder and CEO of FriendliAI. Today, I wanna talk about agentic inference. So I'll first walk through what changed, uh, why it matters, and how we rebuilt the inference cloud for agents."
  },
  {
    "startMs": 41020,
    "text": "Before we go deeper, let me briefly introduce FriendliAI. FriendliAI is the frontier AI inference cloud for agents. So we run inference for agents at scale, uh, faster, cheaper, and more reliably. We are born from a research team at Seoul National University, and those research roots still define us. We are the team that invented continuous batching, the inference optimization that is now standard across the industry,"
  },
  {
    "startMs": 71020,
    "text": "and our Orca work inspired vLLM, a widely used open source framework. Today, we operate globally, headquartered in San Francisco, with a team in Seoul to scale frontier inference."
  },
  {
    "startMs": 85460,
    "text": "As you know, 2026 is the year agents go into massive production, and it's driven by two trends coming together. First, agents are going exponential. AI an-agents are driving explosive adoption across software, operations, and knowledge work. Second, open-weight models have reached the frontier and make agents economical. They now rival closed frontier models in capability, which"
  },
  {
    "startMs": 115200,
    "text": "means you can run frontier quality agents on open models with much lower token cost."
  },
  {
    "startMs": 122820,
    "text": "Let me make the open-weight model part concrete. Open-weight models are now strong enough for these types of real agentic workflows. Here, we gave the exact same task, building a tower defense game with a coding agent, to two models. On the left is GLM 5.2, an open-weight model running on FriendliAI. On the right is Anthropic's Opus 4.8. The important point is not that the outputs are identical."
  },
  {
    "startMs": 152980,
    "text": "The point is that both complete the task at a level that is clearly usable. For many agentic workflows, open-weight models have crossed the quality threshold."
  },
  {
    "startMs": 166400,
    "text": "But the economics are very different. For the same te-task, Opus 4.8 costs about $1.50, GLM 5.2 on FriendliAI costs 27 cents, about 5.6 times cheaper. So this is the promise I mentioned earlier. Open-weight models give you frontier quality agents at a fraction of the cost. But model cost is only one part of the story. To make"
  },
  {
    "startMs": 196040,
    "text": "agent actually fast and reliable, the inference stack itself has to change. So let's look at what actually happens inside an agentic workload. So first, let's look at changes in the workload. In the past, the dominant usage was chat. The basic unit was a request. A person asks a question, the model answers, and the person reads it. Latency meant how fast did I get one response."
  },
  {
    "startMs": 226820,
    "text": "Agents are different. The basic unit is a task. A task may involve many model calls, many tool calls, and they may run autonomously for a while. So the user does not really care about the latency of one individual request. The user cares about when the whole task is completed. That means we have to optimize for tasks, not just individual requests."
  },
  {
    "startMs": 256000,
    "text": "Let's look at agentic workloads more closely. An agent really runs a session made up of tasks. Each task typically runs in a loop. First it plans, which usually means an LLM call. Then it acts, maybe by calling a tool. Then it observes the result and adds the, that back into the context, and it repeats this until the task is done. So we are constantly alternating between LLM inference"
  },
  {
    "startMs": 285960,
    "text": "and one or more non-LLM tool executions. So there is a gap between LLM calls. An agent can also create sub-agents and run them in parallel."
  },
  {
    "startMs": 301180,
    "text": "Agent inputs also look very different from chat. The graph here shows the prompt and completion length dis-distributions of our internal coding agent runs with GLM 5.2, which we use day to day. They are much longer. They grow as the task progresses since every observation gets appended back into the context. There is an important pattern here. Consecutive agent steps usually share a"
  },
  {
    "startMs": 331240,
    "text": "huge prefix. If we recompute the same prefix every time, we are burning a lot of compute on work we already did. So this is one of the biggest opportunities in agentic inference."
  },
  {
    "startMs": 346760,
    "text": "So how token hungry are agents? Now let's look at a long-horizon task example like deep research. We ran Explain the Spec decoding framework in vLLM using code code with GLM 5.2 on FriendliAI. There are multiple stages, and each stage is composed of sub-agents which run multiple inferences and tool calls. So it might run tens or even hundreds of inference steps,"
  },
  {
    "startMs": 377252,
    "text": "sometimes over minutes or hours, and the shared context keeps going the whole time. For the user, what matters is not the latency of a single token or one call, what matters is when is my task completed."
  },
  {
    "startMs": 395192,
    "text": "So agentic inference is not just chat with more requests. It's a different problem. The context grows over time. Tool work is interleaved between model calls. The number of model calls depends on the input, so you can't really plan around a fixed request rate plan. And the real metric is end-to-end task latency, not a single request latency."
  },
  {
    "startMs": 423032,
    "text": "This is where FriendliAI comes in. We rebuilt the frontier inference cloud specifically for agentic workloads around the challenges I just walked through. And we set one goal: optimize end-to-end task latency, uh, the task, not just the request. So how do we do that? Let me show you the key engineering behind it."
  },
  {
    "startMs": 447252,
    "text": "Here's the engineering map for how we think about it. We built the stack layer by layer around agentic workloads. There are four big pillars I'm gonna cover today: prefix caching, key value, in short, KV, cache management, cache-aware routing, agent-aware optimization. And of course, underneath, we need model layer optimization like sparse attention for long context, techniques to"
  },
  {
    "startMs": 476992,
    "text": "reduce errors, fast corners, resilient serving, and more. In this talk, I'm gonna focus on the four pillars."
  },
  {
    "startMs": 488212,
    "text": "Let's start with prefix caching. Since agent steps share a large prefix, we compute key value for the prefix once and cache it. Then on later steps, we reuse the cache key value and only process the new suffix. Reading from cache is much cheaper than recomputing prefill, so this improves time to first token and reduces compute on every step. And the longer the task runs in agents, the more valuable this"
  },
  {
    "startMs": 518072,
    "text": "becomes."
  },
  {
    "startMs": 521352,
    "text": "But caching only works if the k- KV cache actually fits and can move around efficiently. So we need strong KV cache management. We use frugal memory management to pack more active context onto each GPU memory. We use KV contiguation to reduce the memory footprint. We u- we use hierarchical caching across GPU memory, host memory, and disks, so"
  },
  {
    "startMs": 551312,
    "text": "we can go beyond GPU limits. And we also use disputed caching, so one prefix can be served across replicas, not just inside one instance."
  },
  {
    "startMs": 566252,
    "text": "At global cluster scale, routing becomes really important. A naive load balancer may spread requests evenly across GPU clusters, but it can destroy cache locality. A cache-aware router at a global scale does something smarter. It sends a request to the path that already has the right prefix cached, turning a cold prefill into a warm cache hit. At the same time, it still has to balance load, so one path doesn't become a"
  },
  {
    "startMs": 596312,
    "text": "hotspot. In this example, the two requests of task A go to the same path one for cache locality."
  },
  {
    "startMs": 607952,
    "text": "The next piece is agent-aware optimization, and this is the next frontier of agentic inference. Today, most systems schedule each LLM call as if it were independent. They don't really understand that this call is part of a l- longer agent program. But if the optimizer knows the agent-level context, it can make better decisions. For example, preempting the right work,"
  },
  {
    "startMs": 637752,
    "text": "speculatively prefilling context for a likely next step, or making a better cache eviction decision based on agent-level context. So the goal is to reduce end-to-end task latency, not just make one call look fast."
  },
  {
    "startMs": 657952,
    "text": "When we put all of this together, this is the payoff. We are using the same model, GLM5.2, with Kilo Code to create a simple mobile game. We ran the same task with model APIs of FriendliAI and another well-known inference provider. As you can see, FriendliAI completes the same task end to end 2x faster thanks to our agentic-centric cloud design."
  },
  {
    "startMs": 687332,
    "text": "So what does this unlock in practice? A stronger production agent stack. Take an agent you already like. Now plug in open-weight frontier models like GLM5.2, MiniMax, and Kimi served on FriendliAI. The model gives you frontier quality capability and better economics. FriendliAI gives you the speed, reliability, and end-to-end task performance needed in production."
  },
  {
    "startMs": 717552,
    "text": "That combination, quality, speed, reliability, and cost, is what makes agents actually useful and economical in production"
  },
  {
    "startMs": 729400,
    "text": "FriendliAI is currently powering teams in production from AI native startups to global enterprises. I'd like to highlight a couple here. Kilo is a hugely popular agentic AI coding tool serving millions of users."
  },
  {
    "startMs": 748020,
    "text": "LG is a global enterprise whose businesses range from electronics to healthcare to energy. Very different companies, but they all need the same thing: fast, reliable, cost-effective agentic inference."
  },
  {
    "startMs": 765820,
    "text": "This testimonial from our client Kilo says it all. Over the past year, Kilo Code has tested several inference providers hosting both open and closed models. In a split test of GLM5 usage compared against other third-party providers and direct usage from the model lab g.ai, FriendliAI was consistently seven times faster with a significantly lower error rate. Today, FriendliAI is a core"
  },
  {
    "startMs": 795380,
    "text": "component of the Kilo stack."
  },
  {
    "startMs": 800440,
    "text": "And you can consume this however fits your stack. Model API is the fastest way to start. Call Frontier open-weight models through our serverless API, dedicated endpoints give you your own isolated deployment with guaranteed SLAs for production workloads. And BYOG, bring your own GPU, lets you run Friendli inference on your own infrastructure. Same stack, three ways to deploy."
  },
  {
    "startMs": 829960,
    "text": "To wrap up, there are three things to remember. First, Frontier open-weight models make production agents economically scalable. Second, agents are not just chat with more calls. Agentic inference requires optimizing end-to-end task latency with the challenges I mentioned. Third, FriendliAI is built as an inference cloud for that world. Fast, reliable, cost-effective agentic"
  },
  {
    "startMs": 859360,
    "text": "inference. Thank you for attending my session. If you're building agents, give, uh, Frontier open-weight models a try on FriendliAI today. You can get started at friendli.ai in minutes. And, uh, thank you. I'll be around after the session. Um, thank you."
  }
]