AI Engineer World's Fair 2026
The State of Model Routing — NVIDIA, Cognition, OpenRouter
Read the talk
Model Routing: Delegation, Context, and the Cost of Finishing the Task
Routing an agent means deciding who plans, who executes, what context they share, and when to switch—not merely choosing the cheapest model for a prompt.
From a talk by Walden, Carter, Tanay, Alex Atallah and Nav
Before you start: Familiarity with LLM tokens, context windows, and agents that call tools will help; the article explains how caching and delegation affect routing.
Which tasks justify the expensive model?
Which engineering tasks deserve a frontier model, and which can a cheaper model handle without sacrificing the result? That question becomes unavoidable when an application uses several models, including models running locally. NVIDIA frames its Nemotron releases—described here as including datasets, weights, and customization recipes—as infrastructure for that increasingly customized, multi-model world.
For Walden, co-founder of Cognition, the question comes directly from customers deploying Devin: how should they measure model ROI, and where should engineers spend their expensive-model budget? Carter, a developer technology engineer at NVIDIA, describes the same pressure among individual developers, startups, and small companies. More available intelligence is useful only if developers can afford to keep using it. Tanay approaches routing through model evaluations: understanding accuracy, efficiency, cost, and behavior well enough to build systems in which models cooperate. The tooling is still young, with no settled architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep frontier judgment, delegate the implementation
Devin Fusion starts from a failure mode of simple routing: send the user to a weaker model, discover that it cannot do the task, and then pay for the stronger model anyway. Cognition instead keeps a frontier model responsible for planning and difficult decisions while delegating implementation. Walden qualifies the claimed performance improvement over Fable: it is not a capability jump comparable to the gap between a frontier model and a substantially weaker one. Walden reports a 40% reduction in the cost of Fable-level intelligence with Devin Fusion. The mechanism is continued frontier judgment combined with cheaper execution, rather than replacing that judgment outright.
The implementation model can be an open model or a cheaper mini model. Its lower token price buys room to investigate more thoroughly. Walden illustrates this with three agents exploring a codebase: their combined exploration might cover more ground than one frontier model following a single path through its context. This is an example of the economic opportunity; his more precise description of Fusion’s persistent sidekick comes later. Carter makes the budget argument explicit: sufficiently cheaper tokens let a system spend more tokens while remaining within the original frontier-model budget.
Model size is only one dimension of the decision. Tanay points to jagged capabilities within coding itself. A data-science workflow can involve model building and prediction with scikit-learn, followed by visualization with matplotlib. Different training corpora can produce different strengths across those steps. A higher aggregate coding score does not establish that one model is better at every constituent task; useful routing requires understanding those differences.
After Alex Atallah joins from OpenRouter, the discussion extends to scientific discovery: a difficult, sometimes one-shot problem spanning many subdomains. Different post-training teachers and subtasks can create complementary failure patterns. Once those patterns are understood, orchestration can exploit them. Tanay tentatively attributes up to 10% higher accuracy to routing techniques, depending on the model pool and task. He calls the benchmark “LM router bench”; the discussion does not establish the exact experiment or whether the percentage is relative or measured in percentage points. The useful mechanism is complementary expertise, not a guaranteed gain from adding models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The task changes while the agent is working
Walden confirms that cheaper models can use more tokens, but warns against turning task-specific benchmark wins into a permanent initial assignment. Consider the progression of a developer session:
- Ask the agent to explain how a codebase works.
- Ask it to implement a feature.
- Ask it to live-test the feature and debug difficult cases.
The task category and difficulty change during the same conversation. A model selected for code explanation may be a poor choice for the later debugging work. Frontier models are attractive partly because they can move between those domains without requiring the user to diagnose the transition.
Routing must follow the session, not just classify its first prompt. The difficult part is recognizing when the current model has become inadequate. A small model may not reliably know that it is out of its depth. Cognition’s answer is to keep a frontier agent supervising even when it is not executing the work. That agent can notice that the delegated task has become harder and redirect it.
Supervision introduces a second problem: what should the supervising model see? Passing it the entire worker trace can erase the savings. Walden’s example is a file read duplicated across three models, creating three separate context charges. Fusion instead sends most context to one model by default. The supervisor receives selected information—such as filenames, a high-level account of the work, or a carefully produced summary—rather than every tool result. Long-running agents already need context compaction; the same techniques can support these handoffs.
This makes context design part of routing design. Adding models increases the opportunities for redundant tokens, so collaboration needs a deliberate information boundary. Walden expects models co-designed for collaboration to improve on these early techniques. Tanay adds that the natural units of routing are subtasks and session phases: exploration, implementation, and later checks can each demand different expertise and levels of intelligence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cheap tokens can produce an expensive task
Alex describes an unresolved question inside OpenRouter: should the outer model doing the orchestration be large or small? Even the pricing consequence is not obvious. A large orchestrator may repeatedly reuse cached context, changing its effective cost. More broadly, fusion tries to combine the benefits of training data, compute, and reinforcement learning from multiple labs. A small model can be economical when the task lies comfortably within its training distribution; outside that distribution, repeated tool calls and unproductive loops can consume the apparent savings.
Alex reports that on Terminal-Bench, Opus performs about three times better at one-tenth the cost of Haiku, despite Haiku’s lower per-token price. He does not specify model versions, the benchmark version, the harness, the performance metric, or the cost accounting in this discussion. His explanation is that an undersized model can spend far more effort trying to solve an unfamiliar task. By contrast, distinguishing a person’s name from an organization’s name is his example of familiar classification work suitable for a small model.
OpenRouter Fusion explores both distribution fit and the arrangement of inner and outer models, alongside primitives such as sub-agents and advisor tools. Asked whether domain fit determines which model should lead, Alex does not claim an established rule. In the published OpenRouter Fusion work he describes, the stronger outer model produced the best deep-research results. Coding was not yet well optimized, and the preferred arrangement for other workloads remained unclear. Dollars per successfully completed task is the relevant efficiency target—not the advertised price of an individual token.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A sidekick keeps its own running context
Walden points out that caching can benefit the side agent as well as the main agent. Devin Fusion—distinct from OpenRouter Fusion—uses a persistent sidekick with a continuously running context. The main agent does not need to reconstruct the sidekick’s earlier context every time it delegates another task.
| Collaboration pattern | Context behavior |
|---|---|
| Fresh sub-agent | Earlier context must be supplied again |
| Persistent sidekick | Earlier context remains in its running session |
Walden describes cached input tokens as ten times cheaper in the setup under discussion. The operational distinction is prefix reuse: current Claude prompt-caching documentation separates discounted reads from cache-write charges, so neither that multiplier nor a later supervisory call should be treated as universally free. With both contexts maintained, the smarter model can move between leading and assisting.
Persistence also changes what a model should learn. Cognition is experimenting with reinforcement learning in both collaboration roles:
- Orchestrator: Decide what to delegate and issue useful instructions to another model.
- Executor: Carry out another model’s instructions effectively as its sidekick.
The goal is to co-design models with the orchestration system instead of only assembling independently trained models. The moderator connects this to Nemotron post-training: if a model will operate inside a routing-heavy harness, that environment should shape its training.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Routing can happen inside a model artifact
Tanay introduces Flextron, describing a main model distilled into smaller footprints, with the task determining which footprint performs decoding. The within-artifact mechanism is flexible activation of model capacity: nested elastic subnetworks can provide different amounts of computation without requiring a collection of independently deployed models. This moves part of the routing decision from choosing an endpoint to choosing how much of a model to activate.
Access to training recipes and data can help characterize whether a question is familiar to the model. Training studies can also expose distillation gaps between a teacher and its smaller student. Those gaps need not be uniform: including a domain in the training data does not guarantee that the student absorbed it as effectively as other domains. Task-sensitive model footprints therefore need evidence about retained capabilities, not just parameter counts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remember where the evidence lives
Tanay returns to context through abstract syntax trees, Headroom, and RTK. Ordinary compaction is lossy. Structured representations that an agent can later expand offer something closer to recoverable compression, though he stops short of calling it fully lossless. The distinction is whether omitted detail is gone or remains accessible somewhere else.
Walden uses a human-memory exercise: how many spoken numbers can someone retain before losing track? Limited working memory does not prevent effective work because people use external records. An agent likewise need not remember a whole file if it remembers that the file exists, where it is, and which parts matter. A well-designed harness should give the agent what it needs to find the information, even when the information is not currently in its context window.
A sidekick can therefore return file references instead of dumping all its findings. The main model can inspect selected passages or decide that one verification command will answer its question. For a concrete implementation of that handoff, suppose codebase exploration produced a note about an authentication path. Store the full note and pass only its location and a short summary:
python
import json
from pathlib import Path
def save_finding(root: Path, finding_id: str,
summary: str, details: str) -> str:
root.mkdir(parents=True, exist_ok=True)
path = root / f"{finding_id}.md"
path.write_text(details, encoding="utf-8")
return json.dumps({"summary": summary, "path": str(path)})
handoff = save_finding(
Path("agent-notes"),
"auth-path",
"Review the token-expiry check before changing login behavior.",
"Exploration note: inspect the token-expiry check and its callers.\n"
"Status: investigation pending; no code change or test result recorded.\n",
)
print(handoff)
The message contains a pointer and the work’s status; the file retains the fuller record. Walden’s broader observation is that smarter models can be selective about tool use and reading, so adding an expensive model can sometimes reduce the total system’s token consumption and cost. Tanay similarly emphasizes token efficiency as a reason model size alone is an incomplete cost predictor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cache-aware routing meets the always-running agent
There is a boundary between model orchestration and inference infrastructure. Alex explains that OpenRouter is a marketplace and generally cannot inspect a provider’s KV cache unless it runs the model itself, which he says is rare. It can optimize for cache hits and pass their savings to customers. It can also consider remaining cache lifetime when deciding whether to keep using a model.
A better model match may justify abandoning a still-useful cache. Alex’s example is a switch with two minutes of cache life remaining: the routing gain must outweigh the lost reuse. OpenRouter had begun experimenting with that tradeoff, including how much tolerance a customer might allow, but had not exposed those controls to customers at the time of the discussion.
The demand for automatic routing did not initially come from such fine-grained optimization. Alex says OpenRouter’s auto router had existed for almost two years with little adoption because people preferred selecting specific models. He describes an adoption surge around January of the recording’s year, associated with OpenClaw. His example is an agent sending a heartbeat about every ten minutes to its configured model: if Opus is the default, expensive intelligence gets spent on routine background checks. That is a deployment example, not a universal cadence; current OpenClaw heartbeat documentation makes cadence and heartbeat-model choice configurable.
Heartbeats and substantive agent work have sharply different intelligence requirements. As open models improved, separating those workloads became practical, and Alex says more agents and applications began using OpenRouter’s routing options.
| Option | Role described in the panel |
|---|---|
| Auto router | Select a model for the request |
| Pareto Code | Select a coding model using a tunable tradeoff threshold |
| OpenRouter Fusion | Orchestrate models into a fused result |
OpenRouter wants to support both developers building sophisticated orchestration and users who simply set a model slug in an existing harness.
Two changes reinforce each other. Breaking a problem into planned steps can improve the work itself, including code quality. Meanwhile, the workload has moved from question-and-answer exchanges to reasoning and then to continuously active agents. A long-running agent mixes important decisions with routine background activity, making one fixed level of intelligence an increasingly awkward fit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Local capacity and shorter context change the calculation
Carter adds two routing objectives beyond answer quality and cloud-token savings:
- Privacy: Detect sensitive information and process it on device, potentially anonymizing it before sending a more demanding workload to a cloud model.
- Owned-compute utilization: Send suitable work, such as background heartbeats, to an underused DGX Spark rather than leaving purchased capacity idle.
His economic framing is incremental: after buying the hardware, additional local work incurs electricity costs while cloud requests still incur their token charges. He describes hybrid local/cloud routing as an emerging area of NVIDIA’s work.
For self-hosted models, Tanay points out that growing context can lower throughput. Compaction can recover some of that throughput without changing the model. This creates two different levers:
| Lever | What changes |
|---|---|
| Compaction | Fewer context tokens to process |
| Model routing | A different cost and capability per token |
Walden responds that compaction alone cannot eliminate large differences in model intelligence and cost. It also changes context prefixes, potentially sacrificing cache reuse. In Walden’s compaction example, newly uncached input costs ten times the previous cached-input price. The tradeoff depends on which prefixes remain reusable and how much context is removed.
For Cognition, Walden says the primary reason to compact is preserving intelligence. Walden recommends avoiding contexts above 200K tokens and staying below 100K when possible, despite advertised million-token windows. That is his operational guidance from agent work, not a measured limit established here for every model. A natural opportunity to compact is during a model switch: when losing the old model’s cache reuse is already unavoidable, the handoff can start with a smaller context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use supervision opportunities to detect trouble
Alex asks whether a small model generating many tokens might signal that it should hand work back to a larger one. Carter sharpens the question: what detects the need for help, and what mechanism triggers it? Walden says small-model self-detection is imperfect, so the large model often has to make that judgment.
One opportunity comes from cache maintenance. Walden describes a roughly five-minute default cache lifetime in the provider setup being discussed. If the system is already making a call to refresh that cache, it can ask the frontier model to examine the worker’s progress: is it pursuing a useful path, or disappearing into a rabbit hole? The idea is to combine necessary cache maintenance with a useful supervisory decision, reducing the incremental overhead of the check.
The lifetime itself reflects an operational policy. GPU memory can retain only so much cached state; inactive caches may be evicted or offloaded. Self-hosting gives the operator more control over retention, subject to available resources. Tanay explains that compute-to-memory ratios, unified memory, and systems such as Vera Rubin create different choices. Five-minute and one-hour windows are not physical constants.
That control matters when a workload is predictable. Tanay compares these illustrative request shapes:
| Workload | Cached context | New input | Output |
|---|---|---|---|
| A | 32K tokens | 1K tokens | 1K tokens |
| B | 64K tokens | 1K tokens | 1K tokens |
An API provider prices and optimizes across a mixture of customers. A self-hosted deployment can optimize for its own recurring shape, which Tanay argues can lower cost. The table is a workload comparison, not a measured savings result.
Walden recalls the early economics of Devin in 2024, before API cache discounts made repeated context cheaper to users. Sending the same large context again could mean paying full input price again. Cognition bought direct compute capacity from providers instead of paying per token, capturing the underlying benefit of context reuse. He would like longer-lived cache storage, potentially backed by something like S3. Tanay points to NVIDIA Dynamo for prefix-cache optimizations, while explicitly qualifying that recommendation as less relevant to a DGX Spark setup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure confusion, then inspect the trace
Returning to Alex’s proposed token-count trigger, Walden says Cognition has not yet explored it. Some small models are less token-efficient than others, and models may be trained on their own characteristic traces. Whether a long output indicates productive work or confusion needs empirical testing.
Tanay proposes another source of signals: probes over internal model states. Magnitude analysis, linear probes, or other classifiers could estimate hallucination tendency or aspects of uncertainty, providing a proxy for how lost the model has become. Prefill and cache states contain vector representations on which such classifiers could operate. These are candidate indicators for routing; the panel does not present a validated detector or threshold.
Switching models also raises prompt portability. Different architectures may respond differently to the same instructions. Alex’s answer emphasizes the accumulated recovery knowledge inside an agent product: domain-specific edge cases, repeated failure loops, and the prompts that help escape them. Calling an advisor requires different instructions from assigning a subtask executor. Engineers and agents can inspect those traces, adjust the prompts, and observe whether accuracy improves over time. The mechanism is an observable repair process, rather than a universal translation rule between models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn from the routing decisions users correct
Walden describes a production-feedback capability he wants for Devin Fusion but says is not yet available. Dataset tuning is useful, but real usage supplies a different stream of evidence: a user upgrades the model, downgrades it, or the system discovers that its initial assignment was wrong. Those events could connect the original prompt, the actual route, and a better route. An internal research loop could then revise the router against the workload people actually bring to the product.
Tanay asks about prompt tuning and GEPA. Walden responds with a preference for diagnosis by a capable model over the low-level mechanical prompt-tuning approaches he describes. His preferred repair procedure is:
- Give the model the routing decision and its context.
- Ask why it chose that route instead of the desired one, with references to the relevant prompts.
- Have Devin update the prompts.
- Rerun the failed case as a regression test and check that the behavior changes.
He acknowledges that this is a heavier process. Its appeal is that the system reasons about the failure before editing the instructions that caused it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What remains when models become better collaborators?
The closing discussion returns to local hardware. The moderator argues that when a DGX Spark has high memory utilization but spare compute capacity, concurrent collaborating agents can help use more of that compute. Collaboration therefore matters both to cloud orchestration and to extracting useful work from edge hardware. That raises the architectural question: will routing remain a product, or become ordinary behavior inside models and harnesses?
Walden says Cognition is already training for collaboration and describes newer frontier models—including Fable and the GPT-5.5 and GPT-5.6 models he names—as increasingly capable delegators. Carter sees the model and harness developing together: a strong harness cannot be separated cleanly from the model behavior it relies on, and a strong model still needs the surrounding system.
Tanay expects an arbitration layer to persist. Applications built from nondeterministic models operate with limited trust, and no individual model has complete visibility into every other model’s behavior. Improvements can be distributed across models and harnesses while a controller still decides among them. He compares this with the centralization of traffic-routing controls in web infrastructure: better participants do not eliminate the need for coordination.
Alex tests the strongest counterexample: suppose one future large model can solve every task better and more cheaply than Haiku. Why would it delegate? Another model might already hold the relevant context in cache. An orchestrator might know more about that context placement than the model doing the work. Limited memory and the location of useful state can therefore make collaboration worthwhile even when capability differences shrink.
There is also a more immediate mismatch between capability and spending. Alex says Opus ranks first on OpenRouter for classification tasks by dollars spent. That is a spending ranking, not a ranking of classification quality or request volume. He sees substantial room to move easy, familiar tasks to smaller models, and expects those tasks to grow as a share of the workload while frontier models concentrate on more valuable work. Routing’s continuing opportunity lies in that allocation: matching intelligence, available context, and compute to what the next piece of work actually needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Multi-model panel experiments on 100 DRACO deep-research tasks, including methodology and limits for coding applications.
An elastic model architecture that routes tokens through subnetworks to meet latency and accuracy targets.
A context-compression layer for tool outputs, logs, files and retrieved content.
A CLI proxy that reduces the command output passed into model context.
Reflective optimization tools for prompts, code and other text artifacts.
NVIDIA's framework for distributed inference serving at datacenter scale.
Further reading
- LLMRouterBenchPaper
A unified evaluation of model routing, complementary capabilities, cost tradeoffs and gaps to oracle selection.
Updates since the talk
Current cache pricing, prefix-matching rules and cache-control configuration.
Configure heartbeat cadence, model choice and context handling.
Read the complete timestamped transcript
- 0:00
[upbeat music] Have been really exciting. We've tried to get a bunch of the industry leaders together to talk about some of the problems that are-- we're facing as we try to run more on local.
- 0:21
If you guys were here for the first panel, one of the things that we talked about was model routing. We firmly believe that we're in a mo- a multi-model world.
- 0:29
I think you heard this from many of the panelists. Anyone who is deploying AI in production and who is doing so locally is seeing that multi-model world. That's why we released these Nemotron models at NVIDIA.
- 0:40
Everything is released from the datasets to the weights with recipes so that you can customize them. We do that because we know that people customizing models is gonna be huge.
- 0:49
And so this panel is really exciting 'cause we're gonna talk specifically about model routing. So as you are picking which model to use, how does that-- how-- essentially, how does that tooling itself look?
- 1:00
Um, do you guys wanna introduce yourselves?
- 1:03
Yeah, sure. Um, I'm Walden. I'm the co-founder of Cognition. We build Devin AI software engineer. Uh, in addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models, uh, and these agents.
- 1:15
And one of the things they're constantly asking us nowadays is basically, "How do I know the ROI of our models? And how do I know which tasks I can actually let our engineers spend the most expensive models on versus, you know, letting them use a more cost-efficient model?"
- 1:29
And so that's why we're, we're also thinking a lot more about multi-model routing nowadays.
- 1:32
Totally. Yeah.
- 1:33
I'm Carter. You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at NVIDIA, and, uh, ultimately, I spend a lot of time thinking about how to get intelligence into as many developers' hands as possible.
- 1:46
And, uh, something that is continually becoming a-- not an issue, but something that is top of mind for a lot of developers is, uh, as you use more intelligence and the frontier models get more expensive, uh, it becomes somewhat cost-prohibitive to, to use the best, uh, tools-- what feels like the best tools as much as you would
- 2:03
like to use them. And so this has become a recent, uh, you know, focus is how can we, you know, still get the same desired outputs, but actually, uh, both as an individual developer, but also imagine startups and small companies, how can you leverage this incredible tool without totally breaking the bank?
- 2:21
Uh, I'm Tanay. I work on model evaluations, uh, both in terms of its accuracies and efficiency and cost understanding of the model. Uh, and then I try and understand those-- uh, implement those learnings and, uh, help build a router.
- 2:38
So it's basically, uh, my job is to understand the behavior of, um, of the model on an intimate level and then use those learnings to both improve the model and try and design a system of model that can work together with each other.
- 2:52
Totally. Um, yeah, I love a lot of the research that you're doing at NVIDIA as we kind of see the space through. I think what's really interesting is model routing itself is pretty new still.
- 3:01
And so what you'll notice is there isn't a very clear solution here. That was something that came up on the first panel, is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here 'cause we're still figuring out how to best do these patterns.
- 3:15
And I think, Walden, I wanna kinda ask you. So, um, Cognition just released Fusion, your guys' model router.
- 3:21
Yeah.
- 3:21
And when you guys released it, you in your blog said that you're actually getting better performance than Fable, um, than these frontier models. And I feel like that was a very surprising statement to hear because we're thinking that you're getting as good or close enough usually when we're running on Edge, when we're running local in these compute
- 3:38
strain, smaller footprint models, but you guys are getting better. Can you explain how? [chuckles]
- 3:44
Yeah, absolutely. Um, so I, I, I also wanna be, be clear about something here is like, you know, we're not saying that like we gap above Fable-level performance-
- 3:50
Totally. Yeah, yeah
- 3:51
... in the same way that maybe Fable-level performance gaps above other models. I think actually there's this really unintuitive, um, dynamic where smarter models actually get better and better at delegating work.
- 4:03
And so one of the philosophies we had with building a model router is we don't want to route people to a dumber model, and then suddenly you're stuck with a model that doesn't know how to do your task, and next thing you know, you're, you're switching yourself back to smarter model anyways and now taking that expensive cost.
- 4:18
Uh, in general, we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago. And so we really wanted to put out a new framework that actually lets people still feel like and, and still have a frontier model in their system while getting all these
- 4:33
like cost benefits. So yeah, we, we're reducing the cost of Fable-level intelligence by forty percent. The way we do that is we allow Fable to still do like the planning and the, the hard decision-making, but delegate a lot of the work to an implementation model.
- 4:46
Mm.
- 4:47
Um, and the implementation model can be, you know, one of these open source models, be it a, a cheaper mini model. Um, the unintuitive thing is, even though it's cheaper, because you're delegating the work to another model, you can let that model go at the task with much more depth and intensity than you might otherwise.
- 5:01
Like, you can spin off like, you know, three sub-agents to go and like explore the code base. And maybe that's actually more, uh, comprehensive than if you had just let Fable explore the code base itself.
- 5:10
So you're actually getting-
- 5:11
Mm
- 5:11
... this like nice trade-off where it's both more cost efficient and also more comprehensive overall.
- 5:15
Interesting. I see. So you're saying, um, by using a bunch of smaller models, you're essentially like, for one example, scouring the code base. You can, you, you can explore it potentially better than if you were to just have one model, I don't know, figure out with what it's limited-- with its limited context, with whatever path it's on.
- 5:31
Yeah, totally.
- 5:32
But, but also if you think about the budget of if you were to say the frontier model costs this amount per token and the smaller model is this amount per, per token and it's significantly cheaper, then you can use a lot more tokens from the smaller model-
- 5:45
Oh, that's right. Yeah, I see
- 5:45
... with still within the budget that it would've been from the frontier model.
- 5:48
I would also like to encourage everyone to think, uh, there are jagged capabilities in most models, right? So coding is not one domain. Within, let's say, data visualization, uh, there'll be scikit-learn, there'll be matplotlib, there'll be something else.
- 6:02
Uh, it largely come downs-- comes down to the- ... cor- training corpora that, and, uh, that went into each of the models, so, right? So one model, while you're trying to do X type of work, let's say data visualization, and the other type is Y.
- 6:18
That means, let's say, model building. Let's say you're trying to have a data science work stream, or you're trying to optimize for some kind of prediction and then visualizing, uh, your results.
- 6:29
Within that task, uh, different models will have different strengths. So not nec- it's not necessary that model A, if it scores higher on a coding benchmark, is just plain better at every task there is.
- 6:42
So routing is a task of intimately, in- intimately understanding the behavior of and strengths and weaknesses of different models and then applying them thusly, right? I would, I would encourage everyone to think, "Hey, models are strong at different things," rather than like, there's one model to rule them all.
- 7:02
I see. And by the way, real quick, thank you, Alex, for joining. Yeah. [laughs]
- 7:05
Sorry I'm late.
- 7:05
No. Oh, is that still...
- 7:08
I might need yours.
- 7:09
Sorry I'm late. [laughs] [laughs] I'm Alex from OpenRouter. Um, thanks for having me, Nav.
- 7:14
Yeah, of course. Thank you so much. He came right from the airport, so this is perfect. I think, um, Tanay, that's, that's super interesting. So, um, the way that you're thinking through, through model routing, it's not even just, uh, delegating to necessarily a smaller model, but like...
- 7:26
And maybe this is kind of what you're saying, is can you, can you put essentially a swarm of agents to accomplish the same task and suddenly routing the task between them is, is much, is a, is a problem to solve in and of itself.
- 7:36
Yeah. So if you look at, like, let's say, let's take an easy example. Let's take a science or, like, scientific discovery a- as an example, right? Usually these are one-shot problems.
- 7:46
It's ex- incredibly hard. You have, uh, models think throughout this process, right? So in that, you have tons of subdomains, like tons and tons and tons. So in that aspect, if you think about post-training, uh, like the post-training process of a model, they'd be tuned with different teachers, they'd be tuned on different subtasks.
- 8:09
Uh, so those, those, um, those overlapping strengths will be readily apparent when you're trying to understand failures of each models on different, different subtasks. Once you understand that, you can orchestrate your system to leverage that arbitrage essentially, and that essentially becomes free.
- 8:28
So, uh, I think this is on, uh, LM router bench. There was, there are tons of benchmarks out there, uh, but if you use these techniques, you can get like up to 10% higher accuracy even, right?
- 8:39
It depends on the model pool, depends on the task at hand. But I would encourage, uh, to think about the complementary nature of models.
- 8:48
I see. Do you see-- So, um, in kind of the way that you were describing the way the task gets broken up, do you see that the, some of the smaller models, because the token cost is cheaper, are they using more tokens?
- 8:58
Like is it, are, are you specifically routing so that they do, uh, that they, so that they are chattier?
- 9:03
Uh, oh, yeah. I mean, they, they absolutely do use more tokens. Um, I, I actually, I, I wanna, uh, kind of like riff on something that Tanay was saying, which is like, you know, a, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than like even the
- 9:17
frontier models in certain cases. Um, I think a lot of people, they look at this, and they immediately jump to, "Oh, how can we just route like, you know, the task where the smaller models do better just straight to the smaller mo- models?"
- 9:28
I, I think that one of the themes we really want to emphasize with our recent, uh, blog post and recent dev infusion was that this kind of like naive, like initial routing to...
- 9:37
based on the task type is extremely fragile, uh, especially the more agentic the task you, you work on is. So for example, like a real developer, you might ask your agent first, "Oh, how does this code base work?"
- 9:48
And then you, you go deeper and you're like, "Okay, actually, can you implement some features for me?" And then you go deeper, and it's like, "Oh, can you like now go do like a live test of this feature and debug deep cases?"
- 9:56
Uh, the complexity changes and the type of task changes over time, and you don't want to be left with some subpar model for the, the task that you're now on.
- 10:04
I think this is why people like frontier models so much, is they're, they're just like generally intelligent, and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks.
- 10:15
Um, and, and so the challenge is: How do you get a small model to know that it's out of its depth, and you need to now like go switch to another model or go like, you know, go to a smart, smarter model?
- 10:26
And, and our solution to this is you kind of just always have this like main frontier agent that's watching, even if it's not the one doing the work. It should at least be keeping tabs to figure out, "Okay, wait, like the, the agent I delegated to now is like out of its depth.
- 10:38
I need to kind of like move it to, to something else." Um, and, and overall, just the guarantee of always having frontier intelligence present, I think reduces the, the fragility of, of these systems quite a lot.
- 10:48
How does the sharing of context between one of those smaller agents who has basically completed up to some level of a task and decides, "Actually, I don't think I'm the right person for this.
- 10:59
I need to hand it back to the, the foundational model." Um, of course you don't wanna have the entire trace of that-
- 11:05
Yeah
- 11:05
... smaller agent be passed back to the larger model. So how do you get that level of specificity while, uh, basically providing the information it needs, but not more?
- 11:14
Yeah, absolutely. So I, I, I think like the, the context here is it's actually very easy to accidentally create a system that's more expensive as soon as you're running like, you know, multiple models together.
- 11:23
'Cause, oh no, like, you know, this one file reading, now every, every one of these models is now reading this one file reading, so you, you're getting charged like three times as much.
- 11:30
That's-
- 11:30
Um, the, the, the trick that we, we spend a lot of our time on is, um, most of the context by default will only be going to like one model.
- 11:37
So like most of the context, let's say, will be going to the small model. But the thing you need to then tune very well is, okay, like maybe you still show like what files it's reading.
- 11:44
Maybe you show like the high-level thinking of what it's doing back to the main model. Maybe you have the small model, um, you tune its ability to present the context back to the main model.
- 11:53
Um, and actually the, a lot of these problems already have been well studied in many domains already, like context compaction is something you already have to solve if you want to do like really long-running agents.
- 12:03
And so this problem of taking long context, compacting it in a way that is now understandable is the one that you can also apply to this domain and just kind of give the compacted context back to, back to the main agent.
- 12:13
Context compaction is something that, you know, I, I'm familiar with, but I hadn't really thought about as you're doing model routing and as you're trying to share context across now potentially many models, you're, you're expanding the amount of what could be seen as wasteful tokens or redundant tokens just be- just because you have to process that across
- 12:29
the many models.
- 12:30
Yeah. Yeah. Um, I, I, I think there's-- Uh, the way I describe it is I think we are early in, in the, in the model routing domain. Um-
- 12:38
Totally.
- 12:38
I, I, I hope that a year from now that even the techniques we kind of used for Devin Fusion, you'll be able to look back on that and like, "Oh, these are some, like, really legacy ideas," and, and now we have, like, much better methods at routing between models.
- 12:49
Um, and when people actually start co-designing their models with this in mind, we're, we're gonna ha- be, be in a much better world.
- 12:55
Yeah. I, I echo, uh, what you say, right? Um, I think routing
- 13:02
will, uh, evolve as the task evolves when, when you start the task, right? So, um, it's more useful to see things in terms of sub-tasks and sessions, uh, than individual problems that you're trying to solve.
- 13:15
Because more than likely you're a- when you're working through a problem, you're asking a lot of questions, you're, you're exploring different things. Um, and it, it is imperative that you try-- Like, uh, people who design routers, it's imperative that they try and understand these phases, uh, of different complexities and then try and a-apply some logic, uh, for
- 13:39
essentially, uh, sidekicking, uh, uh, tasks or, or, uh, leveraging expertise from other models. Uh, that's, that's pretty on point.
- 13:49
Yeah. I, I'd love to hear from the, the, the router guy. [laughs] [laughs] [laughs]
- 13:53
Yeah. Um, I, I think those are-- these are, like, important points, and one of the biggest, um, debates I think we have internally is whether that outer model that's doing the orchestration should be the big model or the small model.
- 14:07
Yeah.
- 14:07
Mm-hmm.
- 14:08
Um, you get, like, very different results depending on your choice. Um, and in fa- and it's not even clear what the pricing impact would be. 'Cause if your outer model that's doing the orchestration is the big model, it can leverage its caching to, like, make more of its decisions-
- 14:22
Mm.
- 14:22
Um, and its caching is going to be, like, a dramatic price savings compared to the small model's caching a lot of the time-
- 14:29
Yeah
- 14:29
... um, especially for, like, perform-- for, you know, uh, issues that are on the bright line. Like, zooming out a little bit, I think the-- what, what you want from all the models out there when you do model fusion is to benefit from all the data that is being trained on across all the labs and, and not
- 14:50
just the data from one lab, right, or one source. And a, a model is just, like, a combination of, like, the data and its understanding of the data, both its compute and the quality of its RL.
- 15:02
So, um, long term, I think you want, you want mo- uh, models where they know that, "Oh, this is, like, in distribution. Like, this is in my data." You can use small models pretty easily, um, and get a cost savings.
- 15:18
But if it's out of distribution, small models may actually increase your cost because of how often they'll, like, call tools and how crazy the loops will be. Like, if you run Terminal Bench on Opus and Haiku, like, Opus will do about three times better at one-tenth the cost of Haiku- [laughs] [laughs] ...
- 15:37
even though Haiku is significantly cheaper per token. So it really becomes a huge problem if you use a too small of a model, particularly on tasks that are out of domain for the training data.
- 15:49
When you're doing something like classifying text, like, "Hey, is this, like, a person's name or is this an organization's name?" That's super in domain, so you would ne- you don't want that kind of task to go to a large model.
- 16:00
You want it to go to a s- a small model. Everyone has that in their domain. Um, so d- like, being able to understand in domain, out of domain is a lot of work that we're doing for, for, um, Open Router Fusion and, uh, and then also figuring out, like, what, like, how to orchestrate the outer and
- 16:16
inner models, um, for different types of tasks. And, like, um, you know, it's, uh, it's an early industry. It's, like, an early field of research. Uh, most research on model fusion has not been very detailed, not been very, like, you know, optimistic sometimes.
- 16:32
Um, it's only, like, just, you know, recently getting, uh, more optimistic, and I think, um, I'm personally very optimistic about it. And, uh, you know, it, like-- We-- We're a very, like, ecosystem-driven collaborative company, and a lot of our-- Like, we work with a lot of partners to try to, like, help improve their orchestration pipelines with good
- 16:52
primitives, like the, you know, sub-agent and, like, uh, the advisor tool, um, which is kinda similar to what you were talking about.
- 16:59
I'm curious. So, uh, help me understand. Y- When, uh-- If a s-- It makes total sense that, um, a small model, if it's in domain, would be cheaper, but if it's not, then it's gonna thrash around, uh, as it tries to get an answer.
- 17:12
When you're describing whether the, uh, con-- like, the main agent should be the lo- the, the local model or the cloud model,
- 17:22
does that-- is that d- a decision that's then dependent on whether the task is something that it, that's in domain or not? Does my question kinda make sense?
- 17:30
I don't know. Um, I basically, I don't, uh-- It's kinda early to-
- 17:36
Yeah
- 17:36
... um, to say. I, I think the, uh, what, what-- The results that we published, which are, uh, a couple weeks ago, which were focused on deep research, not coding-
- 17:47
Mm-hmm
- 17:47
... um, we had the smart model be the wrapper model and be the outer model, and we got the best results from doing that. Um, but it, it, uh, for deep research, it works the best.
- 18:01
Um-
- 18:01
Mm
- 18:02
... for other tasks, it's, like, kind of unclear. Like we-- There-- It's-- Uh, fusion is not super well optimized for coding, and, uh, it might be that, like, a smaller model ends up being, like, a higher, um, efficiency per-- or, you know, fewer dollars, um, per, like, completed, successfully completed task.
- 18:21
But it's kinda early to say.
- 18:22
Yeah.
- 18:23
Yeah.
- 18:23
One thing you said earlier is, like, oh, you get the caching benefit from, like, the mainline agent. Um, you actually can get the caching benefit from the side agent.
- 18:30
And, um, this is actually one of the key things we talked about with our Devin Fusion launch, is that, um, you kind of are leaving a lot on the table if you do a main agent and sub-agent type system- Um, s- so we don't use sub-agents.
- 18:43
We use what we call a sidekick, which is, um, one sub-agent that continually has a running context. So the main agent doesn't need to re-provide a context from earlier.
- 18:52
Um, it's all still in the KV cache, right? Like, it's 10 times cheaper on, on all those cache tokens. Um, and then if you want to, like, switch the smart model to be, like, the one on the side or the one, uh, in charge, um, it's actually totally fine, and you can kind of like do, do the
- 19:06
swapping back and forth. We're also spending a lot of time right now thinking about
- 19:11
how do you train models to actually work collaboratively with other models. I actually, I think there's a lot of literature out there on how you RL one model to do a task end-to-end.
- 19:20
How can you RL a model to also be good at collaboration? And when we think about it, we actually try both of these setups where let's RL the model being the orchestrator and the one deciding what gets delegated to other models, see how well that performs.
- 19:32
Um, and we also orchestrate it in a way where the model we're training is actually the executor, the sidekick, and see how well is it at executing other models' instructions.
- 19:42
Um, and we expect that to, to be a, probably a big lift in this next step of, like, multi-model orchestration is don't just, like, take models as they are and orchestrate them, but, like, can you actually co-design your models with the-
- 19:52
Interesting
- 19:52
... orchestration system.
- 19:53
Yeah, that makes sense. I mean, with Nemotron and, you know, with all the foundational models, we're, um, we're essentially post-training them for the harnesses that they're getting used.
- 20:00
Mm-hmm.
- 20:00
If the harness is gonna include a lot of routing, then that makes sense that that makes its way into the post-training.
- 20:05
Yeah. Um, are you guys thinking a lot about the kind of like the model training at, at NVIDIA for these kind of purposes?
- 20:12
Yeah. So we have a technology called Flextron.
- 20:16
Hello?
- 20:16
Oh, the mics go.
- 20:18
Trying.
- 20:18
Hello? [laughs]
- 20:20
Testing. Oh, this one works.
- 20:21
Oh, wow.
- 20:21
All right. [laughs] [laughs]
- 20:23
Okay. Uh, so we have a technology called Flextron. So you have, we, we have a setup where, uh, there's a, the, uh, there's a main model, then we distill it, uh, into smaller, uh, footprints.
- 20:36
And then based on the, based on the task at hand, you can switch which model does the decoding, right? So there's a lot of fancy stuff you can do, uh, within a model artifact to, uh, to essentially only activate a class of model or a section of weights depending on the task at hand or the complexity at
- 20:58
hand. Most, in most cases, you can essentially understand the novelty of a question to a model if you have access to the recipe, uh, with which it was trained, uh, trained.
- 21:11
So this works very well for open models, right? Like, or any model you have access to its data for, right? Because you can literally decide if it's in, like, see if it's in distribution or not.
- 21:21
Uh, again, if you have, uh, studies from when it was trained, you can also see how much, uh, essentially, uh, how much was your distillation gap across teachers and, uh, the artifact that you trained, right?
- 21:34
Because sure, you have domain data from all, uh, all the different domains you're tuning, but it's not guaranteed that it, uh, absorbed all the da- data evenly across the model, right?
- 21:44
So, um, it, it becomes, it, it becomes very interesting, uh, to start thinking about, uh, these flexible weights and, uh, flexible model sizes, uh, essentially. There's also... I wanted to add about the context piece, right?
- 21:59
So, uh, how do you think about co-, uh, ASTs and, uh, and context compression rep- representations? Uh, compaction in its very nature is lossy, right? So, uh, just like Headroom is there, RTK is there, right?
- 22:14
Uh, these, uh, these bases, code bases are usually designed to have representations, uh, that we carry forward through life, and you essentially give models the capability to further expand on them.
- 22:28
It's, it's more, more like loss-less-ish compression, which can retain states of models or, or states for agents. What do you think about that?
- 22:37
Yeah. It, I think this gets to, like, kind of like a, a fundamental philosophy of how agents and context should work. Um, one, one exercise I like to do is like, you know, as a human, like, how many numbers can you...
- 22:46
Like, if I just start spitting out numbers now, right? Like, how many can you remember before, you know, like, you, you start losing track of them. I think it's actually very few, right?
- 22:54
So in some ways, you could argue that your context window is actually shorter than these language models. Um, and yet you can actually be very effective-
- 23:00
Totally
- 23:00
... at that, right? [laughs] Um, your, your context is, is very lossy. Um, I think one of the nice things that people are starting to realize with agents is like you have a lot of non-lossy systems that you can fall back to.
- 23:10
So you have a file system. Like, uh, if in your memory all you remember is that you read some file earlier, you don't need to remember the whole file.
- 23:16
You maybe remember the important parts, but you can still have the full version of the file on your system. And that's kind of m- my goal when I'm thinking about how do we build a good context engineered harness is the harness should have everything it needs to find what it needs to have, even if it doesn't have
- 23:33
everything immediately available.
- 23:35
In that case, uh, do you think that the context sharing problem will become cheaper and less of a problem in future?
- 23:45
Yeah, it's de- de- definitely possible as well. I've, I've seen cases where, um, you know, the sidekick agent does a bunch of work. It tells the main model, "Oh, yeah, like, here's, here's all the things I found," instead of dumping the full thing, it just references them by file.
- 23:58
And then the main model is actually generally... You find these larger, smarter models, they're actually more token efficient with how they use tools and how they read, and so they actually read the files in a way where they only see the important parts, right?
- 24:09
Or they decide that, "Oh, actually I only need to look at a subset of this," or, "Oh, I can run a single command and just know if everything i- is done properly."
- 24:16
Um, it, it's, it's actually quite amazing, um, the fact that, you know, these multi-model systems, they actually seem to scale and get better with intelligence, which is, um, not something we should just take for granted, right?
- 24:27
It's not obvious that actually more expensive models are actually creating an overall cheaper system.
- 24:33
Yeah. Like, uh, the scaling laws, uh, if you have a larger model, it's gonna be more, uh, efficient with its tokens. Uh, smaller models, less efficient with its tokens.
- 24:43
Yeah.
- 24:43
I guess I had a question for you, Alex. Uh, do you guys, uh, like f- weigh more importance on, uh, the actual infrastructure side of routing? So for instance, KV cache-aware routing or, uh, like is that where most of the business is right now?
- 24:59
Or, or the- are you seeing strong, uh, pull of like people actually deploying routers in production?
- 25:07
So OpenRouter i-i-is a, like a marketplace for language models. We, we exist at, um, uh, w- like we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare.
- 25:23
Um, we, we, we do spend a lot of time optimizing for cache hits, and we, we like pass through cache hits directly to users. Um, but w- like in terms of KV cache optimizations, we, we, we can't do any like specific work there.
- 25:38
What we do for, um, model routing is we, we try to like find the best model or best combination of models for the prompt, and then when we see a cache hit, we will, we'll like use up the duration of the cache and send the downstream customer like the f- the full savings of the cache hit.
- 26:00
Um, there's more work that we can do here where we could say, "Okay, this looks like something where there's significant benefit to switching the model right now," but you haven't used up the full cache.
- 26:11
You still have like two minutes left. Um, and it-- we think it's probably worth switching the model and losing the rest of your cache, um, and letting people kind of like tweak their tolerance for that behavior.
- 26:25
Um, we've, we've been doing a little bit of that, but we haven't like exposed it to customers yet.
- 26:30
What, what is next for you guys in terms of your, your, your model routing? 'Cause as, as you mentioned, you know, uh, it is kind of like a, a different direction from like the, the marketplace business that exists today.
- 26:39
So I, I'd love to hear.
- 26:41
So I mean, we've been doing-- like we, we've had a, an auto router for like two years almost. Um, but when we launched it, there was like no adoption of it.
- 26:51
It wa-- people really wanted to use specific models. Um, and the, the auto router just had like no real usage. It was-- Uh, we mostly saw it as like a, a discovery play point.
- 27:02
Like, "Hey, this is how you discover which model might be good for your prompt." And then, um, a-a-around like this J- January this year, with OpenClau, it exploded. [laughs] And the reason it exploded is because there was this fundamental, um, idiosyncrasy in OpenClau where it sends heartbeats every like ten minutes to your model of
- 27:27
choice just to see like if the u-- you know, if the, the, the client was still active. [laughs]
- 27:32
Mm-hmm.
- 27:32
And that means that if you set Opus to be your default model, it would be like using a lot of tokens on this like heartbeat process. And so, um, this was the very beginning of like a very popular app with two completely different intelligence needs.
- 27:48
Completely different. And the models, the open source models have improved to a point where, like it makes sense to segment the market in at, in at least those two areas.
- 27:57
And so that's, that was how it got started, and then we saw a lot more segmentation blossom afterwards. And so, um, and now a whole bunch of agents and apps on OpenRouter use the, the different routers that we have, and we have a couple of them.
- 28:11
We have Pareto Code, which gives you the Pareto, Pareto optimal model for coding tasks given a certain like threshold that you can tune. Um, we have Fusion, which orchestrates multiple models and gives you a fused result.
- 28:26
And, um, and we'll have other experiments in the future. What we wanna do is basically like create good primitives that, uh, developers can use to like get really advanced with how they use model orchestration.
- 28:39
Um, kinda like sidecar-
- 28:41
Yeah
- 28:41
... sort of like that. Um, but also give people like a really easy thing that they can just like set a slug to that works with all harnesses, um, and just like gets the job done.
- 28:52
I feel like it's super interesting how much of a perfect storm there is for model routing right now. Because on one hand, you know, ignore agents, ignore OpenClau for a second, like just to squeeze better performance, it seems like we should be smarter about how we tackle problems.
- 29:06
That's obvious, right? If you make a plan, if you make a strategy, that's a, that's a better way to, to go about your day. So I'm not surprised that you're gonna see, uh, you know, better code get written or more performant code get written, less buggy code get written if you break the problem down.
- 29:19
And so routing specifically for that, uh, use case makes a ton of sense. But then hearing this, yeah, like the profile of workloads changed with agents, right? They're very-- They-- It went from like I ask questions, I get a response.
- 29:31
Then it went to reasoning, where I ask questions, it reasons, and then it comes back. And then it went to, uh, yeah, this heartbeat, right? Like if, um, if my agents are running optimally, uh, there's a token being s- generated every second.
- 29:43
Um, and suddenly that is its own need for model routing, and it, it feels like, um, hearing the different solutions, uh, to kind of tackle each of those is very interesting.
- 29:52
Even-
- 29:53
Well, I was just gonna say, I think that r- yeah, the, the use cases for model routing are, are, there are many of them. And so one could be getting a better answer.
- 30:02
One could be, you know, saving money and trying to get the same answer. One that we haven't even talked about yet, which is probably the most relevant maybe even to this crowd, is, uh, when do you wanna actually run a model locally versus when do you actually need something like a frontier model to, to perform that task?
- 30:18
And that might be something to the effect of like, uh, for privacy protecting information. You know, when I'm running some local-- Can you detect that my prompt has sensitive information?
- 30:28
And if so, do that on, on device. Um, but then maybe even anonymize some of that information to go do the more advanced workloads up on top of that information in the cloud.
- 30:38
Um, another example would be, again, for the cost savings, where it's like, "Hey, I bought this DGX Spark-" And I'm n- I know I'm not at 100% utilization, how can I make sure that as part of my workloads, whether it's the heartbeat in OpenCL or what have you, that I'm leveraging that compute, um, to the fullest of
- 30:54
its ability because I'm only paying for the electrons that are coming in for my power bill, um, but I'm paying full price for the tokens in the cloud? And I think that's a whole other area of, of model routing that I know that we're doing some work with at NVIDIA, um, that I think will be, will be
- 31:07
really cool as it starts to, uh, the hybrid of local and cloud-
- 31:11
Totally
- 31:11
... starts to, to really emerge as its own sector.
- 31:14
Yeah. I'd be curious to know what you guys take, uh, take is on. Like, like if you self-host a model, the cost dynamics change, right? You have a considerably higher cost at a higher context length because your throughput slows down as the context gets deeper.
- 31:29
So rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction, uh, to bring your throughput back up. Uh, have you guys, uh, thought about this paradigm, like compaction versus, uh, just routing?
- 31:44
Because one is you have fewer tokens to work with, one is we have cheaper tokens.
- 31:51
Y- yeah. I, I think in practice, um, compacting alone doesn't solve the, the cost or, or throughput problems because a lot of times it's just like the, uh, the differential in like model intelligence and cost is just so big where...
- 32:02
Also, by the way, when you compact, you're taking a cache miss, so you're actually now like paying 10 times as much for the, for those input tokens if you didn't compact.
- 32:09
Um, the main reason we compact is actually intelligence. Um, a- a- all these like model providers, they advertise some like insane context window, like a million tokens. Um, I would like never recommend using like these models past like- [laughs] ...
- 32:19
200K tokens. Under 100K if you can. Um, the, the, the intelligence, uh, just kind of like falls off a cliff at some point. Um, sorry, Anthropic, if you're, if you're watching.
- 32:28
But [laughs] I, I think that, uh, you know, compaction is like a very useful tool if you are gonna have to take a cache miss anyway, um, one way or another, like when you're routing to another model and you want to just like minimize the, the window there.
- 32:43
Do you find that, um, in the sidecar-
- 32:46
Yeah
- 32:47
... um, when small models are generating lots of tokens, um, is that like an, is that like one of the best reasons to switch it to a larger model?
- 32:58
Like basically, when, when small models generate lots of tokens, I wonder if that's like a-
- 33:03
Indicator
- 33:03
... crux of root, you know, the root cause of, of intelligence problems down the road.
- 33:08
Mm-hmm.
- 33:08
You want your big model to generate the big token chunks, the small models to generate smaller token chunks, right?
- 33:13
And on that question, you mentioned like a, a small model essentially needing to flag that it needs help from the larger model.
- 33:19
Yeah.
- 33:19
What is that mechanism? 'Cause that seems like the... What's the indicator, and then what's the mechanism for it to do so?
- 33:25
Yeah, totally. Uh, so there are a lot of mechanisms we, we talk about in our blog post about how, uh, we just detect that we need to like change the model up.
- 33:33
Um, to, to... Okay, I guess to answer your question first, how does the small model detect? Um, actually the, the thing that we spend a lot of time on is, um, how do we make sure the small model is like good at detecting it?
- 33:43
Unfortunately, there's a lot of cases where you do need the big model to detect it. Um, one thing that we d- don't go into the blog post is, um, you have some kind of cadence on which you're refreshing the cache anyways, 'cause by default there's some like five-minute lifetime on these caches.
- 33:56
If you're gonna go refresh a cache anyways, you basically can get a free like big, like frontier model call, uh, if you kind of like ask the right question.
- 34:04
So it's at that point where you might say, "Hey, just take a look at what the small model is doing. Does it feel like it's kind of like going into some rabbit hole and, and needs some help now?"
- 34:12
Um-
- 34:12
What's the need for the five-minute refresh?
- 34:15
Uh, it's just like a practical, like you, you have to pay some kind of like cost to like keep these like KV cache-
- 34:20
Oh, yeah
- 34:20
... caches warm. Um, and so most caches just get evicted on some kind of cadence.
- 34:24
How it works is like, at inference time, you only have so man- so many cache you can keep, uh, kept loaded in the GPU. So once, uh, if a cache is not being used again and again, it's offloaded.
- 34:35
I see.
- 34:35
So it's lost essentially. So that's why the inference provider asks you for money. Uh, but if you self-host it, you can, uh, get around this problem. You can, you can make it as long as you want based on your big business logic.
- 34:48
Well, do, do you see a world where we'll have like much more dynamic cache durations rather than just the five minute, one hour?
- 34:57
It depends on who's do, who's deploying the model where, right? So if you have a GPU which has, uh, like a lot of memory, uh, which, like the ratio of, let's say, SMs to memory, um, is memory more, uh, heavily skewed.
- 35:14
Or if you're working with unified memory, um, and you have systems like Vera Rubin, uh, you, you have a lot of tricks to play here, right? Uh, the five-minute, uh, window is what a lot of providers right now put, but that's, uh, that's, that's more an operational, uh, o- operational, uh, operational determination rather than a, like a
- 35:36
science-based or like a core physics law determination. So you can technically see over time maybe some, uh, some APIs are priced differently. Uh, but, uh, if you de- self-deploy, again, you can, you can get past a lot of this.
- 35:53
The, the cost economics really change when you move from self-hosted models to, uh, API providers, uh, because you have a lot more control and, uh, you don't have to guess the shape of your workload.
- 36:08
So let's say if your workload is 32K, uh, uh, like on average 32K cache, 1K input, 1K output, um, and someone else is like, let's say 64K, uh, 1K, 1K.
- 36:22
Uh, if you use some provider, they are amortizing everyone's use, uh, use case, and then giving you a price, right? And they have optimized, quote-unquote, for general use. If you're self-host, you can optimize specifically for your use, and you'll likely pay much less.
- 36:39
Yeah. Uh, and th-this is, like, kind of like the level of like, you know, h-hardware, software frontier that we kind of like thinking about. When, when we started Cognition and we were working on the first agents, uh, I think one reason why no one else worked on agents is they were just extremely expensive.
- 36:52
This was before cache tokens was a thing that API providers paid for. Like, if you were sending 100,000 tokens and the same 100,000 tokens, you were paying full price for those tokens-
- 37:01
Mm
- 37:01
... uh, back in 2024 when we started. Um, one of the key things that let us build Devin and build these first agents was we actually bought direct c-compute capacity from these providers, and instead of paying on a per token basis, we just paid for the underlying compute, knowing that the economics of the compute was that we
- 37:19
were actually paying far less for, for the cache tokens that we'd send over. Um, and, and then nowadays, you know, uh, uh, you know, there's, there's like similar, you know, things I would like about, like, you know, having a, a version of the cache that maybe, like, you can just back out to, like, storage in S3 or
- 37:32
something and just, like, hold for much longer.
- 37:34
Yeah. Now, this is not extremely relevant to a DGX Spark organization, uh, like setup, but, uh, if anyone's looking to do what you guys are wanting to do, try out Dynamo.
- 37:44
Uh, we have a lot of prefix cache optimizations in there.
- 37:48
Yeah. Um, and then, yeah, uh, uh, going back to your question, Alex, I think you said like, oh, like you ... When a small model is, like, going off and generating a ton of tokens, is that, like, a interesting time to back off?
- 37:58
Um, to be honest, we haven't explored that right yet. So that, that might actually be a very interesting thing to, to take a look at. Um, it, it is weird.
- 38:05
I think, like, some small models d-do tend to kind of be less, like, token efficient than others, but, um, they also seem to be trained on, like, their own traces, so maybe in a way it ends up, like, balancing out.
- 38:15
A lot of these things I, I feel like we have to be, like, very empirical about, uh, to actually know.
- 38:20
So, uh, just to add on, uh, uh, uh, uh, that, um, you have a lot of... Like these days, there are a lot of hallucination probes, so probes that work on either, uh, the internal state, like internal state of the models, uh, uh, directly.
- 38:35
So you can have some form of either magnitude analysis done or l-linear probes or just, uh, the n types of probes that you can see. And you can essentially rate, like, how, how much you think is, is tending towards hallucination.
- 38:51
Uh, so that kind of gives you a proxy for how lost it is, uh, like how lost a model is in its thinking. Uh, so you can use, like, different kinds of probes to understand, uh, like the perplexity within a model.
- 39:05
Oh, that's interesting. So yeah, instead of using the quantity of tokens that, uh, as indicative of a, of a model being lost, it's the, it's hallucinating more. And you're-
- 39:14
Yeah. So, uh, so essentially what is cache, right? It's, it's, it's the pref- uh, it's the prefill, uh, states, right? So you c- uh, what is a prefill state?
- 39:21
It's just a vector at the end of the day. So you can do tune all kinds of classifiers to understand, uh, different aspects of those, uh, collections of vectors.
- 39:32
So, uh, with those kind of probes, you can guesstimate a lot of, uh, states, uh, of a model.
- 39:41
I see. One, one question I have is, um, you know, different models behave, uh, differently. And, um, th-that kind of means that these prompts aren't portable. So as you're doing model routing, how do you handle es- essentially if you're, if you're going to a different model architecture, um, what do you have to do to the prompt, and
- 39:58
how much is that a factor into either of your guys' model routing solution?
- 40:03
Like what, like how, how is the prompt itself a factor-
- 40:06
Yeah, like how much-
- 40:07
... into the routing?
- 40:08
Yeah.
- 40:08
Yeah. Um, well, I think with building agents, there are all kinds of paper cuts and edge cases that are domain specific, and you-- like, the, the value of an agent company, um, like the value of Devin is all these, like, doom loops that you've discovered that a-are across all industries and the best ways to recover from them.
- 40:32
Um, and like man- here, I mean, it manifests big time in what the prompts are going to be, both for, like, you know, how the, the advisor model gets called, um, uh, you know, the smart friend, the [laughs]- [laughs] ...
- 40:47
the, uh, how the, like, subtask agents get called. And, uh, and the best thing is that, like, anyone can, like, ins- like any engineer or any, like, agent can inspect the traces and, like, adjust the prompt and then see the, like, live accuracy, uh, long time.
- 41:02
So I mean, basically, I just think that that's part, the prompt is part of the, the startup building process and, uh, is also really easy to observe and-
- 41:12
Yeah
- 41:13
... um, like, and have, and cl- and have, like, multiple people and agents collaborate on them.
- 41:16
Yeah. One thing I'd love to do, uh, with, with our Fusion product, and we don't have this yet, and so this is k- maybe a preview of some, some things we work on, is, you know, you can tune it against a data set, but the, the real thing you want when you're building a real agent someone uses
- 41:29
is to just, like, tune it against what actual people use it for and, and what actual models they get routed to. And so there's a lot of signals for this.
- 41:36
Um, like if, if someone sends a prompt and then Devin is working, and then you see that the user decides themselves to, like, upgrade to a different model, or they decide to downgrade, or the system detects that we originally sent to the wrong one, we now got to replace, like that's actually a really useful stream of signals.
- 41:51
And we're actually getting to this world of, like, auto research, where, like, maybe we can just have, like, a constant stream of prompts, what it should have been, what it was instead, and build a system internally that's just capturing all of this and then reiterating on our routing system until it eventually kind of, like, fits the real
- 42:07
production data. Um, that's kind of, uh, now that we-- it's, it's, it's public and, and people are using it, this is now something that we're, we're thinking about.
- 42:15
Have you guys looked into prompt tuning, and do you find it useful, like, say, Japa?
- 42:21
Yeah. So, uh, there are, like, these prompt tuning frameworks from, like, a few years ago that, uh, tried to do some kind of like, like gradient descent type thing.
- 42:29
Um, I'm like, I'm actually personally less bullish on, um, these kind of like low-level mechanical prompt tuning harnesses versus just telling, like, a smart model, like- Here is the decision that was made, um, and the context.
- 42:44
Uh, figure out why it went wrong. Sometimes you, you can do something as dumb as asking a model, "Why did you do this instead of this?" And cite the, the prompts.
- 42:51
Um, and then just have your agent, your dev, and just go and, like, update the prompts, rerun the, the test as a regression, make sure, make sure it changes.
- 42:58
Like, it-it's a lot heavier weight of a system, but I, I kind of trust the intelligence of a system like that a lot more.
- 43:04
So we're, we're running out of time, so we're gonna wrap up real quick. But I think what's really interesting is just from talking to you guys, we can kind of see how, uh, new this space is, right?
- 43:13
How much of this is actually just research. Um, we're starting to see new products come in. I'm, I'm really excited about your guys' solutions as you guys enter the space.
- 43:20
The, the ways and the needs that you need routing for. You know, even on a DGX Spark, when you're doing local inference, um, you have, uh, more compute. And if the memory's filled, one w- uh, or if it's, if the memory utilization is high, one thing you need to do is increase the compute utilization.
- 43:35
And so one way you can do that is by spawning multiple agents that are working collaboratively. So that collaborative piece is something that not only is optimal for all of these cloud workloads that you guys are doing, but, but specifically that is how you extract more performance out of this edge hardware.
- 43:49
And I think, um, you know, a question here is, uh, and maybe s- to, to end on, is a router gonna be something that we see as a product, or is that gonna be seen as part of the plumbing here?
- 44:00
Are models gonna get good at routing to other models because they know they need to be collaborative, or are harnesses gonna know that they are working across multiple models?
- 44:11
I, I think we already see this. Like, uh, you know, w- at Cognition, we're training our, our models to be able to be good collaborators. Um, I think, uh, it's very clear that new frontier models, like dif- fable models and GPT 5.5, 5.6 models are, like, themselves, like, naturally collaborative and, and better at delegation.
- 44:26
So I think we're already there at that point.
- 44:27
Hmm. Interesting.
- 44:28
Yeah, I, I think that the systems are, are kind of becoming not muddied in some sense, but I think that ultimately we're understanding that as we step up the, the abstraction ladder and build more, more things to create this smarter, uh, blob, um, which obviously we should hopefully, and we do understand how we're building it and why
- 44:47
we're building it, um, that it's gonna, it's gonna become a system that you look at kind of both the, the different components of the system. But, um, it's not just gonna be just models.
- 44:56
There's not gonna, there's not gonna be a thing as, like, a really great harness that is in absence of a really great model and vice versa.
- 45:03
Yeah. Makes sense.
- 45:05
I think, uh, applications, especially, uh, built on, uh, non-deterministic systems like models ha- operate in a very low-trust environment. So yes, most of the, uh, improvements will likely be distributed across both models and the harnesses.
- 45:23
But I think, uh, overall, it's, it's, it's mostly, it's, it-- there, there will, there will have to be some form of controller, uh, trying to have some form of arbitration, because even from the model perspective, you aren't in a perfectly visible world.
- 45:37
You don't know the behavior of every model, so it's, it's gonna be at the orchestration level where you have these kind of things. And this has traditionally been shown by other industries.
- 45:48
Like when web, uh, when web launched, you know, uh, you had traffic-based routing. Sure, it's different. Uh, but all the sort of routing controls have been centralized over time.
- 45:59
Makes sense.
- 46:01
I think it's most likely gonna be good news in the future. Um, and, and I, I think, like, caching is a big reason for that. Even if you-- I think, like, a-- To take the flip side of, of this argument, um, the f- you know, it might be that in the future we have, like, one big model
- 46:18
that's like, "I know I am the, like, most efficient at everything, and I'm, like, way more efficient than Haiku. I'll solve every task better than Haiku can at, like, a lower price.
- 46:29
Um, why should I ever delegate to Haiku?" You know, something like that actually could, could be a model that we have in the future. Um, but you're always gonna have these, like, you know, for example, caching.
- 46:41
It could be that, like, you tell the model that this other model, like, does have the right context in cache and, uh, you know, the, the orchestrator model just always has more context, and the models have to be aligned. [chuckles]
- 46:54
So I, I think, like, it's-- I, I don't really see a world where, like, we wouldn't be able to get models to collaborate really well and, and I think they're gonna get better over time, um, in part because they're, you know, they just have limited memory. [chuckles]
- 47:09
So I, I think that's kind of one, one deciding factor, and another is that, um, there will be, like, uh-- there will continue to be... Like, if you just look at, like, the, the, the rankings on OpenRouter, if you look at our, our public data and you look at, like, the top model being used by dollar spent
- 47:30
on classification tasks, like, guess what it is. It's Opus. [laughs]
- 47:35
I think there are, there are, there are big opportunities for, like, using small models for in-distribution easy tasks and the-- and, like, as time goes on, that's gonna be a larger and larger percentage of tasks relative to, like, the most valuable tasks that, um, very smart models spend most of their time on.
- 47:54
Totally. Well, I wanna thank you guys so much. Can we all give everyone a round of applause? [clapping] Thank you. [outro jingle]