The State of Model Routing — NVIDIA, Cognition, OpenRouter

Which engineering tasks deserve a frontier model, and which can a cheaper model handle without sacrificing the result? That question becomes unavoidable when an application uses several models, including models running locally. NVIDIA frames its Nemotron releases—described here as including datasets, weights, and customization recipes—as infrastructure for that increasingly customized, multi-model world.

For Walden, co-founder of Cognition, the question comes directly from customers deploying Devin: how should they measure model ROI, and where should engineers spend their expensive-model budget? Carter, a developer technology engineer at NVIDIA, describes the same pressure among individual developers, startups, and small companies. More available intelligence is useful only if developers can afford to keep using it. Tanay approaches routing through model evaluations: understanding accuracy, efficiency, cost, and behavior well enough to build systems in which models cooperate. The tooling is still young, with no settled architecture.

Devin Fusion starts from a failure mode of simple routing: send the user to a weaker model, discover that it cannot do the task, and then pay for the stronger model anyway. Cognition instead keeps a frontier model responsible for planning and difficult decisions while delegating implementation. Walden qualifies the claimed performance improvement over Fable: it is not a capability jump comparable to the gap between a frontier model and a substantially weaker one. Walden reports a 40% reduction in the cost of Fable-level intelligence with Devin Fusion. The mechanism is continued frontier judgment combined with cheaper execution, rather than replacing that judgment outright.

The implementation model can be an open model or a cheaper mini model. Its lower token price buys room to investigate more thoroughly. Walden illustrates this with three agents exploring a codebase: their combined exploration might cover more ground than one frontier model following a single path through its context. This is an example of the economic opportunity; his more precise description of Fusion’s persistent sidekick comes later. Carter makes the budget argument explicit: sufficiently cheaper tokens let a system spend more tokens while remaining within the original frontier-model budget.

Model size is only one dimension of the decision. Tanay points to jagged capabilities within coding itself. A data-science workflow can involve model building and prediction with scikit-learn, followed by visualization with matplotlib. Different training corpora can produce different strengths across those steps. A higher aggregate coding score does not establish that one model is better at every constituent task; useful routing requires understanding those differences.

After Alex Atallah joins from OpenRouter, the discussion extends to scientific discovery: a difficult, sometimes one-shot problem spanning many subdomains. Different post-training teachers and subtasks can create complementary failure patterns. Once those patterns are understood, orchestration can exploit them. Tanay tentatively attributes up to 10% higher accuracy to routing techniques, depending on the model pool and task. He calls the benchmark “LM router bench”; the discussion does not establish the exact experiment or whether the percentage is relative or measured in percentage points. The useful mechanism is complementary expertise, not a guaranteed gain from adding models.

Walden confirms that cheaper models can use more tokens, but warns against turning task-specific benchmark wins into a permanent initial assignment. Consider the progression of a developer session:

1. Ask the agent to explain how a codebase works.
2. Ask it to implement a feature.
3. Ask it to live-test the feature and debug difficult cases.

The task category and difficulty change during the same conversation. A model selected for code explanation may be a poor choice for the later debugging work. Frontier models are attractive partly because they can move between those domains without requiring the user to diagnose the transition.

Routing must follow the session, not just classify its first prompt. The difficult part is recognizing when the current model has become inadequate. A small model may not reliably know that it is out of its depth. Cognition’s answer is to keep a frontier agent supervising even when it is not executing the work. That agent can notice that the delegated task has become harder and redirect it.

Supervision introduces a second problem: what should the supervising model see? Passing it the entire worker trace can erase the savings. Walden’s example is a file read duplicated across three models, creating three separate context charges. Fusion instead sends most context to one model by default. The supervisor receives selected information—such as filenames, a high-level account of the work, or a carefully produced summary—rather than every tool result. Long-running agents already need context compaction; the same techniques can support these handoffs.

This makes context design part of routing design. Adding models increases the opportunities for redundant tokens, so collaboration needs a deliberate information boundary. Walden expects models co-designed for collaboration to improve on these early techniques. Tanay adds that the natural units of routing are subtasks and session phases: exploration, implementation, and later checks can each demand different expertise and levels of intelligence.

Alex describes an unresolved question inside OpenRouter: should the outer model doing the orchestration be large or small? Even the pricing consequence is not obvious. A large orchestrator may repeatedly reuse cached context, changing its effective cost. More broadly, fusion tries to combine the benefits of training data, compute, and reinforcement learning from multiple labs. A small model can be economical when the task lies comfortably within its training distribution; outside that distribution, repeated tool calls and unproductive loops can consume the apparent savings.

Alex reports that on Terminal-Bench, Opus performs about three times better at one-tenth the cost of Haiku, despite Haiku’s lower per-token price. He does not specify model versions, the benchmark version, the harness, the performance metric, or the cost accounting in this discussion. His explanation is that an undersized model can spend far more effort trying to solve an unfamiliar task. By contrast, distinguishing a person’s name from an organization’s name is his example of familiar classification work suitable for a small model.

OpenRouter Fusion explores both distribution fit and the arrangement of inner and outer models, alongside primitives such as sub-agents and advisor tools. Asked whether domain fit determines which model should lead, Alex does not claim an established rule. In the published OpenRouter Fusion work he describes, the stronger outer model produced the best deep-research results. Coding was not yet well optimized, and the preferred arrangement for other workloads remained unclear. Dollars per successfully completed task is the relevant efficiency target—not the advertised price of an individual token.

Walden points out that caching can benefit the side agent as well as the main agent. Devin Fusion—distinct from OpenRouter Fusion—uses a persistent sidekick with a continuously running context. The main agent does not need to reconstruct the sidekick’s earlier context every time it delegates another task.

Walden describes cached input tokens as ten times cheaper in the setup under discussion. The operational distinction is prefix reuse: current Claude prompt-caching documentation separates discounted reads from cache-write charges, so neither that multiplier nor a later supervisory call should be treated as universally free. With both contexts maintained, the smarter model can move between leading and assisting.

Persistence also changes what a model should learn. Cognition is experimenting with reinforcement learning in both collaboration roles:

• Orchestrator: Decide what to delegate and issue useful instructions to another model.
• Executor: Carry out another model’s instructions effectively as its sidekick.

The goal is to co-design models with the orchestration system instead of only assembling independently trained models. The moderator connects this to Nemotron post-training: if a model will operate inside a routing-heavy harness, that environment should shape its training.

Tanay introduces Flextron, describing a main model distilled into smaller footprints, with the task determining which footprint performs decoding. The within-artifact mechanism is flexible activation of model capacity: nested elastic subnetworks can provide different amounts of computation without requiring a collection of independently deployed models. This moves part of the routing decision from choosing an endpoint to choosing how much of a model to activate.

Access to training recipes and data can help characterize whether a question is familiar to the model. Training studies can also expose distillation gaps between a teacher and its smaller student. Those gaps need not be uniform: including a domain in the training data does not guarantee that the student absorbed it as effectively as other domains. Task-sensitive model footprints therefore need evidence about retained capabilities, not just parameter counts.

Tanay returns to context through abstract syntax trees, Headroom, and RTK. Ordinary compaction is lossy. Structured representations that an agent can later expand offer something closer to recoverable compression, though he stops short of calling it fully lossless. The distinction is whether omitted detail is gone or remains accessible somewhere else.

Walden uses a human-memory exercise: how many spoken numbers can someone retain before losing track? Limited working memory does not prevent effective work because people use external records. An agent likewise need not remember a whole file if it remembers that the file exists, where it is, and which parts matter. A well-designed harness should give the agent what it needs to find the information, even when the information is not currently in its context window.

A sidekick can therefore return file references instead of dumping all its findings. The main model can inspect selected passages or decide that one verification command will answer its question. For a concrete implementation of that handoff, suppose codebase exploration produced a note about an authentication path. Store the full note and pass only its location and a short summary:

import json
from pathlib import Path


def save_finding(root: Path, finding_id: str,
                 summary: str, details: str) -> str:
    root.mkdir(parents=True, exist_ok=True)
    path = root / f"{finding_id}.md"
    path.write_text(details, encoding="utf-8")
    return json.dumps({"summary": summary, "path": str(path)})


handoff = save_finding(
    Path("agent-notes"),
    "auth-path",
    "Review the token-expiry check before changing login behavior.",
    "Exploration note: inspect the token-expiry check and its callers.\n"
    "Status: investigation pending; no code change or test result recorded.\n",
)
print(handoff)

The message contains a pointer and the work’s status; the file retains the fuller record. Walden’s broader observation is that smarter models can be selective about tool use and reading, so adding an expensive model can sometimes reduce the total system’s token consumption and cost. Tanay similarly emphasizes token efficiency as a reason model size alone is an incomplete cost predictor.

There is a boundary between model orchestration and inference infrastructure. Alex explains that OpenRouter is a marketplace and generally cannot inspect a provider’s KV cache unless it runs the model itself, which he says is rare. It can optimize for cache hits and pass their savings to customers. It can also consider remaining cache lifetime when deciding whether to keep using a model.

A better model match may justify abandoning a still-useful cache. Alex’s example is a switch with two minutes of cache life remaining: the routing gain must outweigh the lost reuse. OpenRouter had begun experimenting with that tradeoff, including how much tolerance a customer might allow, but had not exposed those controls to customers at the time of the discussion.

The demand for automatic routing did not initially come from such fine-grained optimization. Alex says OpenRouter’s auto router had existed for almost two years with little adoption because people preferred selecting specific models. He describes an adoption surge around January of the recording’s year, associated with OpenClaw. His example is an agent sending a heartbeat about every ten minutes to its configured model: if Opus is the default, expensive intelligence gets spent on routine background checks. That is a deployment example, not a universal cadence; current OpenClaw heartbeat documentation makes cadence and heartbeat-model choice configurable.

Heartbeats and substantive agent work have sharply different intelligence requirements. As open models improved, separating those workloads became practical, and Alex says more agents and applications began using OpenRouter’s routing options.

OpenRouter wants to support both developers building sophisticated orchestration and users who simply set a model slug in an existing harness.

Two changes reinforce each other. Breaking a problem into planned steps can improve the work itself, including code quality. Meanwhile, the workload has moved from question-and-answer exchanges to reasoning and then to continuously active agents. A long-running agent mixes important decisions with routine background activity, making one fixed level of intelligence an increasingly awkward fit.

Carter adds two routing objectives beyond answer quality and cloud-token savings:

• Privacy: Detect sensitive information and process it on device, potentially anonymizing it before sending a more demanding workload to a cloud model.
• Owned-compute utilization: Send suitable work, such as background heartbeats, to an underused DGX Spark rather than leaving purchased capacity idle.

His economic framing is incremental: after buying the hardware, additional local work incurs electricity costs while cloud requests still incur their token charges. He describes hybrid local/cloud routing as an emerging area of NVIDIA’s work.

For self-hosted models, Tanay points out that growing context can lower throughput. Compaction can recover some of that throughput without changing the model. This creates two different levers:

Walden responds that compaction alone cannot eliminate large differences in model intelligence and cost. It also changes context prefixes, potentially sacrificing cache reuse. In Walden’s compaction example, newly uncached input costs ten times the previous cached-input price. The tradeoff depends on which prefixes remain reusable and how much context is removed.

For Cognition, Walden says the primary reason to compact is preserving intelligence. Walden recommends avoiding contexts above 200K tokens and staying below 100K when possible, despite advertised million-token windows. That is his operational guidance from agent work, not a measured limit established here for every model. A natural opportunity to compact is during a model switch: when losing the old model’s cache reuse is already unavoidable, the handoff can start with a smaller context.

Alex asks whether a small model generating many tokens might signal that it should hand work back to a larger one. Carter sharpens the question: what detects the need for help, and what mechanism triggers it? Walden says small-model self-detection is imperfect, so the large model often has to make that judgment.

One opportunity comes from cache maintenance. Walden describes a roughly five-minute default cache lifetime in the provider setup being discussed. If the system is already making a call to refresh that cache, it can ask the frontier model to examine the worker’s progress: is it pursuing a useful path, or disappearing into a rabbit hole? The idea is to combine necessary cache maintenance with a useful supervisory decision, reducing the incremental overhead of the check.

The lifetime itself reflects an operational policy. GPU memory can retain only so much cached state; inactive caches may be evicted or offloaded. Self-hosting gives the operator more control over retention, subject to available resources. Tanay explains that compute-to-memory ratios, unified memory, and systems such as Vera Rubin create different choices. Five-minute and one-hour windows are not physical constants.

That control matters when a workload is predictable. Tanay compares these illustrative request shapes:

An API provider prices and optimizes across a mixture of customers. A self-hosted deployment can optimize for its own recurring shape, which Tanay argues can lower cost. The table is a workload comparison, not a measured savings result.

Walden recalls the early economics of Devin in 2024, before API cache discounts made repeated context cheaper to users. Sending the same large context again could mean paying full input price again. Cognition bought direct compute capacity from providers instead of paying per token, capturing the underlying benefit of context reuse. He would like longer-lived cache storage, potentially backed by something like S3. Tanay points to NVIDIA Dynamo for prefix-cache optimizations, while explicitly qualifying that recommendation as less relevant to a DGX Spark setup.

Returning to Alex’s proposed token-count trigger, Walden says Cognition has not yet explored it. Some small models are less token-efficient than others, and models may be trained on their own characteristic traces. Whether a long output indicates productive work or confusion needs empirical testing.

Tanay proposes another source of signals: probes over internal model states. Magnitude analysis, linear probes, or other classifiers could estimate hallucination tendency or aspects of uncertainty, providing a proxy for how lost the model has become. Prefill and cache states contain vector representations on which such classifiers could operate. These are candidate indicators for routing; the panel does not present a validated detector or threshold.

Switching models also raises prompt portability. Different architectures may respond differently to the same instructions. Alex’s answer emphasizes the accumulated recovery knowledge inside an agent product: domain-specific edge cases, repeated failure loops, and the prompts that help escape them. Calling an advisor requires different instructions from assigning a subtask executor. Engineers and agents can inspect those traces, adjust the prompts, and observe whether accuracy improves over time. The mechanism is an observable repair process, rather than a universal translation rule between models.

Walden describes a production-feedback capability he wants for Devin Fusion but says is not yet available. Dataset tuning is useful, but real usage supplies a different stream of evidence: a user upgrades the model, downgrades it, or the system discovers that its initial assignment was wrong. Those events could connect the original prompt, the actual route, and a better route. An internal research loop could then revise the router against the workload people actually bring to the product.

Tanay asks about prompt tuning and GEPA. Walden responds with a preference for diagnosis by a capable model over the low-level mechanical prompt-tuning approaches he describes. His preferred repair procedure is:

1. Give the model the routing decision and its context.
2. Ask why it chose that route instead of the desired one, with references to the relevant prompts.
3. Have Devin update the prompts.
4. Rerun the failed case as a regression test and check that the behavior changes.

He acknowledges that this is a heavier process. Its appeal is that the system reasons about the failure before editing the instructions that caused it.

The closing discussion returns to local hardware. The moderator argues that when a DGX Spark has high memory utilization but spare compute capacity, concurrent collaborating agents can help use more of that compute. Collaboration therefore matters both to cloud orchestration and to extracting useful work from edge hardware. That raises the architectural question: will routing remain a product, or become ordinary behavior inside models and harnesses?

Walden says Cognition is already training for collaboration and describes newer frontier models—including Fable and the GPT-5.5 and GPT-5.6 models he names—as increasingly capable delegators. Carter sees the model and harness developing together: a strong harness cannot be separated cleanly from the model behavior it relies on, and a strong model still needs the surrounding system.

Tanay expects an arbitration layer to persist. Applications built from nondeterministic models operate with limited trust, and no individual model has complete visibility into every other model’s behavior. Improvements can be distributed across models and harnesses while a controller still decides among them. He compares this with the centralization of traffic-routing controls in web infrastructure: better participants do not eliminate the need for coordination.

Alex tests the strongest counterexample: suppose one future large model can solve every task better and more cheaply than Haiku. Why would it delegate? Another model might already hold the relevant context in cache. An orchestrator might know more about that context placement than the model doing the work. Limited memory and the location of useful state can therefore make collaboration worthwhile even when capability differences shrink.

There is also a more immediate mismatch between capability and spending. Alex says Opus ranks first on OpenRouter for classification tasks by dollars spent. That is a spending ranking, not a ranking of classification quality or request volume. He sees substantial room to move easy, familiar tasks to smaller models, and expects those tasks to grow as a share of the workload while frontier models concentrate on more valuable work. Routing’s continuing opportunity lies in that allocation: matching intelligence, available context, and compute to what the next piece of work actually needs.