# Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

Archana Kamath, DigitalOcean · Tyler Gillam, DigitalOcean

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/FvxY8oPoI8o-preferences-over-benchmarks-model-routing) · [Video](https://www.youtube.com/watch?v=FvxY8oPoI8o)

## Preferences Over Benchmarks: Model Routing

Archana Kamath and Tyler Gillam show how per-request routing can trade a small amount of measured quality for lower cost and latency, while adding failover and an evaluation loop that a single-model setup lacks.

### Why one model is the wrong unit of choice

[0:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=12s)

The familiar model-selection habit starts with a leaderboard: find the highest-ranked model and send every request to it. Archana Kamath argues that this optimizes the wrong unit. An application does not have one uniform workload; it receives individual requests with different difficulty, latency requirements, and consequences for failure.

Three pressures break the one-model habit. **Cost:** paying frontier-model rates for routine work wastes inference budget. **Fit:** classification, code generation, and security review do not demand the same capabilities. **Risk:** when one provider or model degrades, a product built around that single dependency has no failover path. Kamath compares the emerging discipline of model orchestration with FinOps because teams must actively govern where inference spending goes rather than treating every token as equivalent.

Her examples establish a rough capability ladder. Classification and labeling may fit a small open model. Code generation and bug fixing may work well on a mid-sized open-weight model. Accuracy-sensitive work such as code review or security analysis may justify a frontier model. Inline code completion adds another constraint: even an accurate answer can be unsuitable if routing and generation are too slow for an interactive editor.

A public benchmark cannot encode the complete decision because the model is only one component of the deployed system. The relevant variables include the request’s task, the system prompt and tools around the model, the acceptable cost, the required latency, and the end user’s preferences. Routing turns those application-level requirements into a per-request model-selection problem.

### Turn preferences into routing policy

[4:26](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=266s)

Tyler Gillam begins with the main objection to automatic routing: a black box can make a poor choice without giving builders a practical way to correct it. DigitalOcean’s proposed answer sends each request through the open-source [Plano-based routing architecture](https://www.digitalocean.com/blog/inference-router-architecture). A purpose-built routing model interprets the request against configured tasks, while explicit preferences and rules constrain what it may choose.

Builders can describe tasks in natural language, set priorities such as cost, latency, or quality, name preferred models, and add hard decision-tree rules. Presets provide a starting point, but the important feature is editability: the routing policy can be changed and then tested against the application’s own evaluation set. Gillam describes the operating loop as route, evaluate, adjust, and feed the result back into the policy.

The software-engineering configuration makes the two-stage decision concrete. First, the router matches the request to a task such as bug fixing, code generation, or test writing. Then the task’s policy selects from its eligible model pool. A manual ranking can prefer one model and use another only as failover. A `fastest` policy can instead choose whichever eligible model has been fastest during roughly the previous 30 minutes.

This separation matters. Intent matching answers, “What kind of work is this?” Selection policy answers, “Given this task and its allowed models, what should we optimize now?” Keeping those decisions distinct lets a team change model availability, priority, or failover behavior without rewriting the calling application’s task logic.

From request to selected model
The router first identifies the task, then applies that task’s model pool and selection policy.
- Incoming request — Includes the prompt and surrounding workflow context.
- Declared preferences and rules — Cost, latency, quality, preferred models, task descriptions, or hard rules.
- Matched task — For example: code generation, bug fixing, or test writing.
- Eligible model pool — One or more models configured for the matched task.
- Selection policy — Manual ranking, failover order, or fastest recent model.
- Selected model — Receives this request; another request may take a different route.
- Incoming request → Matched task (classify intent)
- Declared preferences and rules → Matched task (define task descriptions)
- Matched task → Eligible model pool (select eligible set)
- Declared preferences and rules → Selection policy (set optimization and rules)
- Eligible model pool → Selection policy (rank candidates)
- Selection policy → Selected model (choose or fail over)

### A playground shows behavior; an evaluation tests it

[8:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=481s)

The first side-by-side demonstration sends three related prompts through a fixed premium model and the software-engineering router. “Write a basic Fibonacci function” matches a code-snippets task and goes to a comparatively inexpensive model. “Optimize my function” matches performance optimization and takes a different route. “Write some unit tests” matches test writing and code verification, selecting yet another configured model.

The playground makes the mechanism visible: related turns in one coding conversation need not use the same model. It also displays cost and speed per request. But Gillam calls the output comparison a “vibe check,” which is the right limitation to preserve. A few plausible-looking answers can reveal routing behavior; they cannot establish dependable quality.

The accompanying evaluation compares the router with the fixed premium model over a dataset. The router scores 90% correctness and the single model scores 95%. Gillam characterizes the difference as close to the uncertainty of the LLM-based judge, while the router uses substantially fewer tokens and returns faster. That interpretation should remain conditional: the talk does not provide the dataset size, judge calibration, repeated-run variance, or task-level breakdown needed to establish statistical equivalence.

### The session-level cost gap compounds across agent steps

[9:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=581s)

The larger demonstration moves from isolated prompts to two coding-agent sessions. Both receive the same request: “Build me a spinning wheel app.” The left terminal uses a fixed premium-model configuration; the right sends requests to the software-engineering router. A lower observability panel reports token use, selected models, matched tasks, and accumulating cost as the agents work.

On the routed side, code-generation requests go to the model configured for that task, and the session uses two models over the course of the initial build. On the fixed side, requests largely continue to the premium model, although Gillam notes that the coding tool itself appears to make an occasional different-model choice. That caveat means the comparison is not a perfectly isolated experiment between a literally single-model agent and a router; it is a live comparison between the two demonstrated configurations.

Both builds finish at a similar time, and Gillam judges their visible output as comparable, while explicitly acknowledging the subjectivity of that assessment. After the first feature request, the routed session has spent $0.08 and the other session $0.25—about a threefold difference at that point. This is useful evidence of the session’s observed economics, not proof that every coding workload will preserve the same ratio or quality.

The next prompt asks both agents to write unit tests. The router recognizes test writing and code verification, then uses the model configured for that task. A final prompt asks for README documentation. After those additional steps, the routed session totals $0.14 and the comparison session totals $0.44. The absolute savings are small in one short demo, but agent workflows multiply requests; repeated per-step savings can therefore accumulate across sessions and users.

One coding session, two selection strategies
The demonstrated router changes models by task, while the comparison configuration sends most work to one premium model.
- Fixed premium-model configuration — Most requests retain premium-model cost and latency; demonstrated final session cost: $0.44.
- Software-engineering router — Matches each step to a task and its configured model; demonstrated final session cost: $0.14.
- Build spinning wheel app — Both configurations finish at a similar time with subjectively comparable output.
- Write unit tests — The router matches test writing and code verification.
- Write README documentation — The final task extends the measured session cost comparison.
- Fixed premium-model configuration → Build spinning wheel app (mostly one premium model)
- Software-engineering router → Build spinning wheel app (code-generation route)
- Software-engineering router → Write unit tests (test-writing route)
- Build spinning wheel app → Write unit tests (next agent step)
- Write unit tests → Write README documentation (next agent step)

### Routing is the foundation, not the finish line

[13:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=824s)

Kamath closes with the implementation claims that make routing practical in an application’s request path: a decision in under 200 milliseconds per request, a purpose-built mixture-of-experts routing model, no application-code changes beyond adopting the router configuration, and no additional router charge. The routing components are presented as open source through Plano, allowing teams to inspect or operate the mechanism rather than depending entirely on an opaque proprietary selector.

Routing alone only chooses where work goes. Kamath places three layers above it. **Evaluations** test whether those choices work for the application. **Caching** avoids paying repeatedly for an answer that can be reused. **Personalization** adapts routing toward what works for a particular team. Together they turn static configuration into a proposed improvement loop rather than a one-time model-picking exercise.

The strongest version of the talk’s conclusion is also the most bounded: benchmarks remain useful signals, but they describe only part of a deployed system. A router still needs representative evaluations, well-written task descriptions, sensible model pools, and explicit cost and latency goals. The live demo shows promising savings and comparable-looking output under its conditions; it does not remove the need to measure failures, judge uncertainty, or verify quality on the workload that will actually ship.

That leaves a practical replacement for leaderboard climbing: define what “right” means for each request, route within those constraints, evaluate the results, and revise the policy. There may be no single best model, but there can be a measured selection process that reflects the application’s real preferences.

The routing improvement loop
Routing becomes more useful when observed results change future configuration and behavior.
- Declare preferences — Describe tasks, cost and latency priorities, model pools, and hard rules.
- Route requests — Choose a model separately for each matched task.
- Evaluate outcomes — Compare correctness, token use, latency, and application-specific quality.
- Adjust policy — Revise descriptions, rules, priorities, or eligible models.
- Learn workload preferences — Use repeated routing and evaluation to adapt to the team over time.
- Declare preferences → Route requests (configure)
- Route requests → Evaluate outcomes (measure results)
- Evaluate outcomes → Adjust policy (identify changes)
- Adjust policy → Learn workload preferences (accumulate what works)
- Learn workload preferences → Declare preferences (update preferences)

## Key takeaways

- Choose models per request rather than per application: task, tools, cost, latency, risk, and user preference all affect what “right” means.
- Separate task matching from model selection. After identifying the task, apply its eligible model pool, optimization policy, and failover order.
- Do not confuse a convincing side-by-side demo with proof. Use representative evaluations to compare quality, tokens, latency, and cost, and preserve uncertainty in LLM-judged scores.
- In the demonstrated coding session, routing reduced cost from $0.25 to $0.08 after the initial build and from $0.44 to $0.14 after tests and documentation, with subjectively similar output.
- Routing is a foundation layer. Evaluations, caching, and personalization determine whether it improves a real workload over time.

## Resources

- [How We Built DigitalOcean Inference Router](https://www.digitalocean.com/blog/inference-router-architecture) — A first-party technical article expanding on Plano, intent matching, model ranking, selection policies, failover, routing models, and the proxy request path discussed in the presentation.

## Related talks

- [The State of Model Routing — NVIDIA, Cognition, OpenRouter](https://ai.engineer/talks/QHBjufYK8TA-state-model-routing-nvidia-cognition-openrouter) — Extends the discussion to application quality, inference cost, local hardware, context compaction, KV-cache-aware routing, and collaborative agents.
- [20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna AI](https://ai.engineer/talks/hqHC6Z_lXyo-20-days-compute-vs-7-hours-rethinking) — Develops the same critique of single-metric leaderboards through a Pareto-front view of task quality, latency, cost, and efficiency.
- [LLM Evals That Work IRL](https://ai.engineer/talks/nbZzSC5A6hs-llm-evals-that-work-irl) — Explains how application-specific evaluations and trace inspection can replace subjective output checks when comparing models and routing behavior.

## Slides

### Preferences > Benchmarks

This talk explores model routing for how teams build, challenging the instinct to choose a model by chasing the top of a benchmark.

[0:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=20s) [DigitalOcean title slide: Preferences > Benchmarks, Model Routing for How Teams Actually Build.](https://assets.ai.engineer/sites/aiecode2025/sha256/28a08c1251cdd1c35b7f2f5ce9511cebfef00d25c9251800e20f66b2661d771c.webp)

AI ENGINEER WORLD’S FAIR • 2026

DigitalOcean

Preferences > Benchmarks
Model Routing for How Teams Actually Build

2026 DigitalOcean. All rights reserved
1

### Meet your hosts

Archana Kamath leads engineering for inference and AI infrastructure. Tyler Gillam built parts of the router and will demonstrate it live.

[0:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=30s) [Portraits introduce Archana Kamath and Tyler Gillam from DigitalOcean’s Inference Router team.](https://assets.ai.engineer/sites/aiecode2025/sha256/df1dba1c175bc78e7d5ec686d0af37a19b0dac2b8ee10c2dfe4968a4923348c0.webp)

Meet your hosts
From the team building DigitalOcean’s Inference Router

Archana Kamath
VICE PRESIDENT, ENGINEERING

Tyler Gillam
SENIOR SOFTWARE ENGINEER II

DigitalOcean
2026 DigitalOcean. All rights reserved
2

### The DigitalOcean AI-Native Cloud

DigitalOcean places its inference router in the middle of a five-layer stack, between core cloud and data and learning, with infrastructure below and managed agents above.

[0:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=49s) [Five layers: managed agents, data and learning, inference engine, core cloud, and infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/86230f3d7c7b80d2c547f5cb70a0a7ce65b677c2fecae8175ca5482cc35a22a6.webp)

The DigitalOcean AI-Native Cloud
Five layers. One beautifully integrated stack. Zero lock-in.

Managed Agents
Open Harness · Sandbox · Toolbox · State · Plano

Data & Learning
DBaaS · Advanced Edition (PG/MySQL) · Knowledge Bases · Weaviate

Inference Engine
Serverless · Dedicated · Batch · Router · BYOM · Model Catalog & Services · Multi-modal · Guardrails

Core Cloud
Kubernetes · Droplets (CPU & GPU) · VPC · Storage (object, block and file system) · High performance NFS

Infrastructure
20 data centers · NVIDIA: H100, H200, HGX™ B300 - AMD Instinct™: MI300X, MI350X, MI355X · Liquid-cooled

DigitalOcean
2026 DigitalOcean. All rights reserved
3

### Everyone is reaching for model routing

Three pressures drive interest in routing: rising inference costs, using frontier models for work smaller models can handle, and having no failover when a single model degrades.

[1:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=82s) [Three reasons for model routing: rising cost, model fit, and the risk of models going down.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bffb85d020ad4e2e37294209985c5fc0785bf67006b6c7b45b4d4b4c4a1baa1.webp)

Everyone is reaching for model routing
Three forces are breaking the one-model habit.

COST
Spend is exploding
Uber  Microsoft  Walmart

FIT
One model is overkill

RISK
Models go down

DigitalOcean
2026 DigitalOcean. All rights reserved
4

### Model orchestration is the new FinOps

The speaker compares model orchestration to cloud cost optimization: FinOps took about 15 years to mature, while model orchestration is arriving in months.

[2:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=133s) [Model orchestration is described as the new FinOps, alongside an abstract orbital diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/da703b7ff95d97639170d6905e48667a74eaa2fa2014f6b14bacf76b38d0bf25.webp)

THE SHIFT

Model orchestration is the new FinOps

Cloud cost optimization took fifteen years.

This isn't waiting.

x183.15 y131.2
x183.15 y234.9

DigitalOcean

### There’s no single best model

The slide pairs cost-sensitive labeling with a small open model, inline completion with fast routing, code generation and bug fixing with a mid open-weight model, and accuracy-critical review with a frontier model.

[2:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=156s) [A diagram maps classification, code completion, code generation, code review, and bug fixing to different model types.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a2c7c8d9e3dc4e953dbb4230be157720749911174fd3d435ea9a1fb8b3accff.webp)

There’s no single best model
The right one depends on the request

Your tasks
Classification & Labeling
High volume, cost-sensitive

Code Completion (Inline)
Constant, latency-sensitive

Code Generation
Most traffic, open-weight

Code Review & Security
Accuracy-critical

Bug Fixing
Reasoning-heavy

Right model for each
Small open model
Fast routing model
Mid open-weight model
Frontier model

DigitalOcean
2026 DigitalOcean. All rights reserved
6

### What makes a model right

Choosing a model depends on the task and its surrounding tools, as well as cost, latency and what users want. No public leaderboard captures that mix.

[3:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=211s) [Five factors in model selection: task, system prompt and tools, cost limits, latency, and end-user preference.](https://assets.ai.engineer/sites/aiecode2025/sha256/cfc0da4afe6110a00a87d30d86f222f57bde5be4c86f7ea17fba207060ac6a75.webp)

What actually makes a model “right”
It’s a mix no public leaderboard encodes.

The task - what you’re actually asking the model to do.
System prompt & tools - the scaffolding around the call.
Cost limits - what this request is actually worth to you.
Latency - how fast the use case needs an answer.
End-user preference - what your users actually want.

DigitalOcean
2026 DigitalOcean. All rights reserved
7

### Routing architecture

Requests pass through Plano to RouterService, which connects to a routing model and cost and latency metrics. The diagram distinguishes full-proxy and routing-only paths.

[4:34](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=274s) [Diagram connecting a client and Plano to a router service, routing model, model metrics service, and LLM provider.](https://assets.ai.engineer/sites/aiecode2025/sha256/12911e02a6f9478c6f7dc25a84a2daaac4221de3010164fcacdbc36d3801d8a7.webp)

So, we built it differently

LEGEND
Shared path
Full Proxy Mode/v1/*
Routing Only/routing/v1/*

Routing Model
Client
Plano
RouterService
LLM Provider
ModelMetricsService
Cost
Latency

DigitalOcean
2026 DigitalOcean. All rights reserved
8

### Set your own rules for routing

Describe the task and priorities, layer decision-tree rules over presets, and validate the model choice with your own evaluations. Then adjust and feed the results back in.

[5:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=344s) [Three panels describe declaring preferences, controlling routing rules, and validating choices with evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/d896a212aa41123c2033c8215d17f0e0bd3bec6ad1bce6da4f5bc9a6849c754e.webp)

Set your own rules for routing

DECLARE
We honor your preferences
Describe the task in natural language and set what matters: cost, latency, task type.

CONTROL
Determine your own rules
Layer decision-tree rules on top. Change anything in one line. Start from presets.

TRUST
Validate with evals
Confirm the model choice against your own evals, not someone else’s leaderboard.

DigitalOcean
2026 DigitalOcean. All rights reserved
9

### Watch it decide, live

The live demo will change a playground preference to show model selection, then route a coding flow: bulk to open models, frontier models where they count, with no application code changes.

[6:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=380s) [Two demo panels introduce changing a playground preference and routing a coding workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/c1cf8b832ce45197d165ba5f2ded8c286cb550f5a77a22ab9bb2c2fc40f1e026.webp)

Watch it decide, live

The benefit shows up on screen. It doesn’t just get claimed.

PLAYGROUND
Set a preference
Change one preference and watch the router pick a different model per request.

OPENCODE
Route a real coding flow
Send the bulk to open models, frontier where it counts. No application code changes.

DigitalOcean
2026 DigitalOcean. All rights reserved
10

### Inference Router inventory

The demo starts with a custom software-engineering router alongside four presets. The custom router has six tasks; the software engineering preset has nine.

[6:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=387s) [DigitalOcean Inference Router page listing a custom router and several preset routers.](https://assets.ai.engineer/sites/aiecode2025/sha256/a97dc02f578556a85daffe360b93b3b730f71cd6c4f5f3e0d24e7159392580f9.webp)

Inference Router (Public Preview)
Create Router

Getting Started   Analyze   My Routers   Playground
Learn More

Name | Tasks | Date created
software-engineering-router | 6 tasks | 2 months ago
This router is optimized for software engineering workflows including bug fixing, code generation, testing, inline completion, a...

Software Engineering  PRESET | 9 tasks | —
Best for development workflows and coding assistance

General  PRESET | 7 tasks | —
Best for broad everyday language tasks

Writing  PRESET | 4 tasks | —
Best for content creation and editing

Knowledge Base & Document Intelligence  PRESET | 4 tasks | —
Best for grounded retrieval and document-heavy reasoning

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Thu Jul 2  12:12 PM
DigitalOcean
Preferences > Benchmarks: [illegible]
Ask Gemini
https://cloud.digitalocean.com/model-studio/router/inventory
Search by resource name or public IP (Cmd+B)
Create
Signup Credit: $5.00
Deploy Demo 2026
AI Assistant
Back to overview
Home
Launchpad  NEW
PROJECTS
FAVORITES
Star Your Favorite
INFERENCE
Overview
Inference Router  NEW
Dedicated Inference
Serverless Inference
Batch Inference  NEW
Evaluations
Model Catalog  NEW
Agent Platform
Manage
COMPUTE
DATA SERVICES
INSIGHTS
NETWORKING
Support  Status  Docs  Tutorials  Blog  Pricing  Careers  Privacy  Terms  Refer your friends for $
Share Feedback

### Software engineering router tasks

The demo ranks GLM-5.2 first for code generation, with GPT-5.2 as fallback. For bug fixing, the Fastest policy selects from the model pool based on roughly the last 30 minutes.

[7:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=423s) [Router details show tasks, assigned models and selection policies alongside a partially clipped chat-completions code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a20c0c522f1413699577d957bcaaa2470d3423f6b8bd28059bbbbfdb348bc94.webp)

Router Tasks

bug-fixing
Identify, diagnose, and resolve bugs in existing code. Includes interpreting error messages, tracing logic errors, and generating patches for reported issues.
Models: OpenAI GPT-5.4; OpenAI GPT-oss-120b; OpenAI GPT-5.2
Selection Policy: Fastest

code-generation  CUSTOM
Generate standalone functions and complex multi-file implementations, including feature development, refactoring, and optimization.
Models: GLM-5.2; OpenAI GPT-5.2
Selection Policy: Manual ranking

test-writing-code-verification
Write unit tests, generate test cases, or verify code correctness via test output prediction.
Models: Anthropic Claude Sonnet 5
Selection Policy: Fastest

code-performance-optimization
Optimize code for efficiency, refactor for performance, or reduce resource usage.

Chat completions
Pydo

import os
from pydo import Client

client = Client(
    api_key=os.environ.get("MODEL_ACCESS_KEY"),
)

stream = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Are there any syntax issu[illegible]
        }
    ],
    model="router:software-engineering-router",
    max_tokens=512,
    stream=True,
)

for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    piece = delta.get("reasoning_content") or del[illegible]
    if piece:
        print(piece, end="", flush=True)
print()

Copy

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
DigitalOcean  Preferences > Benchmarks  Ask Gemini
https://cloud.digitalocean.com/model-studio/router/inventory/[illegible]
Search by resource name or public IP (Cmd+B)  Create  Signup Credit: $5.00  Deploy Demo 2026  AI Assistant
Home  Launchpad NEW
PROJECTS  FAVORITES  Star Your Favorite
INFERENCE  Overview  Inference Router NEW  Dedicated Inference  Serverless Inference  Batch Inference NEW  Evaluations  Model Catalog NEW  Agent Platform  Manage
COMPUTE  DATA SERVICES  INSIGHTS  NETWORKING
Share Feedback

[7:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=430s) [Router configuration lists tasks, models, and selection policies beside a partially visible code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d5398fd110b6a16f475857e4dc4edd53d6b5adccaa25167a5989996fa71f701.webp)

Router essentials

Router Tasks

bug-fixing
Identify, diagnose, and resolve bugs in existing code. Includes interpreting error messages, tracing logic errors, and generating patches for reported issues.
Models: OpenAI GPT-5.4; OpenAI GPT-oss-120b; OpenAI GPT-5.2
Selection Policy: Fastest

code-generation  CUSTOM
Generate standalone functions and complex multi-file implementations, including feature development, refactoring, and optimization.
Models: GLM-5.2; OpenAI GPT-5.2
Selection Policy: Manual ranking

test-writing-code-verification
Write unit tests, generate test cases, or verify code correctness via test output prediction.
Models: Anthropic Claude Sonnet 5
Selection Policy: Fastest

code-performance-optimization

Chat completions
Pydo

import os
from pydo import Client

client = Client(
    api_key=os.environ.get("MODEL_ACCESS_KEY"),
)

stream = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Are there any syntax issu[illegible]
        }
    ],
    model="router:software-engineering-router",
    max_tokens=512,
    stream=True,
)

for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    piece = delta.get("reasoning_content") or del[illegible]
    if piece:
        print(piece, end="", flush=True)
print()

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
DigitalOcean  Preferences > Benchmarks  Ask Gemini
https://cloud.digitalocean.com/model-studio/router/inventory/[illegible]
Search by resource name or public IP (Cmd+B)  Create  Signup Credit: $5.00  Deploy Demo 2026  AI Assistant
Home  Launchpad NEW
PROJECTS  FAVORITES  Star Your Favorite
INFERENCE  Overview  Inference Router NEW  Dedicated Inference  Serverless Inference  Batch Inference NEW  Evaluations  Model Catalog NEW  Agent Platform  Manage
COMPUTE  DATA SERVICES  INSIGHTS  NETWORKING
Share Feedback

### Optimization comparison metrics

For the optimization request, the router selects GPT-5.2. The comparison reports it as 77% cheaper and 14% faster to first byte than Claude Opus 4.8. The presenter begins typing a unit-test request.

[8:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=523s) [A table compares token counts, cost and time to first byte for Claude Opus and the routed GPT-5.2 response.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f5df947fbe28f9898d5abd0940ad624e48da0f2f2a9ff460c8770f007ed6bc0.webp)

return (c, d)

return _fib(n)[0]

Go ahead and share your function, and I’ll tailor the optimization to your exact code! 🚀

Model: anthropic-claude-opus-4.8
Tokens: 592 in / 489 out · Cost: $0.02 · Time: 7.4s

if n < 0:
    raise ValueError("n must be a non-[illegible]
if n < 2:
    return n
return fibonacci(n - 1) + fibonacci(n[illegible]

Task: Code Performance Optimization
Model: openai-gpt-5.2 · Tokens: 494 in / 193 out
Cost: $0.004 · Time: 4.6s

Write some unit tets
Learn more about Inference Router Playground

Router Metrics    Router Evaluation

Anthropic Claude Opus 4.8 - 3P | Model | openai-gpt-5.2
n/a | Task | Code Performance Optimization
592 | Tokens In | 494
489 | Tokens Out | 193
$0.02 | Cost | $0.004 CHEAPER 77% cheaper
1.7s | Time to First Byte | 1.4s FASTER 14% faster

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Thu Jul 2 12:13 PM
DigitalOcean
Preferences > Benchmarks: [illegible]
Ask Gemini
https://cloud.digitalocean.com/model-studio/router/playground?i=[illegible]&left=anthropic-claude-opus-4.8&right=router%3Asoftware-engineering-router
Search by resource name or public IP (Cmd+B)
Create
Signup Credit: $5.00
Deploy Demo 2026
AI Assistant
Home
Launchpad NEW
PROJECTS
FAVORITES
Star Your Favorite
INFERENCE
Overview
Inference Router NEW
Dedicated Inference
Serverless Inference
Batch Inference NEW
Evaluations
Model Catalog NEW
Agent Platform
Manage
COMPUTE
DATA SERVICES
INSIGHTS
NETWORKING
Share Feedback

### Evaluation comparison

The evaluation compares the software engineering router with Claude Opus 4.7: the router scores 90% on correctness, versus 95% for Opus. Both use the same judge model.

[9:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=548s) [The evaluation comparison shows a 90% correctness score for the software engineering router and 95% for Claude Opus.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf147e2d8bd13430514695a2d6bb9fffbe8d3da494e439a2afe9e0e9433f71c3.webp)

Compare Evaluations

Evaluation A: model-router-mq04u6q2-ohompu
Evaluation B: model-router-mq04u6pz-kfvsv7

Overall Score
model-router-mq04u6q2-ohompu vs model-router-mq04u6pz-kfvsv7
90% | Overall Score | 95% HIGHER 5.0%
Correctness | Star Metric | Correctness
80 | Pass Threshold | 80

Configuration
software-engineering-router  View Router Config | Candidate | Anthropic Claude Opus 4.7
DeepSeek R1 Distill Llama 70B | Judge Model | DeepSeek R1 Distill Llama 70B

Interface text:
Back to Evaluations
Search by resource name or public IP (Cmd+B)
Create
Signup Credit: $5.00
Deploy Demo 2026
AI Assistant
Home
Launchpad NEW
PROJECTS
FAVORITES
Star Your Favorite
INFERENCE
Overview
Inference Router NEW
Dedicated Inference
Serverless Inference
Batch Inference NEW
Evaluations
Model Catalog NEW
Agent Platform
Manage
COMPUTE
DATA SERVICES
INSIGHTS
NETWORKING
Share Feedback
Chrome File Edit View History Bookmarks Profiles Tab Window Help
DigitalOcean
Preferences > Benchmarks: [illegible]
Ask Gemini
https://cloud.digitalocean.com/model-studio/model-services/model-evaluation/compare?runs=[illegible]

[9:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=563s) [A model evaluation comparison showing configuration and runtime, partly covered by a browser menu.](https://assets.ai.engineer/sites/aiecode2025/sha256/44308e3ac13f591197cc7853de26867da50bfbafb026bb9741ec0c64b9505cbe.webp)

90% | Overall Score | 95% HIGHER 5.0%
Correctness | Star Metric | Correctness
80 | Pass Threshold | 80

Configuration
software-engineering-router  View Router Config | Candidate | Anthropic Claude Opus 4.7
DeepSeek R1 Distill Llama 70B | Judge Model | DeepS[illegible]
coding_agent_eval_20_rows_exact_format.csv | Dataset | coding[illegible]mat.csv
-- | Temperature | --
1024 | Max Tokens | 1024
3m 31s | Runtime | 2m 51s

Router task summary
Bug Fixing & [illegible]

Interface text:
Back
Forward
Reload
Ask Gemini
Save As...
Print...
Cast...
Search this tab with Google Lens
Open in Reading Mode
Send to Your Devices
Create QR Code for this Page
Translate to English
View Page Source
Inspect
Search by resource name or public IP (Cmd+B)
Create
Signup Credit: $5.00
Deploy Demo 2026
AI Assistant
Home
Launchpad NEW
PROJECTS
FAVORITES
Star Your Favorite
INFERENCE
Overview
Inference Router NEW
Dedicated Inference
Serverless Inference
Batch Inference NEW
Evaluations
Model Catalog NEW
Agent Platform
Manage
COMPUTE
DATA SERVICES
INSIGHTS
NETWORKING
Share Feedback
Chrome File Edit View History Bookmarks Profiles Tab Window Help
DigitalOcean
Preferences > Benchmarks: [illegible]
Ask Gemini
https://cloud.digitalocean.com/model-studio/model-services/model-evaluation/compare?runs=[illegible]

### Evaluation performance

In this evaluation, the software engineering router used 28% fewer candidate output tokens and had 59% lower average latency than Claude Opus.

[9:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=571s) [A comparison table shows token counts and latency for the router and Claude Opus.](https://assets.ai.engineer/sites/aiecode2025/sha256/d52a7c1337ecb5d9f4c127a98b9c58dcd8c3dac36eae220883eb7c1b0089d678.webp)

software-engineering-router | Anthropic Claude Opus 4.7

Performance

1,117 | FEWER | 13% fewer | Candidate Input Tokens | 1,290
10,512 | FEWER | 28% fewer | Candidate Output Tokens | 14,536
33,083 | FEWER | 10% fewer | Judge Input Tokens | 36,578
17,258 | FEWER | 0% fewer | Judge Output Tokens | 17,338
3929.8 ms | FASTER | 59% lower | Avg Latency | 9500.9 ms
2797.3 ms | FASTER | 71% lower | P50 Latency | 9580.1 ms
6698.8 ms | FASTER | 43% lower | P90 Latency | 11831.8 ms
9192.3 ms | FASTER | 25% lower | P95 Latency | 12233.8 ms

Per-Entry Results

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
DigitalOcean
Preferences > Benchmarks: [illegible]
Ask Gemini
https://cloud.digitalocean.com/model-studio/model-services/model-evaluation/compare?[illegible]
Search by resource name or public IP (Cmd+B)
Create
Signup Credit: $5.00
Deploy Demo 2026
AI Assistant
Home
Launchpad NEW
PROJECTS
FAVORITES
Star Your Favorite
INFERENCE
Overview
Inference Router NEW
Dedicated Inference
Serverless Inference
Batch Inference NEW
Evaluations
Model Catalog NEW
Agent Platform
Manage
COMPUTE
DATA SERVICES
INSIGHTS
NETWORKING
Share Feedback

### Diagnosing the startup error

The router-side session hits a Python unicode escape error, identifies JavaScript backslashes as the cause, and prepares an edit. Its usage panel shows four requests to GLM-5.2 and four code-generation hits.

[10:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=657s) [The router-side coding session identifies an escaping issue and prepares an edit.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b1b176de9f6d0f6af5c710fb7ebc7aff7bb97ce8588585f586ca2c27af55c95.webp)

197 if __name__ == "__main__":
198     app.run(host="127.0.0.1", port=5040, debug=False)
199

# Todos
[✓] Create Flask app with spinning wheel HTML/JS
[•] Start server on port 5040
[ ] Verify server is running

$ lsof -i :5040 2>/dev/null || echo "port free"
port free

log 2>&1 &
echo "PID: $!"
sleep 2
cat /tmp/wheel_server.log

PID: 42914
  File "/Users/tgillam/code/deploy-demo/20260702-114404/router/wheel.py", line 5
    HTML = """
           ^^^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 5266-5267: truncated \uXXXX escape

Thought: 2.4s
The \u{...} in the JavaScript is being interpreted by Python. I need to escape the backslashes.

Preparing edit...

Session totals by routed model
Model | Req | Input | Output | Cache write | Cache read | Cost | Run rate
anthropic-claude… | 6 | 76,995 | 3,018 | 14,876 | 62,108 | $0.199534 | $0.3333/min
anthropic-claude… | 1 | 657 | 596 | 0 | 0 | $0.003637 | $0.0061/min

Router task hit %
Route | Hits | %
unknown | 7 | 100.0%

Session totals by routed model
Model | Req | Input | Output | Cache write | Cache read | Cost | Run rate
glm-5.2 | 4 | 42,647 | 2,103 | 0 | 17,216 | $0.039571 | $0.0714/min

Router task hit %
Route | Hits | %
code-generation | 4 | 100.0%

Interface text:
iTerm2  Shell  Edit  View  Session  Scripts  Profiles  Window  Help
Thu Jul 2 12:16 PM
OC | Python spinning wheel on localhost:5041
OC | Spinning wheel visualization port 5040 (opencode)
OC | Python spinning wheel on localhost:5041 (opencode)
Build · Claude Opus 4.7
Build · Claude Opus 4.7 Anthropic
Build · Inference Router - Software engineering
Build · Inference Router - Software engineering DO Proxy
esc interrupt
15.0K (6%) · $0.20  ctrl+p commands
9.7K (4%)  ctrl+p commands
cd (Python*)
session=[illegible]
HTTP Request: POST [illegible] "HTTP/1.1 200 OK"
127.0.0.1:62635 - "POST /v1/chat/completions HTTP/1.1" 200 OK

### Completed app comparison

Both sessions report running spinning-wheel apps. The router used two models; the speaker puts session cost at about $0.08 versus $0.25 for Opus, with similar output quality.

[11:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=675s) [Two completed spinning-wheel coding sessions appear above model usage and routing statistics.](https://assets.ai.engineer/sites/aiecode2025/sha256/b02e98c5300b8058653959ead97541bfdcd6fda3e3e7e9e55ab380de00eb216f.webp)

Left session:

Server is running on http://127.0.0.1:5040 (HTTP 200, PID 42924).

What was built — app.py:
- Flask app serving a single page with an HTML5 canvas wheel
- 6 default options: Pizza, Burger, Sushi, Tacos, Salad, Pasta
- Wheel slowly spins on load (~0.01 rad/frame = 1 rev every ~10 sec)
- Each option is a clickable button — clicking any one stops the wheel and displays the result under the pointer
- A "Spin Again" button resumes spinning

How to interact:
- Visit http://127.0.0.1:5040/ in your browser
- Logs: /Users/tgillam/code/deploy-demo/20260702-114404/opus/server.log
- Stop server: kill 42924

Build · Claude Opus 4.7 · 1m 1s

Session totals by routed model
Model | Req | Input | Output | Cache write | Cache read | Cost | Run rate
anthropic-claud… | 10 | 138,321 | 3,784 | 15,728 | 122,578 | $0.254264 | $0.2657/min
anthropic-claud… | 1 | 657 | 596 | 0 | 0 | $0.003637 | $0.0038/min

Router task hit %
Route | Hits | %
unknown | 11 | 100.0%

Right session:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Spinning Wheel</title>
<style>
  * { margin: 0; padding: 0; box-sizing: border-box; }
  body {
    display: flex;
…
Click to expand

The server is running and serving the spinning wheel visualization on http://127.0.0.1:5041. To stop the server, you can use kill 42926.

Build · Inference Router - Software engineering · 56.0s

Session totals by routed model
Model | Req | Input | Output | Cache write | Cache read | Cost | Run rate
glm-5.2 | 7 | 102,196 | 2,494 | 0 | 46,848 | $0.078927 | $0.0880/min
llama-4-maverick | 1 | 10,828 | 39 | 0 | 0 | $0.003823 | $0.0043/min

Router task hit %
Route | Hits | %
code-generation | 7 | 87.5%
fallback | 1 | 12.5%

Interface text:
iTerm2  Shell  Edit  View  Session  Scripts  Profiles  Window  Help
Thu Jul 2 12:16 PM
OC | Python spinning wheel on localhost:5041
OC | Spinning wheel visualization port 5040 (opencode)
OC | Python spinning wheel on localhost:5041 (opencode)
Build · Claude Opus 4.7 Anthropic
16.0K (6%) · $0.25  ctrl+p commands
Build · Inference Router - Software engineering DO Proxy
10.9K (4%)  ctrl+p commands
cd (Python*)
task
min
session=[illegible]
session=[illegible]

### Spinning wheel result

The software engineering router’s app shows a food-choice wheel stopped on Ramen, with a prompt to click again to resume spinning.

[11:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=682s) [A colorful food-choice wheel has stopped on Ramen.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fba5b2b195266f7b69fe4839b1c1be76aadf2aa2fe7dfbb3f340fa3b989d45f.webp)

Spinning Wheel

Ramen
Dessert
Pizza
Sushi
Burger
Salad
Tacos
Pasta

🎯 Stopped on: Ramen!
Click again to resume spinning

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
DigitalOcean
DigitalOcean
DigitalOcean
Preferences > Benchmarks: [illegible]
Spinning Wheel
Ask Gemini
http://127.0.0.1:5041
Thu Jul 2 12:16 PM
iTerm

### Spinning wheel result

Opus 4.7’s wheel displays Sushi as the result, with food buttons and a Spin Again control. In a subjective comparison, the presenter prefers the router’s version and questions how well this approach works.

[11:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=692s) [A food-selection wheel stopped on Sushi, with option buttons below.](https://assets.ai.engineer/sites/aiecode2025/sha256/35aa136a3843a69c8df2eb5dfda78eb08ea2f3ec74d9f9f054b5f509952bf273.webp)

Spinning Wheel

Wheel labels: Sushi, Tacos, Salad, Pasta, Pizza, Burger

Pizza
Burger
Sushi
Tacos
Salad
Pasta
Spin Again

Result: Sushi

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Thu Jul 2 12:16 PM
DigitalOcean
DigitalOcean
DigitalOcean
Preferences > Benchmarks: [illegible]
Spinning Wheel
Spinning Wheel
Ask Gemini
http://127.0.0.1:5040

## Full transcript

[0:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=12s) >> Hello everyone.

[0:14](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=14s) So, preferences over benchmarks.

[0:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=17s) The talk today is about model routing

[0:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=20s) and specifically why the way most people

[0:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=23s) think about picking a model, which

[0:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=25s) usually is chasing, you know, to the top

[0:26](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=26s) of a benchmark, is actually the wrong

[0:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=29s) instinct.

[0:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=32s) I'm Archana, VP of engineering for

[0:34](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=34s) inference engine and AI infrastructure

[0:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=36s) at DigitalOcean. And I'll be joined by

[0:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=38s) Tyler, who built parts of the router and

[0:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=41s) will actually do a live demo for us

[0:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=42s) today.

[0:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=44s) We both work on the managed agent

[0:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=45s) orchestration and inference engine

[0:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=47s) products at DigitalOcean.

[0:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=52s) So, you may know DigitalOcean as

[0:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=54s) droplets, databases, and app platform.

[0:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=57s) All of that is true. We are also the AI

[0:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=59s) native cloud.

[1:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=61s) This is five integrated layers, starting

[1:04](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=64s) from infrastructure all the way up to

[1:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=66s) the managed agents, with the inference

[1:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=68s) engine right in the middle.

[1:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=70s) And that's why we are here talking about

[1:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=72s) inference router. The routing lives in

[1:14](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=74s) the inference engine, and if you want to

[1:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=76s) know more about our stack and the full

[1:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=77s) story, please come find us at the booth.

[1:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=84s) So, everybody is reaching out for the

[1:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=85s) model routing. And let's look at three

[1:28](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=88s) reasons why the three reasons that are

[1:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=90s) breaking the one model habit for most

[1:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=92s) users.

[1:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=93s) The first one I want to talk about is

[1:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=95s) cost.

[1:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=96s) Spend is exploding, and even companies

[1:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=98s) like Walmart, Uber, Microsoft, they're

[1:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=101s) actively capping usage to control the

[1:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=103s) inference bills.

[1:46](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=106s) Second one is fit.

[1:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=108s) One model for every task is likely an

[1:50](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=110s) overkill.

[1:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=111s) You're essentially paying frontier rates

[1:53](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=113s) for a work that a much smaller model

[1:55](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=115s) will be able to handle really well.

[1:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=118s) And the third one, which for me is the

[1:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=119s) most important one, is the risk.

[2:02](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=122s) The risk associated with one single

[2:04](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=124s) model.

[2:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=125s) Models can go down, and if you bet your

[2:07](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=127s) entire product and production on one

[2:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=130s) model, you have no failover when

[2:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=132s) something degrades.

[2:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=136s) And model orchestration is actually the

[2:18](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=138s) new FinOps.

[2:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=140s) As you all know, cloud cost optimization

[2:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=143s) took us about 15 years for it to

[2:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=145s) actually become a real good discipline

[2:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=147s) and for companies to get it right.

[2:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=149s) This one actually is arriving in months

[2:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=151s) and not years. And here's the premise

[2:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=153s) that I think everybody gets wrong about

[2:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=155s) this.

[2:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=157s) We all think of like what is the best

[2:39](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=159s) model for a job. Here's the thing. There

[2:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=162s) is no single best model.

[2:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=165s) The right one depends on the actual

[2:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=167s) request.

[2:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=168s) For example, if you're doing

[2:50](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=170s) classification and labeling,

[2:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=172s) a small open model may very well work

[2:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=174s) really well for you and will give you

[2:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=176s) really good cost optimizations.

[2:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=178s) However, if you're running code

[3:00](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=180s) completion in line, you will likely need

[3:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=183s) really fast routing, and that is where a

[3:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=185s) faster, larger routing model comes into

[3:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=188s) picture.

[3:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=189s) Think about code generation and bug

[3:11](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=191s) fixing. You're likely good with an mid

[3:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=193s) open weight model, uh and again, it'll

[3:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=196s) bring you like really good cost

[3:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=197s) optimizations over using a frontier for

[3:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=199s) something that is likely an overkill in

[3:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=201s) this situation.

[3:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=203s) But then you're looking at like really

[3:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=204s) accuracy-critical tasks, like code

[3:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=207s) review and security, you're likely going

[3:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=209s) to lean towards a frontier model.

[3:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=213s) So, essentially, what makes a model

[3:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=215s) right for a request?

[3:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=217s) It's a mix that no public leaderboard

[3:40](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=220s) can actually encode for you.

[3:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=222s) Because it's the task itself. What are

[3:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=224s) you actually trying to achieve? What is

[3:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=225s) your model trying to achieve?

[3:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=227s) The system prompts and tools around it,

[3:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=229s) that is the methodology by which you're

[3:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=231s) getting something done using a model.

[3:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=234s) The cost you're willing to spend, this

[3:55](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=235s) is a very, very important aspect.

[3:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=238s) And latency the use case needs, not all

[4:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=241s) use cases need the same amount of

[4:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=243s) latency. So, depending on what you're

[4:04](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=244s) trying to do, this can vary widely. And

[4:07](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=247s) finally, the end user preference. All of

[4:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=248s) this is driven by what the end user

[4:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=250s) really wants out of your application.

[4:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=253s) So, I'm going to welcome Tyler onto

[4:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=255s) stage so that he can actually show you

[4:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=257s) the Inference Light Router live in

[4:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=259s) action and show you how it can really

[4:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=261s) help with all of these key aspects that

[4:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=264s) I'm calling out here.

[4:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=271s) >> Testing. All right. Thank you, Archana.

[4:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=275s) Okay. So, many builders

[4:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=278s) have tried auto routing before.

[4:40](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=280s) But the problem was that it feels like a

[4:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=282s) black box.

[4:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=284s) The router makes a choice, and if that

[4:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=285s) choice results in poor performance, you

[4:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=287s) really have no way of improving it.

[4:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=289s) We built ours differently.

[4:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=291s) At the architecture level, which is what

[4:53](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=293s) you can see on the screen,

[4:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=294s) a request runs through our open proxy

[4:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=296s) planner

[4:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=297s) and our purpose-built routing model,

[4:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=299s) both open source.

[5:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=301s) There's no vendor lock-in, which is a

[5:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=303s) key DigitalOcean value.

[5:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=306s) You describe what matters for your

[5:07](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=307s) workflow,

[5:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=309s) costs, latency, quality, preferred

[5:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=312s) models,

[5:14](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=314s) or hardware. Then the router uses that

[5:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=316s) context to pick the right model per

[5:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=319s) request.

[5:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=320s) Because the routing model is specialized

[5:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=322s) for this job, it's super fast, under 200

[5:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=324s) milliseconds, and it costs customers

[5:26](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=326s) nothing extra. In our evaluations, it

[5:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=329s) actually has beaten frontier models like

[5:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=330s) the GPT-5 series models at routing tasks

[5:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=333s) itself with a fraction of the latency.

[5:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=337s) So, the difference is simple.

[5:39](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=339s) This is routing you can customize,

[5:40](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=340s) evaluate, and improve without vendor

[5:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=343s) lock-in.

[5:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=348s) So, you bring your preferences and we

[5:50](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=350s) honor them. You describe a task in

[5:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=352s) natural language and set what matters,

[5:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=354s) cost, latency, and task description. You

[5:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=357s) bring your rules and we execute them

[5:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=359s) intelligently.

[6:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=361s) Layer decision tree rules on top, start

[6:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=363s) from presets, change anything you want

[6:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=366s) in a single line of code.

[6:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=368s) And you validate with your own

[6:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=369s) evaluations, not someone else's

[6:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=370s) leaderboard. Route, evaluate, adjust,

[6:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=375s) then feed that back in. That loop is

[6:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=377s) key.

[6:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=379s) Okay, we're going to switch gears here.

[6:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=381s) We're going to do a live demo.

[6:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=385s) Bear with me here.

[6:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=389s) All right, I'm going to show you a

[6:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=390s) couple of things. First, I'll show you

[6:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=391s) router configuration in the UI, how to

[6:34](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=394s) use it, and then how you can use

[6:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=395s) evaluations to measure and improve your

[6:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=398s) router's performance.

[6:39](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=399s) And then I'll show you a real router

[6:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=401s) that I created inside a coding agent

[6:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=403s) workflow.

[6:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=405s) So I'm here in the cloud console, the

[6:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=407s) DigitalOcean cloud console. And you can

[6:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=409s) see my routers. We have several presets.

[6:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=411s) You can see software engineering in

[6:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=412s) general writing,

[6:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=414s) knowledge bases and document

[6:55](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=415s) intelligence. In this case, I've

[6:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=417s) actually created my own. So I I

[6:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=419s) customized our preset software

[7:00](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=420s) engineering. Uh if we click into this,

[7:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=423s) we can see that I have several several

[7:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=425s) different tasks here.

[7:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=426s) I have bug fixing, code generation, test

[7:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=429s) writing, and a few others.

[7:11](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=431s) This also shows that you can specify

[7:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=433s) more than one model per task

[7:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=435s) in the bug fixing case and code

[7:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=436s) generation case.

[7:18](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=438s) Um in the code generation, I have GLM

[7:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=440s) 5.2 and GPT 5.2.

[7:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=442s) And because I really want to always

[7:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=444s) route to GLM 5.2 unless it's down, I use

[7:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=447s) this manual ranking option. So it'll

[7:28](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=448s) always go to GLM 5.2. If GLM fails,

[7:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=451s) it'll fail over to GPT 5.2.

[7:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=455s) In the bug fixing one, you can see a

[7:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=456s) little bit of a different one. In this

[7:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=457s) case, I have selection policy fastest.

[7:40](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=460s) So out of this model pool, if it matches

[7:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=463s) to bug fixing, it'll pick whichever

[7:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=464s) one's been fastest in about the last 30

[7:46](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=466s) minutes.

[7:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=468s) Okay, let's do this in action a little

[7:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=469s) bit. Here's our playground. We're all

[7:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=471s) show a couple of examples side by side.

[7:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=474s) First, I'll start with this is a simple

[7:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=476s) prompt, write a basic Fibonacci

[7:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=477s) function.

[8:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=481s) And as this runs, we can see on the left

[8:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=483s) we're writing to Opus.

[8:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=485s) On the right we're using our software

[8:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=486s) engineering router that I just showed

[8:07](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=487s) you.

[8:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=488s) And you're going to see that it picks

[8:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=489s) different models on the right. So, in

[8:11](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=491s) this case, it matched to the code

[8:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=492s) snippets task and just used the llama

[8:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=495s) format router model that I had

[8:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=496s) configured for that one. And if we

[8:18](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=498s) scroll down, I mean this is this is

[8:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=499s) obvious, right? But this model is

[8:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=501s) extremely fast and extremely cheap

[8:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=503s) compared to Opus.

[8:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=507s) Now, let's say optimize my function.

[8:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=510s) And we'll see the same thing happen. In

[8:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=512s) this case, it matched to the code

[8:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=513s) performance optimization task using

[8:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=515s) GPT-5.2.

[8:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=516s) And again, it's obviously significantly

[8:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=518s) faster. If we scroll down here, we can

[8:40](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=520s) also see that it's significantly

[8:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=522s) cheaper.

[8:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=524s) We'll do one more, write some unit

[8:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=525s) tests.

[8:50](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=530s) Okay, and in this case it matched to

[8:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=532s) Claude 5 Summit on the test writing and

[8:53](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=533s) code verification.

[8:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=536s) And again, we're going to see faster and

[8:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=537s) cheaper.

[8:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=538s) So, it's a pattern. It matches my, you

[9:00](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=540s) know, vibe check, right? It still vibes

[9:02](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=542s) though. How you actually prove it is

[9:04](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=544s) working it through evaluations.

[9:07](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=547s) So, I have an evaluation that I ran

[9:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=549s) here.

[9:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=550s) Comparing Opus on the left or actually

[9:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=553s) on the right hand side to my router on

[9:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=555s) the left hand side.

[9:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=557s) You can see that the scores, 90% for my

[9:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=559s) router, 95% correctness for Opus, are

[9:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=562s) very, very close. In fact, that's pretty

[9:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=564s) much within oh, it was a judge uh margin

[9:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=567s) of error.

[9:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=569s) But we what's really interesting is if

[9:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=570s) we scroll down here,

[9:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=572s) we can see that the router used

[9:34](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=574s) significantly less tokens and was

[9:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=577s) significantly faster than Opus.

[9:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=581s) Okay, let's jump into a real workflow

[9:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=583s) here. This is where the inference router

[9:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=585s) really becomes impactful.

[9:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=587s) Here I have two terminals running open

[9:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=589s) code.

[9:50](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=590s) On the left, I have a single model

[9:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=591s) approach using quad Opus. So, I have

[9:53](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=593s) Opus set up or open code set up with

[9:55](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=595s) Opus.

[9:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=596s) On the right, I've configured open code

[9:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=598s) to send requests to our software

[9:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=599s) engineering router

[10:00](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=600s) that I just showed you configure.

[10:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=603s) Um below, I kind of have this custom

[10:04](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=604s) built open code where you'll be able to

[10:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=606s) see live uh

[10:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=608s) observability essentially.

[10:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=609s) So, let's go go ahead and get these

[10:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=610s) started. It's just a simple

[10:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=612s) feature request preloaded into here.

[10:14](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=614s) Build me a spinning wheel app.

[10:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=616s) I'll run the same prompt in both.

[10:18](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=618s) And as this runs, we can focus on the

[10:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=620s) bottom panel. So, it'll start to show up

[10:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=622s) here. Hopefully, you can see that on the

[10:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=623s) screen.

[10:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=624s) Uh you'll be able to see token usage in

[10:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=625s) real time, which models are being

[10:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=627s) selected,

[10:28](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=628s) what task those map to, and the cost

[10:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=630s) accumulating live.

[10:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=632s) So, on the right, we can already see

[10:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=633s) that we're starting to route to Gemini

[10:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=635s) 5.2 because our requests are starting to

[10:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=638s) match the code generation.

[10:40](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=640s) And on the left, of course, we're just

[10:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=641s) routing to quad Opus. I think open code

[10:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=643s) sometimes routes to to Haiku by itself.

[10:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=645s) So, that's what you see there.

[10:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=648s) And we'll notice the latency, too, how

[10:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=649s) quickly things start to come back. In

[10:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=651s) this case, it wants me to

[10:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=654s) create a temporary directory.

[10:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=658s) So, the key difference here is that on

[10:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=659s) the left, we'll see every single request

[11:01](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=661s) that I write goes to the same premium

[11:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=663s) model. Cost and latency are going to

[11:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=665s) stay high for pretty much every single

[11:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=666s) task. On the right, the router is

[11:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=669s) selecting models based on the task.

[11:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=673s) >> [snorts]

[11:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=673s) >> So, we're optimizing both cost and

[11:15](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=675s) speed.

[11:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=676s) And we can see that uh our software

[11:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=677s) engineering router already finished. And

[11:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=679s) if we look here, it actually matched to

[11:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=681s) two models throughout. So, let's go

[11:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=682s) ahead and open this app and see how it

[11:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=683s) looks.

[11:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=685s) Okay, this actually looks really solid

[11:26](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=686s) to me. And Opus 4.7 finished at a

[11:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=689s) similar time. Let's take a look at that.

[11:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=692s) We can compare them. I mean, it's This

[11:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=693s) is a vibe check, right? But honestly, I

[11:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=695s) would say the software engineer router

[11:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=696s) did better cuz this is an interesting

[11:38](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=698s) approach that you

[11:39](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=699s) I'm not even sure it works too well. So,

[11:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=701s) in this case,

[11:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=702s) the router did a little bit better.

[11:45](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=705s) So, now that last step is done, you

[11:46](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=706s) know, we get similar outputs, but if we

[11:48](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=708s) look here,

[11:50](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=710s) the software engineer router has only

[11:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=712s) spent 8 cents on the session,

[11:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=714s) while Opus directly has spent 25 cents.

[11:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=716s) So, we have a about a 3x in cost and

[11:59](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=719s) very, very similar quality so far.

[12:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=723s) Let's try another another prompt here.

[12:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=725s) What What comes next in a software

[12:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=726s) engineering life cycle? Probably writing

[12:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=728s) some unit tests, right?

[12:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=732s) So, we'll write this in both.

[12:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=736s) Start with this first. On

[12:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=737s) [clears throat] the right, we have the

[12:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=739s) router again. And we can see that it got

[12:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=741s) matched to the test writing and code

[12:23](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=743s) verification, which picked the Claude 5

[12:25](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=745s) Sonnet

[12:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=747s) model because that's what I configured

[12:28](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=748s) earlier.

[12:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=749s) And we'll see the same pattern. It's

[12:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=750s) going to be significantly cheaper

[12:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=752s) overall across the entire session than

[12:34](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=754s) going straight to Opus.

[12:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=757s) So, we'll let this finish here.

[12:42](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=762s) Okay, and that one finished. Let's just

[12:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=764s) queue up one more. Write some

[12:46](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=766s) documentation in a read me.

[12:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=774s) And then we'll compare

[12:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=776s) the total session cost.

[13:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=783s) Okay, and as this runs, we'll wait and

[13:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=785s) see what it does.

[13:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=789s) Okay, it created the read me. And well,

[13:11](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=791s) if we look here, we can see that the

[13:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=793s) total session cost

[13:14](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=794s) for the router was 14 cents, while the

[13:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=797s) total session cost for Opus was 44

[13:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=799s) cents.

[13:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=801s) So, at this point, we can see the cost

[13:22](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=802s) is significantly lower,

[13:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=804s) latency is optimized per step,

[13:26](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=806s) and the quality remains pretty similar

[13:28](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=808s) across. So, you can see as you scale

[13:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=810s) this, the cost

[13:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=812s) performance really add up.

[13:35](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=815s) Okay. Archana, thank you.

[13:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=821s) >> [applause]

[13:44](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=824s) >> Thank you so much, Tyler. And that was

[13:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=827s) actually a live demo that we ran here.

[13:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=829s) So, thanks to Tyler for setting it up

[13:51](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=831s) and taking us through that.

[13:53](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=833s) So,

[13:54](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=834s) now that you've seen it work, let's look

[13:56](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=836s) at some quick facts.

[13:58](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=838s) Routing decision in under 200

[14:00](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=840s) milliseconds per request.

[14:03](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=843s) It runs on a custom mixture of experts

[14:05](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=845s) model purpose-built for routing.

[14:08](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=848s) Zero application code changes needed

[14:10](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=850s) from you to get it to adopt.

[14:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=852s) And it's free and included, so you do

[14:13](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=853s) not have to roll out your own router.

[14:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=856s) And we open source the whole routing

[14:18](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=858s) model via plain old, so you can actually

[14:20](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=860s) check how that looks as well.

[14:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=864s) The last thing I wanted to talk about

[14:26](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=866s) was a bit about

[14:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=867s) routing is the foundation layer. It's

[14:29](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=869s) not really the destination.

[14:31](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=871s) And there are three things that we

[14:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=872s) usually build on top of it.

[14:34](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=874s) The first one is evals to prove that the

[14:37](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=877s) right model works with your use case and

[14:39](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=879s) your test well.

[14:41](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=881s) Caching, so that you can stop paying

[14:43](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=883s) twice or more for the same answer each

[14:46](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=886s) time.

[14:47](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=887s) And personalization, so that the router

[14:49](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=889s) learns what works for your team over

[14:52](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=892s) time.

[14:53](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=893s) This is a continuous improvement loop

[14:55](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=895s) maturing over time. That means that the

[14:57](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=897s) more you route and evaluate, the better

[15:00](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=900s) the router does for your workload.

[15:04](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=904s) So, to summarize, where does this leave

[15:06](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=906s) you?

[15:07](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=907s) There is no single best model. There's

[15:09](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=909s) only the right model for the request.

[15:12](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=912s) And benchmarks will only tell you part

[15:14](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=914s) of the story. Your preferences will tell

[15:16](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=916s) you the rest.

[15:17](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=917s) And we built the router to honor your

[15:19](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=919s) preferences and stay open, so that

[15:21](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=921s) you're never locked into a single stack.

[15:24](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=924s) And that's how teams actually built.

[15:27](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=927s) We are DigitalOcean, any i-native cloud.

[15:30](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=930s) Come find us at the booth and route your

[15:32](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=932s) next workload with us. Thank you so much

[15:33](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=933s) for being here.

[15:36](https://www.youtube.com/watch?v=FvxY8oPoI8o&t=936s) >> [applause]
