# fun stories from building OpenRouter and where all this is going

Alex Atallah

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/84Vtz2IL1Ug-fun-stories-from-building-openrouter-where-all) · [Video](https://www.youtube.com/watch?v=84Vtz2IL1Ug)

## fun stories from building OpenRouter and where all this is going

OpenRouter grew from a question about winner-take-all AI into a marketplace whose hardest problems include provider differences, streamed middleware, and cancellation.

Familiarity with language-model APIs and streamed responses will help with the middleware and provider-routing examples.

### Will inference be winner-take-all?

[0:25](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=25s)

Will one company supply the intelligence behind almost every application? After ChatGPT arrived in late 2022, Alex Atallah began investigating that question. OpenAI had a commanding lead, and inference looked to him like a contender for the largest market in software. At the beginning of 2023, he started building prototypes with alternative models and exploring open source. [OpenRouter](https://openrouter.ai/) began as that experiment before becoming a marketplace.

![Slide stating “Inference may be the largest market ever in software” and asking “Will it be winner-take-all?” beside the speaker.](https://ai.engineer/api/talk-essay-media/7155cff61a78ad8ffc4b28b61a6b57f1b3e18b0d84c524820cfe1bf2c0e30e26)

Will inference be a winner-take-all market?

The first evidence for model choice came from moderation. In January, users wanted to understand which outputs a provider allowed and whether their application risked being deplatformed. Consider a detective novel that reaches chapter four: the detective encounters a murder involving a shooting, and OpenAI sometimes refuses to continue—or leaves the author unsure whether continuing would violate its terms. Role-play exposed the same uncertainty across a broader range of interactions. **A model could lead on capability and still fail to meet a particular user's requirements.**

### Open weights arrive before usable infrastructure

[2:38](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=158s)

Early alternatives included **BLOOM 176B** and Facebook's **OPT**. They attracted experimentation, but practical usefulness remained limited. Atallah's informal audience poll captures how narrow that early community was: roughly ten people raised their hands for BLOOM and five for OPT. Then Meta released **LLaMA 1** in February 2023.

The [LLaMA paper](https://arxiv.org/abs/2302.13971) reported that LLaMA-13B outperformed **GPT-3 175B** on most benchmarks. The parameter sizes made this striking: a much smaller model offered a path toward laptop inference instead of requiring the expensive server infrastructure associated with the larger model. But benchmark performance did not make LLaMA a ready replacement for a chat assistant. It was primarily a text completion model, local execution was difficult, and the supporting infrastructure was immature.

### Alpaca makes a long tail imaginable

[4:29](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=269s)

The turning point for Atallah was Stanford's March 2023 release of [**Alpaca**](https://crfm.stanford.edu/2023/03/13/alpaca). Its recipe transferred instruction-following behavior into an already pretrained smaller model:

1. Generate instruction demonstrations with **`text-davinci-003`**, the teacher Atallah describes broadly as GPT-3.
2. Fine-tune LLaMA 7B on those demonstrations.
3. Expose the resulting model's learned behavior through inference.

Stanford reported demonstration generation below $500 and fine-tuning below $100; the combined below-$600 figure excluded base-model pretraining. The distinction matters: this was an inexpensive way to adapt existing weights, not to train a foundation model from scratch.

Atallah saw both style and knowledge moving from a large model into a small one. Instead of assuming that creating a useful model required a $10 million training budget, developers could begin imagining their own data delivered as a language-model service. That made a future with tens or hundreds of thousands of specialized models plausible to him.

But distributing weights did not solve discovery. Seven billion floating-point numbers do not tell a developer which tasks a model handles well or how to use it. **Open weights can still be opaque in behavior.** Even Alpaca had reached only a small portion of this audience: Atallah counted roughly twelve raised hands. OpenRouter's initial purpose was to collect these models in one place and help people understand what they could do.

![Slide titled “March, 2023: The first distillation success” showing the Alpaca article heading and Stanford Alpaca logo.](https://ai.engineer/api/talk-essay-media/59e781b2a4c2e1a0e3b5529962b480037c76772dd124b7a5a144f0e340acbad9)

March 2023: Stanford Alpaca, a turning point for Atallah.

### Let the user choose the intelligence

[6:20](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=380s)

Before building the marketplace, Atallah tested a more direct question: could a user bring a preferred model to a website whose developer had never chosen that model? In April 2023, he launched [Window AI](https://github.com/alexanderatallah/window.ai), an open-source Chrome extension that let the user select the model while the web application consumed it. The demonstration used Together's deployment of **GPT-NeoX** to replace OpenAI inside a webpage. The application and its source of intelligence could be selected independently.

OpenRouter launched the following month. Atallah co-founded it with Louis, the founder of Plasmo, the framework used to build Window AI. It began as a collection of models and a way to understand their uses, then expanded toward better prices, better uptime, no subscription, and more choice. Atallah reports 10–100% month-over-month growth over the preceding two years, without naming the growth metric. Its [public rankings](https://openrouter.ai/rankings) also became a way to see which models developers were using and how that usage changed.

The developer proposition was one paid API with near-zero interface switching costs between models. At talk time, Atallah reported over 400 models and over 60 active providers. OpenRouter accepted multiple payment methods, including crypto, and took on the work of normalizing tool calls and caching across providers. The original collection and usage-data explorer was becoming infrastructure for using the models, not just finding them.

### The same model becomes a marketplace

[9:07](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=547s)

Initially, an open model might have only one or two providers on OpenRouter. A primary provider handled requests; a fallback covered failures. The providers were not even named. As more companies began hosting the same weights, that arrangement stopped expressing the choices developers faced. Providers differed in both economics and supported behavior:

| Dimension | Provider differences |
|---|---|
| Price and performance | Different prices, latency, and throughput |
| Sampling | Some supported `min-p`; most did not |
| Caching | Available on some deployments |
| Tool calling | Supported on some deployments |
| Structured outputs | Supported on some deployments |

Selecting the model was no longer enough. A developer also needed an appropriate deployment of that model. Aggregating those deployments at different prices turned OpenRouter into a marketplace.

Atallah gives **LLaMA 3.3 70B Instruct** as an example with approximately 23 providers on OpenRouter. Closed models created a related opportunity when their services could not keep up with demand: aggregating providers gave developers additional routes to obtain the same model and improved availability. Real-world latency and throughput graphs helped developers evaluate the alternatives for both open and closed models.

The surrounding product made those choices easier to inspect. Developers could compare models using their own prompts with an interaction Atallah likens to texting in iMessage. Fine-grained privacy controls, API-level overrides, and consolidated usage observability brought model selection, data policy, and monitoring into the same workflow.

### Different models, interchangeable supply

[11:30](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=690s)

The platform's traffic led Atallah away from a winner-take-all forecast. Atallah reports that **Google Gemini** grew from roughly 2–3% to 34–35% of OpenRouter's processed tokens over about twelve months, beginning in June of the previous year. Anthropic was also highly popular, and OpenAI grew too. These are platform token shares, not global market shares: Atallah says OpenAI was underrepresented because developers often came to OpenRouter to obtain an OpenAI-like interface for other models.

Two ideas fit together here. **Model choice remains diverse, while serving a particular model can become a commodity.** Customers use different models for different purposes. But **Claude** served through Bedrock should look the same to an application as Claude served through Vertex, even when the providers offer different rates and performance. Making that interface consistent lets a developer choose inference supply without rewriting the application around the supplier.

Atallah expects inference to become a dominant operating expense, making selection and routing increasingly consequential. He also describes model usage as sticky: the number of active models kept growing rather than each new release simply replacing its predecessor. The engineering goal was therefore to make an expanding, heterogeneous ecosystem easier to use through a consistent interface.

### Middleware on both sides of inference

[13:40](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=820s)

The next problem was adding capabilities across models. OpenRouter did not have an MCP marketplace at the time of the talk, but it wanted features such as web search and PDF parsing to work across its model catalog. Atallah characterizes the MCP usage he was considering then as pre-flight access to APIs and behaviors. OpenRouter also needed to transform results on their way back to the user. That additional requirement suggested **middleware**, familiar from authentication and caching in web frameworks such as Next.js.

![Slide titled “We built a middleware for inference called ‘Plugins’” above a TypeScript code screenshot showing FileParserPlugin and its complete method.](https://ai.engineer/api/talk-essay-media/089ca3c96d049126c9b1ddd12a2bbb4498a36de633294c724447b699de61ea1e)

Inference middleware implemented as plugins.

A plugin could call MCPs internally, then augment the inference result on the return path. The web-search example makes the distinction concrete: search supplies information to the model, while output processing adds web annotations to the response. Atallah describes those annotations arriving as results stream to the user, without waiting for all tokens to be generated.

In TypeScript, an asynchronous generator expresses the return-path mechanism directly. Here, a search result carries a `sourceId`; response chunks retain their text while middleware attaches annotations for cited sources:

```typescript
type SearchResult = {
  sourceId: string;
  title: string;
};

type Chunk = {
  text: string;
  citedSourceIds: string[];
};

type AnnotatedChunk = Chunk & {
  annotations: SearchResult[];
};

async function* annotateStream(
  chunks: AsyncIterable<Chunk>,
  results: SearchResult[],
): AsyncGenerator<AnnotatedChunk> {
  const sources = new Map(
    results.map((result) => [result.sourceId, result]),
  );

  for await (const chunk of chunks) {
    const annotations: SearchResult[] = [];
    for (const id of new Set(chunk.citedSourceIds)) {
      const source = sources.get(id);
      if (source) annotations.push(source);
    }
    yield { ...chunk, annotations };
  }
}
```

This expresses the middleware pattern rather than OpenRouter's internal plugin contract. Each incoming chunk can produce an annotated outgoing chunk; the function never collects the entire response. The key architectural capability is interception on the return path as well as preparation before inference.

### A dropped stream is not necessarily canceled inference

[16:02](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=962s)

A common interface also has to account for operational differences. Atallah reports approximately 30 ms latency after extensive custom caching, but does not specify measurement boundaries, percentile, region, or request conditions. He believes the result is industry-leading; the talk does not establish a comparative benchmark or a complete model-response latency.

Cancellation is a particularly consequential difference because stopping delivery does not necessarily stop billing. Atallah describes three reported outcomes after the client drops the stream:

- **Full completion billed:** the provider charges for the entire completion.
- **Not the full completion:** the provider does not charge for the entire completion; the talk does not specify the exact cutoff.
- **Additional unseen tokens:** the provider bills for tokens the client never receives—twenty in his example.

These charges do not by themselves establish whether or when generation stopped. Standardizing streaming therefore requires investigating each provider's cancellation and billing semantics, then determining when those differences matter to developers. A uniform stream format alone cannot resolve them.

### Extending the interface to images

[16:50](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1010s)

Standardizing models and providers had become a substantial architecture problem. The next planned expansion was into additional modalities, including language models that generate images. Atallah uses “transfusion” to describe a combination of transformers and diffusion, loosely invoking **Stable Diffusion**. The related [Transfusion research](https://arxiv.org/abs/2408.11039) combines next-token prediction and image diffusion within one transformer; the approach does not inherently require the Stable Diffusion product.

The expected benefit was richer world knowledge in generated images and the ability to converse about an image while creating or changing it. Atallah illustrates the possibility with a secondhand customer story about generating a whole menu for a delivery app. He does not identify the model or show its output. The example points toward the kind of multimodal application he expected to become significant in the coming year.

### Route to the right GPU, discover the right model

[17:54](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1074s)

More powerful routing was also on the roadmap. Geographic routing was minimal at talk time; Atallah wanted to route requests to the right GPU in the right location and add enterprise-level optimization. That extends provider selection beyond a model name and a price to the infrastructure serving the request.

Better prompt observability and finer model categorization would make selection more specific too. Instead of asking for the best model in general, a developer might ask which models best turn Japanese instructions into Python code. That is a discovery problem organized around a concrete input, task, and output.

![Slide reading “Better prompt observability. Better model discoverability/routing.” above an OpenRouter model page with category tags and navigation tabs.](https://ai.engineer/api/talk-essay-media/9445e6be17f8c78cf87eac2a7d29de788a9bbebf9ea3b2b3fc4e7b316b2d4873)

Better prompt observability and model discovery through categorization.

Atallah closes with the promise of better prices and an invitation to build a durable ecosystem through collaboration. Low vendor lock-in is the connecting goal: applications should be able to choose useful intelligence, obtain it from suitable infrastructure, and keep changing those choices as models and providers evolve.

## Resources

- [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) — Original LLaMA paper describing its model family and comparison with GPT-3.
- [Stanford Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca) — Training recipe, cost breakdown, preliminary evaluation, and limitations of Alpaca 7B.
- [Window AI](https://github.com/alexanderatallah/window.ai) — Browser extension source and integration instructions for applications using user-selected AI models.
- [OpenRouter provider routing](https://openrouter.ai/docs/guides/routing/provider-selection) — Current controls for provider selection, fallbacks, performance preferences, and data policies.
- [OpenRouter web search](https://openrouter.ai/docs/guides/features/plugins/web-search) — Current instructions for adding web search and citation annotations to model responses.
- [OpenRouter model rankings](https://openrouter.ai/rankings) — Live model usage rankings based on tokens processed through OpenRouter, with measurement methodology.
- [Transfusion](https://arxiv.org/abs/2408.11039) — Research combining text prediction and image diffusion in a single multimodal transformer.

## Slides

### Inference as a software market

Alex Atallah started OpenRouter in early 2023 asking whether the emerging inference market would be winner take all. He saw its potential size as making that question critical.

[0:29](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=29s) [Slide states that inference may be the largest market ever in software.](https://assets.ai.engineer/sites/aiecode2025/sha256/5bf98f9c5dc1c892e7c215d994594d89a72c4ae0312ff25be0c8653361ce0753.webp)

Inference may be the largest market ever in software

AI Engineer
World’s Fair

Microsoft
ALEX ATALLAH
CEO
OpenRouter

### Inference as a software market

Alex Atallah started OpenRouter in 2023 with a question: would the emerging inference market be winner-take-all? He saw it as potentially the largest market ever in software.

[0:43](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=43s) [Slide states that inference may be the largest market ever in software.](https://assets.ai.engineer/sites/aiecode2025/sha256/a0c700f6910d1a58dbd8372f6c16fb202803c3506ddedff541fc3482ed9deae6.webp)

Inference may be the largest market ever in software

AI Engineer
World’s Fair

aws

FUN STORIES FROM BUILDING OPENROUTER AND WHERE ALL THIS IS GOING

### Will inference be winner-take-all?

When he started OpenRouter in early 2023, the speaker wanted to know whether inference would be winner-take-all. He says many assumed it would.

[0:48](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=48s) [Slide asks whether the inference market will be winner-take-all.](https://assets.ai.engineer/sites/aiecode2025/sha256/c70bce39a9a133ff312485b9afe4b95dfe248297e4cb97a878aa64d298b7537f.webp)

Inference may be the largest market ever in software

Will it be winner-take-all?

Interface text:
AI Engineer World’s Fair
aws
FUN STORIES FROM BUILDING OPENROUTER AND WHERE ALL THIS IS GOING

### Will inference be winner-take-all?

The talk asks whether one player will dominate a potentially enormous inference market. OpenRouter began as an experiment and evolved into a marketplace.

[1:28](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=88s) [Slide asks whether the inference market will be winner-take-all.](https://assets.ai.engineer/sites/aiecode2025/sha256/e87591b141bec243ef7316b9f5fe0a528a9c6cdd7d17cf262c8c3735255d3720.webp)

Inference may be the largest market ever in software

Will it be winner-take-all?

Interface text:
AI Engineer World’s Fair
AWS
FUN STORIES FROM BUILDING OPENROUTER AND WHERE ALL THIS IS GOING

### January 2023: Moderation

A Reddit user reports losing API access and suspects fictional roleplay triggered it. The speaker cites moderation uncertainty as an early reason people sought other models.

[2:09](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=129s) [A slide shows a Reddit post discussing an API account suspension and moderation.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ca73ba1a8f8942b48b2436b063fa49595c2124153ebbb5f4a55f3f7968569e8.webp)

January, 2023: Moderation

OpenAI Banning accounts.

Gone Wild

Hi there,

After a thorough investigation, we have determined that you or a member of your organization are using the OpenAI API in ways that violate our policies.

Due to this breach we are halting access to the API immediately for the organization Personal. Common reasons for breach include violations of our content policy, repeated attempts at disallowed use-cases, or accessing the API from an unsupported location. You may also wish to review our Terms of Use.

If you believe this is in error and would like to appeal, please contact us through our help center. We will review appeals within one business day and will contact you if we reinstate access to the API.

Best,
The OpenAI team

That was the E-mail I received just a couple of minutes earlier.

It seems OpenAI does not appreciate people using it's program for D&D/ Text based/ Roleplaying/ Fictional based adventure games that might entail forms of Violence/ Gore/ Fantasy/ Erotic/Fictional gameplay.

I had just purchased an account for the Beta Playground and shortly after I was removed. I'm not going to be supporting OpenAI anymore after this action. I don't appreciate the weights and aggression they have placed on the filter precautions, and I feel this is highly unfair.

Interface text:
r/ChatGPT · 2 yr. ago
Noobrage2112
AI Engineer World's Fair
Engineering the future of AI

### Language models to try

A Reddit user lists models accessible without a GPU cluster or more than $20, including ChatGPT, BLOOM and OPT. The speaker recalls BLOOM and OPT as early open-source options.

[2:44](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=164s) [A Reddit post lists accessible language models, including ChatGPT, BLOOM and OPT-175B.](https://assets.ai.engineer/sites/aiecode2025/sha256/93e4feb1aeefecdbf85437a7522e2f21645622f66a50a3eff453a4aadb1b6f95.webp)

[D] List of Large Language Models to play with.

Hello! I'm trying to understand what available LLMs one can "relatively easily" play with. My goal is to understand the landscape since I haven't worked in this field before. I'm trying to run them "from the largest to the smallest".

By "relatively easy", I mean doesn't require to setup a GPU cluster or costs more than $20:)

Here are some examples I have found so far:
1. ChatGPT (obviously) - 175B params
2. OpenAI api to access GPT-3s (from ada (0.5B) to davinci (175B)). Also CodeX
3. Bloom (176B) - text window on that page seems to work reliably, you just need to keep pressing "generate"
4. OPT-175B (Facebook LLM), the hosting works surprisingly fast, but slower than ChatGPT
5. Several models on HuggingFace that I made to run with Colab Pro subscription: GPT-NeoX 20B, Flan-t5-xxl 11B, Xlm-roberta-xxl 10.7B, GPT-j 6B. I spent about $20 total on running the models below. None of the Hugging face API interfaces/spaces didn't work for me :(. Here is an example notebook I made for NeoX.

Does anyone know more models that are easily accessible?

Interface text:
r/MachineLearning · 2 yr. ago
sinavski
Discussion

### February 2023: Open source race begins

A Reddit post lists models accessible without a GPU cluster or more than $20. The speaker recalls their limited utility before Meta’s first LLaMA launch in February.

[3:14](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=194s) [A slide shows a Reddit post listing accessible language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/90cbaf918aa99125a0288f8de5c016a053eca50bc30715fc2f38462420783629.webp)

February, 2023: Open Source race begins...

[D] List of Large Language Models to play with.

Hello! I'm trying to understand what available LLMs one can "relatively easily" play with. My goal is to understand the landscape since I haven't worked in this field before. I'm trying to run them "from the largest to the smallest".

By "relatively easy", I mean doesn't require to setup a GPU cluster or costs more than $20:)

Here are some examples I have found so far:

1. ChatGPT (obviously) - 175B params
2. OpenAI api to access GPT-3s (from ada (0.5B) to davinci (175B)). Also CodeX
3. Bloom (176B) - text window on that page seems to work reliably, you just need to keep pressing "generate"
4. OPT-175B (Facebook LLM), the hosting works surprisingly fast, but slower than ChatGPT
5. Several models on HuggingFace that I made to run with Colab Pro subscription: GPT-NeoX 20B, Flan-t5-xxl 11B, Xlm-roberta-xxl 10.7B, GPT-j 6B. I spent about $20 total on running the models below. None of the Hugging face API interfaces/spaces didn't work for me :(. Here is an example notebook I made for NeoX.

Does anyone know more models that are easily accessible?

Interface text:
r/MachineLearning · 2 yr. ago
sinavski
Discussion
AI Engineer World's Fair
Engineering the future of AI

### LLaMA: Open and Efficient Foundation Language Models

Meta’s paper reports that LLaMA-13B outperformed GPT-3 (175B) on most benchmarks. The speaker highlights that the smaller model could run on a laptop.

[3:25](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=205s) [LLaMA paper abstract with the LLaMA-13B benchmark claim highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/1c1d82cad5df8fd77aa3ecc6cc4095f0ebe11d4bff7a76beeb3f6a380baf6b13.webp)

LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron*, Thibaut Lavril*, Gautier Izacard*, Xavier Martinet
Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal
Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin
Edouard Grave*, Guillaume Lample*

Meta AI

Abstract

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community¹.

performance, a smaller one trained longer will ultimately be cheaper at inference. For instance, although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens.

The focus of this work is to train a series of language models that achieve the best possible performance at various inference budgets, by training on more tokens than what is typically used. The resulting models, called LLaMA, ranges from 7B to 65B parameters with competitive performance

### Alpaca: A Strong, Replicable Instruction-Following Model

Stanford fine-tuned LLaMA 7B on 52K instruction-following demonstrations to create Alpaca. The speaker describes generating the training outputs with GPT-3 and producing the model for less than $600.

[4:39](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=279s) [Stanford Alpaca page introducing Alpaca 7B and showing the Stanford Alpaca logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e2d1c28150113a2bf3b86418e29ff799f691630d045be14cf2d324f2139d597.webp)

Alpaca: A Strong, Replicable Instruction-Following Model

Authors: Rohan Taori* and Ishaan Gulrajani* and Tianyi Zhang* and Yann Dubois* and Xuechen Li* and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto

We introduce Alpaca 7B, a model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. On our preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to OpenAI’s text-davinci-003, while being surprisingly small and easy/cheap to reproduce (<600$). Checkout our code release on GitHub.

Update: The public demo is now disabled. The original goal of releasing a demo was to disseminate our research in an accessible way. We feel that we have mostly achieved this goal, and given the hosting costs and the inadequacies of our content filters, we decided to bring down the demo.

Stanford Alpaca

Overview

Instruction-following models such as GPT-3.5 (text-davinci-003), ChatGPT, Claude, and Bing Chat have become increasingly powerful. Many users now

### Alpaca: A Strong, Replicable Instruction-Following Model

Stanford fine-tuned LLaMA 1 on GPT-3 outputs to create Alpaca for less than $600. The speaker saw it as a transfer of style and knowledge from a large model to a smaller one.

[4:44](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=284s) [Stanford Alpaca page describing Alpaca 7B and showing the Stanford Alpaca logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/aff67909388156c363329b4d7ed345367fa45b2769e69a07226eb6810b56c518.webp)

Alpaca: A Strong, Replicable Instruction-Following Model

Authors: Rohan Taori* and Ishaan Gulrajani* and Tianyi Zhang* and Yann Dubois* and Xuechen Li* and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto

We introduce Alpaca 7B, a model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. On our preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to OpenAI’s text-davinci-003, while being surprisingly small and easy/cheap to reproduce (<600$). Checkout our code release on GitHub.

Update: The public demo is now disabled. The original goal of releasing a demo was to disseminate our research in an accessible way. We feel that we have mostly achieved this goal, and given the hosting costs and the inadequacies of our content filters, we decided to bring down the demo.

Stanford Alpaca

Overview

Instruction-following models such as GPT-3.5 (text-davinci-003), ChatGPT, Claude, and Bing Chat have become increasingly powerful. Many users now

### March 2023: The first distillation success

Alpaca 7B fine-tuned LLaMA 7B on 52K instruction-following examples. In a preliminary evaluation of single-turn instruction following, it behaved qualitatively like text-davinci-003. OpenRouter began as a place to collect models like it.

[6:13](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=373s) [Presentation slide showing the Stanford Alpaca page beneath a March 2023 distillation heading.](https://assets.ai.engineer/sites/aiecode2025/sha256/05ede43c91e2c280ab46a963ffaea6474a09cbe2df65c287a9fb3f7448f3589b.webp)

March, 2023: The first distillation success

Alpaca: A Strong, Replicable Instruction-Following Model

Authors: Rohan Taori* and Ishaan Gulrajani* and Tianyi Zhang* and Yann Dubois* and Xuechen Li* and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto

We introduce Alpaca 7B, a model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. On our preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to OpenAI’s text-davinci-003, while being surprisingly small and easy/cheap to reproduce (<600$). Checkout our code release on GitHub.

Update: The public demo is now disabled. The original goal of releasing a demo was to disseminate our research in an accessible way. We feel that we have mostly achieved this goal, and given the hosting costs and the inadequacies of our content filters, we decided to bring down the demo.

Stanford Alpaca

Overview

Instruction-following models such as GPT-3.5 (text-davinci-003), ChatGPT, Claude, and Bing Chat have become increasingly powerful. Many users now

### Window.ai permission request

The webpage calls Window.ai for completions while a permission dialog requests access to Together’s GPT NeoXT 20B. The speaker describes swapping an open-source model for OpenAI inside the page.

[7:11](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=431s) [A Window.ai browser example shows code and a permission request to access an AI model.](https://assets.ai.engineer/sites/aiecode2025/sha256/2fb4fa90688072b0e9d72b12c0607e9ef34e67c2d47eec9a7f72608d448eec7f.webp)

window.ai
Use your own AI models on the web

In this example, a simple chat bot is implemented in one file, with no backend. Using the window.ai Chrome E[illegible]
what model to use in your favorite apps, including ones from OpenAI, Together, Cohere, or even an AI running [illegible]

// Get the active model from the window.ai API
await window.ai.getCurrentModel()

// Get completions from the window.ai API
await window.ai.getCompletion(
  {
    messages: [...last10messages],
  },
  {
    onStreamResult: (result: any, error: any) => {
      console.log(result.message.content)
    },
  })

AI
Say

Permission Request
windowai.io
This app is requesting permission to access
Together: GPT NeoXT 20B
View Request
Ask every time
Deny
Allow

Interface text:
https://windowai.io
window.ai

### May 2023: OpenRouter

OpenRouter launched in May 2023 to collect models in one place. Its interface shows models from Google, OpenAI and Anthropic alongside usage, latency and weekly growth.

[7:29](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=449s) [An OpenRouter slide presents a unified interface for language models and featured model listings.](https://assets.ai.engineer/sites/aiecode2025/sha256/d53b471224e69156318bfbaa05174d3846f74458f7fb97ceaa6e7e647258dbb3.webp)

May, 2023: OpenRouter

The Unified Interface For LLMs
Better prices, better uptime, no subscription.

Featured Models
View Trending

Gemini 2.5 Pro Preview
by google
215.0B Tokens/wk
2.4s Latency
+7.02% Weekly growth

GPT-4.1
by openai
49.1B Tokens/wk
815ms Latency
+4.23% Weekly growth

Claude Sonnet 4  New
by anthropic
242.8B Tokens/wk
1.7s Latency
+47.29% Weekly growth

Interface text:
Start a message...
AI Engineer World's Fair
Engineering the future of AI

### OpenRouter growth and model rankings

OpenRouter reports 10–100% month-over-month growth over two years. Its rankings chart shows usage rising sharply, with GPT-4o-mini leading the displayed weekly leaderboard at 443B tokens.

[8:00](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=480s) [A growth claim beside a chart of model token usage and a leaderboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/21cb83f9af9bb3a015e6d481562677ca2f7af65350ee7cc1bec91e87ba83f807.webp)

10 to 100% growth
MoM for the last 2y

LLM Rankings
Compare models for all prompts

All Categories  Programming  Roleplay  Marketing  Marketing/Seo  Technology  Science
Translation  Legal  Finance  Health  Trivia  Academia

Y-axis ticks: 0, 650B, 1.3T, 1.95T, 2.6T
Timeline: Jun 10, 2024; Aug 5; Sep 30; Nov 25; Jan 20; Mar 17; May 12

Leaderboard
Token usage across models
Top this week

1. OpenAI: GPT-4o-mini — 443B tokens, ↓8%
GPT-4o mini is OpenAI's newest model after [GPT-4 Omni](/models/...
2. Google: Gemini 2.0 Flash — 239B tokens, ↑9%
Gemini Flash 2.0 offers a significantly faster time to first token (TTFT)...
3. Anthropic: Claude Sonnet 4 — 207B tokens, ↑25%
Claude Sonnet 4 significantly enhances the capabilities of its predec...
4. Google: Gemini 2.5 Pro Preview — 183B tokens, ↑1%
Gemini 2.5 Pro is Google's state-of-the-art AI model designed for ad...

### One API and switching costs

OpenRouter offers a single paid API for switching between models with near-zero switching costs. The Python example specifies a model in the request payload.

[8:24](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=504s) [OpenRouter quickstart documentation showing a sample API request.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab441cab9799d69aa2d7e689c9de9ee6ee5a862438029ff32b5f0d51a39303d9.webp)

One API, near-zero switching costs

Using the OpenRouter API directly

You can use the interactive Request Builder to generate OpenRouter API requests in the language of your choice.

Python  TypeScript  Shell

import requests
import json

response = requests.post(
  url="https://openrouter.ai/api/v1/chat/completions",
  headers={
    "Authorization": "Bearer <OPENROUTER_API_KEY>",
    "HTTP-Referer": "<YOUR_SITE_URL>", # Optional. Site URL for rankings
    "X-Title": "<YOUR_SITE_NAME>", # Optional. Site title for rankings
  },
  data=json.dumps({
    "model": "openai/gpt-4o", # Optional
    "messages": [
      {
        "role": "user",
        "content": "What is the meaning of life?"
      }
    ]
  })
)

Interface text:
Overview
Quickstart
FAQ
Principles
Models
Features
Privacy and Logging
Model Routing
Provider Routing
Prompt Caching
Structured Outputs
Tool Calling
Images & PDFs
Message Transforms
Uptime Optimization
Web Search
Zero Completion Insurance
Provisioning API Keys
API Reference

### Models, providers, and tokens

OpenRouter reports 8.4T monthly tokens across 400+ models and 60+ active providers. The slide shows three steps to get started: sign up, buy credits, and create an API key.

[8:31](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=511s) [OpenRouter usage metrics above signup, credit purchase, and API key setup steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/4cb5de4faa9e0dc4b18aa65c4e2fc137efa4226f82d2476d3bd50f159b8ff34e.webp)

The most models, providers, and tokens

8.4T
Monthly Tokens

2.5M+
Global Users

60+
Active Providers

400+
Models

1 Signup
Create an account to get started. You can set up an org for your team later.

2 Buy credits
Credits can be used with any model or provider.

3 Get your API key
Create an API key and start making requests. Fully OpenAI compatible.

Interface text:
Apr 1  $99
Mar 30  $10
OPENROUTER_API_KEY
[redacted]

### Initially, it wasn't a marketplace

OpenRouter began as a collection of models and a way to explore data about who used each one.

[8:59](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=539s) [A presentation slide stating that OpenRouter initially was not a marketplace.](https://assets.ai.engineer/sites/aiecode2025/sha256/e9d4ee9a05e1eeef690fb24a762d48b12c9b0f377b60ba488c8123e1daed07a4.webp)

Initially, it wasn’t a marketplace

Interface text:
AI Engineer World’s Fair
aws
[illegible]eo4j
Microsoft
smol ai

### A marketplace for both

OpenRouter compares providers for open- and closed-source models, using real-world latency and throughput data to help developers choose.

[10:43](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=643s) [A provider comparison lists Claude providers above throughput and latency charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f2b46a3f48d6b5202f0306f77e0b7e0f562240489d779c434beebdd4f299b18.webp)

So we became a marketplace for both...

Google Vertex
Context: 200K | Max Output: 64K | Input: $3 | Output: $15 | Latency: 1.38s | Throughput: 62.51 tps | Uptime

Anthropic
Context: 200K | Max Output: 128K | Input: $3 | Output: $15 | Latency: 1.90s | Throughput: 55.13 tps | Uptime

Google Vertex (Europe)
Context: 200K | Max Output: 64K | Input: $3 | Output: $15 | Latency: 1.90s | Throughput: 50.55 tps | Uptime

Amazon Bedrock
Context: 200K | Max Output: 128K | Input: $3 | Output: $15 | Latency: 1.61s | Throughput: 41.08 tps | Uptime

Throughput
Latency

Interface text:
Overview | Providers | Versions | Apps | Activity | Uptime | API
Sort by
US

AI Engineer World's Fair
Microsoft
aws
smol ai
neo4j

## Full transcript

[0:00](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=0s) [upbeat music]

[0:25](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=25s) All right. When I started OpenRouter at the beginning of 2023, I had one major question in mind. I was looking at this new market that was coming online and, and it was incredible.

[0:38](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=38s) Like, at the, the very end of, uh, 2022, we all saw ChatGPT, and I got bitten by the AI bug. Um, and I decided to look into answering this question: Will this market be winner take all?

[0:52](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=52s) Inference might be the largest market ever in software, and this seemed like a critical thing that everybody was assuming the answer to, the answer to it would be yes.

[1:02](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=62s) Um, OpenAI was just far and away the leading model. There were a few others that were coming up on its tail, and I, I built a couple prototypes to look into what they could be g- used for, and also wanted to investigate open source.

[1:19](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=79s) So in this talk, which Swyx named, um, I'm gonna talk about the founding story of OpenRouter and, uh, and go through a little bit of the hoops that we jumped through and sort of the investigation that we did as we, we put together this product that started as an experiment and kind of evolved into a marketplace over

[1:39](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=99s) time. In January, we saw the first signs of people wanting other types of models, and the, the first evidence was moderation.

[1:55](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=115s) This, this was like a very clear interest from users in looking for models where they could understand why, whether they'd be deplatformed or what the, the moderation policy of the company was.

[2:07](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=127s) And, and we saw some people, like, generating novels where, like, it would be a detective story and in chapter four, um, the detective would find someone who, like, commits a murder and shoots the victim.

[2:20](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=140s) And, and OpenAI at the time sometimes refused to generate that output, or it was, like, questionably against the terms of service. And of course, we saw role play and a, and basically a big gray area emerge around what models were willing to generate.

[2:38](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=158s) So, uh, in, in the next month, we saw the open source race begin. And that... Uh, I'm gonna do a little bit of an OG test here. Uh, raise your hand if you ever used BLOOM 176B.

[2:56](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=176s) There's like... Ah, there's like 10 hands raised. Um, or OPT by Facebook. This was like one of the earliest open source language models. About five hands raised. Uh, there were a couple of these emerging, and there were some very interesting projects to help people access them.

[3:13](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=193s) And, uh, and, and early days, they weren't really useful for very much. So, uh, we kept digging and, uh, and eventually, like, the open source community, um, rou- like, ran into Meta's first launch, which was LLaMA 1 in, in February.

[3:34](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=214s) And LLaMA 1, in their abstract, advertised that it outperformed GPT-3 on most benchmarks, you can see the highlighted part here, which blew everyone away. This was huge. An open-weights model better than GPT-3.

[3:50](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=230s) And, uh, and especially a smaller model. This was the 13 billion parameter version, one that you could run on your laptop, um, outperforming a large server-only, only like, you know, tons of money required to run inference company's model.

[4:07](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=247s) It, and it was beating it on some benchmarks. Everyone lost their minds. And LLaMA kicked off a huge storm. It still was not very useful, I have to say.

[4:18](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=258s) It was like a text completion model for the most part, and it was very difficult to run locally. The infrastructure just wasn't there. Um, and people were struggling to figure out what to do with it.

[4:29](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=269s) Which is when we found, when we had the greatest moment of all, I think, for the birth of the long tail of language models, which was the first successful distillation in March of 2023: Alpaca.

[4:45](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=285s) Uh, a group at Stanford took LLaMA 1, generated a bunch of outputs on GPT-3, and fine-tuned LLaMA 1 on those outputs and created Alpaca for less than $600 in to- like, total.

[4:59](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=299s) And this was an incredible moment. It was the first time I saw the transference of both style and knowledge from a large model onto a small one. And this me- this was a huge unlock 'cause it meant that not only do you not need a $10 million training budget to create your own models, but you could also

[5:19](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=319s) for the first time make unique data available as a service in the form of a language model.

[5:26](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=326s) And I immediately began to wonder, like, well, w- what are... There's, there's gonna be tens of thousands of these, maybe hundreds of thousands. Um, and they seem incredibly important.

[5:37](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=337s) This is knowledge finally being distilled into software. Uh, there needs to be a place on the internet to discover these and understand what they do. 'Cause even this open-weights model was still closed in a way.

[5:49](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=349s) It's a black box. You get seven billion floating point numbers. You don't know what it's good at or what to do with it. Ve- uh, very few people used Alpaca.

[5:58](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=358s) Raise your hands if you used Alpaca.

[6:02](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=362s) I see about fif- maybe 12. So it's, like, only double the number of people who used the, like, almost unusable open source models on the previous slide. So OpenRouter initially started as a place to collect all these things.

[6:20](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=380s) Um, but before we got there, I wanted to check out people's willingness to bring their own model to generic websites. Like, what if the developer didn't even know which model a user wanted to use?

[6:34](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=394s) How would a user bring their choice of model to the software that they want?

[6:40](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=400s) And, uh, in April, I launched Window AI, which was a, an open source Chrome extension, uh, that let a user choose their model and let a web app just kind of suck it in.

[6:54](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=414s) And so you can see from the Chrome extension here, if you look really closely, um, this user is using Together's open source deployment of GPT-NeoX.

[7:07](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=427s) I can't, I can't read it from the- here. But, like, an open source model that, um, swaps out OpenAI directly inside the webpage.

[7:17](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=437s) So the next month, OpenRouter launched. And, uh, I, uh, co-founded it with the founder of the framework that, that Window AI was built on, Plasmo, um, Louis. And we started OpenRouter as first a place to collect all the models in one spot and f- and help people figure out what to do with them.

[7:39](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=459s) And it eventually grew into a place that gives you the, like, better prices, better uptime, no subscription, and, uh, and the most choice for figuring out which intelligence your so- your, uh, software should run.

[7:55](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=475s) So let's talk a little bit about what it is, 'cause not everyone here might be familiar with it. Um, we, we have been growing ten to 100% month over month for the last two years.

[8:09](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=489s) It is an API that lets you access all language models and, uh, and it, it's also become kind of the go-to place for data about who's using which model, um, and how that is changing over time, which you can see on our public rankings page here.

[8:26](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=506s) It's a single API that you pay for once. You get near zero switching costs to go from model to model.

[8:33](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=513s) Uh, and we have about 400, over 400 models, over 60 active providers, and, uh, you can buy with lots of different payment methods, including crypto. And, and we basically do all the, the tricky work of normalizing tool calls and caching for you so that you get the best prices and the most features, uh, and you don't have

[8:53](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=533s) to worry about what the provider supports. Another story. Initially, OpenRouter was not a marketplace really. It was just kind of a collection of all the models and a way to explore data about who was using each one.

[9:07](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=547s) So how did we get here? Initially, when the first open source models emerged, uh, we only had, like, one or two providers for each one, and so we had, like, a primary provider and a fallback provider.

[9:21](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=561s) And i- initially that was it, and, and we didn't even name the providers. Um, but it became clear that there were going to be a bunch of companies that wanted to host these provi- these models and at very different prices and performances.

[9:37](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=577s) The, the number of features ballooned. Um, there were companies that supported the min-p sampler and most didn't. There were some that supported caching, some that supported tool calling and structured outputs, and others that didn't.

[9:49](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=589s) And suddenly the ecosystem was just ballooning into this kind of out of control heterogeneous monster. And we wanted to tame the monster. So we aggregated all providers in one spot and at different price points it became a marketplace.

[10:06](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=606s) And you can see, like, this model, LLaMA 3.3 70B instruct, um, it has one of-- it, it's one of the models with the most providers on the platform. Um, and it has, like, 23.

[10:20](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=620s) Um, closed source models also had something interesting happen to them, which is that they just-- they couldn't keep up with the demand. And, uh, and, and so we helped developers basically get uptime boosting, and you can see, like, the delta, uh, and how much we can boost uptime just by aggregating lots of different providers for a model.

[10:41](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=641s) And this became really helpful for people using open sour- or closed source. And we became a marketplace for both, um, showing graphs about latency and throughput and helping people figure out using real world data what the latency and throughput is on each model.

[10:57](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=657s) Um, and that's how OpenRouter became a marketplace and an-- uh, one optimized for language models, which I thought would be proper for, for inference, po- potentially the biggest market in software.

[11:11](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=671s) Uh, you can-- obviously a couple other things that we support, comparing models w- using your own prompts with the ease of just texting an iMessage, um, fine grain privacy controls with API level overrides, the ability to see, like, your usage of all models in one place and have great observability.

[11:30](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=690s) And back to the original question here of whether will intelligence be winner take all. Uh, I, I-- we've come to the most likely bet that that is not the case.

[11:42](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=702s) Um, here's our data broken down by model author, um, how much-- how many tokens have been processed by each one. And you can see Google Gemini started pretty low, like roughly two, 3% in June of last year, and just has grown to 34, 35%, uh, pretty steadily over the last 12

[12:06](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=726s) months. Um, oh, uh, Anthropic, uh, is, is like one of the most popular models on our platform. OpenAI is a little bit underrepresented in this data, um, because a lot of developers use us to get OpenAI-like behavior for all of their models.

[12:22](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=742s) But OpenAI has, has grown a lot here as well.

[12:27](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=747s) So here's what we believe about the market after all of the, you know, backstory that I just gave you. Um, the future's gonna be multi-model. Ton-- all of our customers, tons of customers use different models for different purposes and realize they can unlock huge gains by doing so.

[12:46](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=766s) Inference is also a commodity. Claude from Bedrock, we wanna make look exactly the same as Claude from Vertex. And we do that because, like, the two hyperscalers have fundamentally, uh, you know, the same commodity being delivered at different rates, different performances, and for a developer, you just wanna be able to, like, select that without worrying about who's

[13:09](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=789s) serving it. Um, we think inference will be, like, a dominant operating expense, and selecting and routing will be crucial. Um, you can see the number of active models on OpenRouter has just steadily grown.

[13:23](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=803s) It's not the case that people just hop from model to model. Like, it tends to be sticky, and, uh, and we tr- what's-- we're trying to just make this wild ecosystem a lot more homogeneous and easier to work with as a developer.

[13:40](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=820s) Um, to honor Swyx's title for this presentation, uh,

[13:46](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=826s) let's give a technical story, um, something that we've worked on in the process of building the company, and that was our own idea for how to do an MCP within OpenRouter.

[13:59](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=839s) So we don't have MCPs. We don't have an MCP marketplace. Um, but we did run into the need to expand inference with new features and new abilities. For example, searching the web for all models, PDF parsing for all models, um, you know, other interesting things coming soon.

[14:22](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=862s) And what we really wanted to do was give these abilities to all models, but that involves not just the pre-flight work that MCPs do today, where you can kind of get in-- you know, like call another API, get a bunch of behaviors, and then have the inference process access those behaviors as it goes.

[14:42](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=882s) We also need the ability to transform the outputs on the way to the user. And so what we really, really needed was something more like middleware. Middleware, um, is kind of a common concept in web development.

[14:57](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=897s) You set up middleware when you're setting up authentication, for example, or, or, or caching for a web app. And so we came up with a type of middleware that's sort of-- that's AI native and optimized for inference, um, and that looks not totally dissimilar from the way middleware looks in, in Next.js or, or web development.

[15:18](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=918s) You know, so pardon the code on the screen, but this is a little bit about how our pl- our plug-in system looks, and it, you know, it can call MCPs from inside a plug-in.

[15:27](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=927s) But importantly, it can also augment the results on the way back to the user. So here's an example of our web search plug-in, which augments every language model with the ability to search the web.

[15:40](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=940s) Um, every language model can just kinda tap into this plug-in and get web annotations as results are being fed back to users in real time, and this all happens in a stream.

[15:53](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=953s) See? There's no kind of like, you know, requirement that you get all of the tokens at once. It can just happen in-- live in the stream.

[16:02](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=962s) We got-- We solved a bunch of other tricky problems, uh, while building OpenRouter. We, we really wanted to get extremely low latency, um, and we got it down to about thirty milliseconds, uh, the best in the industry, I believe, um, using a lot of custom cache work, and we, we also need to make streams cancelable.

[16:23](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=983s) All these different providers have completely different stream cancellation policies. Sometimes if you just drop a stream, the, the, the, the inference provider will bill you for the entire thing.

[16:35](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=995s) Sometimes it won't. Sometimes it'll bill you for the next twenty tokens that you never got. And, um, we ki- we, we work a lot to try to figure out these edge cases and understand when developers are going to care about them, too.

[16:50](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1010s) And standardizing all these providers and models, uh, became, like, a big tricky architecture problem that we spent a while working on. So here's where all this is going. Uh, we're gonna add more modalities to OpenRouter, and I think this is, like, a big change in the industry as well.

[17:06](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1026s) We're gonna start seeing LLMs generate images. We already have, uh, a few examples on the market, but tr-- like, some people call it transfusion models, a transformer mixed with Stable Diffusion.

[17:20](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1040s) Um, these are gonna give images way more world knowledge and the ability to have a conversation with the image, which we think is just critical for growing that industry and making it really work.

[17:31](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1051s) Imagine... I just ran into somebody today who is using a transfusion model, uh, or who told me about their customer using a transfusion model to generate menus. Imagine doing that, like, a whole menu, like, in a delivery app generated by a transfusion model.

[17:48](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1068s) Um, it's going to be really exciting and, and a big deal in the coming year.

[17:54](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1074s) We're also gonna work on much more powerful routing. Like, routing is our bread and butter. And so doing geographical routing right now, we-- it's pretty minimal, but routing people to the right GPU in the right place and doing enterprise-level optimization's coming.

[18:09](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1089s) Um, better prompt observability, better discovery of models, like really fine-grained categorization. You know, imagine being able to see, like, the best models that take Japanese and, and create Python code.

[18:21](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1101s) And of course, even better prices coming soon. So, you know, we, we believe in, in collaboration, um, and, and building an ecosystem that's durable and with low vendor lock-in.

[18:34](https://www.youtube.com/watch?v=84Vtz2IL1Ug&t=1114s) So, you know, collaborate with us. Um, here's our email, and if you're interested, join us too. [outro music]
