fun stories from building OpenRouter and where all this is going

Will one company supply the intelligence behind almost every application? After ChatGPT arrived in late 2022, Alex Atallah began investigating that question. OpenAI had a commanding lead, and inference looked to him like a contender for the largest market in software. At the beginning of 2023, he started building prototypes with alternative models and exploring open source. OpenRouter began as that experiment before becoming a marketplace.

The first evidence for model choice came from moderation. In January, users wanted to understand which outputs a provider allowed and whether their application risked being deplatformed. Consider a detective novel that reaches chapter four: the detective encounters a murder involving a shooting, and OpenAI sometimes refuses to continue—or leaves the author unsure whether continuing would violate its terms. Role-play exposed the same uncertainty across a broader range of interactions. A model could lead on capability and still fail to meet a particular user's requirements.

Early alternatives included BLOOM 176B and Facebook's OPT. They attracted experimentation, but practical usefulness remained limited. Atallah's informal audience poll captures how narrow that early community was: roughly ten people raised their hands for BLOOM and five for OPT. Then Meta released LLaMA 1 in February 2023.

The LLaMA paper reported that LLaMA-13B outperformed GPT-3 175B on most benchmarks. The parameter sizes made this striking: a much smaller model offered a path toward laptop inference instead of requiring the expensive server infrastructure associated with the larger model. But benchmark performance did not make LLaMA a ready replacement for a chat assistant. It was primarily a text completion model, local execution was difficult, and the supporting infrastructure was immature.

The turning point for Atallah was Stanford's March 2023 release of Alpaca. Its recipe transferred instruction-following behavior into an already pretrained smaller model:

1. Generate instruction demonstrations with text-davinci-003, the teacher Atallah describes broadly as GPT-3.
2. Fine-tune LLaMA 7B on those demonstrations.
3. Expose the resulting model's learned behavior through inference.

Stanford reported demonstration generation below $500 and fine-tuning below $100; the combined below-$600 figure excluded base-model pretraining. The distinction matters: this was an inexpensive way to adapt existing weights, not to train a foundation model from scratch.

Atallah saw both style and knowledge moving from a large model into a small one. Instead of assuming that creating a useful model required a $10 million training budget, developers could begin imagining their own data delivered as a language-model service. That made a future with tens or hundreds of thousands of specialized models plausible to him.

But distributing weights did not solve discovery. Seven billion floating-point numbers do not tell a developer which tasks a model handles well or how to use it. Open weights can still be opaque in behavior. Even Alpaca had reached only a small portion of this audience: Atallah counted roughly twelve raised hands. OpenRouter's initial purpose was to collect these models in one place and help people understand what they could do.

Before building the marketplace, Atallah tested a more direct question: could a user bring a preferred model to a website whose developer had never chosen that model? In April 2023, he launched Window AI, an open-source Chrome extension that let the user select the model while the web application consumed it. The demonstration used Together's deployment of GPT-NeoX to replace OpenAI inside a webpage. The application and its source of intelligence could be selected independently.

OpenRouter launched the following month. Atallah co-founded it with Louis, the founder of Plasmo, the framework used to build Window AI. It began as a collection of models and a way to understand their uses, then expanded toward better prices, better uptime, no subscription, and more choice. Atallah reports 10–100% month-over-month growth over the preceding two years, without naming the growth metric. Its public rankings also became a way to see which models developers were using and how that usage changed.

The developer proposition was one paid API with near-zero interface switching costs between models. At talk time, Atallah reported over 400 models and over 60 active providers. OpenRouter accepted multiple payment methods, including crypto, and took on the work of normalizing tool calls and caching across providers. The original collection and usage-data explorer was becoming infrastructure for using the models, not just finding them.

Initially, an open model might have only one or two providers on OpenRouter. A primary provider handled requests; a fallback covered failures. The providers were not even named. As more companies began hosting the same weights, that arrangement stopped expressing the choices developers faced. Providers differed in both economics and supported behavior:

Selecting the model was no longer enough. A developer also needed an appropriate deployment of that model. Aggregating those deployments at different prices turned OpenRouter into a marketplace.

Atallah gives LLaMA 3.3 70B Instruct as an example with approximately 23 providers on OpenRouter. Closed models created a related opportunity when their services could not keep up with demand: aggregating providers gave developers additional routes to obtain the same model and improved availability. Real-world latency and throughput graphs helped developers evaluate the alternatives for both open and closed models.

The surrounding product made those choices easier to inspect. Developers could compare models using their own prompts with an interaction Atallah likens to texting in iMessage. Fine-grained privacy controls, API-level overrides, and consolidated usage observability brought model selection, data policy, and monitoring into the same workflow.

The platform's traffic led Atallah away from a winner-take-all forecast. Atallah reports that Google Gemini grew from roughly 2–3% to 34–35% of OpenRouter's processed tokens over about twelve months, beginning in June of the previous year. Anthropic was also highly popular, and OpenAI grew too. These are platform token shares, not global market shares: Atallah says OpenAI was underrepresented because developers often came to OpenRouter to obtain an OpenAI-like interface for other models.

Two ideas fit together here. Model choice remains diverse, while serving a particular model can become a commodity. Customers use different models for different purposes. But Claude served through Bedrock should look the same to an application as Claude served through Vertex, even when the providers offer different rates and performance. Making that interface consistent lets a developer choose inference supply without rewriting the application around the supplier.

Atallah expects inference to become a dominant operating expense, making selection and routing increasingly consequential. He also describes model usage as sticky: the number of active models kept growing rather than each new release simply replacing its predecessor. The engineering goal was therefore to make an expanding, heterogeneous ecosystem easier to use through a consistent interface.

The next problem was adding capabilities across models. OpenRouter did not have an MCP marketplace at the time of the talk, but it wanted features such as web search and PDF parsing to work across its model catalog. Atallah characterizes the MCP usage he was considering then as pre-flight access to APIs and behaviors. OpenRouter also needed to transform results on their way back to the user. That additional requirement suggested middleware, familiar from authentication and caching in web frameworks such as Next.js.

A plugin could call MCPs internally, then augment the inference result on the return path. The web-search example makes the distinction concrete: search supplies information to the model, while output processing adds web annotations to the response. Atallah describes those annotations arriving as results stream to the user, without waiting for all tokens to be generated.

In TypeScript, an asynchronous generator expresses the return-path mechanism directly. Here, a search result carries a sourceId; response chunks retain their text while middleware attaches annotations for cited sources:

type SearchResult = {
  sourceId: string;
  title: string;
};

type Chunk = {
  text: string;
  citedSourceIds: string[];
};

type AnnotatedChunk = Chunk & {
  annotations: SearchResult[];
};

async function* annotateStream(
  chunks: AsyncIterable<Chunk>,
  results: SearchResult[],
): AsyncGenerator<AnnotatedChunk> {
  const sources = new Map(
    results.map((result) => [result.sourceId, result]),
  );

  for await (const chunk of chunks) {
    const annotations: SearchResult[] = [];
    for (const id of new Set(chunk.citedSourceIds)) {
      const source = sources.get(id);
      if (source) annotations.push(source);
    }
    yield { ...chunk, annotations };
  }
}

This expresses the middleware pattern rather than OpenRouter's internal plugin contract. Each incoming chunk can produce an annotated outgoing chunk; the function never collects the entire response. The key architectural capability is interception on the return path as well as preparation before inference.

A common interface also has to account for operational differences. Atallah reports approximately 30 ms latency after extensive custom caching, but does not specify measurement boundaries, percentile, region, or request conditions. He believes the result is industry-leading; the talk does not establish a comparative benchmark or a complete model-response latency.

Cancellation is a particularly consequential difference because stopping delivery does not necessarily stop billing. Atallah describes three reported outcomes after the client drops the stream:

• Full completion billed: the provider charges for the entire completion.
• Not the full completion: the provider does not charge for the entire completion; the talk does not specify the exact cutoff.
• Additional unseen tokens: the provider bills for tokens the client never receives—twenty in his example.

These charges do not by themselves establish whether or when generation stopped. Standardizing streaming therefore requires investigating each provider's cancellation and billing semantics, then determining when those differences matter to developers. A uniform stream format alone cannot resolve them.

Standardizing models and providers had become a substantial architecture problem. The next planned expansion was into additional modalities, including language models that generate images. Atallah uses “transfusion” to describe a combination of transformers and diffusion, loosely invoking Stable Diffusion. The related Transfusion research combines next-token prediction and image diffusion within one transformer; the approach does not inherently require the Stable Diffusion product.

The expected benefit was richer world knowledge in generated images and the ability to converse about an image while creating or changing it. Atallah illustrates the possibility with a secondhand customer story about generating a whole menu for a delivery app. He does not identify the model or show its output. The example points toward the kind of multimodal application he expected to become significant in the coming year.

More powerful routing was also on the roadmap. Geographic routing was minimal at talk time; Atallah wanted to route requests to the right GPU in the right location and add enterprise-level optimization. That extends provider selection beyond a model name and a price to the infrastructure serving the request.

Better prompt observability and finer model categorization would make selection more specific too. Instead of asking for the best model in general, a developer might ask which models best turn Japanese instructions into Python code. That is a discovery problem organized around a concrete input, task, and output.

Atallah closes with the promise of better prices and an invitation to build a durable ecosystem through collaboration. Low vendor lock-in is the connecting goal: applications should be able to choose useful intelligence, obtain it from suitable infrastructure, and keep changing those choices as models and providers evolve.