Trends Across the AI Frontier

How should you compare AI systems when model intelligence, API performance and hardware performance measure different things? Artificial Analysis benchmarks all three: model capability, endpoint speed and cost, and the accelerators serving inference. Its coverage extends beyond language to vision, speech, image generation and video generation. At the time of George Cameron’s presentation, the company covered more than 150 models, published most results freely, and offered reports and enterprise subscriptions to support production adoption. Those separate measurements provide the basis for choosing a model that fits an application.

The launch of ChatGPT and GPT-3.5 began the historical arc: more releases followed, from more labs, with increasingly crowded competition at the frontier. In the presentation’s ranking, o3 led, followed closely by o4-mini with high reasoning. Other leading models included DeepSeek-R1, Grok 3 Mini with high reasoning, Gemini 2.5 Pro, and Claude 4 Opus Thinking.

The ranking used the Artificial Analysis Intelligence Index, a weighted composite of seven evaluations intended to provide a generalist view of capability. These are historical measurements using that seven-evaluation index and the configurations and prices reported in the talk; the current methodology and live rankings have since changed.

The most intelligent model is not always the right model. Accessing intelligence entails trade-offs, so a useful comparison must examine several frontiers. The progression here is reasoning versus non-reasoning models, open weights versus proprietary models, cost, and speed. Each changes the set of applications that is practical to build.

Plot Intelligence Index scores against the output tokens required to complete the evaluations, and two groups emerge. Non-reasoning models use fewer output tokens but achieve lower measured intelligence in this comparison. Reasoning models achieve higher scores while generating substantially more tokens. The additional generation matters twice: it adds expense and extends the time required to finish a request.

The reported evaluation runs show the scale of that difference. GPT-4.1 required millions of output tokens; the reasoning models required tens or hundreds of millions.

These are totals across the seven evaluations, not the length of an individual response. Cameron describes an order-of-magnitude difference between reasoning and non-reasoning models—large enough that token volume must be treated as an application constraint alongside capability.

Across Artificial Analysis’s tested API requests, GPT-4.1 had a median full-response latency of 4.7 seconds; o4-mini with high reasoning took over 40 seconds in the reported comparison. The slide uses a 500-output-token workload, including reasoning time. That is roughly an order of magnitude more waiting. Full-response latency measures the wait for the complete answer, rather than time to first token or the rate at which tokens arrive. The slide separates input processing, thinking and output generation, making visible why a fast output stream alone does not describe the entire interaction.

That wait affects enterprise chatbots as well as consumer applications. A user does not necessarily need the strongest reasoning model for every interaction, particularly when responsiveness is part of the product’s value. Cameron invokes Facebook studies of latency-related user drop-off as supporting motivation, without giving a specific study or numerical result. The actionable question is whether the additional capability justifies the delay for the interaction being built.

For an agent making successive requests, latency accumulates along the chain. Cameron uses 30 sequential queries as a plausible workload. Faster responses can change the workflow itself: instead of merely completing the same 30 calls sooner, an application might fit something closer to 100 calls into its waiting budget.

In the illustrative agent calculation, 30 sequential calls at 10 seconds each take 300 seconds, or five minutes; at one second each, they take 30 seconds. This is arithmetic for a sequential workload, not a measured agent benchmark. A small TypeScript calculation makes the dependency explicit:

const calls = 30;

const scenarios = [
  { secondsPerCall: 10 },
  { secondsPerCall: 1 },
];

console.table(
  scenarios.map(({ secondsPerCall }) => {
    const totalSeconds = calls * secondsPerCall;
    return {
      calls,
      secondsPerCall,
      totalSeconds,
      totalMinutes: totalSeconds / 60,
    };
  }),
);

The calculation isolates model-call time. An actual workflow must also account for whatever happens between calls, but the sequential dependency alone already creates a substantial difference.

A contact-center application makes the product constraint concrete: a 30-second wait might be acceptable, while five minutes on the phone could make the interaction unusable. Likewise, imagine waiting five minutes every time you invoked a function in Google. The latency budget determines which workflows are viable, so measure the trade-off in the application rather than assuming a higher intelligence score will compensate for the wait.

The open-weights frontier has moved unevenly. Around the release of GPT-4, Llama 65B and Llama 2 70B were substantially behind proprietary capability. Later releases such as Mixtral 8x7B and Llama 405B narrowed the gap. Then o1 pulled ahead again in late 2024, reopening the separation.

DeepSeek-V3 arrived in December, prompting Cameron to interrupt his Christmas holiday to read its technical report. R1 followed in January. By the presentation, Cameron reported that the recent R1 release was only a couple of Intelligence Index points behind the leading models. The progression matters: open weights were catching up not only to an older proprietary generation, but also to the newer reasoning frontier.

China-based labs led both the reasoning and non-reasoning open-weights rankings shown in the talk, with DeepSeek first in each. Alibaba’s Qwen 3 series followed in reasoning. Meta and NVIDIA’s Nemotron fine-tunes of Llama were also competitive. The resulting choice was broader than a single leading proprietary API: open-weights models had become close contenders on the measured capability axis.

Cost changes what can be built just as latency does. At the prices used for the presentation’s evaluations, o3 cost almost $2,000 to complete the Intelligence Index. The cost of running evaluations had become substantial enough to attract TechCrunch coverage. Cameron then compared GPT-4.1 and GPT-4.1 Nano with more expensive reasoning models:

The denominators differ: GPT-4.1 is compared with o1, while Nano is compared with o3. These ratios describe the evaluation-suite workload.

An application’s cost structure may therefore determine both which model it can use and how many times it can call that model. Cameron pushes the idea toward agents, suggesting that hundreds of inexpensive calls could cost less than one expensive call. That is a design possibility to investigate, not a universal per-query ratio established by the suite totals; actual prompts, responses and reasoning lengths determine the bill.

Price per token is only half of the cost calculation. The other half is how many tokens the model consumes and produces. In the billing behavior discussed here, reasoning tokens count as output tokens even when the provider does not expose their contents. A model can therefore have an attractive advertised price per million tokens and still be expensive to use because it generates much more internal reasoning or a longer answer.

Verbosity also varies among non-reasoning models. Compare the cost of completing the same application workload, including billed reasoning tokens, instead of assuming that every model will generate the same amount of text. The evaluation-suite comparison is useful precisely because it measures token quantity and token price together.

To study cost over time, group models into intelligence bands. This asks what it costs to access a comparable level of capability, rather than what the newest frontier model costs. Cameron reports that the cost of accessing GPT-4-level intelligence fell by more than 100 times from mid-2023 to the presentation. Declining costs appeared across the plotted quality bands.

Even a newly reached band could become cheaper quickly. Cameron reports that the cost of accessing the intelligence band reached by o1-mini in late 2024 halved within a few months. That creates a useful planning exercise: what would you build if the current inference cost were no longer a barrier? An application whose economics fail today might become feasible six months later. That possibility is a reason to examine future cost scenarios, not a guarantee that a particular project’s economics will work on that schedule.

The speed frontier measures output tokens per second. Because capability and speed often trade off, intelligence bands again provide the useful comparison: how quickly can an application receive output at a given capability level? The plotted bands all show rising output speed.

Cameron recalls GPT-4 generating about 40 output tokens per second in 2023 and reports access to comparable intelligence at over 300 tokens per second by the presentation. The practical memory is familiar: waiting for ChatGPT to finish emitting code before copying it into an editor and trying it. Faster generation shortens that wait even when the model is not performing an extended reasoning step.

Several changes contribute to faster inference:

• Model sparsity: Mixture-of-experts models activate only a fraction of their parameters during inference. Less computation per token can mean faster generation, and increasingly sparse models extend that advantage.
• Smaller, more capable models: Distillation helps smaller models, including the cited 8B examples, deliver more intelligence without requiring the same model size.
• Inference software: FlashAttention and speculative decoding are two named optimization techniques.
• Hardware: The progression from A100 to H100 and then B200 systems provides another source of improvement.

Cameron reports that newly published Artificial Analysis B200 benchmarks exceeded 1,000 output tokens per second. The talk does not specify the model or workload for that result, so it is not a controlled hardware comparison with the earlier GPT-4 figure. Specialized accelerators from Cerebras, SambaNova and Groq add further options.

Lower costs at a fixed intelligence level, better hardware and greater system output throughput all improve efficiency. Yet Artificial Analysis’s house view is that aggregate compute demand will continue to rise. Efficiency gains change what is affordable, while the desired scale of model use continues to expand.

Larger models are one driver. Cameron points to DeepSeek having more than 600 billion total parameters, explicitly correcting himself from active to total parameters. That distinction matters for sparse models: the full parameter count and the amount activated during inference are different quantities.

The other multipliers follow directly from the earlier comparisons. Demand for greater intelligence encourages more inference-time reasoning, which consumes additional compute. Agents turn one user task into sequences of 20, 30 or more than 100 model requests. More capable models, longer reasoning and more calls per task can absorb the savings from cheaper, faster inference. The closing expectation is therefore rising net compute demand—a house view about how these forces combine, rather than a quantified forecast.