Trends Across the AI Frontier

[upbeat music] Hi, everyone.

I'm George, co-founder of Artificial Analysis. A quick background to who we are before we dive into things.

Do you see that? We see it here.

Sorry, I think my clicker's not working. Click.

Oh, there we go. Great. So a quick background to who we are. We're a leading independent AI benchmarking company. We benchmark a broad spectrum across AI. So we benchmark models for their intelligence.

We benchmark API endpoints for their speed, their cost. We also benchmark, uh, hardware and all the AI accelerators out there. Uh, and we also benchmark a range of modalities, not just language, but also vision, speech, image generation, video generation.

And we publish essentially nearly all of it, uh, for free on our website, artificialanalysis.ai, whereby we benchmark over a hundred and fifty different models, uh, across a range of metrics.

We also publish reports, many of which are publicly accessible, and we also have, uh, a subscription for enterprises looking to, uh, enter, uh, or bring AI to production in their environments, um, in an efficient, uh, and effective way.

Let's start off with AI progress. Let's set the scene. So it's been a crazy two years. I think that we've all felt it in this room, whereby OpenAI, uh, kicked off the race, uh, with the ChatGPT and GPT-3.5 launch, and since then, it's only gotten more hectic.

There's been f-- more and more, uh, model releases by more and more labs pushing the AI frontier.

So the current state now of frontier AI intelligence. I think this will be-- this order of models will be familiar to a lot in this room. o3 is the leader, but followed closely by o4-mini with reasoning mode high.

DeepSeek-R1, the release in the last week or two. Grok 3 Mini, reasoning high. Gemini 2.5 Pro. Claude 4 Opus Thinking.

This benchmark is our Artificial Analysis Intelligence Index. It's made up of a composite-- it's a composite index of seven evaluations, which we then weight to develop our Artificial Analysis Intelligence Index, which just provides a generalist perspective on the intelligence of these models.

We all have an understanding of what f-frontier AI intelligence is, but what I want to explore with you today is that there's more than one frontier in AI. There's trade-offs to accessing this intelligence.

You shouldn't always use the leading, most intelligent model. And so what we wanna do is we wanna explore the different frontiers out there, and as an AI benchmarking company, we're gonna bring some numbers to the fore to help you reason about this.

First, we'll be looking at reasoning models. Next, we'll be looking at the open weights frontier. Third, the cost frontier. And lastly, the speed frontier. There's other frontiers out there that we benchmark, but we'll focus on these key ones today.

Starting with reasoning models. What we've done here is we've taken our intelligence index and looked at that relative to the output tokens used to run the intelligence index. So we've measured all of how many tokens each model took to run our seven evaluations, and we've plotted it on this chart.

And you can see two distinct groups. It's helpful to think about these separately. So non-reasoning models, which offer less intelligence but, uh, require fewer output tokens.

And reasoning models, which use more output tokens but offer greater intelligence.

And the more out-- this is important to look at because more output tokens comes with trade-offs, both for request latency as well as cost. We're gonna bring some numbers to draw that out and look at the real differences here.

Starting with output tokens and the verbosity of these models, just how yappy these reasoning models are. We can see that there's an order of magnitude difference between reasoning and non-reasoning models.

It's not just that feeling, "Oh, this is taking a long time." It's real. It's an order of magnitude. So between GPT-4.1, it c-- uh, it required seven million tokens to run our intelligence index evaluations.

But then o4-mini-high took seventy-two million tokens. And the yappiest of them all, Gemini 2.5, uh, Pro, took a hundred and thirty million tokens to run our intelligence index.

And as mentioned, this has implications for cost as well as end-to-end latency responsiveness.

So looking at latency, we benchmark the API latency of how long it takes to receive a response when accessing these models via their APIs. Here we can see that GPT-4.1 on median across our request took four point seven seconds to return a full response.

o4-mini-high took over forty seconds, roughly another ten x or order of magnitude increase. This has implications for applications and uses which require responsiveness, even enterprise, uh, kind of chatbots.

You don't always reach for o3 in ChatGPT.

And it-- And, you know, Facebook's done a lot of studies on this where they've looked at the-- for consumer apps, where they've looked at, uh, user drop-off by latent-- uh, application latency, which clearly demonstrate this.

Sorry, do you mind if we jump back a slide?

And it also has up, uh-- implications for how we're building. So I think particularly with agents whereby thirty, uh, queries in succession is not uncommon. It has m- it's a multiplier effect on the latencies, uh, for your application and how you can build.

If you have faster responses, maybe you can make that thirty, uh, one hundred queries, for instance. And so putting numbers to that, in terms of agents, thirty is normal.

And so even less than the o4-mini, maybe you're at ten seconds for a reasoning model. If you're running thirty queries, that's three hundred seconds that a user might be waiting for a response or an application might be waiting for a response.

That's five minutes. If with the order of magnitudes that we're dealing with here, if that ten seconds was one second, then those thirty queries takes thirty seconds. Thirty seconds versus five minutes impacts what you can build.

Think of a contact center, uh, application. That might-- maybe thirty seconds is okay there, but five minutes, uh, definitely not. Who likes waiting on the phone, uh, that long?

Or imagine if you had to, uh, use Google, and each time that you wanted to use a function, it took five minutes. This impacts how we can build with these models.

And so I think bringing numbers to these trade-offs is really important. I'd encourage everybody, uh, to measure them.

Next, we're gonna move to the open weights frontier.

Around the time of GPT-4, there was a huge delta in terms of open weights intelligence versus proprietary intelligence.

Llama 65B or Llama 2 70B wasn't close to the intelligence of GPT-4. What I'd like to show here is where we plot our intelligence index by release date is that that gap had closed until with, with great models like Mixtral 8x7 and, uh, Llama 45B.

But o1 broke away in late twenty twenty-four.

But then, of course, I think we remember DeepSeek released V3, I think December twenty-sixth. Ruined some of my Christmas holiday plans. [laughs]

Had to tell my family, "I, I need to go read this paper. It's really exciting."

And then, of course, R1 in January. The gap between open-weights intelligence and proprietary model intelligence is less than it's ever been, particularly with the recent R1 release in the last couple of weeks, which is only a couple of points different in our intelligence index to the leading models.

You can't talk about open weights intelligence without talking about China. The leading open weights models across both reasoning models and non-reasoning models are from China-based AI labs. DeepSeek's leading in both.

Alibaba with their Qwen three series is leading-- is coming in second in reasoning.

But you also have other labs such as Meta, uh, and NVIDIA with their Nemotron fine-tunes of Llama coming in close as well.

Let's look at the cost frontier. This is really important, and I think similar to reas-- to, uh, end-to-end latency impacts what you can build. So bringing some numbers here, we can really see these order of magnitudes play out.

So o3 cost us almost two thousand dollars to run our intelligence index. TechRun actually wrote an article about how much money we were, we were spending on running evals.

We di- we didn't wanna read it. You can see four point one, a great model. It's thirty times roughly cheaper in terms of, uh, the cost to run our intelligence index compared to o1.

And four point one Nano, over five hundred times cheaper to run our intelligence index than o3.

You should think about these when building applications. The kinda co- uh, cost structure of your application might dictate what you can use here

and how you use them. Those thirty, uh, sequential, uh, API calls for your agentic application could be, uh, five hundred and still be cheaper than an o3 query.

A key point to note here with this cost to run intelligence index and why we don't just look at the per token price is that, and the labs maybe don't want you to think this way, [laughs] but you're paying for the cost per token, but then you're also paying for how verbose the models are.

All the reasoning tokens that are output when these models are in their thinking mode.

You pay for those as output tokens, even if some of the labs hide them. And so you need to think about this and measure it in your application, not-- and benchmark not just by the cost per million tokens, but also considering how many reasoning tokens there are and how verbose these models are.

You can see even amongst the non-reasoning models, there's big differences between how verbose these models are in responses.

So for instance-- Ah, we'll go to the next slide.

Do you mind if we go back one, please?

So what we've done here is we have-- now-- we're now gonna look at the trends in terms of cost. And so what you can see here is we've bucketed models by how intelligent they are, intelligence, uh, bands, if you will.

And what we can see here is that accessing GPT-4 level of intelligence has fallen over a hundred times since mid '23.

This is the case across all quality bands.

You can see that even when a new quality band, a new frontier is reached, o1-mini in late '24.

Quickly, within only a few months, the cost of accessing that level of intelligence halved. This is moving quickly. And so what I would say to you is when building applications, think about what if cost wasn't a barrier when you're building.

It's a, it's a very important kinda cost, uh, exercise because it might well be that if you build for a cost structure that doesn't work now, then maybe in six months' time that will be, uh, possible and it will be, uh, feasible.

Next, we're gonna look at the speed frontier. So this is how quickly you're receiving tokens, the output speed, output tokens per second that you're receiving after sending an A- API request.

This has been increasing and has increased dramatically since early '23 as well. So similarly, we've-- because there's a trade-off typically between intelligence and speed, we've grouped models into certain buckets.

And we can see here that they've all increased in terms of how quickly you can access a level of intelligence.

So 4o, I believe, was around forty output tokens per second. Now you can access-- That was in 2023. Who remembers hitting-- It wasn't a reasoning model, hitting Enter in ChatGPT and just waiting for it to output, especially code, which you wanna just copy straight into your editor and, you know, hit Run [laughs] See if it works.

Now you can access that level of intelligence at over three hundred tokens per second. A few drivers here that I'll go through. It's not the focus of the talk, but important to, to reference.

Model sparsity, so we're seeing more MoEs, Mixture of Experts models, and they activate only a proportion of, uh, parameters at inference time. Less compute per token, which means it can go faster, essentially.

And MoEs were around back then, but they're getting more and more sparse. A smaller proportion of active parameters. Next, smaller models. Smaller models are getting more intelligence, uh, intelligent, particularly with distillations, you know, 8B distillations, et cetera.

Inference software optimizations like FlashAttention and speculative decoding.

And lastly, hardware improvements. So H100 was faster than A100. Now that we've recently launched benchmarks of the B200 on our Artificial Analysis website, and it's getting over a thousand output tokens a second.

Think about that relative to the forty output tokens per second of GPT, uh, four in '23.

There's also specialized, uh, accelerators like Cerebra, SambaNova, Groq.

I wanna share a house view here to frame things.

Yes, things are getting more efficient. Yes, the cost of accessing the same level of intelligence is decreasing, and hardware is getting better. We're getting more system output throughput

on our-- on the chips. But our view is that demand for compute is gonna continue to increase.

We're gonna see larger models. I mean, DeepSeek, it's over six hundred billion active, uh, sorry, not active, total parameters.

And the demand for more intelligence is insatiable. Reasoning models, as we saw, the Yappi models, they require more compute at inference time. And lastly, agents, whereby twenty, thirty, a hundred plus,

uh, sequential requests to models is not uncommon. These act as multipliers on the demand for compute. And so the house view, playing with these numbers is net net. We're going to continue to see commute-- compute demand increase.

Thanks, everyone. [outro music]