AI Engineer World's Fair 2025
Measuring AGI: Interactive Reasoning Benchmarks
Read the talk
Measuring AGI Through Games No One Has Seen Before
ARC-AGI-3 proposes a harder test than winning a familiar game: discover unfamiliar rules, learn through interaction, and compare that learning efficiency with humans.
From a talk by Greg Kamradt
What does beating Pokémon prove?
Watching Claude play Pokémon makes autonomous agents tangible: a small character moves through a familiar world while a model decides what to do next. OpenAI joined the game-playing demonstrations, and Kamradt points to a reported Gemini Pokémon completion. If an agent can finish the game, how much evidence of general intelligence does that provide?
The failures matter as much as the finish. Kamradt reports that Claude remained stuck in one place for three days, needed human interventions, and hallucinated actions. The accompanying post identifies the location as Cerulean City.
There is also the question of prior knowledge. Kamradt argues that Pokémon material was already present in the model’s training data. Exploring a game is impressive, but a familiar game cannot cleanly separate learning during play from recalling information acquired beforehand. The benchmark needs to test how an agent acquires a new skill, not merely whether it can display an existing one.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Humans as the reference point
Greg Kamradt leads ARC Prize, a nonprofit founded by François Chollet and Mike Knoop to guide progress toward open AGI. Its work includes participating in OpenAI’s December livestream announcing o3 preview results on ARC-AGI. Its benchmark design starts from a specific reference point: humans are the only demonstrated example of general intelligence available to us.
That reference gives benchmark builders a practical search strategy: find problems people can solve but AI cannot. The remaining problems define a measurable capability gap. They also direct research toward particular failures, rather than making the benchmark merely a scoreboard. A useful task reveals something that a human can learn and a machine still struggles to acquire.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Intelligence as skill acquisition efficiency
Kamradt introduces unfamiliar, unprepared-for tasks through a definition he attributes to John McCarthy. McCarthy’s own definition concerns the science and engineering of making intelligent machines; the emphasis on unseen tasks is best understood here as Kamradt’s framing. For evaluation, the distinction is consequential: retrieving a familiar solution does not establish that the system learned a new skill during the test.
Chollet’s formulation is more compact: “skill acquisition efficiency.” It asks both whether a system can learn something new and how efficiently it does so. His 2019 paper, On the Measure of Intelligence, connects that definition to a concrete benchmark. Human learning efficiency supplies the motivating comparison; ARC supplies tasks on which learning and subsequent application can be observed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn a transformation, then apply it
An ARC-AGI-1 task separates learning from demonstration using colored grids:
- Inspect example input grids and their corresponding output grids.
- Infer the transformation that explains those pairs.
- Apply that transformation to a new input, constructing the answer in a grid editor.
A correct answer demonstrates some generalization: the solver has carried a rule from the examples to a new case. The examples are the task’s training portion, not an instruction to retrain a model’s weights.
Kamradt describes ARC-AGI-2 as containing over 1,000 tasks. He emphasizes that each task targets a novel skill rather than asking the solver to repeat another task’s transformation. The intended unit of measurement is therefore a succession of small learning problems, not repeated execution of one mastered procedure.
Kamradt reports testing over 400 people in person in San Diego to establish human solvability. The team rented computers and conducted testing in person to protect task data. The important claim is task-level coverage: every included task could be solved by people, not that every participant solved every task. The ARC-AGI-2 announcement provides the corresponding human-validation account.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a single-turn benchmark leaves out
The grid format still leaves substantial abilities untested. All necessary information is supplied at test time. The solver does not need to explore an environment to discover what information matters. An interactive test adds that missing responsibility: choose actions, inspect their consequences, discover goals, and interpret rewards.
This direction connects to David Silver and Richard Sutton’s Welcome to the Era of Experience. Agents collect observations through engagement with the world, use those observations to construct world models, and infer rules for themselves. Learning exclusively from human-produced data imposes a ceiling that experience is intended to move beyond.
An interactive reasoning benchmark puts that process inside a controlled environment with defined rules and potentially sparse rewards. The evaluator knows the environment’s structure; the agent must discover enough of it to complete an objective. Sparse feedback makes exploration consequential because an action may yield useful information without immediately producing a reward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Games that are unfamiliar to the developer, too
Games offer complex rules within a bounded scope, with enough design flexibility to create many different environments for both humans and artificial agents. But using games does not automatically produce a clean test of generalization. Kamradt identifies several weaknesses in earlier Atari evaluations:
- Dense rewards: frequent feedback can guide behavior without requiring the same discovery process as a sparse-reward environment.
- Inconsistent reporting: different performance scales make systems difficult to compare.
- No hidden test set: familiar evaluation games permit development against the test itself.
- Developer knowledge: a developer who knows the game can encode useful strategies into the system.
The last issue changes what a score means. Some apparent agent intelligence may have been supplied by the person building it.
Chess and Go establish that AI can master individual games. Systems such as Agent57 and MuZero extend the discussion to suites of known games. Kamradt loosely describes this as beating 50 known games with unlimited compute and training data. The historical Atari suite contains 57 games, and that shorthand should not be read as a literal resource condition or as identical results from the two systems. The relevant distinction is that the games were known during development.
The proposed stronger test is to beat 100 games that neither the system nor its developer has seen before. Success would support a different capability claim: the system can acquire useful skills across unfamiliar environments without game-specific preparation by its creator. This is the ambition behind ARC-AGI-3, presented in the recording as a preview of ARC’s first interactive reasoning benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Private scoring and discovery through play
The preview proposes approximately 40 novel public training games so developers and agents can learn the interface. Kamradt also announces a public evaluation set, while assigning all performance reporting to a private evaluation set; he does not specify the public evaluation set’s separate role. Private evaluation would run without internet access, intended to prevent game data from escaping. The scored games must remain unfamiliar to both the model and its developer.
Novelty also applies to the instructions. A typical game benchmark might tell an agent that it is driving a racing car or playing a first-person shooter, then explain the controls. ARC-AGI-3 instead proposes dropping humans and AI into an environment without that framing. Understanding must emerge through exploration.
The example is Locksmith. Its initial screenshot does not explain the objective. As a player explores, they may discover things to pick up, walls, and possible goals. Those are hypotheses to establish through interaction, not instructions handed over at the start. Humans and AI face the same requirement: construct an account of how the world works, then use it to make progress. Kamradt presents this as a gap in the reasoning models available at the time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Testing the floor of reasoning ability
The games are intended to require only core knowledge priors. Kamradt describes the tasks as lacking language, text, symbols, and trivia. The practical distinction is the absence of required external linguistic or specialist knowledge, rather than the exclusion of symbolic reasoning: visual patterns can themselves carry meaning. Instead of commissioning ever harder expert problems, ARC aims to test the reliability of basic capabilities, even when AI is already superhuman in particular domains.
Kamradt describes four priors as innate or acquired very early in human development:
| Prior | Intended foundation |
|---|---|
| Basic math | Small quantities, such as counting up to 10 |
| Basic geometry | Shapes and topology |
| Agentness | Recognizing other interacting agents; theory of mind |
| Objectness | Recognizing objects as units |
These constrain what a task may assume before it begins testing abstraction and reasoning.
Practice adds another dimension. Kamradt invokes a passage matching Dwarkesh Patel’s Why I don’t think AGI is right around the corner: humans accumulate context, examine their failures, and find small improvements as they repeat a task. An interactive environment can expose that process over a sequence of decisions. ARC-AGI-3 is intended to provide the benchmark setting Kamradt sees as missing for those behaviors.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the learning, not just the win
Finishing a game supplies an outcome, but skill acquisition efficiency also requires accounting for how the solver got there. The proposed evaluation would test hundreds of humans on the same games, recording completion duration and action count, then evaluate AI against that baseline. The comparison encompasses exploration, inference about the environment, creation of intermediate goals, and completion of the objective.
Kamradt sets a demanding interpretation: if AI cannot outperform humans in this process, he would not call it AGI. More broadly, he treats the continued existence of human-solvable problems that machines cannot solve as evidence against having reached AGI. That is the project’s proposed criterion, with skill acquisition efficiency as its main output metric.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building and pressure-testing the benchmark
At the time of the preview, the plan was to release five sandbox games the following month in San Francisco, alongside a small agent competition with prize money. The purpose was to put the environments in front of strong agents early and learn from what happened. Kamradt’s larger release target was approximately 120 games by Q1 2026. These were forecasts in the recording. Each game would contain multiple levels, so the number of games alone understates both the available interactions and the operational work of creating them.
That work requires funding and attempts to break the evaluation. Kamradt invites direct donations, described as tax-deductible, and introductions to philanthropic donors, including individuals and LPs. He also seeks adversarial testers and participants in online or offline agent competitions to pressure-test ARC-AGI-3.
The implementation had already become simpler. The team initially tried Unity, then found it excessive for these small grid games and began building a lightweight Python engine. That choice also shaped recruitment: Python-capable game developers and game designers could help construct the environments.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How long will it last, and what counts as efficient?
Asked how long the new benchmark might take to saturate, Kamradt offers a tentative horizon of years rather than decades. He gives no exact deadline. The benchmark is intended to expose a present capability gap, not to remain permanently beyond AI.
An audience question about wattage and time makes the efficiency requirement more precise. Kamradt would ideally measure intelligence against two inputs:
- Energy: the human brain provides a physical reference for the resources consumed by a generally intelligent system.
- Training data: the amount of prior experience required matters when judging learning efficiency.
Both are difficult to obtain for closed models. Cost is a practical proxy, and interactive evaluation adds observable action counts and completion duration.
The games themselves, however, are turn-based and have no in-game wall clock. That distinction separates actions that change the environment from time spent deciding what to do. The preview leaves the exact duration measure unspecified; its clearest operational measure is how many interactions a solver needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognizing objects—and the boundary of the test
A second audience question asks what objectness means. Kamradt makes it concrete: a pixel surrounded by other pixels that all move together can be understood as part of one body. The useful representation groups their behavior instead of treating every pixel as independent.
A small Python example illustrates the shared-motion cue. Three labeled pixels retain their identities while each moves one column to the right. Their common displacement is evidence for treating them as one moving object:
python
before = {"a": (1, 1), "b": (1, 2), "c": (2, 1)}
after = {"a": (1, 2), "b": (1, 3), "c": (2, 2)}
def shared_displacement(before, after):
if not before or before.keys() != after.keys():
return None
shifts = {
(after[key][0] - row, after[key][1] - column)
for key, (row, column) in before.items()
}
return next(iter(shifts)) if len(shifts) == 1 else None
shift = shared_displacement(before, after)
assert shift == (0, 1)
This captures one cue for grouping, not a complete object-perception system.
Kamradt’s everyday analogy is a tree: its parts are understood as belonging to the same tree. The prior is a basic way of organizing perception into entities on which reasoning can operate.
The final question asks about tasks that neither humans nor AGI can perform. Those fall outside ARC’s stated aim. Its reference remains the set of tasks humans can solve, and the gap where machines cannot yet match them. Kamradt places human-impossible tasks in a separate discussion of superintelligence, leaving this benchmark focused on learning abilities for which human performance supplies a concrete standard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
François Chollet's framework for measuring intelligence through efficient skill acquisition and the original proposal for ARC.
David Silver and Richard Sutton argue for agents that learn through interaction and experience beyond human-generated training data.
Dwarkesh Patel's essay on continual learning and the practical importance of improving through experience.
DeepMind explains Agent57's performance across all 57 Atari benchmark games and why difficult games matter when assessing generality.
The original MuZero paper describes learning a model for planning across Atari, Go, chess, and shogi.
Further reading
Explains ARC-AGI-2's task design, human testing, two-attempt evaluation, and launch-era model results.
Updates since the talk
The later benchmark report explains environment construction, human calibration, dataset splits, and Relative Human Action Efficiency scoring.
Read the complete timestamped transcript
- 0:00
[on hold music] Today, we are gonna talk about why AI benchmarking is about to get a lot more fun.
- 0:21
But before we do, we need to go over some cool demos here. So I love the Claude Plays Pokémon demo. There's something s- about-- really special about seeing this little embodied agent make its own decisions, go and play Pokémon, uh, for us from our childhood here.
- 0:35
Now, OpenAI got in on the same game. I thought this was awesome. And then just the other day, Gemini beat Pokémon. I like seeing the little robots with AGI on top of their head.
- 0:46
That must mean that we're already there, right? Well, if we have these agents playing Pokémon, and it's already doing it and beating it, that means the game's over, right?
- 0:52
We're all done. Well, not quite because with Claude Plays Pokémon, we saw that it would get stuck in the same place for three days. It would need interventions. It would hallucinate different actions.
- 1:04
And not only that, there was a ton of Pokémon training data that was within the model itself. So although this is a really cool example about an agent exploring a world, there are a lot of things that we can go and improve on.
- 1:15
So as Kyle was saying, my name is Greg Kamradt, president un-- president of ARC Prize. We are a nonprofit with the mission to be a North Star guide towards open AGI.
- 1:25
We were founded last year by Mi-- uh, François Chollet and Mike Knoop. And just last December, we were invited by OpenAI to join them on their live stream to co-announce the o3 preview results on ARC-AGI.
- 1:37
Now, there's a lot of AI benchmarking companies out there, but we take a very opinionated approach as to how we should do this. And our opinion is that the best target we should be aiming for is actually humans.
- 1:48
And the reason why we think that is 'cause we see that humans are the one proof point of general intelligence that we know about. And if we use humans as the target, that does two things for us because what we do is we come up with problems that are feasible for humans, but hard for AI.
- 2:04
Now, while we do that, that does two things. Number one is it creates a gap, and when you have that gap, you can start to measure, well, how many problems can we come up with that humans can still do but AI can't?
- 2:15
And then number two is it guides research. So you can quantify that class of problems and then go tell researchers, "Hey, there's something really interesting going on on this side of the problem.
- 2:25
Uh, there's something that we need to go, uh, check out from there." All right? So if we're going to measure artificial general intelligence based off of humans, we need to actually define, well, what is general intelligence?
- 2:38
And there's two definitions that I love to, uh, quote. The first one was by John McCarthy, and he says that AI is the science and engineering of, uh, making machines do tasks, and this is the important part, that they have never seen beforehand and they have not prepared for beforehand.
- 2:55
This is very important because if you've seen a class of problem beforehand, if it's already in your training data, then you're simply just repeating memorization. You're not actually learning anything new on the fly, right?
- 3:06
The second, uh, person I like to quote on this is actually François himself, and he put it very eloquently within just three words, and he calls intelligence skill acquisition efficiency.
- 3:18
And this is, this is really beautiful here because skill acquisition, can you learn new things? And not only that, but how efficiently can you learn those new things? And humans are extremely...
- 3:28
Uh, spoiler, humans are extremely efficient at learning these new things. So François proposed this definition in his two thousand nineteen paper on the measure of intelligence, but he went further than that.
- 3:37
He didn't just define it. He actually proposed a benchmark to see can a human or s- or, uh, an AI, can it learn something new and then go repeat what it learned?
- 3:47
And this is where the ARC-AGI version one benchmark came out. So over here on the left-hand side, this is the learn the skill portion. This is what we call the training portion.
- 3:54
And what we show you is a transformation from an input to an output grid, and then the goal for the human or the AI is to look at it and say, "Hmm, what's going on here?"
- 4:03
And then on the left, we actually ask you to demonstrate that skill. So it's a little mini skill you learn on the left, and we ask you to demonstrate it on the right.
- 4:09
And if you can successfully do it, and this is what it looks like, it's just a grid editor here, then yes, you've learned what the transformation is, and you've actually applied this.
- 4:17
And so you're showing a non-zero level of generalization as you go through this. So our benchmarks, ARC-AGI 2, this is the most recent one, it has over a thousand tasks in it.
- 4:27
And the important part here is each one of these tasks is novel and unique. And what I mean by that is the skills required for one of them, we will never ask you to apply that same skill to another task.
- 4:39
Um, this is very important because we're not testing whether or not you can just repeat the skill you've already learned, but we wanna test all the little mini skills that you can do over time and so you can-- see if you can actually demonstrate those.
- 4:50
And if we're gonna back up that humans can actually do this, well, we need to go get first-party data. So our group as a nonprofit, we went down to, uh, San Diego, and we tested over four hundred people.
- 4:59
So rented a bunch of computers, and we did this in person to prever-- uh, have data privacy. And we made sure that every single task that was included in ARC-AGI was solvable by people.
- 5:11
So this isn't just an aim here. We're actually doing the work to, to, to go and do that. But if we think about it, there's actually quite a bit of human-like intelligence that's out of scope from what we call a single-turn type of benchmark.
- 5:25
With ARC-AGI, you have all the information presented that you need right at test time. You don't need to do any exploring or anything, and it's all through single turn.
- 5:33
So if we're gonna be measuring any human-like intelligence, and I would argue that if you are gonna measure human-like intelligence, it needs to be interactive by design.
- 5:44
And what you need to have is you need to be able to test the ability of an agent, whether that be biological or artificial, to explore an open world, understand what goals it needs to do, and ultimately look at the rewards and go from there.
- 5:58
So this is actually very in line with what Rich Sutton had just published within his paper, Welcome to the Era of Experience, and he argues that if we want agents that will, um, be readily adaptable to the human world, they need to engage with the open world They need to collect observational data, and they need to be
- 6:16
able to take that data to build a world model and make their own rules and really understand what it is, or else you're just gonna have the human ceiling, uh, the human data ceiling going forward from here.
- 6:26
If we're gonna be able to build this, we're gonna need a new type of benchmark that gets out of the single-turn, uh, realm, and this is where interactive reasoning benchmarks are gonna come in.
- 6:36
Now, an interactive reasoning benchmark is gonna be a benchmark where you have a controlled environment, you have defined rules, and you may have sparse rewards where an agent needs to navigate to understand what is going on in order to explore and, uh, complete the objective from here.
- 6:50
Now, there's an open question as to, all right, if our aim is interactive reasoning benchmarks, what is the medium in which we're gonna actually execute these benchmarks in? And it turns out that actually games are quite a v- uh, quite suitable for interactive reasoning benchmarks.
- 7:07
The reason for this is is games, they're a very unique set of, um, intersection of complex rules, defined scope, and you have large flexibility into creating these types of environments that you can then go put different artificial systems in or biological systems with it.
- 7:23
Now, I know what you may be asking here. "Wait, Greg, didn't we already do games? Didn't we do this 10 years ago? We already went through the Atari phase."
- 7:33
Well, yes, we did, but there's actually a huge amount of issues with what was going on during that realm there. Not, um, uh, even just starting with all the dense rewards that come with the Atari games.
- 7:43
There was a ton of irregular reporting, so everybody would report their own performance on these different scales, and it was tough to compare these models with it. There was no hidden test set that came, and then one of my, um, biggest gripes with the Atari phase was that all the developers, they already knew what the Atari games
- 7:57
were, so they were able to inject their own developer intelligence into their models themselves, and then all of a sudden, the intelligence of the performance, that, well, that's getting borrowed from the developer.
- 8:07
That's not actually getting done by the model itself from there. So if we were able to create a benchmark that overcame these shortcomings, well, then we'd be able to make a capabilities, uh, assertion about the model that beat it that we've never been able to make beforehand.
- 8:24
And so to put it another way that's a bit more visual, we know that AI can beat one game. This is proved. We-- AI can beat chess. AI can beat Go.
- 8:30
We've seen this many, many, many times here. And we know that AI can beat 50 games with 50 known games with unlimited compute and unlimited training data. We've seen this happen with Agent57 and MuZero.
- 8:42
But the assertion that we want to make is, well, what if AI beat 100 games that the system has never seen beforehand and the developer has never actually seen beforehand either?
- 8:56
If we were able to successfully p-put a test or put AI to this test, then we could make the capabilities assertion about that AI that we don't, we don't currently have in the market right now.
- 9:08
And I'm excited to say that that's exactly what ARC is gonna go build. So this is gonna be our version three benchmark. Today is a sneak pr-preview about what that's gonna look like, and this is gonna be our first interactive reasoning benchmark that is gonna come from ARC, and I wanna jump into three reasons why it's very
- 9:24
unique here. So the first one is, much like our current benchmark, we're gonna have a public training and a public evaluation set. So the reason why this is important with our public training, call it on the order of about 40 different novel games.
- 9:37
This will be where the developer and the AI can understand the interface and understand kinda what's going on here. But all performance reporting will happen on the private evaluation set, and this is very important because on this private evaluation set, there's no internet access allowed, so no data's getting out about this.
- 9:52
The scores that come out of private evaluation set will have been done by an AI that has never seen these games beforehand, and neither has the developer seen them.
- 10:00
So we can authoritatively say that this AI has generalized to these open domains here. Now, the second important point about what ARC-AGI 3 is gonna have is it's gonna force understanding through exploration.
- 10:13
One of my other gripes with, uh, current game benchmarks out there is you give a lot of instruction to the actual AI itself. "Hey, you're in a racing game," or, "Hey, you're in an FPS.
- 10:22
Go control the mouse and do all these things." We're gonna drop AI and humans i-into this world, and they won't know what's going on until they start exploring. So even as I look at this screenshot, this is actually one of our first games.
- 10:34
We call it Locksmith. We give all of our games a cool little name like that. As I look at this, it was-- I don't know what's going on, right?
- 10:40
But I start to explore, and I start to understand, oh, there's certain things I need to pick up. There may be walls. There may be goals and objectives. I'm not sure what those goals and objectives are right when I first start, but that's the point.
- 10:53
So not only are we gonna ask humans to explore and make up their own rules as to understand how to do the game, but we're gonna require the same thing for AI as well, and that's something that we're not currently seeing from the, from the reasoning models that we have from there.
- 11:06
Now, the third key point is that we're only gonna require core knowledge priors only. This is something that we carry from ARC-AGI 1 and 2 as well, but what this means is you'll notice the ARC tasks, there's no language.
- 11:18
There's no text that's being involved here. There's no symbols, and we're not asking you any trivia. So I call these, um, o-other benchmarks that rely on these, sometimes we try to make the hardest problems possible.
- 11:28
We go hire the best people in the world, and I call them PhD++ problems, right? And that's great, but AI's already superhuman. It's way smarter than me in a lot of different domains.
- 11:38
We take the alternative approach, which is let's look at more of the floor and look at the reliability side. Let's take all the core, um, anything outside of core knowledge and, and strip those away.
- 11:46
So core knowledge priors, the four of them that there are,
- 11:50
are basic math, and these are things that are humans that we're either born with or hardwired to gather right, immediately after birth. So basic math, meaning counting up to 10.
- 11:59
Basic geometry, so understanding different shapes and topology. And then agentness, which is understanding theory of mind, that there's other types of agents out there in the world that I know that they're interacting.
- 12:08
And then the fourth one is objectness. So as we create our benchmark, these are the four principles that we like to, um, go after when we try to test the abstract and reasoning piece.
- 12:18
Now, I was reading the recent, uh, Darkesh essay, and he actually put it really well in one of his paragraphs here. He was talking about one of the reasons why humans are, are great, and he says, "It's their ability to build up context, interogate- interrogate their own failures, and pick up small improvements and efficiencies as they practice
- 12:35
a task." We don't yet have this type of environment that can go and test this from a benchmark perspective for AI, and this is exactly what ARC-AGI is going to go build.
- 12:46
So before we wrap it up here, I want to talk about how we're going to evaluate AI because it's like, okay, cool, you go- they go play the game.
- 12:51
Well, what does it mean? How do you know if it's doing well or it's not? And we're going to bring it back to François's definition. So we're going to bring it back to s- uh, skill acquisition efficiency, and we're going to use humans, which again, is our only proof point of general intelligence.
- 13:05
We're going to use humans as the baseline. So we're going to go and test, um, at hundreds of humans on these exact ARC tasks, and we're going to measure how long does it take them, how many actions does it take them to complete the game, and then we're going to get a human baseline, and we're going to
- 13:18
be able to measure AI in the same exact way. So can the AI explore the environment, intuit about it, create its own goals, and complete the objectives faster than humans?
- 13:27
Well, if it cannot, I would go as far as to assert that we do not yet have AGI. And as long as we can come up with problems that humans can still do but machines cannot, I would again assert that we do not have AGI with it.
- 13:38
So we're going to be looking at skill acquisition efficiency as our main, uh, our main output metric here.
- 13:45
Today, we're giving a sneak peek about what this looks like. This is World's Fair. Actually, next month, uh, in San Francisco, we're going to give a sandbox preview. So we're going to release five games.
- 13:52
Um, we know better than to try to wait till the end. We're going to make contact with reality. We're going to put out these fives. We're actually going to host a mini agent competition too.
- 13:59
So we want to see what is the best possible agent that people can do. We'll put up a little prize money. Um, and then we're going to look forward to launching about a hundred and twenty games, that's the goal, by Q1 of 2026.
- 14:10
Now, that sounds like it's not that many games, and you think it's not that many data points, but the richness of each one of these games goes really, really deep.
- 14:17
There's multiple levels. It goes deep with each one of them, and it's quite the operational challenge to make all of these, and that's a whole another side of the benchmarking process, which I'm happy to talk about later.
- 14:27
If this mission resonates with you, again, ARC Prize, we are a nonprofit. One of the best ways to get involved is through making a direct tax-deductible donation from that.
- 14:35
If anybody in the room knows any philanthropic donors, whether it be LPs or individuals, I'd love to absolutely talk to them. But then also, we're looking for adversarial testers.
- 14:43
We want to pressure test ARC-AGI-3 as best as we can. So if there's anybody who's interested in participating in the agent competition, whether it's offline or offline, uh, online or offline, let me know.
- 14:53
Happy to chat. And then also, kind of cool story, we originally started with Unity to try to make these games, and we quickly found out that Unity was way overkill for what we needed to do if you're just doing two-by-two, sixty-four-by-sixty-four, uh, games here.
- 15:06
So we're actually making a very lightweight Python engine ourselves. So if an- there's any game developers out there, anybody who wants to get involved with this and knows Python well, we're looking for game developers and game designers as well.
- 15:20
That is all we have today. Thank you very much. [audience applauding]
- 15:29
Kyle, do we have-
- 15:31
Yeah. I think we have time in this case for, for a couple of questions. If anyone wants to come up, there's microphones, one, two, three of them. Um, maybe a couple of questions.
- 15:40
Um, I'm going to kick that off-
- 15:41
Sure, yeah, yeah
- 15:41
... if that's all right.
- 15:42
Yes, yes.
- 15:42
Um, question for you. So, um, and I don't know where, um-
- 15:46
I, I can repeat. Okay.
- 15:48
All right. All right. Um, I, uh, I, I, it's, it's very hard to make estimates about timelines famously, but, um, if you had to guess, how long do you think this new version of the benchmark you're making will take before it gets saturated?
- 16:01
Um, well, the wa- [laughs] the way I think about that is, uh, well,
- 16:08
I would say I'm counting in years, I'm not counting decades. We'll put it that way.
- 16:12
Okay.
- 16:12
Yes.
- 16:13
Interesting. All right. Uh, yeah, we'll, we'll take, uh, one at each mic. Looks like you've... it's well distributed, so starting over here.
- 16:19
Sure. Hi. Um, you mentioned efficiency as part of the requirements, and so I'm wondering for the, uh, the benchmarks if you're considering things like wattage or time or other ways of, of using that as one of the criteria.
- 16:33
Yeah. I, I love that question, and I would've put it in if I had more time, but m- I'm very opinionated about efficiency for measuring AI systems. If I could have two denominators for intelligence on the output, number one would be energy because you know how much energy the human br- uh, the human brain takes, and that's
- 16:47
our proof point of general intelligence, so you can take how much calories the human brain takes. So I would love to do energy. But the number two denominator is the amount of training data that you need for it, neither of which are very accessible for closed models in the current day, so we use proxies, and the proxy
- 17:00
is the cost. But then with interactive evals like this, you get another proxy, which is, um, action count and how long does it take you to actually do it.
- 17:07
We're not going to have a, uh, a wall clock within these games. It's going to be turn-based, um, so we won't have a wall clock to do it.
- 17:14
Awesome. All right. Question two and then we'll do three. Please keep them both very short. Yeah. Um, yeah, very quick question. Um, could you define more what do you mean by o- o- objectness?
- 17:23
Um-
- 17:23
Yes. It, it, that one's actually quite simple. It's just understanding that when you look out into the world, that there's, um, a mass of things that may act together.
- 17:32
So the crude way would be you have one pixel, but then it's surrounded by a whole bunch of other pixels, and they all move together. You understand all those pixels as one and it kind of acting as a one body rather than individual.
- 17:42
And, uh, really, evolutionary-wise, that's the same, that's a tree over there. All this is part of the same tree, that, that kind of thing.
- 17:48
Thanks.
- 17:49
Um, final question, I'll keep this one super short. Uh, how do you distinguish between tasks that, um, in the games that you gu- you guys are developing, how do you distinguish between tasks that humans cannot do and, and AGI also cannot do?
- 18:03
Like, what is the North Star there?
- 18:06
It, it's a good question. It, hu- tasks that humans cannot do are a bit out of scope for our thesis on how we want to drive towards AGI. So I would say that's not, that's not really the aim that we're looking, looking for on that.
- 18:16
Um, that's a whole different con- uh, conversation around super intelligence that maybe that's for another time. Thank you. [outro music]