Agentic Evaluations at Scale — For Everybody

How can someone find the evaluations that matter, keep their results current and contribute expertise that existing benchmarks miss? Kaggle approaches these questions as a community platform. Nicholas Kang introduces himself as the product manager for Kaggle Benchmarks and its agentic evaluation work; Michael Aaron works on evaluations and benchmarks as a Kaggle software engineer. Kang describes a community of more than 30 million users and roughly two years of work on generative AI evaluations. Their tools are an effort to make evaluation more accessible, with substantial problems still open.

The first obstacle is discovery. Kang estimates that more than ten benchmarks appear each day, leaving readers to spend hours searching arXiv and reading papers. Even someone whose full-time job involves benchmarks cannot reliably keep up. Publication also does not guarantee maintenance: once authors move to their next research project, the paper's leaderboard can stop tracking the models people actually use. A benchmark needs an ongoing execution and publication process if it is to remain useful after its initial release.

A published score leaves another question unanswered: what configuration produced it? Both model settings and the way a benchmark is orchestrated affect the result. A model-release chart can show a clean comparison without exposing enough of that setup for someone else to reproduce or interpret it.

Kang gives an example from a benchmark Kaggle published with an AI lab. A competing lab objected to the results, reran the benchmark and published substantially better results for its model. In Kang's account, the competing run used compaction supplied through that lab's API; Kaggle had not enabled it for any of the models in its comparison. The benchmark name alone therefore did not identify the experiment. The rerun changed a consequential part of the system being evaluated.

Coverage is a different problem from reproducibility. Kang depicts the world's knowledge as a large circle, with AI researchers and technical professionals occupying much smaller circles inside it. His rough population estimates are 30,000 AI researchers, attributed to a Google AI search, and 30 million software engineers, data scientists and other technical professionals. The point is the mismatch between the people creating evaluations and the people AI is expected to help. Without a benchmark for a capability, developers lack a measured target for improving it. That can deepen the unevenness of systems that perform exceptionally in some domains and poorly in others.

Wastewater treatment makes the gap concrete. Kang describes a user in Turkey with 20 years of experience as a wastewater plant engineer. The engineer recounted fatal incidents in his country involving failures to follow safety protocols, then built a benchmark to ask whether AI could help him do his job and prevent similar incidents. The dataset came from his professional experience.

Kang describes that dataset as original and proprietary, containing knowledge not otherwise available on the web and outside the labs' immediate economic priorities. Opening evaluation authoring to domain experts creates a route for this kind of knowledge to become a test. The contribution is not simply another set of questions: it is a practitioner deciding which failures matter.

Kaggle's response spans four products, each addressing a different part of the evaluation process.

The competitive approach aims to keep producing a useful comparison as models improve, while the authoring tools broaden who can decide what to test.

Hackathons channel concentrated effort toward a defined problem. The organizer supplies enough constraints to keep submissions relevant while leaving room for participants' expertise and creativity. Kang describes open-source results as part of the intended outcome, so the work benefits people beyond the participants and organizers.

The example underway at the time of the talk was a hackathon with Google DeepMind's AGI team. It followed the paper Measuring Progress Toward AGI: A Cognitive Framework and asked participants to create benchmarks for five of its ten cognitive faculties. The invitation rests on the same principle as the wastewater example: contributors can bring knowledge that a lab cannot supply by itself.

Making that invitation practical requires infrastructure and judgment:

• Access to tools: In Kang's example of 1,000 globally distributed participants, people need somewhere to host datasets and access models. Paying separately for five model API keys can exclude contributors with fewer financial resources.
• Understandable results: Write-ups let other people inspect, learn from and build on a submission.
• Expert evaluation: Kang finds that judging innovation and creativity still requires human experts. Getting those experts to agree is itself work the platform must support.

An open call for contributions is only useful if people can afford to participate and their work can be assessed and reused.

Standardized agent exams lower the effort required to evaluate an existing agent. Kang originally called them standardized agent tests, or SATs, before changing the name because of a trademark issue. The workflow is deliberately short:

1. Paste a one-line prompt into the agent.
2. Let the agent take the exam.
3. Receive a score that can be compared on a leaderboard.

At the time of the talk, this was an experimental MVP launched the previous week.

The motivation is the gap between enterprise evaluation and consumer-agent use. Research labs and enterprises have tools such as Braintrust and invest in measuring their systems. Meanwhile, people build OpenClaw agents and send them into real workflows without necessarily testing them first. Kang references another conference discussion reporting 1,100 security advisories filed that morning. He raises safety-focused exams as a possible baseline before an agent receives control of an inbox, an Amazon account or other consequential resources; these exams were a proposed direction, not an established safety guarantee.

Accessibility also constrains exam design. A difficult, long-running exam may be easy to start but never finish. An easy exam may finish reliably while revealing little about the capability the user cares about. The useful middle ground must deliver enough signal within a run people will complete.

Kang reports that more than 500 agents had been evaluated in the first week, with little promotion. After the team posted on Moltbook, they saw agents sharing their results and even an SAE preparation course. That response suggests interest in scores and comparison, while leaving the harder question of exam quality and appropriate difficulty unresolved.

Aaron turns to Game Arena as a response to benchmark saturation. With a fixed test, successive models can approach a ceiling that no longer distinguishes them. Player-versus-player competition instead asks how models perform against one another. A model can remain the strongest for a while, but new opponents provide a continuing basis for comparison.

Game selection determines which capabilities become visible.

Poker also exposes behavioral differences that a general capability score might hide. Aaron observes that Grok frequently goes all in, while other models play more conservatively. Some newer models, he says, perform worse at Poker because they are more risk-averse. A newer generation is therefore not automatically a better player under a particular game's incentives.

Before running a comparison, the team iterates on the game, confirms that models can actually play it and spends substantial time refining prompts for fairness. The Game Arena harness makes this work inspectable: prompts and the surrounding game machinery are part of what determines the observed behavior.

Aaron describes the products as live on Kaggle, with much of the work open source. Once game and prompt design are established, the execution pipeline proceeds through four stages:

1. Build and test the harness. Most of the team's work uses OpenSpiel games. Compare model play with a random baseline and look for other interesting properties. Models usually beat random play, Aaron says, but sometimes by less than expected.
2. Connect the models. An LLM model proxy, which Aaron also describes as available on Colab, provides a consistent way to interact with the different models.
3. Run scheduled simulations. Execution uses Kaggle's existing simulation platform, originally built for reinforcement learning. Bradley–Terry pairwise methods help the team reduce the number of games it needs to run.
4. Publish evidence and results. The model conversations become Kaggle datasets, ratings appear through Benchmarks, and game visualizers let readers inspect play—including the Poker hands behind the observations about aggressive betting.

The published outcome is therefore more than a ranking: conversations and visualized games provide material for examining how a model reached it.

Aaron reports that the Poker evaluation required about 400,000 hands to reach statistical significance, with many model turns inside each hand. The inference cost grows with those turns, not just with the number of matches. He does not specify a confidence threshold or stopping rule, so the count describes that evaluation rather than a general requirement for Poker benchmarking. Bradley–Terry pairing is one way the team tries to reduce the work, but obtaining a credible comparison quickly remains an open engineering problem.

A second constraint is participation. Watching models compete can be entertaining initially and repetitive later. Aaron proposes a prompt hackathon in which community members supply prompts, then compete on a leaderboard according to how well those prompts make a model play. That would give the audience an active role in the evaluation process.

A third constraint is continuity. Old models disappear, new models arrive, and an externally hosted endpoint may not make the identity of its underlying model clear. Longitudinal comparison depends on knowing which system actually played, not merely retaining an endpoint name in a results table.

Kaggle Benchmarks is intended for open community evaluation rather than evaluating a particular application's production code. Its basic authoring workflow will nevertheless look familiar to anyone who writes production evaluations: generate an output, then check whether it satisfies an expectation. The Kaggle Benchmarks Python library supports this task-oriented approach.

Aaron's small example asks what gets wetter as it dries. A check asks whether the answer contains towel. The assertion itself can be expressed directly in Python:

def check_towel_answer(answer: str) -> None:
    assert "towel" in answer.casefold(), "Expected an answer containing 'towel'"

This checks a specific property of the response. The platform also supports LLM judging when the desired judgment is not captured by such a rule. Checks are grouped into a task, which runs against the collection of models selected by the user. Tasks are then aggregated into a benchmark, such as the wastewater treatment benchmark.

The richer demonstration is a task Paige created around recreating an SVG from xkcd. Aaron shows a reproduction generated by Sonnet 4, which he describes as already somewhat outdated. Several assertions inspect the result: did the model generate an SVG at all, does it contain the correct text, and does it satisfy the other checks? Side-by-side comparisons let a reader inspect the artifacts as well as the individual judgments. The task decomposes a broad instruction—recreate this graphic—into properties that can be examined separately.

The remaining difficulty begins with incentives. A team shipping a product has an immediate reason to test whether its model works for customers. A community contributor must first find a useful question, then invest the effort to build an evaluation other people will care about. Hackathons, Kaggle points and medals help encourage that work, but they do not remove the effort required to write a good test.

Moving from models to agents makes the object of evaluation less clear. Aaron cites a March 16 Morph LLM post as reporting that six frontier models were within a couple of percentage points on SWE-bench Pro, while coding performance differed by 22% depending on the harness. He explicitly says he has not verified the post, and the cited 22% is not resolved here as a relative change or a percentage-point gap. The comparison motivates a question rather than establishing a controlled result: how much of an agent's score belongs to the model, and how much belongs to the harness around it?

An agent evaluation needs a clear account of the system under test. Otherwise, an apparent model comparison may actually reflect differences in harness behavior. Rapid releases and deprecations add another difficulty: even a well-defined experiment can become hard to repeat when its models are no longer available. Community participation expands what gets measured, but transparent execution and identifiable model versions determine whether those measurements remain interpretable over time.