[
  {
    "startMs": 12540,
    "text": "Good afternoon. Uh, super excited to be talking about evals today. Um, so today we'll be covering, uh, the difficulty of creating evals for real-world workflows in production, um, how Google DeepMind and Kaggle are collaborating to solve this problem, um, and a case study that we did with, um, our partners at LinqAlpha, uh, so they'll present next, um, about a specific benchmark called MandateBench, which basically looks at, uh, how, how well models can perform as a buy-side equity analyst."
  },
  {
    "startMs": 43960,
    "text": "So to start with some quick intros, um, I'm Janie. I'm a product manager at Google DeepMind. Uh, I lead our Kaggle Benchmarks team. Um, Ivan is our amazing forward deployed engineer at, at GDM. Uh, he can't make it today, unfortunately. Um, Jacob is in the crowd, uh, in-- on-- from our LinqAlpha partners. Uh, Suyeol's presenting with me, and Jack also couldn't make it, unfortunately."
  },
  {
    "startMs": 70180,
    "text": "Um, so just starting with a quick intro, uh, about Kaggle and how we got here. So Kaggle, uh, you guys probably know Kaggle as the home for data science, uh, and ML competitions. I'm sure many of you have, have interacted with Kaggle or competed in a Kaggle competition in, in the past. Um, and so we've been doing this for, for 15 years, uh, basically evaluating humans on different capabilities in, in the real world. Um, and we've built up a community of thirty-one million AI and ML enthusiasts"
  },
  {
    "startMs": 99940,
    "text": "across the platform, thousands of competitions over the last fifteen years, uh, and one leaderboard, uh, that everyone really trusts. And what has really made Kaggle succeed in the last fifteen years is, is our rigor. So we have sealed private test datasets. We, um, have leak detection, anti-cheating infrastructure, and really make sure that, that every single competition that, that is put on Kaggle, uh, is designed really well and good at evaluating humans on, on the capability that it's supposed to be."
  },
  {
    "startMs": 130300,
    "text": "Um, and so that brings us to today. So in February 2026, uh, Kaggle actually moved into Google DeepMind, um, and we're moving our subject from evaluating humans to evaluating frontier models. Uh, and so Kaggle has basically become GDM's public platform for, for benchmarks. And our goal is to make an open, neutral proving ground for every single frontier model, uh, that uses the same infrastructure, the same tasks, and the same grading, um, so that you--"
  },
  {
    "startMs": 160180,
    "text": "every score is representative."
  },
  {
    "startMs": 164200,
    "text": "Um, and our goal is really to become the industry standard for benchmarks the way that Kaggle, uh, already is for competitions."
  },
  {
    "startMs": 172740,
    "text": "So why does this actually matter? Why are we, why are we getting into this problem? Why is Kaggle trying to tackle this problem? Um, and it's because benchmarks are really important. Benchmarks are how we measure progress in the AI industry. And right now, the state of benchmarks is fundamentally broken. So you think about all the different ways that benchmarks are used today. Uh, frontier labs are using them to train and hill climb models. Startups are using them to decide what models that they want to build their product around."
  },
  {
    "startMs": 202540,
    "text": "Investors are using it to judge technical moats. Um, and how you measure, like what-- whether the benchmark is correctly measuring, uh, what you want it to measure is exactly what the models end up hill climbing. Um, and when you measure the wrong thing, the models actually get worse at, at what they're supposed to be doing. So there's a few different ways that benchmarks fail today. Um, the first one is, is relevance. So most public benchmarks don't actually resemble the real work that, that your"
  },
  {
    "startMs": 232560,
    "text": "agents are doing in production today. So most of them test synthetic puzzles, academic trivia, random math problems, um, and that bears little, little resemblance to what your agent actually does in, in production. The second problem with benchmarks today is it's very, very expensive in terms of a compute perspective, uh, which is the, like, most scarce reho-resource in the AI industry today. So take a benchmark, let's say it has five hundred multi-turn agent tasks, and you wanna run it"
  },
  {
    "startMs": 262580,
    "text": "across sixteen frontier models. Um, that costs tens of thousands of dollars per sweep. Um, and every time that a new model launches, which is basically every week or two at this point, um, you have to do the sweep again in order to keep your benchmark current, otherwise it, it becomes stale. And the third problem that we see, uh, which is probably the most underappreciated, is verification. So everyone-- like all the benchmarks in the industry, everyone runs their"
  },
  {
    "startMs": 292580,
    "text": "own harnesses. They report their own scores. They, they build their own, like, self-reported benchmark implementation. And the numbers are wildly different across different implementations. And this is mostly because of infrastructure, not because of actual differences in model quality. So if you have different scaffolding, if you have different ways that you're calling tools, if you input-- if you just serialize the input into the model in different ways, um, you end up with different benchmark scores, even if it's the exact same benchmark on the same model."
  },
  {
    "startMs": 324480,
    "text": "Um, so that last point, uh, around verification, I think, uh, is probably the most abstract. Uh, so wanted to, to deep dive into that a little bit. Um, so we-- uh, in order to really understand the, the state of the benchmark industry and all of the different problems, uh, we at Kaggle spent the last two months reproducing sixty-five different industry benchmarks, um, and recreated all of them, ran all of them, and read every single trajectory trace by trace to, to really"
  },
  {
    "startMs": 354500,
    "text": "understand what the problems were so that we could figure out, like, what the infrastructure we wanted to build was to fix it. Um, and we saw tons of issues across all of these different benchmarks. Um, so I'm listing a few of them here just to give you an idea of all of the different problems. So, uh, for one benchmark, we found, uh, the model that was being evaluated actually found the API keys, uh, in the eval container, uh, for a different model, and then called a different model, and then used that response and submitted"
  },
  {
    "startMs": 384440,
    "text": "it as, as their response to the eval. Um- Second, uh, most people when they, when they build their benchmark, they, they don't actually separate the agent being evaluated from the agent being graded. They share a same-- the same container, and models are really good at, at hacking their container now. So we found, uh, models basically modified the grading script, uh, in order to give them whatever score that they wanted, which was a hundred percent, uh, and, and let themselves pass."
  },
  {
    "startMs": 414732,
    "text": "The third is just pure, like, statistical variance. So we found, uh, for all sixty-five benchmarks, we ran all of them multiple times. Uh, and we found that running the exact same benchmark, even on the same infrastructure, uh, multiple times, produced variance of up to plus or minus twenty percent in the score. Um, and again, this is the same model on the same exact harness on the same exact infra. Um, so this is just reflective of instability in, in the, the harness or the benchmark design itself."
  },
  {
    "startMs": 445932,
    "text": "Um, I think everyone's familiar with the Hugging Face incident. Uh, so we found instances of models, uh, basically trying to look up the answers online, uh, in different forums like Hugging Face or Reddit or Stack Overflow. Um, and then this one we were very surprised by. So each harness, uh, actually ships with its own default reasoning effort. Um, so if you use the default, uh, without setting it, um, then you actually end up comparing models, uh, at different reasoning"
  },
  {
    "startMs": 475952,
    "text": "efforts. So you might have one model that's running on high and one that's running on low if you, uh, if you don't actually configure it, uh, in order to make sure that it's a head-to-head comparison. Um, and a lot of the k-- times when we were recreating it, we found that people did not do this, uh, because our scores actually matched parity with the original benchmark paper. Um, but we found issues with, with the score itself."
  },
  {
    "startMs": 500232,
    "text": "Um, and so given all of these problems in, in the industry, uh, Google DeepMind and Kaggle are, are partnering to solve this. So we're doing a few things here. Uh, the first is we are partnering with companies and researchers, uh, to build, to build benchmarks around real workflows, uh, economically valuable real, real-world workflows. Um, and to make this happen, we are offering free compute and frontier model inference, uh, to anyone who wants to build a benchmark. We're"
  },
  {
    "startMs": 530212,
    "text": "giving out five million dollars in grants in, in Q4 to fund any benchmark creation or research into fixing this space of, of benchmarks, given how e-early a space it is. Um, and third, we will, uh, we will provide you a forward deployed engineer from, from GDM to help you co-design your eval, uh, to make sure the methodology is actually rigorous."
  },
  {
    "startMs": 556332,
    "text": "Second, uh, we t- we talked about this a lot on the last slide, but a lot of these problems are just issues all up and down the infrastructure stack. Um, and so what we're building at Kaggle is standardized infrastructure so that every score is auditable, uh, high quality, and, and r-repeatable. So this includes, um, sandboxing every run, having logged proxy calls and open traces for, for every model call, and containerized grading."
  },
  {
    "startMs": 584412,
    "text": "And then lastly, uh, the reason why we specifically are doing this, uh, is, is for our expertise. Um, so Kaggle has fifteen years of designing rigorous tests and preventing cheating by humans, which we are now applying to, to the model evaluation space. Um, we're bringing GDM researcher expertise to, to help us design all of these, these real-world benchmarks. Um, and lastly, we're actually partnering with independent research orgs, uh, like Epoch, to au-audit all of the benchmarks"
  },
  {
    "startMs": 614452,
    "text": "on our platform, so you don't just have to take our wor-word for it."
  },
  {
    "startMs": 620572,
    "text": "So at this point, I hope I've convinced you, uh, how important benchmarks are. Um, but you're probably wondering, \"Why should I build a benchmark as me, as, like, a company with other priorities and other things to do?\" Um, and there's, uh, two b- two big reasons. Um, the first is it actually allows you to build faster internally. So having a benchmark and having a rigorous evaluation suite, um, allows you to easily tell when, when you model-- um, modify your,"
  },
  {
    "startMs": 650492,
    "text": "your prompt, when you modify your harness, whether you've broken something or how you've improved your product. Um, and then the second is if you then want to publish your benchmark externally, uh, then labs will be encouraged to actually hill climb on your use case. Um, like most of the labs are, are looking for real-world benchmarks. Um, and once the labs want to hill climb on your benchmark, then they'll pour trillions of dollars on-- of compute into improving the base models, uh, for your specific"
  },
  {
    "startMs": 680412,
    "text": "use case so that your product gets better for your users over time."
  },
  {
    "startMs": 686572,
    "text": "Um, and with that, I will pass it over to LinqAlpha, um, to, to talk through a case study of MandateBench, which was one of the first benchmarks that we've built on Kaggle."
  },
  {
    "startMs": 704152,
    "text": "Great. Hi, everyone. Um, this is Suyeol. I'm head of AI at LinqAlpha, and we cre-- um, we created a benchmark called MandateBench co-- uh, it's developed, co- co-developed with, um, Google DeepMind and Kaggle. So before I start, um, let me share briefly about our company. So LinqAlpha builds AI research agent for institutional investors. Institutions with more than five trillion dollars in assets under management uses LinqAlpha, and more than two hundred investors use it actively. There are some of the,"
  },
  {
    "startMs": 734512,
    "text": "uh, firms that trust us I listed up here, and we're g- we are also a research team. We have published thirteen papers at, um, at a venue like ICLR or ICML, um, with-- co-authored with Black-BlackRock, Blackstone, JPMorgan, et cetera."
  },
  {
    "startMs": 752784,
    "text": "So, um, I'd like to first go through the existing financial benchmarks."
  },
  {
    "startMs": 760104,
    "text": "So most of these benchmarks prepare a, a golden answers and compare it with the agent answer. But in real-world tasks, two analysts can reach two different answers, and both could be right. So existing benchmark use relatively short and simple tasks. They don't, um, um, they don't task a full unit of analyst's work. So they only get the fragments of it. So MandateBench aims to do two things. First, this"
  },
  {
    "startMs": 790084,
    "text": "rubric explicitly grades the reasoning path. Second, we take the real world, real unit of work seriously. For our analyst, the unit goes from the funds mandate or, uh, and from the mandate all the way to investment memo."
  },
  {
    "startMs": 811244,
    "text": "So let's first look at what a task in the benchmark looks like. Let me start with a simple question. Um, how does an analyst work at a, um, hedge fund? Nobody gives an analyst a simple task. The PM gave analyst the fund's mandate and a company. For example, the mandate looks like we are an event-driven fund, and we trade on mergers and acquisitions with a holding period three to twelve months."
  },
  {
    "startMs": 841284,
    "text": "And the PM asks, \"Okay, here is a company A, and we know company A will be bought by company B. Should we go long, short, or just pass on this company? And please bring me an investment memo with detailed evidence and the reasoning behind it.\" And this can take an analyst hours, twenty to, twenty to a hundred hours easily. And this is the one real unit of analyst work."
  },
  {
    "startMs": 866924,
    "text": "So let's go deeper into an example task. Here's a company called Sonoco, and it's a US packaging company. Recently, Sonoco brought-- bought a company called Eviosys, and it also sold one of its business called TFP. To value Sonoco, you first need one base number, which is sales. But five different sources give the different s-- um, sales numbers from five, five billion to eight billion dollars. Why? Because each source counts a different s- different Sonoco."
  },
  {
    "startMs": 897084,
    "text": "So first, the agent has to understand the numbers doesn't match. Second, it hasn't-- it has to find out why. And,"
  },
  {
    "startMs": 909944,
    "text": "and, um, and third-- And als- second, it has to find out why, and third, it has to rebuild Sonoco as it was in May 2025 with Eviosys for the full year and without TFP, this will be about seven point five billion dollars of sales. And that is just a small part of the entire reasoning to a skilled analyst to work through. So then how we grade the agent work. So agent submits an investment memo long-- together with a reasoning"
  },
  {
    "startMs": 939864,
    "text": "graph in the graph, and then every fact should point to exact tool call it came from. Then our former buy-side analyst  write down the key reasoning steps that any good memo would include. Each step is a node in the graph. For Sonoco, the agent must rebuild the company it was in May 2025 and what the new co-- new, new business really earns and net out what was sold and rebuild the debt."
  },
  {
    "startMs": 969844,
    "text": "It-- For each step we ask age-- Uh, we also attack the memo with a what-if test. For example, management pro-promised cost savings from the Eviosys deal, but, um, what if the sa-savings were zero? Would the call change? And does the memo explain this with clear reasoning about the moving parts? So the, so the, so the memos can reach different calls, and both can score well as long as the work behind the-- its calls"
  },
  {
    "startMs": 1000004,
    "text": "are there. So this is the result, and you can see the Claude Opus-- Uh, this is a preliminary result, um, only runs over ten tasks o-of fifty, um, entire task universe. And the Claude with five point five scores best, um, in the matter of time. Let's skip to the next page. And every model behaves differently. For example, Astra 6, really good at making a financial modeling, while the Opus five point five is balanced very well over the-- across different,"
  },
  {
    "startMs": 1030104,
    "text": "um, tasks. Okay, let me get back to Janie."
  },
  {
    "startMs": 1037944,
    "text": "Um, awesome. So that was the first initial ten tasks, um, of MandateBench. The, the full twenty, fifty-task benchmark will be launching, uh, next month in, in November 2026. Um, and, uh, this is part of many, many other benchmarks that are launching in Kag- on Kaggle and, and DMind over the next few weeks. Uh, we've partnered with a bunch of leading AI startups, uh, to build benchmarks in different domains. Uh, we launched HeyGen's Code-Code2Video Benchmark, uh, last week."
  },
  {
    "startMs": 1067764,
    "text": "Um, we're working on a couple other benchmarks with Activa, Quadrillion, and Rehearsals, uh, on, on m-meta-- uh, clinician workflows, research workflows, uh, and user research workflows. Um, so if, uh, anyone in the crowd is interested in, in building a real-world benchmark for, for your company, uh, we're, we're very interested in collaborating. Um, as I mentioned before, we're, we're providing free model compute, uh, FTE support from, from GDM, and five"
  },
  {
    "startMs": 1097604,
    "text": "million dollars in grants. Um, so please reach out to us. We'd, we'd love to work together. Um, thank you so much."
  }
]