AI Engineer World's Fair 2026
Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers
Read the talk
Stop Counting Correct Answers: Calibrating LLM Evaluations with Item Response Theory
A benchmark score hides which questions a model gets right. Item response theory turns those response patterns into estimates of ability, item quality, and unexpected behavior.
From a talk by Alejandro Vidal
Before you start: Familiarity with benchmark accuracy and basic probability is enough; item response theory and its parameters are introduced as they are used.
What disappears inside an accuracy score
Take a benchmark leaderboard and split each model’s accuracy bar into the individual questions behind it. The total tells you how many answers were correct. It does not tell you whether those answers came from easy questions, difficult questions, or questions whose answer keys are wrong. Alejandro Vidal, founder of MindMakers, begins with this transformation using real benchmark data from Epoch AI. His background in psychology and computer science suggests another way to evaluate the resulting patterns: borrow the measurement tools used in psychometrics.
In psychometrics, a question is an item. Summing correct responses gives every item the same contribution to the score, despite differences in difficulty and usefulness. Vidal describes this approach as classical test theory and asks what becomes possible when the individual responses remain available.
The bars become a response matrix: one row per model, one column per item, and a correct or incorrect response in each cell. Now each item can be treated as its own variable. The matrix begins to expose structure that a leaderboard conceals, including the concentration of failures where weaker models meet difficult questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put difficulty and ability on the same scale
At the other end of the matrix, strong models tend to answer easy questions correctly. To describe this pattern, assign each item a difficulty parameter, b, and a response curve mapping model ability to the probability of a correct answer. In the introductory curve, b is the ability level where the probability crosses 50%. Moving the curve right makes the item harder.
The model’s ability is θ, or theta—what Vidal calls intelligence. Black dots on the response matrix mark those estimates. Questions to the right of a model’s dot are harder than its estimated ability, and their responses are mostly incorrect. This is a probabilistic pattern, not a boundary beyond which success is impossible.
The highlighted row in the demonstration is labeled GPT-5.5 Pro. The frame shows θ = 1.20, b = −1.23, and a predicted success probability of 99%; Vidal rounds the difficulty to approximately −1.2 in his narration. The useful relationship is that the model sits well above the item’s difficulty. Reproducing the displayed probability also requires the curve’s slope and scaling convention; the two location values alone do not determine it.
Calibration supplies a reference population as well as an estimate. Vidal describes normal distributions for ability and difficulty and uses b = 0 to explain average difficulty. Normality is a modeling assumption, however, rather than a property guaranteed by item response theory. On a scale centered at zero, the precise interpretation is that an item with b = 0 gives a model with θ = 0 a 50% success probability. Models above and below that point have different probabilities; half the models do not each have exactly a 50% chance. This shared scale makes the reference behind an ability estimate explicit and supports uncertainty estimates.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discrimination determines how much an answer tells you
This framework is item response theory, or IRT. Difficulty is only the first item parameter. The second, a, is discrimination: how sharply success probability changes with ability.
| Discrimination | Response curve | Measurement value |
|---|---|---|
Large positive a | Steeply increasing | Separates nearby ability levels |
Near-zero a | Nearly flat | Carries little ability signal |
Negative a | Decreasing | Stronger models succeed less often |
Negative discrimination is especially interesting because it runs against the intended direction of the test. It can point toward a problem with the item rather than a weakness in the stronger models.
A standard mathematical expression for this mechanism is the two-parameter logistic model:
Here, Pᵢ(θ) is the probability of answering item i correctly. At θ = bᵢ, the exponent is zero and the probability is one half. The discrimination parameter controls the slope around that point. This formula makes the roles of the parameters explicit without assuming that it reproduces every numerical setting in Vidal’s interface.
Once the item curves are calibrated, a new model’s answers provide evidence about its ability. Vidal displays each item curve above a relative likelihood over θ. A correct answer contributes Pᵢ(θ); an incorrect answer contributes 1 − Pᵢ(θ). Under the usual conditional-independence assumption, these contributions multiply:
The response yᵢ is one for a correct answer and zero for an incorrect answer. As the demonstration adds responses, their curves combine into a likelihood that identifies plausible ability values and supports a likelihood interval. More informative responses can narrow that interval; the result is more than a point on a leaderboard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two more correct answers can conceal a larger difference
Vidal next compares Claude Opus 4.1 and Gemini 3 Pro on the same set of questions:
| Model | Correct answers in Vidal’s example |
|---|---|
| Claude Opus 4.1 | 245 of 337 |
| Gemini 3 Pro | 247 of 337 |
The totals look very close. Vidal reports that the fitted IRT ability estimates are separated by almost one standard deviation, favoring Gemini 3 Pro. He interprets this as a substantial ability difference and notes that Gemini is the later model. The comparison belongs to this calibration and evaluation setup, rather than establishing an unconditional ranking between the models.
Which questions a model answers correctly matters, not just how many. To explain the gap, Vidal contrasts answering harder questions with accumulating correct answers on easier ones. In principle, a model could even answer fewer questions correctly while receiving a higher fitted ability estimate. That hypothetical reversal is distinct from the displayed totals, where Gemini has two more correct answers. Item parameters and likelihood intervals help make the comparison interpretable before the talk moves into practical applications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find questions that punish better answers
The first application is benchmark maintenance. With response data already available, Vidal proposes using a shared skill to inspect the item parameters, retain useful questions, and find candidates for repair. Plotting difficulty against discrimination separates informative positive-a items from nearly flat items and those whose relationship with ability runs backward.
The inspection procedure is straightforward:
- Flag items whose estimated discrimination is significantly below zero.
- Inspect the question and its gold answer, optionally asking another LLM to help.
- Decide whether the label needs correction, the wording needs improvement, or the item should be removed.
The statistical flag prioritizes review; it does not itself establish the correct answer. In Vidal’s first example, he does not know the answer personally, and ChatGPT apparently disputes the benchmark’s gold label.
The second example is more revealing. A question asks for the total number of passengers, but its gold answer is 583—the total people killed, including crew. An answer can therefore match a familiar number while failing to answer the question actually asked. A model that distinguishes passengers from crew may be marked wrong precisely because it read more carefully. The repair is to align the question and label; the passage does not supply a verified replacement passenger count. Other flagged items may have correct labels but poor construction, requiring revision or removal instead.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend evaluation tokens on informative items
Organizations often maintain their own benchmarks to choose models, particularly open-source models. Running every item against every candidate costs time and tokens. Calibrated items make it possible to ask how small an evaluation can become while still serving its purpose. Vidal uses a deliberately simplified selection rule: start with the item having the highest discrimination, then add items in descending order of a.
For the demonstrated benchmark, Vidal reports reaching a target correlation of 0.99 with the original ranking using about 97 of 484 items—almost a fivefold reduction in question count. Random selection performs substantially worse in his comparison. This is a ranking-preservation result for that example; the talk does not specify the correlation definition or a held-out evaluation protocol.
Why can so many questions be omitted? Vidal points to overlapping response curves and weak discrimination. Similar curves can provide similar information about the ability range being measured, while nearly flat curves contribute little ability signal. The objective is not to maximize the number of questions asked, but to obtain enough useful evidence. Rather than automatically deleting the rest of the benchmark, keep the item bank and administer subsets on different runs.
The benefit depends on the benchmark. In Vidal’s analysis of GPQA, selected and randomly chosen subsets perform similarly. He attributes that result to highly discriminative items with little overlap. This counterexample matters: item selection offers less advantage when the bank already contains useful questions throughout.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look for answers the fitted model did not expect
An item curve supplies an expected probability for every model–question pair. Compare that prediction with the observed response and you obtain a residual. These differences make it possible to investigate unusual behavior, including possible leakage, benchmark overfitting, or other contamination.
In the response matrix, Gemini 3 Pro answers an item incorrectly despite a fitted success probability of 86%. A simple raw residual subtracts the predicted probability from the binary observation:
The calculation for that response is:
python
predicted_probability = 0.86
observed_correct = 0
residual = observed_correct - predicted_probability
print(f"{residual:.2f}") # -0.86
The negative sign indicates performance below the prediction. It identifies an answer worth investigating, without explaining why the model failed.
DeepSeek-R1 supplies the opposite case: a correct answer where success was not expected. That positive surprise does not prove overfitting either. The completed matrix highlights both kinds of anomaly, making it easier to distinguish unusual successes from unusual failures.
A single binary response can be noisy. Vidal suggests asking the same question multiple times and averaging the outcomes. Comparing an empirical success rate with the fitted probability may resolve an apparent outlier that was merely one unexpected sample.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use response consistency to investigate inference failures
The same analysis can operate across an entire model row. Vidal identifies o4-mini as the least consistent model in the displayed example: its mixture of correct and incorrect answers fits the difficulty ordering less neatly than expected. That pattern suggests a question for operations as well as model research: is the inference system behaving correctly?
An inference platform running the model incorrectly, or a faulty quantization, could produce unusual response patterns. These are possible causes to investigate, not diagnoses established by the plot. Some noise is inevitable; a useful response model should make success and failure approximately predictable, not turn the matrix into a perfect staircase. Residuals provide a way to inspect deviations from that expectation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give each organization a different exposure pattern
Suppose a benchmark is expensive to build and valuable precisely because its questions remain private. Sharing the whole bank risks exposing it to future training. Vidal introduces a selective-exposure scheme under adaptive testing: evaluate models through a common anchor while giving each participating organization a distinct set of additional questions.
The proposed procedure preserves a common reference and records who has seen which difficult items:
- Randomly select a representative anchor set and use it with every participating model and organization.
- Assign each organization its own exclusive fingerprint set.
- Choose extremely difficult items for those fingerprint sets.
- When organizations release new models months later, run evaluations again and inspect residuals on the previously exposed questions.
The signal of interest is unusually strong performance on the particular fingerprint set an organization received, beyond what the new model’s estimated ability would predict.
Vidal demonstrates the comparison with synthetic data. One organization has an unusually large average residual on a specific fingerprint set relative to the other organization. This provides a targeted contamination signal rather than merely noticing that a newer model improved overall. The scheme is not bulletproof: it creates evidence for investigation, not proof that an organization trained on the benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare groups at the same estimated ability
Psychometrics also asks whether an item behaves differently for different groups. Vidal transfers that question to model evaluation by splitting the dataset into open-weight and closed-weight models. Other grouping variables, or more than two groups, could be used when they answer a useful research question.
For each item, fit a separate response curve for each group. Compare the curves at matched ability values: if the item behaves similarly across groups, the curves should overlap and the probability gap should be close to zero. A gap indicates that group membership carries information about success beyond the shared ability estimate. In the demonstration, some items favor closed-weight models and others favor open-weight models.
Vidal withholds the actual questions to avoid leaking them. He observes common features among some questions favoring open-weight models, but cannot identify the cause. Such patterns could guide research into training differences; the curve comparison itself does not reveal how a model was trained.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use shared residual patterns to explore model relationships
The final application treats a model’s residual vector as a behavioral fingerprint. Instead of asking only how capable two models are, ask whether they make similar unexpected successes and failures after accounting for ability and item difficulty. Vidal constructs a correlation matrix and a simple projection to explore those relationships.
The display groups some models from the same lab or model lineage. It also shows patterns among DeepSeek distillations and Qwen models. The intuition is that shared history may leave shared error patterns, even when the models’ overall abilities differ.
Vidal reports a residual correlation of 0.38 for an example pair sharing a base model. He then considers several related comparisons:
- Distillation and base model: Similar residuals could help investigate a suspected distillation relationship, including unauthorized distillation.
- Different effort settings: Runs of the same model at different effort levels may retain a recognizable response pattern.
- Successive versions: Relationships among Llama or Gemini versions may remain visible as the models evolve.
The pair behind the reported correlation is not named in the spoken example. Vidal frames this application as an area for research into model relationships, rather than an established everyday test for model ancestry.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beyond one ability number
A single ability axis is a starting point. Vidal closes by proposing multidimensional and hierarchical models that could distinguish skill levels across task types. A model might then have a profile of abilities rather than one scalar standing in for every kind of performance.
He also proposes combining benchmarks through IRT to improve their estimates and recommends a paper he calls Meta-Benchmark. The underlying opportunity is to relate evidence across tests instead of treating every benchmark score as an isolated measurement.
Other extensions would broaden what is measured: latency and token usage could contribute additional signals, while psychometric models could be used to study alignment and interpretability. Vidal also suggests that mechanistic interpretability could help explain the behavioral structure psychometrics uncovers. He is working on these directions, invites collaboration and requests for benchmark help, and closes by offering materials, skills, and benchmarks for experimentation. The practical starting point remains the response matrix: retain the individual answers so that evaluation can measure more than their sum.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Public benchmark results with data access and licensing information.
Introduces expert-written science questions and the validation process behind GPQA.
Further reading
- Epoch AI benchmarking methodologyDocumentation
Explains evaluation settings, result provenance, and access to model response logs.
A worked Stan example connecting item difficulty, discrimination, and latent ability.
- metabenchPaper
Research on a sparse benchmark for measuring reasoning and knowledge in language models.
Python tools, small benchmark datasets, and tutorials for estimating performance with fewer evaluated examples.
Read the complete timestamped transcript
- 0:02
Hi everyone, I'm Alejandro Vidal, the founder of MindMakers, and my background is psychology and computer science, which is kind of weird, but for today is going to be extremely helpful because I'm going to show you how you can borrow ideas from psychology and psychometrics to improve the way that you are evaluating models right now.
- 0:21
Because at this moment, the state in the industry is counting the number of right answers. That actually has a name, it's Classical test theory, and we have by far better tools to do that.
- 0:33
So it makes sense to borrow ideas from IQ tests and related stuff so we can apply them to LLMs. Let me start with a very simple example here. Um, we are using real data from Epoch.ai.
- 0:47
If you don't know them, um, their project is amazing, and they have quite open datasets so you can actually use them. And here we have a random selection of models with a real benchmark.
- 1:00
As you can see here, we have an accuracy for each one of them. That's the current state of the art. So if we split each bar into different questions, as one of them is going to be a different question or a different item.
- 1:14
I'm going to use item for, um, uh, for, for the same idea of question. In psychometrics, we use item instead of question. Um, if you sum all together, we are using a very strong assumption.
- 1:27
We are saying that every question is equally important. They should weight the same, which is kind of insane if you think about that. We have better questions, more complicated questions that may-maybe we should pay more attention to.
- 1:42
And also, we, we can have questions that are mislabeled or something like that. So we are going to improve this. What are we going to do is we are going to use each item.
- 1:52
Each column here is going to be one item, and we are going to treat them as individual variables. Okay? So we are going to have this matrix here. As you can see here, um, on the top right corner, we have difficult questions for weaker models, and on the other side we have
- 2:10
very easy questions for strong models. So makes sense that we observe this pattern. Okay? But we are going to estimate for each question, for each item a difficulty level.
- 2:21
That is going to be called B. The B parameter is going to be the difficulty of each one of them, and we are going to create a function for each question.
- 2:32
That function maps the LLM intelligence, okay, to the probability of getting that answer right. Okay? So very easy items are going to be here, and extremely complicated item, items are going to be there.
- 2:47
As you can see here, B is the point that, uh, crosses fifty percent chance in that curve. Okay? Which is going to be useful later. Also, B is going to be distributed by a normal distribution, which is going to be also helpful to use that for interpretation.
- 3:03
Okay? So with that in mind, we can actually estimate also theta. Theta is going to be the level of intelligence for each model. That's going to be that dot, that black dot.
- 3:15
So as you can see on the right side of each dot, mostly, uh, all the questions are going to be red, which makes sense. If they ha-- they are extremely complicated or more complicated than the level of intelligence of that model, the model is going to fail them.
- 3:32
Okay? So we are going to model that way. So for example, here, if I click on this button, I'm going to see that, uh, GPT 5.5 here is going to be able to answer that question because GPT 5 has a theta value of 1.2 and the difficulty of that item is -1.2.
- 3:52
So the probability of the right answer is ninety-nine. Okay? So with that in mind, what are we doing right here is actually calibrating each question, each item. So we are going to improve a lot our estimations.
- 4:04
We are going to improve also our confident intervals and many other properties. S- the other thing that I want to explain here is theta and b is going to be, uh, a pair of numbers that are distributed with normal distributions, so we can actually interpret them.
- 4:21
For example, uh, item of b equals zero means that it's going to be average. Half of the, mm, models in my dataset are going to be able to answer that question fifty percent of the time.
- 4:34
So that's going to be extremely helpful because right now, to evaluate benchmarks, we need that reference compared with other models. So it's better to have one, uh, by default with this methodology.
- 4:46
Actually, this model is called item response theory, which is the evolution of classical test theory. Okay? So on top of that, I'm going to have another parameter. It's going to be the, a slope, the discrimination of that item.
- 5:01
So high discrimination are going to have steeper functions. Also, we can have random functions, items that are not related with intelligence, which I don't want, but something happens. And even worse, we are going to have items that have negative correlation with the actual intelligence of the model.
- 5:22
But then keep that in mind, because each item is going to be a difficulty level, b, and it's going to have a discrimination, uh, parameter here, a.
- 5:31
So last thing that you need to understand is how can we estimate the intelligence level of a given model if we have IRT modeling, item response theory. We're going to actually, um, go by one by one for each, uh, question here.
- 5:48
So I'm going to plot here on top its item curve, and below, you are going to see the relative likelihood of theta. That's the estimation of the intelligence. So as you can see here We are going to use all curves to combine them in one distribution over time.
- 6:04
Obviously, with more questions, we're going to have, have a better es- better estimation. So if we add all of them at the end, I'm going to have my final, final estimation that is extremely nice to have a distribution here because I'm can have a li- likelihood interval with, uh, with classical test theory is more complicated to have.
- 6:25
Okay? So even with that, with a sure scale with B and theta, even with the likelihood interval, if you think that this not... this is not useful at all, let me show you one last example.
- 6:39
Okay? Again, this is real data. So I'm going to compare two models. Here is going to be on the left side, Claude Opus, uh, 4.1. Yeah. That has two hundred and forty-five right answers.
- 6:54
On the other side, we are going to have Gemini 3 Pro, which has two hundred and forty-seven. Okay? So as you can see, the difference here is quite small for three hundred and thirty-seven questions.
- 7:08
But if you use item response theory, you can see that this... the difference between all of them is almost one standard deviation. That means Gemini 3 Pro is by far more intelligent, okay?
- 7:18
Which makes sense because, uh, it's a, uh, later model. Okay.
- 7:24
So as you can see, counting the number of right answers is, is not a good approach because I can create benchmarks that are not calibrated, and even if I get a lot of right answers, I'm not more intelligent than other models.
- 7:37
That could happen, for example, if Gemini is able to answer by far harder questions, uh, even if, uh, Claude is able to answer more of them, maybe Claude was able to answer only the easiest one.
- 7:51
So the theta level should be different. Okay? That's the... that's actually what happens with IRT. You have better estimations, you have more parameters to define each item which is going to be helpful later, and also you have, as you can see here, likelihood intervals.
- 8:06
So, uh, on top of everything, uh, I'm going to show you a few applications that you can use with IRT, um, and I hope that you find them interesting.
- 8:19
Okay, so the first application is going to be one of my favorite ones actually, because you can apply this with, uh, one skill that I'm going to share. It's going to take a few minutes if you have the data, and you can actually, um, pick the best items, the best questions on your benchmark and remove the other
- 8:39
ones that are not working or even fix them. Okay. So we can update our benchmark. So remember that we have two numbers that represents each item, each question. We have B, difficulty, and A, which is the slope or the discrimination of the item.
- 8:53
On the right side here, we have very good items, very informative ones. Here, close to zero, we have kind of noisy with little signal, and on the other side, we have items that we don't like at all, negative items.
- 9:06
Here, for example, you can see that we have items that correlates, uh,
- 9:12
the other way around, that, uh, better models are actually getting that, uh, answer wrong, which makes no sense. So we can use this to actually find items that are significantly below zero, and we can actually flag them.
- 9:26
With that, I'm going to use another LLM, um, to actually evaluate them. So here you, you can see two items that I detect with that technique. The first one is actually, I don't, I don't know the answer.
- 9:37
I ask ChatGPT, and apparently, the gold answer, the answer that is on the benchmark, is not right one. Okay? So this is okay. But the next example is by far more interesting for me because the answer is kind of right, so it's something that, um, a lot of people can miss.
- 9:54
So here it's asking, "What is the total number of passengers?" The gold answer, the answer that is on the benchmark, is five hundred and eighty-three, which is the total people killed, passengers par...
- 10:06
plus crew. But the right answer, if you pay attention, I'm asking only about passengers, which is another number. So again, with, uh, very little effort, if you have the dataset, you can actually find items that are mislabeled.
- 10:19
This is the most common thing. But sometimes they are not mislabeled, they are, like, bad items. So you should remove them or even improve them.
- 10:28
This second application for me is amazing. It saves a lot of time, a lot of tokens, and therefore a lot of money. Okay? So, um, it's quite common for organizations to have their own benchmarks to evaluate which model is better for them, especially with open source models.
- 10:45
With that in mind, it's quite important to reduce the size of the benchmark to find the optimal size of the benchmark. Before, we couldn't do that because we didn't have any property of the item.
- 10:56
But right now, with item response theory, we can actually, uh, pick the best items. Okay? So this is a simplification, but we are going to say that items with high levels of discrimination are going to be the best ones.
- 11:10
Okay? With that in mind, I can do this. So again, real benchmarks with real data.
- 11:18
Uh, for, for this benchmark, I'm going to target a ninety-nine correlation with the original ranking. So it's going to be, like, almost perfect for many uses cases. And what are we going to do is pick o- one by one, starting with the best item, the item with the high- highest A.
- 11:38
So with that methodology in mind, we are going to get that around ninety-seven items compared with four hundred and eighty-four. That's almost 5x. We are going to have the same ranking than before or almost the same ranking as before.
- 11:54
To be fair, I tried to do the same thing but randomly. So if you pick items by random, you are going to observe, uh- B-by far worse performance. So it's extremely helpful.
- 12:07
Maybe this is kind of, um, weird for you because, uh, if we can evaluate items with by far less questions, why are actually using all the benchmark? So... And the answer for that is that we are not used to calibrate benchmarks.
- 12:23
We are, like, uh, assuming that more questions, uh, means better estimation, which is not true. For example, I can have two questions that kind of overlap. Their curves are eq- are more or less the same.
- 12:36
So even if I ask two of them, I'm going to get more or less the same information about your intelligence. Okay? And also there are a few items that you can notice that, uh, they are not extremely informative.
- 12:47
They c- don't correlate with actual intelligence, so you can reduce them or using them less. I'm not saying that you should remove, uh, like the, the, the rest of the data set, but you can use this for another application that we are going to talk, that you can use a subset of items each time that you apply
- 13:06
the benchmark, which is going to be extremely useful. But, uh, be careful because that, uh, this does not happen on every benchmark. Okay? So for example, GPQA, which is a extremely well-designed, uh, dataset.
- 13:20
The, um, benchmark here, as you can see, even if you pick at random, you are going to get more or less the same result. And the reason for that is that every item here is extremely discrimi- discriminative, and also they don't overlap, so they are...
- 13:34
all of them are useful at every time along the benchmark. Okay?
- 13:40
So now, given that we have for each item one function, we can also calculate the error, the unexpected behavior, the outliers for each question, which is going to be amazing for many applications.
- 13:54
We can actually find out if we are leaking information, if we are overfitting with the benchmark and other kinds of contaminations. So let me go back to the metrics that we started with. [clears throat]
- 14:06
And as you can see, every question is going to have the estimation and the actual answer. And that's good because we can observe a few outliers here. For example, Gemini 3 Pro should be able to answer that question.
- 14:20
Actually, our model says that eighty si- eighty-six percent of the times Gemini should be able to answer that. But even with that,
- 14:30
uh, that question is wrong. I cannot say why, but I can actually detect those outliers, those weird patterns. Okay?
- 14:41
And with that in mind, we can actually calculate the residuals, the error for that specific question, which is going to be, uh, useful for different applications that I'm going to explain later.
- 14:51
The other way around, we can actually see here that, for example, DeepSeek R1 is having here a right answer even if it's not expected. Okay? Again, that does not mean that, uh, we are overfitting or anything.
- 15:06
Actually, we can sample the same question more than once, so we can average them together, and maybe with that we can, like, resolve that outlier. But in any case, we have a new tool that we can use to analyze the data.
- 15:20
On top of that, actually, in psychometrics, we have a few techniques to see if the behavior is consistent, if that makes sense. Okay? So for example, here I can observe that o4-mini is the less consistent one.
- 15:36
You can see that has a lot of red, uh, a lot of, of green questions here in the middle, which makes no sense. So you should be able to use that, uh, to, for example, detect if your inference platform is not working well.
- 15:51
Because, uh, if your... for whatever reason, your inference platform is not actually running the models or the quantization is actually wrong, you are going to observe things like that.
- 16:01
Behaviors that are not expected, okay, should be consistent. It's... this matrix is not perfect. It's going to have noise, but we should be able to more or less estimate the level of intelligence given that we can predict more or less what questions are you going to get right and what question are you get it-- are you going
- 16:19
to get wrong. Okay? Again, residuals are going to be extremely helpful for the next application, but only with this I think you can actually look at the data in other way.
- 16:33
Okay. So let's say that you are building a benchmark, a very complicated one, very expensive one, so you don't want that benchmark to be leaked on the internet. Neither you want other organizations to train their models with that benchmark, okay?
- 16:47
Because it's extremely valuable, and if you can protect that benchmark over time, it's going to be more valuable. So with that in mind, we can actually use what we call adaptive testing.
- 16:59
Okay? What is that? I'm going to pick a random set, uh, of items that are going to be representative of my benchmark, and I'm going to call that an anchor set.
- 17:09
An anchor set should be representative of one entire benchmark, and I'm going to use those items with every applicant, with every model from any organization that I'm co- working with.
- 17:21
But for every organization, I'm going to pick, uh, one individual set. I'm going to call that fingerprint set, that I'm going to sew only to that specific organization. And I can do also the same thing with another one.
- 17:37
Pay attention to this because it's important that I pick extremely complicated items from my benchmark for those fingerprint, uh, sets. Over time, let's say a few months later, uh, every organization is releasing their new models and I'm going to observe, I'm going to, uh, like, run the benchmarks again, and I'm going to use residuals to, um, see
- 17:59
if those models are extremely good at those specific questions. So here, this is a synthetic data set. You can see that the average-
- 18:11
residual for that organization for that specific fingerprint set is extremely unlikely. So this is not bulletproof, but this is an extremely good technique that you can use to protect your benchmarks.
- 18:24
As you can see here, the average residuals for one organization is by far bigger than the other one.
- 18:31
In psychology, it's really important to find out if one of our items actually bias against one specific group. We can use the same techniques to research how models behave with items.
- 18:44
So I'm going to show you that with one example. I'm going to split the dataset in two groups. In this case, I'm going to use open weights and closed weights.
- 18:52
You can use whatever variable else is interesting for you, and even you can create more than one group.
- 18:59
After that, I'm going to, for each item, create two different curves for each group. And what we expect if the item is
- 19:10
unbiased is that both lines are kind of overlapped. If not, we are going to calculate this difference, the gap between those ones, and that difference should be zero or close to zero.
- 19:23
In this case, we can detect, we can apply that technique to all items on my benchmark, and actually we can find out that there are a few items that are better for closed weight models and better for open weights models.
- 19:36
I'm not going to show the items because I don't want to leak them on the internet. But if you apply this technique to a real benchmark, you can actually notice a few patterns there.
- 19:47
There are a set of questions that have something in common that are better for open weight models. I cannot pinpoint the real reason for that, but I can say that this technique could be used to understand how they train their models.
- 20:02
The last application that I'm going to show you is kind of interesting. It's the most complicated one, but in a way is the most interesting one. So I'm going to use the residuals to actually have a DNA, a fingerprint for models.
- 20:17
And the idea is that we can actually look at them and see if two different models are related somehow, okay? So I'm going to make a correlation matrix here, and also I'm going to make a super simple projection.
- 20:31
So the first thing that you can notice is that there are a few models that are extremely close here, that they are from the same lab, and also they have the same history.
- 20:40
They are like different versions for the same model. Also here you can observe we have DeepSeek, different distillation of DeepSeeks. We have also Qwen. Um, so we can observe some kind of patterns there.
- 20:55
And it makes sense because if a model has a shared history with other model, we can expect the same kind, kind of errors, okay? And let me show you a few examples.
- 21:06
The first one is going to be this one. Uh, if we have two models that have the same base, we expect high correlations between them. In this case, we have
- 21:17
.38, okay? Also, we can observe that between distillations and its base model, which could be extremely interesting if you want to test that distillation of your model, uh, without consent.
- 21:31
Also, we can det- detect the same model with different effort levels, which makes sense because it's almost the same thing. And also we can say, uh, we can actually detect if we have different versions or evolution of the same model.
- 21:46
Here we have Llama, but also work with Gemini here, okay? So again, this is not as useful as others for day-to-day, but I think this opens an area of research to understand how models are related with each other and even detect distillations.
- 22:06
I really hope that you find inspiring this talk. My goal here was to open the gates of, uh, psychometric research for LLMs. I think we can improve a lot how we benchmark LLMs with very basic maths here.
- 22:24
But there are a lot of, uh, ideas that you should explore because I didn't have time. The first one is multidimensionality and hierarchical models. I'm expecting if we apply them to LLMs to see different skill levels for different kind of task.
- 22:40
Also, we can merge different benchmarks to improve the estimation of each one of them. Makes sense if you have IRT. And also this has been done in a, in a research paper called Meta-Benchmark that I highly recommend you to read.
- 22:55
Also, we can add another signal that we think correlates with intelligence, for example, latency or tokens. And another very promising idea is to use psychometrical models to actually measure, uh, alignment and use that for also interpretability.
- 23:12
I think mechanistic interpretability could help a lot psychometrics here. I'm working on all those areas. If you are working on them or you have any idea or you need help to apply them to your benchmark, let me know.
- 23:26
Thank you for your time, and here you have all the materials, skills, and benchmarks so you can play around with them.