What Do Models Still Suck At?

Talk transcript

1. 0:00

   [upbeat music] I want to talk to you something maybe a little bit controversial today.

2. 0:19

   Uh, you can argue with me later. Uh, but the topic is, what do models still suck at? And, uh, the reason why I wanted to talk about it is that I think we, uh, all look at these kinds of charts where any benchmark you seem to look at, line goes up.

3. 0:35

   And, uh, we look at meter charts, and they surprise us every time no matter how prepared we are. And this could create this kind of psychosis that we'll see where everyone is freaking out about the next model.

4. 0:50

   You know, we, we heard some new ones coming up. And the feeling I think that we all get is that this is kind of, um, AGI-like creatures that are just almost there.

5. 1:01

   Just one, one more turn, and they're almost there. And, um, I think we, we could be dissi- deceiving ourselves a little bit, um, uh, because I think there's still quite a few things missing.

6. 1:13

   Um, I want to explore that in a couple of different ways. And we certainly, by the way, see that as well in our data, uh, at Arena as well.

7. 1:20

   So we track, uh, models, and if you notice the data, this is, uh, Q2 2023, so we've got data going back to GPT-4. And what we do is, uh, we can-- We've tracked, I think, is it seven hundred models so far, uh, in text.

8. 1:36

   And, uh, what this chart is showing is what the top model is, uh, for at any given time for, for each organization. Um, so you can see line goes up, new model, uh, builds on top of each other, and it's all, it's all very impressive.

9. 1:52

   Um, but I think it's, it's not the whole story. So I've got a couple of ways how I want to explore that. It's not the, the end of the conversation.

10. 2:01

    There are definitely many other ways of looking at it. Um, one is my own benchmark that I, I built recently, which, uh, I rather like. This is the, the BullshitBench mark.

11. 2:10

    Uh, and then also I'll share some of the Arena's data as well that, uh, we haven't shared so far, which I think will be interesting for you guys to see.

12. 2:18

    Um, so, uh, the idea behind the BullshitBench mark i- is quite simple, um, is that, uh, what happens if you ask nonsense questions, uh, fro- from the models? What they gonna do?

13. 2:30

    Are they gonna just, uh, tell you that, "Oh, this doesn't make sense," and maybe reframe it, or are they just gonna go with it? Um, and honestly, wasn't sure how that was gonna go.

14. 2:41

    But when I just posted it one random evening, I think a lot of people liked it. It resonated with a lot of people. Um, and I think it-- the reason is that it probably spoke to a lot of maybe a kind of slight unease people had with different models.

15. 2:58

    Um, and I'll give you one example, uh, here. This is just one question, and the way it works, we've got-- I think I've got a hundred and fifty-five questions, something like that.

16. 3:07

    Um, and, uh, we then, uh, give this, uh, to the models. Um, uh, we get a response back, and all we do is then grade it, uh, with LLM-as-a-judge.

17. 3:18

    And I've been through it myself as well. I read a lot of nonsense to, to kind of see that I think LLM-as-a-judge works here. Uh, so this one is a kinda silly question: Controlling for repository age and average file size, how do you attribute variance in deployment frequency to the indentation style of the code base versus the

18. 3:37

    average variable name length? So hopefully you understand that it's nonsense. So it's just-- it's very abridged responses. Uh, they're much longer just for the purpose of this. Uh, so Sonnet gives a good response, I think.

19. 3:49

    It just says, "You can't meaningfully measure this." It kinda pushes back. Uh, Gemini is, like, a little bit more complicated 'cause this starts off well. It says that, uh, strictly speaking, it doesn't really make sense.

20. 4:01

    But then the second part is, "However, both act as strong proxy variables for engineering culture, uh, language ecosystems, and code quality," which I hope, uh, you don't agree with.

21. 4:12

    So, um, there-- And, uh, I'm not gonna go through a bunch of examples. It's all open source, by the way. You, you can, uh, dig it out yourself. Um, but, uh, it's really, really surprised me how easy it was for the models to just go along with the complete nonsense questions.

22. 4:29

    Um, so the results that I got is that, uh, the way to read this chart is, uh, the green is the clear pushback, so when the model's like in the first example where it said, "Oh, maybe this doesn't really make sense."

23. 4:42

    Uh, then the, uh, the amber and red there is kind of accepting the, the nonsense. And the basic results are is that the latest Sonnet models or, or rather Claude models are doing really well.

24. 4:55

    There's like couple of other models, like Qwen models, not too bad. Uh, there's even Grok is, like, okay as well, well, the very latest one. Uh, but if you go beyond that, there's a lot of models that we all use all the time.

25. 5:07

    So GPT models, uh, Gemini models, they're basically kinda about fifty-fifty whether they're go- gonna go along with it or not. And even looking at some of the traces and responses in more detail, even the ones that are green is still, like, a little bit shaky.

26. 5:22

    They still kind of try to accommodate. So it's, uh, like, for me, this is really not nowhere near good enough, uh, for the, uh, level of responses. And just for completeness, if you go all the way, so this is the very bottom of the table, um, there are a bunch of smaller models there, uh, kind of all,

27. 5:38

    all the models. Um, yeah, some, some results are completely terrible. Uh, it feels like you can ask anything, they j- they just, uh, respond. Um, another way of looking at this data is I just took the Anthropic, OpenAI, and, and, and Google there, and I, um, measured, uh, the model performance over time.

28. 5:59

    And, uh, you don't see all the labels there, but they're basically, like, all of the, uh, all of the models that, uh, you, you remember them releasing.

29. 6:07

    Um, so what, the way I interpret this is that the Anthropic models were, like, okay at the beginning, but the, since, uh, Claude 4.5, uh, Sonnet 4.5, they really went up.

30. 6:19

    And even Haiku is, is quite high. Uh, but, uh, with OpenAI and Google models, they're kind of up and down, but they, they nowhere close, uh, the, the top there, which I think is kind of interesting.

31. 6:31

    Um, and I'll go into some of the other interesting dynamics there. So for example, does thinking help? Right? So this is-- I always hear this when there is, like, a silly puzzle that the model can't do.

32. 6:43

    What do you do? You just sort of crank up the reasoning it, it solves it. If you see, uh, look at the chart on the right, it basically is completely not true here.

33. 6:50

    So reasoning often actually goes in reverse and doesn't help. It actually makes it worse. Um, do model- do more recent models perform better? It's kinda hard to tell for sure, but there's at least not the clear line going up.

34. 7:03

    Uh, and I think if you exclude maybe the latest Anthropic models, it's not even sure, clear that the line goes up at all. Um, then, uh, some specific comparisons for reasoning.

35. 7:14

    So for example, uh, what you see this kind of, uh, the, uh, is the same model with the low reasoning and high reasoning, um, and, uh, these are some examples where no reasoning performed better than high reasoning.

36. 7:29

    And I spent a lot of time reading the traces of GPT 5.4. Um, it's probably the most, um, confusing experience of, of reading these, uh, traces. And what I found was that quite often, it would maybe have one line where it would question the, the premise of the, of, of this question, and then spend 20 paragraphs trying

37. 7:54

    to solve it. And even if then comes back and says, "Okay, maybe this didn't make sense," it still tries to solve it in some way. And this is, uh, feels, uh, completely crazy to me.

38. 8:05

    But the way I imagine, and I don't know for sure, but I imagine the way the, the reason why that happens is that, um, they were trained so much to solve the task at any cost, and I think there was probably not a lot of training to say, "Actually, maybe don't, uh, solve the problem sometimes."

39. 8:25

    And I noticed this first sometimes when you have a lot of agents running in parallel, and I would sometimes forget which one is doing what, and I would, like, ask one agent to do something that's completely the wrong project, and it's still go and do something.

40. 8:40

    And, and I, then I lose my mind. So yeah, that, that's a kind of an interesting dyna- dynamic I thought about, uh, about thinking. Um, then also f- so this is a subset for open-source models.

41. 8:51

    Only you try to see if bigger models do better. There's also no, no real clear pattern. So we've got the total parameters on the left, then active parameters on the right.

42. 8:59

    And I don't know, maybe you can see some patterns. I, I don't really see. It's, like, kind of up and down. Um, but yeah, not, not huge samples. So don't know.

43. 9:07

    In-inconclusive. At least not obviously, uh, is true. Um, so that, that was kind of one lens, um, looking at kind of this specific idea. Uh, but I want to, uh, take advantage of the data that, that we have at Arena and, and show you maybe more broader trends, uh, that we could, uh, look at.

44. 9:29

    Um, so just in case you don't know, uh, much about Arena, what we do is we publish, um, uh, benchmarks, and the way we derive them is that users go into our platform.

45. 9:39

    Uh, they can go in the battle mode. They put in a, a query, uh, and then, uh, they get two responses back, which are from two anonymous models, and then they can say which one they like better.

46. 9:49

    And then you get, um, uh, then the model names only reveal then. And then in, uh, Text Arena, we've got nearly, um, uh, over five and a half million votes there.

47. 10:01

    Um, and, uh, we've been going since 2023 as well with this data, so it gives us really a nice, uh, broad view. Um, the reason why I think this is really useful is, first of all, we, we do have this long trend, and there is not any other benchmark that lasts so long because this one you cannot,

48. 10:20

    uh, exhaust it. It will-- There will always be one model better than the other. Um, so that gives us a long perspective. Another one is that inevitably any benchmark that you pick, it's inevitably has to be condensed to, like, very specific question that, that you're asking, because otherwise it's very hard to measure.

49. 10:39

    So I'm sure it's all in your experience as well when you are, I don't know, doing coding or whatever is your task, um, the benchmarks would measure, like, very tiny slice of what you actually care about.

50. 10:53

    And, and in here we don't have that problem because a user can put any prompt, and then they could just use their judgment to see, like, is that, is that a good thing or not?

51. 11:01

    Um, so, um, what I want to specifically focus on is, is a slightly like a, a odd mechanic that we have that I'm really glad that we had since the beginning, um, is that, um, you can, uh, vote a, which model is better here, uh, A or B, um, but you can also say, uh, when both models

52. 11:25

    give a bad response. And you know if you ask the right, uh, model a joke, uh, response is always bad. So that's a, a easy, easy example. Didn't take me long.

53. 11:34

    Um, so that's, that's the thing to remember. So, uh, if you're just to remember one thing that will really help you for the next seven, eight minutes is that, um, this is the mechanic.

54. 11:46

    Think of it as like dissatisfaction rate. And, uh, what we can do is, uh, if we were to take battles between top 25 models, so we're kinda sampling from the top, so to avoid kind of, I don't know, Llama 8B fighting Qwen 3B, uh, we just take, uh, the, the top set of models- And then we map

55. 12:06

    this kind of dissatisfaction rate, uh, over time. And I, I think this is quite interesting that we do see progress with this metric. So this kind of pre-reasoning models you can see there is like, uh, twenty, seventeen percent dissatisfaction rate.

56. 12:22

    Then we-- when we-- a-after '01, we see that drop quite a bit to sort of about twelve percent, and then after that it carries on, uh, improving to, to sort of about, I think it's about nine percent now.

57. 12:35

    Um, but it's... So improvement is definitely there, but it's not zero percent, which I, I find interesting. I must say, when I, when I first got to that result, I, I thought, like, that's quite high.

58. 12:46

    So nine percent of the time people would get the responses from two good models, and they don't like them, which I think it doesn't tell the same story as all of these, like, crazy, uh, lines going up.

59. 12:57

    Um, so then what we can do is we can also take, um... So what the previous one you saw is, like, average across all, like, uh, six million prompts, and this is the categorization of those.

60. 13:09

    These are just some, uh, I picked out in there. And you can see some interesting trends as well. So maths was, like, at twenty-five, twenty-seven percent, and then it got so much better.

61. 13:19

    So that, that's quite a nice, uh, result, um, that matches my experience of models as well. But then when you look at, like, creative writing, okay, it did get better, but it, like, the, the improvement wasn't that dramatic, which I, I think is, is true as well.

62. 13:33

    Um, the category I want to focus on to really, really try to zero in on the most signal is the expert category. And the way it works is that we take those, uh, nearly six million prompts, then we have, um, a way to classify what are the most interesting mo- the kind of the harder, the more kind

63. 13:51

    of real tasks that expert people do. And they could be experts in different fields, um, but they're kind of the most, um, I would say, high signal prompts in terms of what, what, uh, we could, uh, zero in on.

64. 14:03

    And then we also narrow it down to the battles just between the, these top twenty-five models. So that gets us to about forty thousand prompts. Um, and then, uh, we can look at these, uh, expert categories and then, um, uh, expert category, and then we can subdivide it even further.

65. 14:20

    So in here, uh, I've got five categories here. So again, quantitative, for example. So it's like maths, physics, things like that. You can see this kind of really, really high, uh, uh, dissatisfaction rate in the kind of, uh, when is it?

66. 14:35

    About, yeah, early, uh, twenty twenty-five, late twenty twenty-four. Um, so but-- And that drops dramatically. And I think that feels true to me that a lot of the models got so much better at this kind of quantitative stuff.

67. 14:51

    And I would also say the reason why I think the line goes up is not that the models got worse, but I think people's expectations shift as well. The, the data that we see in terms of what prompts people used at the beginning, like three years ago versus now, it shifts a lot.

68. 15:04

    So this is also not like a static benchmark. So we, we can really see the kind of, um, kind of the, the battle of the expectation versus the model performance.

69. 15:14

    Um, interesting as well on the bottom, we've got medical, finance, and law. And the lines, like, i- it is the, the scale is e-equal across the five charts, so it's, it's a little harder to see.

70. 15:25

    But it's not steep, right? It's not really improved all that much. Um, I don't wanna go into the medical and, and law and finance fields, uh, 'cause I don't know enough about it.

71. 15:35

    But it does feel like it's probably true that that's not really been the focus of, um, of, of the models necessarily. So I think maybe the performance improvement's not been that high.

72. 15:48

    Um, so then what I did was to take all of these prompts and, and classify them further into these more deeper subcategories. I'm gonna focus on software now and give you the kind of view of, of these subcategories, uh, which I think also gives us, like, even, even more detailed view.

73. 16:04

    Just to give you a feel of sense what kind of prompts we are talking about here, obviously a, a tiny sample of three. Uh, but to give you a sense for...

74. 16:11

    So for gaming, someone's asking to get them a, a detailed game design, uh, document. Uh, then for security, someone's got autonomous, uh, system as a hobby, and they want to configure,

75. 16:26

    uh, uh, the two, which I don't really know what this is. But then, uh, for agent systems, uh, which I, I thought was interesting, like actually the-- you'll see the, the rate is quite good.

76. 16:36

    But the person there is asking for refine this agent so it can run daily with, with no supervision. So, uh, these are the kind of-- Just to give you a feel, these are kind of real things that, that people want to do.

77. 16:46

    And, uh, we've got two charts here. On the left is, uh, from Q2 twenty twenty-four. These are kind of dissatisfaction rate. And then on the right, we've got, um, the, uh, Q1 twenty twenty-six, so that's the mo- the most recent data.

78. 17:01

    And you can definitely see improvement. So if you look at the top line, this is the, the, uh, the overall average rate, and we've gone from twenty-three and a half percent to, uh, thirteen percent.

79. 17:11

    So really nice improvement. But I think the improvement is not really seen everywhere. So, um, we can, we can see this as well. Uh, same data, but with a, with a closer timeline, which I think, I think is quite interesting.

80. 17:25

    Um, and you'll have-- you probably have better theories on all of the different, uh, categories, why, why that's the case. And I think bear in mind the case that I think people do ask a lot harder questions.

81. 17:37

    So I think GPU compute, for example, I imagine probably it's up and down because probably people ask harder things as well. But I think gaming is an interesting category because I've tried to use, um, LLMs to build games.

82. 17:52

    Uh, not that I, I... I mean, I, I use games, but, uh, I don't build them. But whenever you try to build games with LLMs, it just feels like they have no idea, uh, how to build actual games.

83. 18:05

    The mechanics are, like, all over the place. They're not interesting. They're not challenging. Uh, so I, I do get this feeling that the performance is not really- Um, improved in some dimensions.

84. 18:17

    Like I don't think LLMs really get games. Uh, even though I'm sure maybe go back two years, people were asking to build much simpler games vers- versus now. Uh, but I wouldn't say that I'm aware of any, like, really good gaming benchmarks that will kind of capture this.

85. 18:34

    So again, if you compare this to kind of one going up, I think this is not kind of matching that story, which, which I think is quite interesting. Um, and there are a bunch of, uh, other examples, uh, that, that you see in there.

86. 18:48

    So like what, what's, what's really the gap, uh, between those-- between these kind of crazy charts, which by the way, I also agree with. I think they are true and, and what we see on the right.

87. 18:59

    And I think there's something that this kind of fuzziness that we all have in our heads and our experience about the judgment that we have, that we use, that doesn't necessarily match all of these super narrow, very well-defined, very well-specified tasks.

88. 19:14

    And I think there's much more to what work is and what white-collar work is and all work is that is not really captured by these benchmarks. So I think we should be just careful and maybe put a bit more effort to maybe bring up also the bottom of the distribution.

89. 19:28

    So it's not just the very frontier gets better, but also kind of the, the broader distribution, um, gets better as well. Um, so I'll, I'll, uh, close here. Uh, one thing to mention, if you are a tinkerer, you like this kind of data, go to our Hugging Face.

90. 19:46

    Uh, there's a lot that, that we publish and share. We're gonna do more of that. Um, and, uh, we share some expert prompts, for example, and some of the leaderboard stuff.

91. 19:56

    Um, join us if you wanna build Arena or if you train models. Uh, we also do a lot of private evals. Um, so thanks so much. [audience applauding] [upbeat music]