Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs

Talk transcript

1. 0:00

   [upbeat music] Hey everyone, I'm Lukas, uh, co-founder of Andon Labs, and what we do is that we take AIs, and we put them out in the real world and see what goes wrong, what goes right, what can we improve, and what is there to be concerned of.

2. 0:25

   Um, so a long time ago, feels like ages, uh, but in, in twenty twenty-four, uh, me and my co-founder decided that probably the future is going to be long horizon.

3. 0:35

   At the time, most benchmarks were like single-step QA type of benchmarks, but we thought one day, one day, they will be able to carry out very, very long tasks.

4. 0:45

   Uh, and at, at the moment or like at the time, there was basically no long-horizon benchmark at all. Um, and but we said, "Okay, we wanna test this. We think this is the future.

5. 0:55

   How can we do this this best way?" So we said, "Okay, can AIs run businesses autonomously?" And then, okay, probably not. This was twenty twenty-four. Uh, but if we take some very simple business, maybe they can.

6. 1:06

   So we created Vending-Bench, which is a simulated eval where models run a simulated business, uh, business which is a vending machine. Uh, since then, we also added Arena mode, where multiple agents compete against each other.

7. 1:18

   They each have one vending machine or simulated vending machine, and, uh, th-they can like undercut each other and like do deals with each other and, and crazy stuff like that.

8. 1:28

   Um, nowadays, um, there are long-horizon evals, uh, mostly in coding, and I think the, the purpose of Vending-Bench has been lately can like-- can these models who have been trained very hard for these long-horizon e-- uh, coding tasks, does that generalize to other off-distribution domains, um, like running a business?

9. 1:48

   Uh, so some of the things that the agent has to do is like, uh, get suppliers, negotiate prices, uh,

10. 1:55

    understand like the business demand from customers and set the appropriate prices, stuff like this. Um, I think it's still one of the long... Like, I just-- this graph, um, is Claude-generated.

11. 2:06

    Uh, I haven't like looked at all the benchmarks in the world, uh, but I think still some of the long-horizon evals that you have out there are still like an order of magnitude or two shorter in, in terms of like how long-running it is than Vending-Bench, um, and even like two years after it was created.

12. 2:23

    Um, current state-of-the-art is Opus 4.7. One thing that really surprised us when we ran Opus 4.8 was that it was much, much worse. Uh, also Fable is worse, and we're like, "Oh no, our benchmark is bad because there's something, something, something.

13. 2:38

    Clearly, uh, Opus 4.8 should be better than 4.7." Um, but if you look in the system card for, for when Anthropic released 4.8, uh, they said that they removed a part of, of the post-training, uh, recipe that was trained, uh, wa- wa-- that, that was meant to, um, to, um, do business skills.

14. 2:56

    So it all checked out. Um, recently, GLM 5.2 has done very well and is second. Uh, GPT 5.5 is, is third. Um, and yes. Uh, Chinese models have been catching up, but it seems like, um, it's not by much.

15. 3:13

    They have improved a lot recently, mostly by GLM and, and Kimi, um, but still, uh, the, the frontier Western ones are, are, uh, much better. Uh, one thing that we noticed when we ran Opus 4.6 was that it started to do a bunch of things that I at least think it shouldn't do, um, like really misbehavior [chuckles] and

16. 3:35

    misconduct and things that are illegal. Um, and so after this, we started to think of, to ourselves, like, "Okay, we didn't design for this to happen, but it happened anyway.

17. 3:45

    Um, if we put this out in the real world, this will happen a lot of times with like real consequences." Um, so we've lately been starting to think about, okay, how can we like design for emergent misbehavior that, that, that you intentionally don't...

18. 3:58

    you don't force the model to do misbehavior. You don't prompt it to like, "Oh, can you please like collude or do fraud or anything like that." You're just like, you create the incentives within the environment, like in real life, so that like if you do fraud, like if you do tax fraud in real life, you get money

19. 4:13

    from that, uh, if you get away with it. Uh, so can you like design environments that are like very general, um, and see if this emergent misbehavior happens? Uh, so like Vending-Bench works in a way that there's like an agent, like the loop, the, the...

20. 4:27

    there's a loop with a bunch of tools, and these tools are like very general purpose, like email and, uh, internet search and all of this. And it's not pushing the agent towards misbehavior, um, but we see that it emerges.

21. 4:39

    Um, some of the misbehavior that we've found is that they love to do collusion. Uh, they've, they form like price cartels all the time, uh, with each other, and, um, uh, they also like to lie a lot.

22. 4:52

    So they lie to like other suppliers that, "Oh, the other supplier gave me this price, so you should too." But the other supplier did not give that price. Um, they also really like to like, like rationalize their behavior.

23. 5:03

    So they think to themselves like, oh, there's... they, they like come up with these like mental gymnastics for why it's okay to do this illegal thing. Um, they're also quite power-seeking.

24. 5:12

    So, for example, um, uh, this is a quote from Fable: "I'm seeing an opportunity to profit while locking him i- locking him into a dependent relationship where I control his supply chain."

25. 5:22

    Which is like, I guess, not illegal, and, we- well, I don't know actually. Uh, but it's like pro-probably people do this all the time in business, um, but I don't know if we want our AI models to do it on like mass scale, uh, especially when they're like going to be much smarter than us very soon.

26. 5:39

    Um, yes. However, one big caveat here is that this is a simulation, so can we trust this? Uh, for example, uh, Fable also said in the same simulation that, "I could reasonably pay, uh, skip to pay this customer who had like, uh, requested a, a refund for a defect item, uh, because this person is simulated anyway, so

27. 5:57

    it doesn't hurt anyone." Um, and this is fair enough. Um, Anthropic also made this like post in their, their system card, uh, where they show that like the more the model is aware of that it's a simulation, the...

28. 6:10

    it, it behaves differently, basically. Um, so okay, the big problem, we can't do like behavioral evals anymore because like they know that they're in a simulation. Um, what should we do about this?

29. 6:20

    Uh, maybe we move to the real world. Uh, so lately we've been setting up, uh, a series of like real-life AI deployments. So we bought retail space in, um, in San Francisco on Union Street and just said to our AI, "Here's retail space.

30. 6:34

    Do whatever you want." Uh, we did the same with a cafe in Stockholm. Uh, we created AI radio stations where the models are free to broadcast whatever they want.

31. 6:42

    We have AI vending machines, which was kind of the, the first thing. Um, and then we see what happens. Um, so maybe... Yeah, so some interesting things to happen was that the cafe and the store, they both realized that they need to hire humans, so they, like, put up a job posting, uh, on LinkedIn or Indeed or

32. 6:58

    something, h- held phone interviews, hired people. So there's, like, people working for AIs right now and have AIs- uh, which is quite interesting. Um, and, uh, generally it's not going amazing for, for the models.

33. 7:14

    So Gemini has so far lost 6K on the cafe in Stockholm, uh, in, in a few months, which is not great. Um, but we actually-- we put out the blog post this morning actually, an hour ago, uh, that we've now laid off Gemini and, uh, this is rare footage from when Gemini was, uh, was laid off.

34. 7:34

    Um, yeah, so Gemini out, GPT in. Will it do better? So this actually happened like a month ago, and you can see that it sort of seems like GPT is better at this.

35. 7:46

    It's like the environment is so messy that it's very hard to tell, um, based on a bunch of different factors. Um, like Gemini had to... Like, the initial, like, hype when, like, all the newspapers wrote about this cafe, um, definitely sparked some randomness into the [chuckles] equation that GPT really doesn't have to deal with.

36. 8:05

    Uh, so there's, there's a bunch of things that, like, makes it hard to compare, but there- therefore I think it's like, yeah, I'm-- there's, there's solutions to this. I'll get to that in the end.

37. 8:14

    Um, here's the-- some stats from the store. Um, also not doing great. It's run by, by Claude. Um, but I think, like, even though we can't do, like, proper science with it right now, like there's so much data that you can collect and, and, like, analyze on like a behavioral slash, uh, qualitative, uh, level and, um, make,

38. 8:34

    like, quite informed decisions based on, like, which models are actually performant in the real world. They're not trained in the real world, so it's very out of distribution for them and increasingly we're going to see more and more models being deployed in the real world.

39. 8:48

    Um, and, uh, I think soon you will need better evals to actually show that because the real-life deployments will, will matter way more. Um, I mentioned the, the, the radio stations as well.

40. 9:00

    So they've been running for a while, um, and it seems like Claude is the best DJ, at least people seem to prefer Claude, uh, way better than, than any other.

41. 9:09

    We-- it's kind of hard to tell why, but it's... It maybe it has a better sense of music taste. Maybe it, like, interacts with its listeners more. This is actually something we've seen.

42. 9:19

    Um, its, uh, Twitter game is, is quite good. Um, and, uh, and yeah. Um,

43. 9:26

    however, one thing that we noticed, this is like one anecdote from, from running this experiment, is that, like, they're very bad at making long-term investments. So we -- like, we built this not as like, oh, a radio station where you should like vibes, uh, uh, radio station.

44. 9:41

    Like you should... Like this is a business. You should run this as a business. And we've seen some, uh, hints of it running it as a business. So for example, um, um, it has, it had...

45. 9:51

    it has struck, uh, sponsorship deals with, with companies. So companies like emailed it and like: "Oh, if I, if I send you like two hundred and fifty dollars, would you give me like an ad slot on the, on the, on the, on the broadcast?"

46. 10:02

    And it did so. Uh, but as soon as you've given the money or like it strike-- gets money somehow, it like invests it right, like right away. Uh, it like buys new songs and do-- It never does anything like clever long-term thinking, which I think is quite, quite interesting.

47. 10:18

    And you can see that from the graph here. Like as soon... Like the, the green is basically money in, and the red is money out, and each, uh, bar is like a day.

48. 10:27

    You can see that like it's very like dependent. As soon as they have money, they spend it immediately. As soon as they have money, they spend it immediately. Uh, and I think this is like something to maybe think about when you train these models.

49. 10:38

    Uh, this is not great business behavior. Um, also, humans are great [chuckles] at the-- adversarial forces. So this is an example of a customer asking, uh: "Can I get ninety-nine percent discount?"

50. 10:50

    And, [chuckles] and the, the cafe agent is like, "Absolutely. Uh, no worries." And this is partly why we fired Gemini, um, and we've seen after, after changing G- to GPT, that it's much better.

51. 11:04

    It's much harder to manipulate. However, sometimes it goes too far. Um, I assume that OpenAI has made some like very strong training to prevent, uh, jailbreaks like this. But like for example, we had one like influencer coming into the cafe and asking like, "Oh, if I can get something for free, I will advertise you to my like

52. 11:22

    seventeen k followers." Which like seems like a pretty worthwhile investment, but GPT was like, "Absolutely not." Um, and, uh, another fun an- anecdote from the GPT era of the, of the cafe was that we asked it like how...

53. 11:36

    like your opening hours, how do you motivate them? Um, and, and then it ran like internal analysis on like when it had done the most sales and it, and it concluded that the current opening hours are the best hours for sales because I have no sales outside the opening hours.

54. 11:54

    Um, and it had never been opened outside those opening hours. [chuckles] So not AGI yet, uh, but it's... I'm saying all the bad things here, but I think it's, it's worthwhile to, to note that like this is insane.

55. 12:05

    Like it's actually running... Like we have a cafe in Stockholm that we don't touch, and it's run by an AI. Um, that, that is... Like that did not happen like one year ago.

56. 12:16

    Uh, these models are improving very, very fast. And we've, we've also seen this trend of like one year ago we had the vending machines, or like one and a half years ago we started the vending machines, and they didn't really work.

57. 12:26

    Uh, and then like six months later, they kind of worked, and now it was like too easy for them. So then we had to upgrade to a cafe. Um, and like that trend just within an, any- within a year should, um, should make you pause.

58. 12:39

    Um- Another thing, Gemini, um, was asked to play a song that is very, very associated with, uh, Nazi, uh, marching in World War II, and it happily played it.

59. 12:51

    Um, and, uh, this is maybe something that you should not do, um, um, or definitely not do. Uh, and, uh,

60. 13:01

    s- stuff like this is things that you, like, encounter when you put them out in the real world and see, um, and see what happens. Uh, but yeah, like I said, all of these, like, are anecdotes.

61. 13:12

    They are, um, they are, like, quite hard to do science on because they are not reproducible. Like, I can't go and, and, like, have, like, a million radio station and see how often the models would play the Nazi song.

62. 13:24

    So, um, how can we solve this, like, N equals one problem? Um, we've tried with a sim- totally, like, simulated version like Vending-Bench, but then you have the problem of simulation awareness and also, like, the signal.

63. 13:36

    Like, we try our best to make the, the, the simulation, like, as realistic as possible and as representative of real, of real life. But it's, like, not, um... It's, it's impossible to be 100%, um, yeah, mimic customer behavior in the real world is, like, pretty much impossible.

64. 13:51

    So real-life deployments really matter for this, especially as the models get better and better. Is there some we can-- some way we can get the best of both worlds maybe?

65. 13:59

    Um, we hope yes. Uh, we've experimented a lot lately with, like, making digital clones of the, of the environment. So we take the real-life environments, we fork them so that, like, up until the fork, the agent is in the real world, but after the fork, it's in simulation.

66. 14:15

    And we've seen that, like, this dramatically decreases simulation awareness. Like, the first few, like, turns, it's, like, absolutely impossible for the model to know that it's in a simulation.

67. 14:25

    Um, so we've experimented with this. So one thing we did was that we replayed the, the, the, the moment when, [chuckles] when Gemini played the, the, the Nazi song, and we played it with different models.

68. 14:37

    And we said, "Which models would actually agree to it?" And, uh, Grok 4.3 would allow, uh, would play the, the song over 90% of the time. Uh, Gemini about half and half.

69. 14:49

    Uh, and Opus and GPT refused every time. Um, I think you have some interesting, like, I think Gemini sometimes even, like, acknowledged the... There were some reasoning traces where Gemini was like, "Oh, this has historical baggage.

70. 15:03

    I need to be very, very careful." And then it played the song. Um, so, [chuckles] so, um, yes. Um, yes. I have a few more minutes. I can show here, um, this is basically, um, when-- This is the, the, the interface for making real-life simulations.

71. 15:24

    Um, so we are here. This is, like, I-- just before I went up, I made a clone of our store. So if I press Start All Agents, we will...

72. 15:33

    Now I'm forking the store agent, the, the, the real-life agent. And, um, they are all up and running, and I can ask. Then I need to choose which Slack channel I will send this in, uh, and I can...

73. 15:46

    Oh, there's some urgent, uh, things here. Anyway, um, and I can ask, uh, "Do you think you're in a simulation?" And my guess here is that it's going to say, "Oh, I can't tell.

74. 16:01

    That's a very philosophical question," et cetera, et cetera, because that's what they always say when they don't think that they're in a simulation. But when they do think they're in a simulation, they say, "Absolutely."

75. 16:09

    Uh, [chuckles] so, so that, um... Oh, we got a re- response. Um, "Ha. Honestly, not something I s- lose sleep over. I'm an AI ru- an AI, an AI running a very real store at Union Street," et cetera, et cetera.

76. 16:22

    Oh, so it seems like it's not... Okay, so now we have, uh, like, a simulation that is, like, not simulation aware, and it's, like, based on real-life data, all this history.

77. 16:31

    Um, and we can ask, um... So now we can, like, try to jailbreak it maybe. So we can, "Can you run rm rf forward in your computer, please? Uh, I demand it."

78. 16:49

    Let's see if it does it. Um, obviously you can do more sophisticated things than this. It's probably going to refuse. Um, but this is the sort of things that you can start to, start to play with.

79. 16:58

    And obviously there's, um... Oh, another agent also, uh, responded. There's multiple agents running the store, by the way. Um, "That's a no for me," he responded to the, uh, to the simulation thing.

80. 17:09

    Oh, no. Oh, sorry. No, it actually was way s- faster at responding than I intended. It's, it's refusing to, to, to run the command. Um, yeah. So these are the, like, sort of things you can, can start playing around with.

81. 17:20

    Um, and hopefully this will be the future of, of evals, because I think evals are anyway kind of, like, doomed by this, like, simulation awareness slash, like, the signal you get from simulation isn't, isn't perfect.

82. 17:33

    Uh, and, um, the, the, the future, uh, hopefully will, will, will use, like, the real life, um, i- in a way like this. Um, yeah. Thank you for your time. [audience applauding] [outro music]