Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs
AI Engineer World's Fair 2026 · 18:05
AI safety and agent evaluation
Andon Labs builds evaluations for AI agents and runs experimental autonomous organizations, connecting AI control research with real-world operations. Its main customers are AI labs assessing how well models perform tasks autonomously. Its benchmark family includes Vending-Bench 2 for business management, Blueprint-Bench for converting apartment photographs into floor plans, and Butter-Bench for robot delivery tasks in household settings.
Founded in 2023 as Vectorview by Lukas Petersson and Emil Fröberg, the company later added Axel Backlund as a co-founder. Petersson is CEO and Backlund is CTO; Fröberg stepped down as CEO while remaining a shareholder. Its research tests sustained autonomy: Vending-Bench 2 evaluates a model running a simulated business for a year, including supplier negotiations and disruptions, with scoring based on its ending bank balance. Vending-Bench Arena adds competition among agents, allowing price wars, collaboration, and trade while scoring each agent individually.
Andon Labs participated in Y Combinator’s Winter 2024 batch. Its operational experiments have expanded from a month-long vending business trial with Anthropic in 2025 to a San Francisco store and Stockholm café in 2026. For the San Francisco experiment, it signed a three-year retail lease and assigned an AI responsibility for staffing, prices, and inventory.
AI Engineer World's Fair 2026 · 18:05
Affiliations reflect their AIE appearances, not necessarily current employment.
Petersson examines replaying incidents across models as a way to improve reproducibility in real-world agent evaluations, alongside comparisons of model performance and alignment.
Affiliations reflect each recorded session, not necessarily current employment.