Arithmetic

Arithmetic develops cybersecurity benchmarks and training data to improve AI models’ ability to reason about vulnerabilities. Its research targets a gap between gathering information about a system and finding the sequence of actions needed to exploit it. The company’s stated goal is to make AI defenders capable of outperforming attackers.

Its Mask Off benchmark focuses on access control. Researchers build blackbox environments from vulnerabilities they discover in open-source software; models receive tools to investigate but no source code, internet access or advance knowledge of the flaw. Deterministic graders check intermediate steps and whether a model can perform an action beyond its user privileges, showing where an attempted solution breaks down.