Joey Wang

Every team picking a model right now is reading someone else's leaderboard. We stopped, and the reason was an accident of how we shipped.

Harvey released its cloud agent platform to our own engineers before we shipped long-horizon agents to lawyers. Internally it has now driven 7,000+ merged pull requests across our repositories. That internal deployment was not just a safety net — it became the eval set. Once you have thousands of real agent runs on your own workload, you can measure a model against the work you actually do, instead of trusting a public benchmark that was never scored on your tasks. That measurement is what makes it rational to move to an open model, or to fine-tune one, rather than simply hoping it holds up.

This talk covers the infrastructure that made that possible, and the four places it broke first:

- Durable runs: what happens to a long-horizon agent when the worker dies mid-task, and what we built so the run survives it. - Per-user agent identity: we started on a shared service account and lost the ability to attribute what an agent did. Retrofitting identity was expensive. - Information boundaries: enforcing ethical walls and leaving a reviewable approval trail when an agent acts on a user's behalf. - Context engineering: the case where adding context back made agents measurably worse — a result no external benchmark would ever have surfaced for us.

I will show the failures next to the wins, with real numbers and the denominators included, and demo the platform live. Every pattern here is architecture and engineering method, not a product walkthrough.