AI Engineer Summit
November 11, 2026Online track4 min read

The harness has to finish the job

Joey Wang opens with the scale of Harvey's internal agent platform: more than one agent starting every minute on average, around the clock, and thousands of agent-written pull requests.

Joey WangHarvey

Day 1 livestream · 1:37:50–1:50:46 on YouTube. The player starts and stops at this session’s boundaries.

Introduction

This reading edition uses a descriptive title; the formal talk title was not verified.

Joey Wang opens with the scale of Harvey's internal agent platform: more than one agent starting every minute on average, around the clock, and thousands of agent-written pull requests. The difficult part was not persuading a model to write code. It was keeping everything around those agents working as more people found ways to use them.

The platform, Spectre, had already escaped its original coding use case. Wang describes people asking data questions through Slack and a marketing colleague bringing a prototype and talk track to an agent to prepare material for a product video. Once a harness can reach the company's tools and run useful work, its users will bring tasks its builders did not anticipate.

That expansion is the point of the talk. A harness becomes shared infrastructure, and shared infrastructure has failure modes that a single successful agent run will never reveal.

1:39:24 ↗

Own what happens after the code is written

At 01:39:24, Wang answers the obvious question: why build an internal platform when engineers already have favorite coding agents?

Harvey's answer is to run those agents, not replace them. The platform supplies company-specific secrets, a real test stack, alerts, tickets, shared policies, and capacity. An engineer can start with a preferred local tool and hand work into the company's environment.

The important workflow continues after the first patch. Wang describes an overnight alert becoming a proposed fix, a real stack being started to validate it, and proof being attached to the pull request. If continuous integration fails, a follow-up loop reads the failure and tries again. By morning, the change is waiting in the same review process as other changes, rather than stranded at the first error.

Human approval remains distinct from generating a patch or enabling automatic merging. Wang's recommendation is to build the follow-up loop if you can build only one piece. Its job is to carry a task through the ordinary friction between writing code and landing a reviewed change.

Projects give that work shared context. Teams record how services should be tested, pin trusted skills, and preserve an audit trail. The choice of coding agent can change without throwing away the team's operating knowledge.

1:43:04 ↗

Make every action callable, then expect agents to call it

The second story starts with an interface rule: if a person can do something in the product, an agent should be able to invoke it programmatically.

Spectre exposed actions through a command-line interface. Starting a task, turning on a follow-up loop, and configuring later workflow steps no longer required a person sitting in the web app. That made the platform usable from laptop agents, agents already inside sandboxes, and CI.

Source slide shows Claude Code, Cursor, ChatGPT Dots, and an in-house orchestrator delegating work through Spectre into sandboxes.

At 01:45:00, several different clients share the same execution layer. The interface is useful precisely because no single agent has to own the whole workflow.

Wang describes coordinating a large task into milestones and dispatching many of them to Spectre, each with its own sandbox. He also describes a Dot collecting the questions Spectre is waiting on and drafting answers for his approval. The platform becomes a tool another agent can use, rather than a destination a human must continually visit.

This also changes what counts as a likely workload. A person clicking through a UI has a natural rate limit. A script or coordinator can create hundreds of tasks before anyone notices the cluster is full.

1:46:50 ↗

Deduplicate the work, then make capacity fair

The first overload came from an alert storm. A release bug triggered an alert for each workspace; a workflow tagged the agent for every alert; every tag started a new task. Wang reports roughly nine hundred agents pursuing the same underlying bug over several hours.

The fix was to join a repeated alert to an investigation already in progress. More agents were not more independent evidence. They were duplicate demand.

The next overload came from a legitimate engineer assigning hundreds of tasks through the CLI. There was no alert storm to deduplicate. The platform needed admission control: accepting a request for work should not mean launching it immediately.

A fair-launch-queue slide lists interactive priority, per-person concurrency limits, queue visibility, and results from a 300-task synthetic burst.

The 1:46:50 slide distinguishes getting a place in the queue from getting a worker. Its performance numbers come from a synthetic 300-task load test, not a claim about every production request.

The queue reserves room for interactive work, limits each person's concurrent tasks, allows borrowing when capacity is available, and shows users where their work is waiting. The objective is that one person, agent, or alert cannot take the whole cluster.

Wang is careful about attributing later improvements: several changes shipped together, so he cannot isolate which fix deserves the credit. The stronger evidence is operational. Bursts grew while the platform became better able to keep serving other work.

Fix one bottleneck and meet the next one

Some latency had nothing to do with model reasoning. Repository checkout was expensive enough to justify keeping local copies on workers and lending them to tasks.

After task execution improved, the pressure moved downstream. Wang reports a sharp increase in company-wide pull requests and longer merge waits. A change must rebase, run CI, and remain compatible with a moving main branch. If main breaks, everyone inherits the failure. Generating more changes does not solve that coordination problem, and the talk explicitly leaves it unfinished.

Unattended automations add another source of demand. They need budgets, expiration dates, and a failure policy; Wang describes disabling an automation after five consecutive failures. A workflow that can wake itself must also know when to stop.

The closing questions concern proof and integration: can each proposed change demonstrate that it works before a reviewer sees it, and how does a team merge safely when agents never stop opening pull requests? Spectre's value lies in taking responsibility for those questions, regardless of which agent wrote the first patch.

Full transcript 271 timestamped passages

Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.

1:37:50>> Let me start with one number from last

1:37:52month. On average, more than one agent

1:37:55has started on our platform every single

1:37:58minute, day and night.

1:38:01And honestly, the agents were the easy

1:38:03part.

1:38:05Hi, I'm Joey. I bu the cloud agent

1:38:09platform we built inside Harvey.

1:38:11After 10,000 PRs, the biggest lesson I

1:38:15can share is this. The thing that broke

1:38:17was an agent. It was everything around

1:38:20them.

1:38:23So what does one submit add up to?

1:38:2610,000 PRs that an agent wrote. And it

1:38:30isn't just Tax team playing with a toy.

1:38:34Last month about 460 different people at

1:38:38Harvey use it. And over the summer it

1:38:42stopped being only about code. People

1:38:44started asking a data question in Slack

1:38:47and getting answers back. My favorite

1:38:51recently someone on our marketing team

1:38:53drop a prototype and a talk track into

1:38:55Slack and ask for a story book for a

1:38:58product video. Once the harness exists,

1:39:02people bring the work that you never

1:39:04plan for.

1:39:06So a promise up front. This is not a

1:39:09talk about models, Evos or leaderboard.

1:39:12This is talk about hardness,

1:39:15odd things around the model,

1:39:18checking out the ripple, holding the

1:39:20key, opening the PR, getting the merged.

1:39:24When we first wrote about Spectre back

1:39:26in April, we put it like this. At some

1:39:30point, a harness become the product.

1:39:33Now, whenever I tell people this, I get

1:39:36the same question. Why on earth would

1:39:38you build this when clock or cursor

1:39:41already exist?

1:39:43My answer, we don't compete with it. We

1:39:46run them. And here's how I see it as my

1:39:49own view. Agents on your laptop are

1:39:52powerful, but they don't scale. You can

1:39:55realistically run five to 10 of them.

1:39:58Pass that you need a cloud. And cloud

1:40:01needs our secrets, our real stack, our

1:40:04alerts and tickets, one set of rules and

1:40:08enough capacity that nobody gets

1:40:10starved. Those single two owns that for

1:40:13us. So we build layer and that layer

1:40:16exactly what broke. So I want to tell

1:40:19three stories today. How we stop

1:40:22competing on writing code. How we let

1:40:24any agent drive and were broke once they

1:40:28did. The first story start with the

1:40:31question we keep hearing.

1:40:34More of our engineers were using cursor

1:40:37and they ask how do I get our secret

1:40:40there? How do I test against our real

1:40:42stack? We could have rebuilt all that

1:40:46inside cursor and then again for next

1:40:48two and the one after that.

1:40:51Instead answer became one sentence just

1:40:55ask cursor to use spectre

1:40:58and that's really the whole idea. People

1:41:00will always have a favorite agent and

1:41:03the favorite keeps changing. So pick

1:41:05yours clock codeex cursor dot slack are

1:41:09your own hands.

1:41:11Start on your laptop push the session to

1:41:14the cloud. Put it back. That's how

1:41:17spectra sits underneath all of that.

1:41:20Because once the code was written the

1:41:22real work starts

1:41:24picture appears day alert fires in Slack

1:41:28at 2 in the morning. An agent picks up

1:41:31nobody typing. It writes a fix, spins up

1:41:35real copy of our stack and post the

1:41:38proof on the PR.

1:41:40When C fails, the follow-up loop reads

1:41:43the failure and pushes another fix while

1:41:45everyone still sleep. By morning the PR

1:41:49sits in one review queue under the same

1:41:52rule as other PR whichever agents wrote

1:41:56it. The owner gets a notch and only once

1:41:59they are working day starts and once

1:42:02approved a agent can automerge it.

1:42:07If you only build one of those stages

1:42:10that I described build a follow-up loop.

1:42:13We switched it on for everyone in early

1:42:15October and that months. Four out of

1:42:18five agent PRs merge into our backend PR

1:42:22had been through it.

1:42:24That's the difference between agent that

1:42:26write code and an agent that finished

1:42:29the job.

1:42:31And then there's the knowledge. Every

1:42:33team has that one person who knows

1:42:35exactly how to test the tricky service

1:42:38which flag you never touch. And that

1:42:41knowledge leaves in their head. of their

1:42:44laptop.

1:42:45So we gave it a hope projects.

1:42:49People write down what matters. Adrian

1:42:51write down what they learn. Team pings

1:42:54the skills you trust and there's a

1:42:56record of who rent what. Any agent

1:42:59working in the project read the same

1:43:01page. My personal view whichever agent

1:43:04you pick the audit trail and the tracks

1:43:07should be the same.

1:43:09The second story is about rule.

1:43:13Back in May, if you want a spectctor to

1:43:15do something, you open a browser tab or

1:43:18tacking Slack. The whole mouse, exactly

1:43:22one task came out from terminal.

1:43:25So, we made a rule. If a person can

1:43:27click it, an agent can call it

1:43:30start a task, turn on the follow-up

1:43:33loop, arm, automerge, publish a page,

1:43:36all from CLI,

1:43:39which means your laptop agent can call

1:43:41it A agent inside the sandbox can also

1:43:44call it. Even CI can call it too. And

1:43:47then people ran with it. By October, the

1:43:51command line was starting about 6500 ts

1:43:54a month. The web app didn't shrink.

1:43:58Everyone else just grew around it. Task

1:44:01started from the terminal were merging

1:44:04and task started in browser. And the new

1:44:06kind of user showed up. people who start

1:44:10hundreds of tasks a month and never open

1:44:12the app at all.

1:44:15And this is where it gets fun. In

1:44:17October, I had a big task. So I bought

1:44:21code session and asked to be the

1:44:22coordinator.

1:44:24It split the tickets into 10 different

1:44:27milestones. Get one for me and the rest

1:44:30to spectre with hundreds of tasks.

1:44:34Each has its own sandbox to validate end

1:44:36to end. Then at the end of the October,

1:44:40our it give everyone tragic dots. Mind

1:44:43now watch Spectre for me. It list the

1:44:46question Spectre is waiting on. Draft my

1:44:48answers and once I say yes, send them

1:44:51back.

1:44:53I actually DM to my friend like that. I

1:44:56just use cursor clock code or whatever

1:44:58to spam spectre. Now

1:45:01well that's the dream, but it's also the

1:45:03problem. A quick detour. When we look at

1:45:07where our task time actually went, the

1:45:10slowest step wasn't model syncing. It

1:45:13was get just checking out every ripple

1:45:15we own took over a minute on average.

1:45:20The fix keep a copy of every ripple on

1:45:23every machine and let each test borrow

1:45:26it.

1:45:27The third story is the one where it

1:45:30broke in mid August. A release goes out

1:45:34with a bug. That bug triggers alert that

1:45:37fires once per workspace. So one bug

1:45:40become hundreds of alerts all landing in

1:45:43one single sack channel. Someone has set

1:45:46up workflow that tag our agent on every

1:45:48alert and every tag started task. On

1:45:53normal day those task started about 50

1:45:55task. That day in about 4 hours they

1:45:59started around 900

1:46:02900 agents all chasing the same bug

1:46:04until that wasn't room left for a single

1:46:06note.

1:46:08The fix was almost boring. A repeat

1:46:11alert now joins the investigation that's

1:46:13already open.

1:46:15Two months later it happened again and

1:46:18this time it wasn't a bot. On October,

1:46:22the engineer did exactly what we've been

1:46:25asking everyone to do. Hand work to

1:46:28Spectre.

1:46:30About 950 test from the command line in

1:46:34about 3 hours. Nothing in our platform

1:46:36can work. The cluster filled up and

1:46:40everyone else work set in line behind

1:46:43one laptop. That's not a user problem.

1:46:46That's our problem.

1:46:49Four days later, we had a real fix. A

1:46:52fail launch queue. Now starting a task

1:46:56gives you a place in line. And the

1:46:58dispatcher started when there's room.

1:47:01People sitting at a keyboard goes first.

1:47:04Everyone gets a fair share and can

1:47:06borrow more than the cluster is quite.

1:47:09And whenever you start a task, you can

1:47:12see our placing line. When we hit it

1:47:14with a 300 task bursting low test, the

1:47:18task lit went from almost two second to

1:47:2070 milliseconds.

1:47:22So did it work?

1:47:25When it's the busiest day of every week

1:47:28back in August, two certain task and

1:47:31about one in hand didn't finish.

1:47:35Then in November 4th, more than twice

1:47:37that and much better in the finish line,

1:47:41but we still have some room to grow.

1:47:45I will be modest, we change several

1:47:46things at once, so I cannot tell you

1:47:49which fix mattered the most, but the

1:47:51birds kept getting bigger and the

1:47:53platform kept up.

1:47:56And just when that was on the crunch,

1:47:58the model neck moves.

1:48:00At the end of October, our dev platform

1:48:04team raised an alarm. PRS across the

1:48:07company had jumped at around 60% in a

1:48:10single month. A merge that used to take

1:48:1315 to 30 minutes was taking over an hour

1:48:16now. And merge Q kept having to be

1:48:19switched off. So now a PR rebase onto

1:48:22main rerun CI and merge if it's green.

1:48:27If main breaks, everyone CI breaks with

1:48:29it. Alterately,

1:48:32we haven't cracked this one yet.

1:48:35But before I go, there's one more thing.

1:48:39While we were busy fixing all it,

1:48:42something else was quite growing. Most

1:48:45of what I've been showing you start with

1:48:47a person asking

1:48:49the fastest growing source of the work

1:48:52on Spectre doesn't. These are

1:48:55automations agent that wake up on

1:48:57schedule on an event and just do the

1:49:00work tri alert review the PR write the

1:49:04post.

1:49:06Last month about 110 people had

1:49:09automation running and we learned to

1:49:11give them a guardrail on day one. If one

1:49:14fails five times in a row, it switch

1:49:17itself off and it can carry a budget or

1:49:21expiry date because when no one's

1:49:23watching, an agent can start thousands

1:49:26of tasks, too.

1:49:28So, here's what I'm chasing next, and I

1:49:31don't have an answer yet, honestly.

1:49:34What if every Slack channel had its own

1:49:36agent that never locks off? One that

1:49:39follows competition and start work on

1:49:41its own. What if every pure proof it

1:49:44works before a human even look at it?

1:49:48Today only one merge in 20 carries that

1:49:51kind of proof.

1:49:53And how do you merge when agents never

1:49:56stop opening PRs?

1:49:58Ask me again six months. I will tell you

1:50:01what broke. If you're building something

1:50:04like this when you work, here's three

1:50:06things I do again. Oh, what happens

1:50:09after code is written. Make Eric action

1:50:12something an agent can call and cap what

1:50:16an agent, human or alert can start

1:50:19before they find out what they can do

1:50:21with it.

1:50:23And one last thought, that's my own

1:50:26view. Engineers can write code with

1:50:29whichever agent they like. The work

1:50:32that's lost is the harness that runs it,

1:50:34checks it, and get it merged.

1:50:38Thank you so much for watching.

1:50:41Come find me on linking if you are

1:50:43building similar things.