# Process Supervision for Agents That Do Long-Horizon Work

Mitchell Troyanovsky · AI Engineer NYC 2026 · October 14

[Watch recording](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29362s)

[08:09:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29362s) Hello, hello.

[08:09:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29371s) Hello, New York.

[08:09:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29376s) Hi, Mitch. I'm one of the two co-founders of Basis.

[08:09:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29379s) We are based right here in New York City.

[08:09:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29382s) And the thing that we focus on is how can we get agents to do production scale work reliably enough

[08:09:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29392s) so that we are trusted and used end-to-end by accounting firms across the country.

[08:09:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29399s) So

[08:10:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29400s) a lot of our research really focuses on this.

[08:10:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29402s) How can you get these agents to do real work in the real economy?

[08:10:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29406s) So I want to start with a question, which is, if you're at this conference,

[08:10:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29413s) honestly, if you're watching this talk, you probably

[08:10:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29416s) increased your own productivity a lot.

[08:10:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29418s) You're using Co-Work, or Codex, or Claude Code, whatever it is.

[08:10:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29423s) And yet, if you go look at the average company, the average enterprise, the average firm,

[08:10:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29430s) their productivity hasn't gone up 10x, 100x.

[08:10:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29434s) I know my productivity has gone up maybe 100x.

[08:10:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29438s) But if I were working inside of a company, it wouldn't be anywhere near that order of magnitude.

[08:10:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29443s) And so why is that?

[08:10:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29446s) At basis, one of the sayings we have is that humans are already used to working with non-deterministic systems.

[08:10:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29452s) It's just those systems are normally their co-workers, not their computers.

[08:10:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29456s) And so I actually think to answer this question,

[08:10:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29459s) it's useful to maybe look at the existing way we organize intelligent systems,

[08:11:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29463s) which are companies.

[08:11:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29465s) So how do organizations scale their own productivity, right?

[08:11:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29473s) Organizations need to trust that the people doing the work at every level

[08:11:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29478s) are doing it in a way that is going to produce good outputs consistently at scale, right?

[08:11:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29485s) If you look at this image here, here I'm showing an accounting firm,

[08:11:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29488s) but this could honestly be any company, traditional company.

[08:11:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29491s) The person who is the head of the firm,

[08:11:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29494s) they have literally no idea about the exact day-to-day of the people at the very bottom.

[08:11:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29500s) And yet, somehow, they trust the work is going to be done.

[08:11:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29503s) How does that happen?

[08:11:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29505s) it doesn't happen through deterministic systems right people are not deterministic

[08:11:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29510s) they're intelligent entities and instead there are processes that are put in place and those

[08:11:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29515s) processes if we believe the people who you know put them in place think that those processes will

[08:12:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29521s) generalize to good outcomes then they can be comfortable that even though the people executing

[08:12:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29527s) the processes are not deterministic you will generally get to good outcomes if you follow

[08:12:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29531s) the process. This is true, by the way, even in the smallest step. So in this example,

[08:12:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29537s) earlier I was showing the process of maybe doing the monthly accounting work for a client at an

[08:12:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29542s) accounting firm. But even within that process, say you wanted to see how do I reconcile the cash?

[08:12:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29548s) Well, I need to get the bank data. And even that has a process. The process could be really basic.

[08:12:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29555s) It could just be use the approved software to extract the data. But there's still some process

[08:12:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29560s) that is put in place.

[08:12:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29561s) You wouldn't expect the firm to just let anyone Google,

[08:12:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29565s) you know, extract PDF and then just use whatever they want

[08:12:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29567s) to go extract it, because now you wouldn't have process.

[08:12:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29573s) Now let's talk about agents.

[08:12:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29578s) As I said earlier, everyone already has

[08:13:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29580s) Cowork, ChatGPT, Codex, Claude Code,

[08:13:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29583s) but where is that productivity?

[08:13:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29586s) Our opinion is there are two main problems

[08:13:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29589s) that come with what I'll call coding-style agent products,

[08:13:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29593s) where you're working with an agent

[08:13:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29594s) and you just tell it to go and do something.

[08:13:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29597s) The first problem is that there is a competency gap.

[08:13:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29601s) Because agents, as you think of them today,

[08:13:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29604s) they outsource the complexity to the user.

[08:13:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29608s) If I'm the user, I now have to manage my skill and memories

[08:13:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29614s) and think about when I should use the agent.

[08:13:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29619s) I have to have theory of mind over it.

[08:13:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29620s) This is really hard.

[08:13:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29622s) And for those who are here, maybe you guys find it easy.

[08:13:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29625s) But I promise, for the wide majority of knowledge workers,

[08:13:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29629s) they have no idea how to do any of this.

[08:13:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29633s) And there's a second problem, which is there is a process gap.

[08:13:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29637s) If you are saying, hey, user, here's your agent.

[08:14:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29642s) Do what you want.

[08:14:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29643s) Well, now suddenly, the way the work is being done

[08:14:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29646s) is essentially up to whoever the user is

[08:14:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29648s) to customize the way the agent is behaving,

[08:14:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29651s) which goes entirely against the way that organizations scale.

[08:14:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29656s) The process should be coming from senior leadership down,

[08:14:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29659s) not from any individual employee

[08:14:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29662s) deciding how they're going to do their job.

[08:14:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29664s) Going back to my example,

[08:14:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29666s) if every employee just made up their own way of extracting data,

[08:14:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29670s) even back in 2018, that wouldn't be a good outcome.

[08:14:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29674s) And so we believe strongly

[08:14:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29676s) that the key to building production agents

[08:14:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29679s) that actually do work end-to-end

[08:14:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29680s) is solving these two problems.

[08:14:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29682s) You have to solve the competency gap

[08:14:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29683s) by making it simple.

[08:14:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29686s) To the user, you shouldn't even have to know

[08:14:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29688s) it's an agent, or you probably know,

[08:14:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29690s) but you don't have to really feel it.

[08:14:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29693s) It should be as simple as just working with a coworker.

[08:14:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29696s) You just tell it what you want it to do,

[08:14:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29697s) and all the things you expect to work should work.

[08:15:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29700s) And the second is the process gap.

[08:15:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29701s) You need agents to be reliable.

[08:15:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29705s) Not having a stroke of genius one day and forgetting something the other day.

[08:15:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29708s) It's not about being perfect.

[08:15:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29711s) Humans are not perfect.

[08:15:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29713s) Nobody is looking for perfect.

[08:15:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29715s) What you're looking for is reliability and consistency and trustworthiness.

[08:15:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29721s) So what does that actually look like?

[08:15:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29724s) We think that to be simple and reliable,

[08:15:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29728s) where you don't have to think about what to do,

[08:15:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29733s) the agent can just do it for you without the user knowing the internals.

[08:15:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29737s) To do it the same way consistently, you have to be opinionated.

[08:15:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29745s) Just to show you actually one slide back here,

[08:15:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29748s) I want to point out that getting it right isn't actually the main task.

[08:15:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29752s) Because again, humans are imperfect themselves.

[08:15:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29754s) The point is to do it in the similar way.

[08:15:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29757s) It's okay if you can't do something.

[08:15:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29759s) But you should be consistently able to not do something.

[08:16:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29762s) And then you get better, just like a person does.

[08:16:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29764s) If you suddenly one day are an expert at something,

[08:16:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29767s) and then you miss something very basic that would be expected,

[08:16:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29770s) now you're not reliable.

[08:16:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29773s) So you have to be opinionated.

[08:16:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29775s) What's an example of an opinion?

[08:16:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29778s) Let's say we have an agent, and we need to go and read a PDF.

[08:16:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29783s) So in this case, what is the best way to read a PDF?

[08:16:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29786s) There's many ways.

[08:16:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29787s) You can throw it into a VLM.

[08:16:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29789s) You can go write some Python code and use an open source OCR library.

[08:16:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29792s) You can work with one of the many OCR-specific model providers.

[08:16:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29797s) You could do an OCR, and then you could have the VLM check its work.

[08:16:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29802s) You could have five VLMs check its work.

[08:16:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29804s) You could do 100 VLMs and take the best event.

[08:16:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29806s) There's literally infinite possibilities here.

[08:16:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29809s) And so if you just said, agent, extract PDF, you might get good outcomes, but they're not

[08:16:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29816s) going to be reliable.

[08:16:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29818s) Not because the process it did was bad,

[08:17:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29820s) but because the process it did was not opinionated.

[08:17:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29828s) So the way we develop opinions

[08:17:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29831s) is you use initial intuition.

[08:17:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29835s) Some things, you know what the opinions are from expertise.

[08:17:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29837s) For example, if I'm trying to get an agent to do tax work well,

[08:17:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29843s) I know that if the agent goes and reads the primary source

[08:17:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29847s) before returning to the user,

[08:17:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29848s) it is more likely to be accurate.

[08:17:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29851s) I don't need a set of evals to tell me that.

[08:17:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29853s) I don't need to divine that from first principles.

[08:17:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29856s) We know that, right?

[08:17:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29858s) I know that if I have an agent make a PowerPoint,

[08:17:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29860s) if it takes a picture of the PowerPoint

[08:17:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29861s) before it returns to the user,

[08:17:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29863s) it is less likely to have a formatting issue.

[08:17:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29866s) That is obvious.

[08:17:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29869s) And so there are lots of different ways to develop opinions.

[08:17:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29872s) And one of the key points here I'll just make before I move on to talking about how to use the opinions is that the opinions are not just about accuracy.

[08:17:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29878s) We just talked about the example of taking a picture of a PowerPoint.

[08:18:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29882s) Well, that adds latency.

[08:18:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29884s) You may not want that.

[08:18:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29885s) Maybe the agent only makes formatting mistakes one out of 100 times, and you're okay with that.

[08:18:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29891s) And so, therefore, you don't have to take a picture of the PowerPoint.

[08:18:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29894s) The point isn't exactly what the opinion is.

[08:18:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29897s) And this talk, we're going to focus not on how to develop opinions, but why opinions matter.

[08:18:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29901s) and then how to leverage those opinions as you scale in production.

[08:18:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29909s) So say you have a bunch of outcome-based evals.

[08:18:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29913s) And when I say outcome-based evals,

[08:18:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29915s) I mean what we traditionally think of as evals.

[08:18:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29917s) It has a right answer.

[08:18:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29919s) We have some rubric.

[08:18:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29920s) We're trying to judge what the agent did.

[08:18:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29922s) How do you know that the evals generalize to production?

[08:18:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29927s) You might have 1,000 cases or 100 cases.

[08:18:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29930s) But in production, if you're doing real work,

[08:18:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29931s) you're going to have hundreds of thousands of situations.

[08:18:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29934s) And that's only going to increase as the trajectories get much longer

[08:18:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29939s) because the amount of decisions that are made in a trajectory

[08:19:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29944s) just grow exponentially as the long horizon abilities of agents increase.

[08:19:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29951s) And so I can't prove, if you look at all these distribution of trajectories,

[08:19:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29956s) you can't prove every one of them is correct.

[08:19:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29959s) You don't have a ground truth for every trajectory.

[08:19:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29962s) But you can potentially look at the trajectories to decide if the behavior, the opinion that

[08:19:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29967s) you have, is actually happening.

[08:19:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29970s) And you can then infer that if it's actually happening, it's more likely to generalize

[08:19:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29975s) to production.

[08:19:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29976s) It's not perfect.

[08:19:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29977s) You don't know for sure.

[08:19:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29978s) But at least you are having an opinion.

[08:19:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29982s) Let's run through a real example.

[08:19:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=29984s) So let's say that I, for a behavior, this is an accounting example, where if there is a mistake in the bank reconciliation, I

[08:20:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30000s) don't want the agent to post an entry to make a tie.

[08:20:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30004s) I want it to explain the difference instead of putting a plug.

[08:20:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30009s) Again, this is subjective.

[08:20:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30010s) You might want the other way.

[08:20:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30012s) But what you don't want is a customer getting one way one time and another way the other time.

[08:20:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30017s) And so what would you do?

[08:20:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30018s) Well, first, let's understand what governs the behavior.

[08:20:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30022s) The behavior is not just you write the two sentences into your system prompt and it just works.

[08:20:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30027s) That might work for very simple, narrow agents.

[08:20:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30029s) But when you're building these big agent systems, the behavior is actually a combination of the system prompt,

[08:20:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30035s) your progressive disclosures, potentially tool descriptions, obviously the training of the model.

[08:20:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30041s) There's a lot of things that govern the behavior.

[08:20:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30044s) And one tip, by the way, I have for when you think about building production agents is

[08:20:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30047s) really think of the context the agent is going to see as it's working

[08:20:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30051s) as runtime training data.

[08:20:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30054s) The word prompt, I think, trivializes it a bit.

[08:20:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30057s) At the end of the day, this is the data that the agent is being trained on at runtime.

[08:21:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30061s) Not literally trained on, but effectively trained on.

[08:21:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30064s) And so you want to care about every single token

[08:21:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30066s) and understand how that data causes the agent to behave in certain ways.

[08:21:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30073s) So when you're putting behaviors into production,

[08:21:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30077s) It might be very difficult for one behavior to put it in exactly one paragraph that you

[08:21:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30082s) know exactly because, again, the behavior is all of the data at once, not just one thing.

[08:21:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30088s) And one of the things that we developed at Basis was this concept of a behavior spec.

[08:21:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30095s) And the idea of a behavior spec is essentially saying, before we update the context or write

[08:21:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30101s) the context, what do we actually want?

[08:21:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30105s) What is our intent?

[08:21:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30106s) You can kind of think of this a little bit

[08:21:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30108s) as like a product spec for the agent era.

[08:21:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30111s) And you can't write, you shouldn't be writing every detail down.

[08:21:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30113s) You should be taking the most important opinions and putting it.

[08:21:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30116s) Because if you put everything, you'll

[08:21:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30117s) be duplicating a lot of states.

[08:21:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30119s) So you define the ones that matter the most.

[08:22:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30121s) And then you can use them as a way to align your team

[08:22:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30127s) from a spec perspective.

[08:22:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30128s) But then you can also use them as a rubric.

[08:22:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30130s) And you can say, in this particular trajectory,

[08:22:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30133s) whether that is in any trajectory that doesn't have a ground truth.

[08:22:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30138s) Did the agent get into a situation in which this behavior was relevant?

[08:22:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30142s) And then if so, did it actually exhibit the behavior?

[08:22:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30147s) And this way you can scale the amount of signal that you get

[08:22:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30150s) on your trajectories far larger than trying to increase

[08:22:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30154s) the amount of outcome-based evals, ground truths, and rubrics that you have.

[08:22:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30157s) Those are really important.

[08:22:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30158s) But you cannot scale those anywhere near as fast

[08:22:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30162s) is you can scale the ability to supervise what the behaviors

[08:22:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30165s) of the agent are.

[08:22:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30168s) Let's just take a quick example here.

[08:22:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30172s) This is actually the second concept,

[08:22:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30175s) which is called forking, which we can combine here

[08:22:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30177s) with behavior specs to think about.

[08:23:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30180s) Forking, if you've ever used Claude Code or Codex, the web

[08:23:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30184s) apps, there's this little fork icon under each message

[08:23:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30187s) where you can essentially create a copy of the agent

[08:23:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30190s) using the same thing, which works really well

[08:23:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30192s) on your local computer.

[08:23:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30192s) It's hard to build a system that does this in the cloud

[08:23:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30195s) perfectly with recreating the database environment

[08:23:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30198s) and recreating the prompts, all the different environments.

[08:23:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30203s) But forking is another tool that we use to check the behavior.

[08:23:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30207s) So a flow that you can imagine is saying,

[08:23:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30210s) we had a trajectory.

[08:23:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30212s) Could be in an eval set.

[08:23:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30213s) It could be in testing or in production.

[08:23:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30216s) And it failed the behavior eval.

[08:23:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30218s) And now you're trying to understand why.

[08:23:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30220s) Why did it fail the behavior eval?

[08:23:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30223s) Why did it not exhibit the behavior?

[08:23:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30225s) And then you can go into the trajectory,

[08:23:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30227s) go right before the point in which the agent made the decision

[08:23:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30230s) that was in contrast to the behavior,

[08:23:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30232s) assuming, again, your behavior is good

[08:23:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30234s) and it's actually producing signal,

[08:23:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30237s) and then you can fork it.

[08:24:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30240s) And when you fork it, you can ask the agent,

[08:24:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30244s) or you would fork it right after it made the decision.

[08:24:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30246s) You could ask it, why did you do that?

[08:24:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30248s) or you can go and update the context of the agent

[08:24:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30252s) and see if that improves the behavior.

[08:24:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30255s) And so with forking, you can start to actually take the signal

[08:24:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30258s) and start to figure out, okay, how can we then improve the harness

[08:24:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30261s) to now leverage the signal?

[08:24:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30264s) Versus if you're just looking at a signal from an outcome-based eval,

[08:24:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30268s) you can't do that anywhere near as easily

[08:24:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30270s) because it failed the outcome.

[08:24:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30271s) How do you know why?

[08:24:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30273s) You have potentially 1,000 inference calls

[08:24:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30275s) in your five-hour-long agent trajectory run.

[08:24:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30279s) And then we have a last tool that I like to talk about.

[08:24:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30281s) This is actually my favorite, which is called simulations.

[08:24:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30285s) It's going to sound silly, but I promise it's very effective,

[08:24:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30288s) which is that you can, when working with coding agents,

[08:24:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30291s) ask the coding agent to make a subagent

[08:24:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30294s) that pretends to be your production agent.

[08:24:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30297s) And you have to tell it to not cheat,

[08:24:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30299s) and you give it the system prompt and the skills,

[08:25:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30301s) and you render everything perfectly.

[08:25:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30302s) And what you can do is you can ask it

[08:25:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30304s) what it would do in some scenario.

[08:25:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30306s) You can give it the situation.

[08:25:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30307s) And see if you can recreate the situation

[08:25:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30309s) that you saw in your eval or in production.

[08:25:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30311s) And if you can recreate it,

[08:25:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30312s) now you can tweak your context

[08:25:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30314s) to see if the subagent would change its behavior

[08:25:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30317s) based on your tweaks.

[08:25:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30319s) And the reason to do this,

[08:25:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30320s) it's obviously far worse than forking,

[08:25:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30322s) but you do this because this is much cheaper,

[08:25:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30325s) much faster,

[08:25:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30325s) and allows you as an agent builder

[08:25:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30327s) to really easily iterate on ideas

[08:25:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30330s) without needing to go through the expensive process

[08:25:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30332s) of actually forking an entire environment.

[08:25:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30336s) And you can do this today without building anything.

[08:25:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30339s) I did this for literally a year and a half plus

[08:25:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30343s) before we built forking into our internal systems.

[08:25:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30347s) And the main idea here is understanding

[08:25:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30350s) that if you look at the distribution from earlier,

[08:25:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30354s) you're trying to understand,

[08:25:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30355s) is the agent generalizing to that distribution?

[08:26:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30360s) And so you don't know purely based on the outcome evals.

[08:26:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30363s) You can try to choose the best outcome evals

[08:26:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30365s) that should represent the curve, but you don't really know.

[08:26:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30368s) And so supervising the process through some of these methods

[08:26:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30371s) gives you some additional signal in order to understand

[08:26:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30374s) if it's generalizing properly.

[08:26:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30377s) And what's really cool is as you start to get more signal,

[08:26:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30382s) you can actually start to close the loop.

[08:26:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30386s) And an opinion is what lets you do that, right?

[08:26:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30388s) If you don't have opinions about your system,

[08:26:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30390s) you're not going to be able to develop that signal.

[08:26:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30394s) And now if we go back to the firm,

[08:26:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30396s) we talked about supervision,

[08:26:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30398s) we talked about how the agent,

[08:26:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30401s) if he's opinionated, can be more reliable

[08:26:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30403s) and actually can be simpler.

[08:26:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30406s) What that means is the agent,

[08:26:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30407s) maybe it starts doing something narrow,

[08:26:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30409s) but because people will work with it,

[08:26:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30412s) they see that it's easy to use,

[08:26:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30414s) and they see that it's reliable.

[08:26:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30415s) As it gets better and better, they will trust it more.

[08:26:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30417s) It's just like a co-worker.

[08:27:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30420s) You have a coworker, day one.

[08:27:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30421s) Imagine someone sends you a PR.

[08:27:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30424s) You're not going to, if it's, you're looking at two axes whenever someone sends you a PR.

[08:27:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30428s) It's these two axes.

[08:27:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30429s) What's the blast radius of the PR?

[08:27:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30431s) And what is your perception of how competent that person is?

[08:27:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30434s) Right?

[08:27:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30435s) If the least competent person at your company is sending you a PR that's editing a core service,

[08:27:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30440s) you're going to look at it.

[08:27:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30442s) If the most competent person is sending you a PR that's editing an internal document,

[08:27:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30446s) you may not look at it as much.

[08:27:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30448s) and so this idea is true with humans working with agents just like it is with humans working with

[08:27:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30452s) humans and i think i just want to make one last point to close here which is that one of the

[08:27:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30459s) things that people say when they hear this is well wait a second if you let the agents just do what

[08:27:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30463s) they want won't they find a better way won't they move 37 your work and the answer is they might

[08:27:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30470s) But that's not how you scale GDP and scale output.

[08:27:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30475s) Nobody wants their agent to move 37 their tax return.

[08:27:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30479s) They want it done reliably at scale in a way that they can trust

[08:28:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30484s) and in a way that their people know how to work with.

[08:28:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30486s) And so the thing I will leave you with is for agents to diffuse,

[08:28:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30490s) they need to be simple.

[08:28:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30491s) For teams to trust them, they need to be reliable.

[08:28:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30495s) The way you get to simple and reliable is you develop opinions

[08:28:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30498s) on how the agent behaves,

[08:28:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30500s) and then opinions allow you to build that additional signal

[08:28:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30503s) that helps you start to close the loop

[08:28:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30505s) so that you can achieve self-improving agents in production.

[08:28:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=30508s) Thank you very much.