# The Software Atelier

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[00:40:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2403s) Hello, my name is Armin Ronohar and I want to share with you a little bit the lessons that we have learned from building software with agents.

[00:40:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2413s) And in particular, I want to make a strong case for not so much building the factory as building the artiller or workshop.

[00:40:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2421s) and I want to show some data with this from our own usage but in particular I

[00:40:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2425s) want to explain what I mean by building the atelier or the workshop prior to

[00:40:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2430s) building the software factory and I think everybody here is being used has

[00:40:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2436s) been using agents for quite a while and you have probably noticed that it is

[00:40:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2440s) we're sort of an exponential curve in what we can do with them but the

[00:40:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2445s) comparison that I want to give here is that you can be an amazing artist and

[00:40:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2449s) and use the greatest and latest technology

[00:40:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2451s) from CNC machines to 3D printers,

[00:40:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2453s) which is very good power tools.

[00:40:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2455s) And it can increase the ambition

[00:40:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2457s) of what it can do with your art.

[00:40:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2458s) And I think we can also look at software

[00:41:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2460s) a little bit in that sense.

[00:41:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2462s) So less about building much more software

[00:41:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2465s) and just building way more ambitious

[00:41:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2467s) and high quality software.

[00:41:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2470s) So I'm going to show you a little bit

[00:41:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2471s) of what I mean by that.

[00:41:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2475s) And you might know that Arendelle

[00:41:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2477s) is working on the PyCoding Agent.

[00:41:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2478s) So we have obviously a little bit of a horse in the race here of how we're looking at this.

[00:41:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2483s) But I think what is probably quite interesting is that the PyCoding agent has not really changed in a year on the core harness.

[00:41:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2490s) It still has the same tools today as it did in December last year.

[00:41:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2494s) And I think this is interesting because it has become much more powerful purely as a fact of the models getting better.

[00:41:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2501s) So I'm much more capable today of what I can do with Py without any changes landing in the core functionality of what the harness is doing.

[00:41:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2508s) And I think this is an interesting thing to reflect on because it means that we are all simultaneously getting better even without landing any significant changes in how we're doing this.

[00:41:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2519s) And you might have remembered throughout the last year and a half maybe that there were a lot of ideas of how you can prompt the machines better or how you can do some interesting tricks with the harness.

[00:42:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2530s) Very few of them are still necessary.

[00:42:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2531s) For me, a good example here is that I was in love with Pi Auto Research, which was a

[00:42:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2535s) plugin that basically allowed an agent to drive towards a goal, like improving performance.

[00:42:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2539s) But if you use a present-day state-of-the-art model, you actually discover that the entire

[00:42:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2543s) plugin is no longer necessary.

[00:42:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2545s) So there's a lot that is just happening in the models that is dramatically improving

[00:42:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2549s) how we're doing.

[00:42:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2550s) So these machines are getting better.

[00:42:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2553s) And I just want to show this from a couple of graphs from basically my Pi sessions where

[00:42:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2558s) I filtered them out to have nothing but the default set.

[00:42:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2561s) And the total amount of corrections,

[00:42:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2563s) and I used Jeff to analyze this,

[00:42:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2564s) the total amount of corrections that I had to do

[00:42:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2567s) got down and down and down with every single model

[00:42:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2569s) that got better and better.

[00:42:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2571s) So I'm pushing back a lot less against the agent

[00:42:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2577s) with the latest model than I did in December last year.

[00:43:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2580s) It's the same thing with how often I'm aborting it.

[00:43:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2584s) And I think this to me was the most interesting way

[00:43:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2587s) in which I sort of analyzed my sessions, which is the average, so this is the P 90th percentile,

[00:43:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2591s) but the 90th percentile of how long an agent was running in response to a task that I gave it

[00:43:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2597s) with Opus 4.6 in December last year was about two and a half minutes. And now with the latest

[00:43:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2602s) models, they are on the 90th percentile running for 14 minutes or more. And that does actually

[00:43:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2607s) change something about how you want to work with a harness, but really, again, nothing in Pi has

[00:43:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2612s) changed and still I'm seeing this sort of different behavior of how I'm using

[00:43:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2617s) this. And this probably you have noticed in one form or another if you're still

[00:43:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2623s) using an agent and we're going to see what it's doing you'll probably notice

[00:43:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2626s) that the present-day agents are much more likely to use bash than any of the

[00:43:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2630s) other tools that are being provided but again nothing has changed in how we

[00:43:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2634s) configure the harness it's just that there's a lot more emergent behavior

[00:43:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2637s) coming from the models. What's missing here is Astra and Sol they actually look

[00:44:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2641s) very similar to Opus for my testing, I just didn't have enough data in my test set to

[00:44:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2645s) show this nicely. You will notice that there's a little purple thing here. I want to talk

[00:44:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2652s) about this in the end, which is sort of code mode is sort of slowly creeping up. But the

[00:44:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2657s) core of this is even if you have a piece of infrastructure nowadays that is sort of there

[00:44:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2664s) and is doing a good job, it's still changing as a result of the models. What has not changed

[00:44:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2670s) in the last year, at least this is the case for me,

[00:44:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2672s) and I think this is the case for a lot of people,

[00:44:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2674s) is that we're still humans.

[00:44:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2675s) We're still limited by our fundamental capabilities.

[00:44:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2679s) And as we are being more ambitious

[00:44:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2680s) and we're doing more with these machines,

[00:44:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2683s) I don't want to say we're the bottleneck,

[00:44:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2685s) but we are the thing that paces what they can do.

[00:44:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2689s) And in particular, human attention is, for me at least,

[00:44:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2692s) the limiting factor, and I'm noticing this largely

[00:44:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2695s) in how coding agents are deployed in teams,

[00:44:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2698s) that this is a stable amount of attention that we have to work with.

[00:45:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2707s) And starting new projects is getting incredibly easy with these agents.

[00:45:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2710s) And I think particularly when a new model comes out,

[00:45:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2712s) we are all super excited of trying the latest and greatest thing you can do

[00:45:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2715s) from 3D games to new ways in which we automate things.

[00:45:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2718s) But maintaining them has not really gotten easier.

[00:45:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2722s) But we are also increasingly taxing a lot of external systems

[00:45:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2726s) that actually in many ways are not supposed to be part of this revolution.

[00:45:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2730s) I think we all know that GitHub has a lot more commits being pushed up there, a lot more repositories,

[00:45:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2735s) but I know quite a few people who have well established old websites on the internet that

[00:45:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2740s) found a new life for agents and they are totally not ready for the new volume that's coming from

[00:45:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2745s) from us kind of directly hitting them. So I just want to caution a little bit here in saying that

[00:45:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2751s) when we are pushing things we're also accidentally simultaneously all of us pushing things

[00:45:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2756s) which we're not really ready for what agents are doing.

[00:46:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2761s) And this to me I think is an interesting graph

[00:46:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2764s) because the number of prompts that I'm actively pushing

[00:46:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2768s) over the year is sort of flat.

[00:46:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2770s) I know that in some months it's a little bit higher,

[00:46:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2773s) in some months it's a little bit lower,

[00:46:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2774s) but they're alternating between 120, 190 something like this prompts

[00:46:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2779s) that I'm pushing per day.

[00:46:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2782s) But you can see that the time that the agent was active for my use went from like less than my

[00:46:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2789s) working day to many times my working day. And this November is not even over, right? So you

[00:46:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2795s) can see that this is taking an exponential curve up. And this here is the number of projects,

[00:46:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2801s) this is Git repositories that I've created over the last year. And this is like obviously

[00:46:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2804s) cumulatively going up. So I have 141 different projects that I started. But the number of

[00:46:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2809s) project I'm committing on a regular cadence to, which means I'm committing for more than

[00:46:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2813s) five days to and I'm returning to them also, is pretty much flat with between 10, 15, something

[00:47:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2820s) like this. And I have done this sort of informal survey with a bunch of my friends as well,

[00:47:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2826s) and this seems to be a common trend that actually maintaining something regularly is tricky.

[00:47:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2831s) And I had this discussion with a friend of mine in relation to this graph, and he said,

[00:47:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2836s) like, well, actually, I have an open source project that is now fully automated.

[00:47:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2840s) So there's an issue tracker coming in.

[00:47:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2843s) There's an agent that's set up to auto-review, commit, and send it back.

[00:47:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2846s) And that is maybe a way to scale up, so you take the human out of the whole thing.

[00:47:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2850s) But actually, he also turned it off because eventually he didn't care about the project

[00:47:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2854s) as much anymore, and he wanted his token specs to use them for other things.

[00:47:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2859s) So our attention to an open source project matters even if maybe we have automated it

[00:47:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2863s) entirely.

[00:47:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2863s) So if we stop using something where we have a tendency to no longer actually actively take care of it.

[00:47:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2871s) So if we're pushing towards, and I would make this argument,

[00:47:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2875s) first figuring out how to make a really good workshop to build really high quality software,

[00:47:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2879s) there's a question of what matters.

[00:48:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2881s) And I think for me the three most important things to matter is on the one hand

[00:48:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2885s) to really know when to push back against what an agent is doing,

[00:48:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2890s) telling it that this is the wrong way of doing it.

[00:48:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2892s) And I think that this is a crucial skill because an agent is really rewarded in just completing a goal.

[00:48:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2901s) And it is very easy to look at if it has succeeded in this goal or not.

[00:48:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2906s) And we can sort of see it in the same way as the RL environment maybe is rewarding the model during training.

[00:48:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2912s) But that doesn't mean that what is being actually delivered is something that we are intrinsically happy with over a longer period of time.

[00:48:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2918s) And this becomes tricky the more and more commits we're doing, the larger and larger the software is going.

[00:48:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2925s) It's very easy to basically create the Simpson car of all the features in a software,

[00:48:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2929s) but actually having something that intrinsically fits together where the internal complexity of the system does not explode

[00:48:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2937s) still requires us to work with it as it's working.

[00:49:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2943s) And I like to call this true skill versus agent skill.

[00:49:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2948s) there is clearly a difference between someone who is an experienced software engineer working with an agent

[00:49:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2953s) and someone who is not an experienced software engineer.

[00:49:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2956s) And I think for me, the truest form of this is that my son is now sending me pull requests on a project that we're doing together.

[00:49:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2964s) And there's a very clear quality difference in what his contributions are.

[00:49:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2968s) Because he is not a software engineer.

[00:49:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2971s) We're using the same system to prompt it, right?

[00:49:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2975s) And I have noticed this myself in situations, but I've heard this also particularly from a lot of others.

[00:49:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2981s) It is actually a very frustrating experience if you're in a situation that really matters

[00:49:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2986s) and other people in the room are basically just going to the agent to answer a question

[00:49:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2991s) and no longer have a really good understanding of either what the system is doing

[00:49:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=2995s) or that the fundamental problem that they're working on, they actually comprehend.

[00:50:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3001s) So, there is a lot of value in still understanding what you are doing.

[00:50:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3006s) That also means that you kind of have to moderate the speed in which you are doing things to still work within your understanding.

[00:50:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3012s) And one of the ways in which I started doing this as my commits started going through the roof basically,

[00:50:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3018s) was I looked into how are my trajectories of sessions going.

[00:50:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3022s) And can I hook myself into a part before the end of it to find a consistent way for me

[00:50:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3029s) to evaluate what's going on?

[00:50:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3030s) And so for instance, I started basically prompting the agent that before it's going towards

[00:50:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3035s) the verification, which is typically around 70% of it, to start producing a really nice,

[00:50:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3040s) short executive summaries and a nice typeset HTML page that can really understand what

[00:50:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3045s) it's doing as it's finishing.

[00:50:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3048s) particularly with newer models, it's no longer quite as horrible to read the text that comes

[00:50:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3053s) out of it.

[00:50:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3054s) But this is one way in which I found to balance out that I'm basically no longer reading the

[00:50:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3059s) code in a lot of cases.

[00:51:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3061s) And this I think is something that increasingly is more important to me is to also show up

[00:51:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3068s) with a little bit of a guidance towards not writing all the software over and over.

[00:51:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3077s) I'm a huge believer in using fewer dependencies and writing more code yourself.

[00:51:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3081s) So I really like the Archantic Revolution for that element of it.

[00:51:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3087s) But in German, we have this word called Wildwuchs, which basically means uncontrolled growth.

[00:51:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3092s) And uncontrolled growth in software that is supposed to work together

[00:51:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3096s) can have a lot of really negative consequences.

[00:51:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3099s) You might have seen a blog post that was going around recently

[00:51:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3102s) about a TAR archive that has been crafted in a way

[00:51:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3106s) that different TAR implementations read it in different ways.

[00:51:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3111s) And I think this is a bit of a problem now.

[00:51:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3115s) For us to work together, we really want to have standards.

[00:51:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3118s) We want to have common specifications.

[00:52:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3120s) We want to have common file formats.

[00:52:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3122s) And we now can basically make an agent

[00:52:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3125s) and implement any format we want.

[00:52:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3127s) But there are going to be these differences.

[00:52:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3129s) And what really makes it hard for us to agree on standards

[00:52:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3133s) and what makes us work together on something

[00:52:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3135s) often is not a technical challenge

[00:52:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3137s) as much as it is a problem of how do people agree

[00:52:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3140s) on what is supposed to be in this file format

[00:52:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3142s) and how is it supposed to be implemented

[00:52:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3143s) and how is it supposed to be interpreted

[00:52:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3145s) because there can be multiple interpretations of it.

[00:52:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3148s) And I really think that these open protocols

[00:52:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3150s) and file formats are a really important aspect.

[00:52:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3153s) You can imagine there is a future

[00:52:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3154s) where we all have our own little Photoshop clones,

[00:52:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3156s) but we'll no longer be able to send file formats to each other

[00:52:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3159s) because we have diverged too much.

[00:52:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3162s) And this really comes down to this part

[00:52:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3164s) that collaboration between humans is as hard as it has ever been.

[00:52:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3169s) And this is slowing us in a way down, right?

[00:52:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3173s) And I think it's important to sometimes resist this temptation

[00:52:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3176s) of just having the agent solve the problem

[00:52:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3179s) and just taking a step back and figuring out

[00:53:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3181s) if this is maybe one of those problems

[00:53:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3182s) where we should be working together with other people.

[00:53:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3185s) It is for me coming from an open source background

[00:53:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3188s) fascinating to see how little actual human collaboration

[00:53:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3191s) is happening in open source projects now

[00:53:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3193s) and how many more projects end up with gazillions of forks

[00:53:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3196s) because it's just easier to have a fork happen

[00:53:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3200s) than to try to upstream changes.

[00:53:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3204s) So I wanna make a very strong argument

[00:53:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3206s) for building software that strives towards excellence.

[00:53:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3209s) And I understand that this is hard.

[00:53:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3212s) We have, I think, seen a pretty big explosion recently of verified software.

[00:53:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3217s) It's also very clear that this is in itself hard.

[00:53:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3220s) The technology stack is not right there yet where this works.

[00:53:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3223s) But I think it is one of those areas where it's really worth pushing towards

[00:53:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3226s) and seeing if this is maybe something that can help us over time creating more stable software.

[00:53:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3233s) I really am very happy with seeing so many people wanting to build fundamentally faster

[00:53:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3238s) and more memory efficient software.

[00:54:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3240s) I think it's a great development. For years I was very frustrated with how

[00:54:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3246s) little my RAM usage actually goes to things that I think are worthwhile and

[00:54:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3250s) now there's a significant trend of people trying to rebuild things in a

[00:54:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3254s) native language rather than for instance just building an electron wrapper. So all

[00:54:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3258s) of those things are really cool developments but they are in many ways

[00:54:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3263s) still building up on technology we have created before times. I really like Rust,

[00:54:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3267s) I like that there is now a trend towards building memory-safe software.

[00:54:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3271s) I think you've probably all been exposed to what happens with good models finding security issues.

[00:54:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3279s) So there's a strong argument towards that.

[00:54:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3282s) And it now has also led to people that would never have picked that language before

[00:54:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3286s) to actually give it a try because it delivers some value that now can, in quote, come for free.

[00:54:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3293s) On the last part, I want to talk a little bit about maybe there is something where Harness

[00:54:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3297s) can do something. And this is less so much for me to say, hey, here's a path that we're on,

[00:55:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3303s) is to give you some idea of maybe where some of the challenges are as we're pushing this task to

[00:55:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3307s) be much more long running. The first one is, if you have a task that only takes two minutes to run,

[00:55:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3314s) you're kind of willing to babysit it and if it breaks and it ends early or it

[00:55:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3320s) doesn't run to completion or you have to restart the agent to just restart it

[00:55:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3324s) manually and tell it to continue. Right so it's like we have we have largely been

[00:55:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3328s) working in environment for quite a while but that was acceptable. If you now have

[00:55:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3333s) automated agents running on thousands of issues overnight you want to have an

[00:55:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3338s) agent actually be able to restore itself. And so we're pretty big believers on this

[00:55:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3343s) concept of durable agents, also durable agents that create durable workflows

[00:55:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3346s) where they orchestrate themselves in a way where you can start it, stop it,

[00:55:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3350s) you can suspend it, you can bring it back. And this not just because we have this

[00:55:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3354s) longer-running tasks but also because there's a lot of inefficiency in what

[00:55:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3357s) we're doing and we are all compute constrained, we are all constrained in

[00:56:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3364s) terms of the cost of inference and being able to be a little bit more

[00:56:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3368s) intelligent about how we're doing this I think is really interesting and I wish

[00:56:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3371s) more people would be pushing towards that. For instance, if you pay raw token costs, you get a

[00:56:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3375s) pretty significant discount on flex scheduling. But that means that maybe for minutes, no token

[00:56:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3380s) is coming, right? And so then you want to suspend your agent, you want to suspend also your sandbox

[00:56:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3384s) and actually not waste the other resources that are also expensive. So this is one part where I do

[00:56:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3390s) think beyond the quality of the models, because the models are getting so good now, we might want

[00:56:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3395s) to rethink a little bit of how we are using these resources intelligently.

[00:56:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3401s) And the second thing is you probably know that when Pi came out there was a pretty big

[00:56:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3411s) motivation to say like bash is everything that you have to run and this is still largely

[00:56:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3415s) what we believe.

[00:56:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3416s) But now we also see that there is an emergence of desire to orchestrate sub-agents, a desire

[00:57:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3425s) to use models which are not LLMs, like decision models or classifier models like Jeff.

[00:57:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3431s) And in those cases, you need to start composing workflows.

[00:57:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3436s) And this, I think, originally came from the folks from Cloudflare,

[00:57:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3440s) say, like, instead of having thousands of tools being put into the code,

[00:57:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3446s) into the context, what if we just give the agent, in addition to Bash,

[00:57:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3450s) JavaScript, in addition to bash, JavaScript to run and have all the tool calls that the

[00:57:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3455s) agent would natively do happen via JavaScript.

[00:57:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3458s) Then he can orchestrate them better.

[00:57:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3459s) And this has gone from a curiosity, I think, in many ways, to something that is fundamentally

[00:57:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3464s) shockingly useful.

[00:57:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3466s) And this means that now the harness in many cases, and this is definitely the case already

[00:57:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3470s) in Codex, you can do it in Py and many other harnesses at this point, you can have the

[00:57:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3474s) harness orchestrate very large invocations of tool calls without having to go through

[00:57:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3479s) pretty inefficient context.

[00:58:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3481s) And this does require a little bit of a rethinking

[00:58:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3483s) of how you're doing things.

[00:58:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3485s) And it also allows you to use classifier models.

[00:58:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3488s) And I think this is the other big part

[00:58:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3491s) that really changed how I was looking at

[00:58:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3493s) how far I can push certain technology.

[00:58:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3495s) Because you're not being penalized

[00:58:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3497s) for not having an active KVCache,

[00:58:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3500s) but they're also very cheap and very fast.

[00:58:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3503s) And they allow you now to take certain problems

[00:58:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3505s) that previously were just too expensive or too slow to do,

[00:58:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3509s) to have a decision model do this.

[00:58:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3511s) And the way I think you should be thinking on this

[00:58:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3514s) is that when you have code mode,

[00:58:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3515s) what the harness can do is it can create a little workflow

[00:58:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3518s) that can have decision points along the way

[00:58:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3521s) where if you, for instance, have to crunch through

[00:58:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3524s) 3,000 incoming customer support requests

[00:58:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3527s) and you have to make decisions along the way

[00:58:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3528s) if this is being sent towards having an agent work on it,

[00:58:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3532s) towards requiring more feedback from a customer,

[00:58:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3534s) you can have the agent create basically

[00:58:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3536s) an on-demand workflow and then for every single

[00:58:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3539s) if condition that it's doing, it's just going through Jeff.

[00:59:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3541s) And then you can crunch through thousands of issues for very little time in no time in

[00:59:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3546s) your coding agent, but previously that would have taken you hours.

[00:59:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3550s) So with this, I just want to leave this idea.

[00:59:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3555s) Now that we can build everything, maybe we should not, but we should really, I think,

[00:59:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3560s) push towards much more enjoyable and in many ways much more ambitious software than we

[00:59:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3567s) have been doing before.

[00:59:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3567s) Thank you.