Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
8:15:48What's up, everybody?
8:15:50How are we all doing?
8:15:52Give it up for all the fucking organizers out here.
8:15:55They are killing it.
8:15:56Swixen team, let's get a round of applause.
8:15:58All the speakers, keep it going for the speakers.
8:16:02I knew Quinn was really smart, but that was a really dope talk.
8:16:04I enjoyed watching it from backstage.
8:16:07Hi, everybody.
8:16:08I'm Dex.
8:16:08This is 12 Factor Factories, which started as a joke title, but then everyone started calling it that.
8:16:15And the war on slop.
8:16:16So thank you all for coming.
8:16:19I like to yap about coding agents on the internet.
8:16:22And like a year ago, we did the whole context engineering thing.
8:16:26A lot of views on YouTube.
8:16:27And up until about a week ago, I was the CEO and co-founder at a company called Human Layer, the multiplayer coding agent workspace.
8:16:34space.
8:16:34And for those of you who are blessed to not be quite as terminally online as some of us
8:16:40may be, we are now a part of factory, which I'm very excited about and I will be touching
8:16:44on during this talk.
8:16:45But I want to get right into it.
8:16:47Last summer was all about this, like, you know, three months ago, it feels like forever.
8:16:51Factories were all the hype.
8:16:52And I gave an intuition-based talk back in June of, like, we should probably still read
8:16:56the code and here's how RL and verifiers are working and, like, exactly as our friends
8:17:02at Google were saying, you know, models really like just solving the tests, just getting
8:17:05them to pass. I now have some data beyond just that intuition. There's a benchmark called
8:17:10Slopcode Bench, which we just released actually today a bunch of new data on from the lab
8:17:15at University of Wisconsin. But very unsaturated benchmark, you know, GPT 5.5 got a 15%. Astra,
8:17:22I think, scored 16.3. So like, we are getting better at this, but there's a lot of ways
8:17:27to go. I'll talk all about that data in a sec. We, of course, have the study from Pharos
8:17:31of 30% of PRs merging without review, longer comments, more comments, more incidents per PR,
8:17:37more bugs per developer. And I titled this talk, I don't think it was quite correct. I said,
8:17:42why software factories fail? And everyone said, oh, Dex is the anti-software factory guy. Dex
8:17:46hates software factories. I actually love software factories. I'm a huge fan. The title was probably
8:17:53more along the lines of, there are certain things you could do in your software factory
8:17:57that would cause it to fail.
8:17:59And here's what I think those things might be,
8:18:01and please don't do them.
8:18:04It'd be the dark factory thing.
8:18:07But I think the main takeaway here,
8:18:08and I think we've heard this from a lot of folks,
8:18:10is software factories is just good engineering.
8:18:13A friend of the pod, friend of the conference,
8:18:16Lauren was talking about this yesterday,
8:18:18of a lot of this is just stuff we've been doing for a long time.
8:18:21Actually, Eno from Factory had a really,
8:18:22the post is much longer, but it's like,
8:18:24the factory mindset is about how do we standardize our process
8:18:27not just the output.
8:18:29And there's a whole bunch of stuff we'll dig into there
8:18:31on how we're thinking about that.
8:18:32But I try to kind of make the same point, right?
8:18:34It's like your standard software factory
8:18:36is going to look something like you have some engineers,
8:18:40some product managers, maybe a CEO
8:18:43or someone to drive the vision.
8:18:45And those people are going to come up with stuff
8:18:48that we want to build.
8:18:48And they're going to put it in a tracker.
8:18:50This is, again, this is pre-AI.
8:18:52A lot of the things we hear about software factories today
8:18:55were around for 30 years, right?
8:18:58You take stuff off the tracker, you build it,
8:18:59you run some unit tests, automated tests,
8:19:01maybe you poke it locally.
8:19:02When you're ready, you make a pull request.
8:19:04At that point, someone might pull it down,
8:19:07interact with it, test it.
8:19:09Your CI is going to run a bunch of checks and static linters,
8:19:12and none of this is that new.
8:19:14And if anything is wrong with it, you send it back,
8:19:16you go fix the problems,
8:19:18and when it's actually ready to go to prod,
8:19:21we do some canaries and rollouts and feature flags
8:19:24and all kinds of fun stuff there.
8:19:26And eventually it makes contact with our users.
8:19:28And I love our users.
8:19:29I love customers.
8:19:31But they do like to complain about things that are broken
8:19:33and ask for stuff that we haven't built yet.
8:19:36And so that goes back to the team.
8:19:38And at some point, something breaks at 3 in the morning
8:19:40and the customers are the first one to figure it out.
8:19:43So we add monitoring so that we can wake people up
8:19:45at 3 in the morning and they can solve it
8:19:46before anybody notices.
8:19:48This is your pre-AI software factory.
8:19:50And the basic point is, like,
8:19:52all you need to do to make this agentic
8:19:54is replace someone builds the thing with
8:19:56an agent builds the thing. So you have orchestration
8:19:58I'm not going to go into the details of the sandbox
8:20:00and the model and all this stuff.
8:20:01But you can incrementally build this up by adding what I'll call forward pressure.
8:20:06What causes work to start? Can we take all of the feature requests and just route them straight to our factory?
8:20:11Can we take all of the incidents that happen so that when I get paged at 3 in the morning,
8:20:14I'm looking at a pull request that we think is going to solve it, not just a stack trace about what went wrong.
8:20:21So the forward pressure, what causes new work to start?
8:20:25and back pressure, which is sort of like,
8:20:27okay, cool, now we can build things in minutes or hours,
8:20:29but the review and testing still takes hours or days
8:20:32if we're doing it in the old way,
8:20:33so we bring in agentic code review, agentic testing.
8:20:36Anyone doing agents code reviewing their pull requests?
8:20:40Okay.
8:20:41I thought it would be a little bit more,
8:20:42but that's pretty good.
8:20:43Y'all are catching up.
8:20:45And then we invest heavily into our CICD and our linters
8:20:48and our, anyone doing like lean checks on their code,
8:20:50like formal verification?
8:20:52All right.
8:20:53Hardcore.
8:20:53I like it.
8:20:55So the back pressure, how do we let agents hill climb to a better outcome without needing
8:20:59to involve humans?
8:21:00How do we just overall increase the quality of what is coming out of our build step?
8:21:04And agents help here, but a lot of what makes software factories good is stuff we've kind
8:21:08of had for decades.
8:21:10And just like context engineering, which was the thing that I was talking about last summer
8:21:14in like June of 2025, there was all these ideas of like RAG and memory and state and
8:21:18history and all this stuff, and we tried to like distill out what are the timeless things
8:21:22that go into building agents.
8:21:24And so I want to distill out now some of what I think are the timeless things that go into
8:21:28building and designing a software factory.
8:21:31Because there's a lot of noise and hype.
8:21:32Like I said, Terminally Online, Twitter is going to break your brain if you spend too
8:21:36much time wondering what everyone else is doing.
8:21:38I'm also going to talk about Slop because I have a very fun project that I've been working
8:21:43on with Swix and a handful of other speakers here, but we'll get to that later.
8:21:48We're going to do 12 great practices for software factories.
8:21:51And I think the first time I started talking about this, it was only like 9, but we are at 12 now, so I'm very excited about this.
8:21:56I touched on this, but loops was this concept that was very in vogue.
8:22:00I think this is much more useful if you think about the interplay of forward pressure and back pressure.
8:22:06So, you know, again, very hyped up, but the actual tactical thing is like, how do you start work automatically?
8:22:11And how do you make sure that whatever comes out and is touched by a human by the end, all the low-hanging stuff is already fixed.
8:22:17The outcome is as good as it can be autonomously.
8:22:21Number two, I will call know your enemy.
8:22:23I will still posit that unattended models
8:22:26don't do a great job of maintaining
8:22:28and improving code base quality over time.
8:22:30So if you care about having a system
8:22:31that's going to be still maintainable
8:22:33in three or six months,
8:22:35I mean, again, I gave this talk
8:22:36and I said, please read the code
8:22:37because a lot of it is slop.
8:22:39Agents got really good at solving one-off problems
8:22:41and making video games
8:22:43and Blender slop and all this stuff.
8:22:45I don't think, you know,
8:22:46if you steer a model to fix your code base,
8:22:48it can go fix your code base.
8:22:49But then you have to know what is there and what you want.
8:22:52So if you are just hands-off pushing features into the machine,
8:22:56you may not find that it autonomously will make things better over time.
8:23:00Back then I didn't have a benchmark.
8:23:02Now I have a benchmark that basically has the model solve problems and checkpoints.
8:23:06Like a model can clone all of Excel or Redis in a couple days,
8:23:10maybe even a couple hours now with like agent swarms.
8:23:13But that requires knowing the entire problem in advance.
8:23:16And most of us don't write software that way.
8:23:18Most of us ship a feature, and then ship another feature,
8:23:21and then ship another feature.
8:23:22And if you're building by hand without AI,
8:23:24along the way you say, oh, that's a bad abstraction.
8:23:25We need to centralize this, or we need to remove this.
8:23:27And so this is a benchmark that measures a model's ability
8:23:30to evolve a code base over time.
8:23:33And so Astra, again, is 16%, very unsaturated.
8:23:37There's a bunch of data here.
8:23:38I probably don't have time to look at all of it,
8:23:40but we can look at code quality metrics
8:23:42like cyclomatic complexity, lines per symbol,
8:23:47the percentage of functions that have high complexity.
8:23:49There's more. There's 200 custom rules.
8:23:5250 of them are based on things we've been doing for a while.
8:23:54A lot of them are custom based on looking at code that models write,
8:23:57comparing it to pre-AI repos.
8:23:59I wrote a long article about that. I've read a bunch.
8:24:02Gabe Orlansky, who runs the lab that does this benchmark,
8:24:04just put up this paper today
8:24:06based on some of the research we did with them.
8:24:09There's a bunch of stuff in here.
8:24:10I probably don't have time to go through all of it,
8:24:12but they also test the model's ability to design good libraries and interfaces.
8:24:16The most, the craziest one here is like, models did learn to use exceptions instead of returning
8:24:21tuples in Python.
8:24:23There was a lot of patterns that were identified as like probably not great, that were way
8:24:28less present in the, you know, six months ago models than they are in today's models,
8:24:32so it almost feels like we're like, the slop is amplifying over time.
8:24:37And Claude and GLM are now writing Python 3 in Python 2 style, so we have, you know,
8:24:42%s format strings are back, and they will not use fstrings by default anymore.
8:24:47So things to be aware of. You may not care about this. I care about this.
8:24:50The jury's still out on Fable 5.2. I love reading the Anthropic blog post about how
8:24:55to prompt the new models. And so allegedly, if you tell Fable it owns the code base, it
8:24:59should keep it clean. I haven't tested this yet. We're running the benchmark in the background
8:25:02right now. But for those of us not building FBS games, shouts out to Ben, built a cool
8:25:06game. You should go watch this video. It's insane.
8:25:09you should know your enemy.
8:25:11You should understand that, like,
8:25:12work with this in mind, that if you need your codebase
8:25:15to last a long time, or your codebase has been
8:25:17around for a long time, we should
8:25:19probably maintain understanding of the
8:25:21system and the code.
8:25:23To do this, we like to plan before we
8:25:25build. I highly encourage this. You should do
8:25:27this. We used to do this before in this old software
8:25:29factory. People realize, especially
8:25:31when building things took
8:25:33hours or days,
8:25:35and when testing and reviewing things
8:25:37took hours or days, even if this part is faster,
8:25:39this part is probably still slow if you're doing it properly,
8:25:42we would have architecture review,
8:25:43and we would have sprint planning,
8:25:44and we would talk as a team, and we would align
8:25:46in the hopes that we might decrease the percent chance
8:25:50that we'd have to rework something later.
8:25:51We wanted to be rather sure that we're building
8:25:54the right thing in the right way
8:25:55before we actually go and spend time building it,
8:25:58time in tokens, and then get to review
8:26:00and have to rework it.
8:26:02A lot of people overdo this as well.
8:26:04I've overdone this for sure,
8:26:05where you spend six hours working on a spec
8:26:07and you're like, I feel like I could have just YOLO'd this.
8:26:10Because sometimes you can just YOLO a two-sentence prompt,
8:26:12and if it's small or you don't care,
8:26:13there's maybe a 50% chance you send a couple more prompts
8:26:16and then it's good.
8:26:17You could spend five hours handwriting a detailed spec,
8:26:20and maybe there's a 10% of it you have to change afterwards.
8:26:23Or you could write all the code by hand in 20 hours,
8:26:25and then there's like a 0% chance.
8:26:27Well, there's 0% chance you have to rework it
8:26:29because of something the agent did.
8:26:30You know, people still make mistakes too.
8:26:33But I think about this in terms of expected pain.
8:26:35like how likely will this have to change later
8:26:37times how hard will it be to change.
8:26:41I don't argue with the model during planning
8:26:43about what color the button's gonna be,
8:26:44but I do wanna make sure we get our database schemas right
8:26:47and stuff like that,
8:26:48because it's really hard to change later.
8:26:49So based on your problem and what you're doing
8:26:51and your code base,
8:26:52you should probably decide where on this spectrum
8:26:54you wanna land.
8:26:56Align visually, I published a skill called show me,
8:26:59which is optimized for instead of reading walls of text,
8:27:02you get HTML mockups, you get tables,
8:27:04You get charts. You get all this stuff that is optimized for the human brain, which is very visual.
8:27:10I think Koda said we've been running away from predators and eating fruit for millions of years.
8:27:15We've only been reading text for about 5,000.
8:27:18Anyways, you can get this. It's got about 40,000 installs.
8:27:21You should help agents test their work.
8:27:23Screenshots, browsers, videos, curl, whatever you can do to help the agent.
8:27:26Again, he'll climb against an outcome.
8:27:28If you're building an iOS app, make sure it can build, run, and test that app in whatever environment it's running.
8:27:33You should fight slop with anyone here writing in TypeScript.
8:27:37OK.
8:27:38Some people are feeling lazy.
8:27:39I feel like it should be more than that.
8:27:41But this is a repo from Dylan Mulroy, who's a principal engineer at CloudFlare.
8:27:46Anti-slop.
8:27:46If you're not using this, you should go check it out.
8:27:48It's 100-ish rules on stopping agents from building slop.
8:27:52Factory also has an ESLint plugin that I should probably mention.
8:27:56And also, again, you can go slop fork all of the Python code quality rules out of Slop
8:28:02code bench. This exists for your
8:28:04language. You should use it. Slop
8:28:06is a tool. Not all slop is bad.
8:28:07Vibe coding prototypes, side projects.
8:28:10If you have something behind a feature flag, you just need to like figure
8:28:11out if people even want it before you
8:28:14go put in the time. I think it's like, it's great.
8:28:16I don't think there's like always, always, always,
8:28:18always every line of code has to be perfect.
8:28:20You should give models a feature
8:28:22flagging system. PostHog
8:28:24or Statsig or LaunchDarkly or whatever it is so they
8:28:25can try stuff and see if it performs better
8:28:27and then make decisions about
8:28:29what the right thing to be in your tool.
8:28:32You should compound over time, eval everything,
8:28:34learn from past sessions,
8:28:35and you should unify your forge.
8:28:38What I mean by this is we used to send
8:28:40Git patches around on mailing lists.
8:28:41I think Linus and the Linux people still do this,
8:28:43but most people keep all their code in GitHub.
8:28:46And so before the ANA to forge,
8:28:48we have centralized patches,
8:28:49but we still have people pasting
8:28:52clawed answers back and forth in Slack
8:28:55and hopefully not doing too many rounds of that
8:28:58because it's like, I copy out of mine
8:29:00and send it to my coworker.
8:29:01He pasted it into his agent, and then they sent it back to me,
8:29:03and then I pasted it into my agent, and maybe we get an answer.
8:29:06So centralizing all of your designs and decisions
8:29:08and whatever you have in Notion and Google Docs
8:29:11and whatever it is, you should try to bring those things
8:29:14as close as possible together so you can understand
8:29:16not just the code about what something was built
8:29:18but all the decisions that went into it.
8:29:20You can't buy everything.
8:29:21I think there's things in this space that just cannot be productized.
8:29:25Obviously, you can buy VMs or you can bring your own.
8:29:28Dev environment, I think, is very personal to most teams.
8:29:31The harness, of course, you can go get a Frontier harness,
8:29:33or you can use something minimal like OpenCode or Pi
8:29:35and customize the heck out of it.
8:29:37And then you have this control plane picture of, like,
8:29:39how do you collaborate on sessions and plans?
8:29:41How do you do automations and scheduling?
8:29:43How do you respond to events?
8:29:44All this governance stuff.
8:29:46Part of the reason I'm really excited to be working with Factory on all this stuff
8:29:48is, like, they have a much bigger view of this picture
8:29:51and have a lot more stuff solved here.
8:29:53But there are parts, especially at the top here,
8:29:55is, like, I don't think anyone can sell you verification of your code
8:29:59because only you
8:30:00know, if you have hundreds of engineers and hundreds of repos, you have those opinions and you know how to test your code.
8:30:05Not just a black box you can buy.
8:30:07There are building blocks that are undifferentiated.
8:30:09But the idea of software factory, I think, is a way of thinking about our work.
8:30:14So that's software factories.
8:30:16I also want to talk about slop.
8:30:20Factories are about how do we increase human leverage, right?
8:30:22How do we get the best return on our time?
8:30:24and I've talked a little bit about why humans should be involved
8:30:26and what it means to be a diligent
8:30:28steward of what we're building and what we're doing
8:30:31but I think it's bigger than just software
8:30:33so I'm very excited to announce
8:30:35the Anti-Slop Manifesto
8:30:38which you can go
8:30:39if you go to slopmanifesto.com
8:30:40you can read it, I'll put the URL up later
8:30:43but we built this in collaboration
8:30:44with a lot of people here, both attendees
8:30:46and speakers, special shout out to
8:30:49Swix and Phillip
8:30:50it has signers from Zoom Info
8:30:52and I think XAI and OpenAI,
8:30:56and there's a lot of cool people on there.
8:30:58I encourage you to go read it and check it out
8:31:00and be a part of, you know.
8:31:02But I'll tell you what's in it.
8:31:03We value outcomes over outputs.
8:31:06We value understanding over speed.
8:31:09We value our own thinking
8:31:11over whatever the model came up with as a first draft.
8:31:14We value better work over easier work.
8:31:17We value improving the system
8:31:19over just patching outputs.
8:31:22And the things we believe are that AI is the most useful tool
8:31:25most of us have ever worked with,
8:31:27but it can create a swarm of agents can produce more work in a single day
8:31:32than a human can review in their entire career.
8:31:34Look at some of the math stuff coming out of OpenAI.
8:31:37Slop is when we put less effort into producing something
8:31:39than we ask their consumer on the other end to spend understanding it.
8:31:44And using AI doesn't necessarily make something slop,
8:31:48but it happens when we pass off AI output
8:31:50without taking responsibility for its quality or for the reader's experience.
8:31:55I believe strongly, I've been saying this for a year now,
8:31:58AI cannot replace human judgment.
8:31:59It can only amplify our judgment,
8:32:01or maybe perhaps our lack of judgment in certain cases.
8:32:05And the slop is asymmetric.
8:32:07The cost of producing a thousand-line PR or a ten-page doc is approaching zero,
8:32:12but the cost of reviewing it is still fairly constant.
8:32:16And slop proliferates when we repeatedly work around problems
8:32:20instead of solving them correctly.
8:32:21It comes from tools that love to add
8:32:23and are not biased towards subtracting.
8:32:26I'm not going to name names.
8:32:29But I think a lot of slop comes from pressure.
8:32:31Companies are pushing their people
8:32:33to redefine what is humanly possible.
8:32:36And I think slop compounds.
8:32:39So the more slop sits in our knowledge base,
8:32:41in chat, in code,
8:32:43the more of it the next agent reads and copies.
8:32:46And the more slop flying around our team,
8:32:48the more we lose the will to fight it, and everything gets worse over time.
8:32:52Communication and quality and all of this.
8:32:55So we, the undersigned of the anti-slop manifesto...
8:32:58Guys, is it a little hot in here?
8:33:02I might just have to pop off the jack. There we go.
8:33:06They couldn't give me one of these with a factory logo on it yet,
8:33:08but I'm sure it's coming soon.
8:33:11Thus, we, the anti-slop undersigners,
8:33:14resolve that human time and attention is incredibly valuable.
8:33:19So we optimize for using it wisely.
8:33:22We fight slop because AI should make our work better,
8:33:25not just easier.
8:33:27We optimize our systems and tools and products
8:33:30for the efficient elicitation of human intent
8:33:32and for efficient human understanding.
8:33:35And we do our own thinking when working with a model.
8:33:37We explore resources, code, prior art.
8:33:40We really want to understand the problem
8:33:41before we actually get to work.
8:33:43And we own the intent and the taste and the consequences of what we ship.
8:33:46However much AI helps us to make it, we own it.
8:33:50And when an agent wrote something that we're sending somebody,
8:33:52we make that clear, and we actually make sure it's good
8:33:55before we drop it in Slack or email or whatever it is.
8:33:58And when someone sends us slop, this is also like we have to push back.
8:34:01We have to tell people kindly, empathetically,
8:34:03and then ask them to review it first.
8:34:05And most of us, it's a journey.
8:34:07I've been on both sides of this one, so I've had to have that note.
8:34:11I believe there will always be alpha in reviewing something.
8:34:16As models get better, we may stop reading most of the code.
8:34:19We may still review a plan or a design or a result.
8:34:22But if you don't review anything, if you bring nothing to the table,
8:34:25no thinking, then you're just going to get the same AI output as everybody else.
8:34:30Agent reviewers, linters, verifiers, they're very useful,
8:34:33but they are not sufficient.
8:34:36So for now, we'll still review some subset of the outputs.
8:34:40What subset, of course, depends on the stakes.
8:34:43But what shift, who's using it, and how it's helping them matters far more than lines of
8:34:48code or PRs merge or, God forbid, the number of agents we're running in parallel.
8:34:53It's cool if it gets you to the outcome.
8:34:55But unreviewed output isn't always necessarily bad.
8:34:57Slop is a cool one-shot prompts, vibe coding.
8:35:02These are all high-leverage things that are in the right place.
8:35:04Slop is bad when it undermines our outcome.
8:35:06If we want to make a good product, a maintainable system, we want to save people time and energy.
8:35:11If slop undermines that, then it's bad.
8:35:14And I guess finally, none of us has this figured out.
8:35:17We're each relearning what we will and won't accept, and there's no single right answer here.
8:35:22You've got to calibrate yourself.
8:35:23The world moves fast.
8:35:25My goal here is these are principles that we can use to embrace our important human role
8:35:30in harnessing AI safely, responsibly, in a way that creates value for everyone.
8:35:37for many years to come.
8:35:39So you can read and sign the manifesto
8:35:40at slotmanifesto.com.
8:35:43TLDR, you know, give a shit.
8:35:46Care about what you're doing.
8:35:48I'm going to stick around for Tariq's talk,
8:35:50but then, what is that slide?
8:35:53All right, I'll be hanging in the hallway afterwards.
8:35:55So come say hi.
8:35:56Keep learning, keep giving a shit.
8:35:58I'm excited to keep building with y'all.
8:36:00Thank you so much.
8:36:01Thank you.
At 08:19:55, Horthy reduces the transition to a single substitution: the agent builds the thing. His surrounding argument concerns everything needed before and after that step.