Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff

Talk transcript

1. 0:00

   [upbeat music] They've got it?

2. 0:16

   Cool. Amazing. So welcome, everyone. I'm Vincent. Uh, what do I do? I'm one of the core maintainers at OpenClaw, working with Peter. And as you've heard before, I have a day job as well.

3. 0:28

   Same as Peter, he has a day job at OpenAI. Um, but you know, it's an open source project. Amazing things have been happening. I'm gonna talk about what I call dark factories, and how OpenClaw ships faster than you can read a diff.

4. 0:40

   Um, this meme is absolutely hilarious. So I think Peter posted this, uh, a week or two ago. "I wake up." "There's a new technological advancement." "I wake up." It's this, this joke that we're shipping at insane speed, and the velocity is just absolutely phenomenal.

5. 0:54

   And some of you might think, oh, this is some luck or we're just, like, Ralph looping to the max. Um, I think there's actual engineering work here, and I'm gonna talk about that.

6. 1:05

   Now, as I mentioned, I'm Vincent. I'm your friendly clanker. Uh, this is me using VR goggles back in 2013. So despite my accent that sounds somewhat [REDACTED:origin], I was born and raised in [REDACTED:location], not far from here.

7. 1:17

   I actually went to, to college just down the road in Westminster, and yeah, at some point decided to live in Australia, and my accent changed. But I used to love technology.

8. 1:25

   I used to love being at the edge of technology, and this was like one of the first few sort of early, um, VR goggles that came out. It came in this big box with a big warning sign on it saying, "Hey, use for five minutes at a time," 'cause it didn't have like the anti-motion sickness built into

9. 1:39

   it. And the funny thing with this one here was that I didn't use it for five minutes, I used it for three hours. And I played Team Fortress 2, had an absolute blast, and then I vomited for three hours after that 'cause my vision turned into bvisions.

10. 1:53

    Um, the... What I'm trying to say here is that like anything on the edge is gonna be janky, it's gonna be f- horbi- ho- horrific, it's gonna be, uh, uncharted territory.

11. 2:04

    And working on OpenClaw and being part of the team that ships probably, you know, an insane velocity of commits to a point where I get rate limited by GitHub on an hourly basis, uh, is an interesting experience.

12. 2:18

    And this experience Britain's gone through before. Um, we had the Industrial Revolution when mills and cotton were being produced at extreme amounts of volume, and there's a lot of history here around production and productionization at scale, uh, in the UK and in Europe.

13. 2:36

    And I feel like we're going through this moment again. Uh, we're going through this moment of how do we build at scale? And the ways we used to work before just don't work anymore.

14. 2:46

    And it's kind of strange because in my day job, I kind of work in the space of evals, which everything is sort of structured, and there's telemetry, and it has to be all perfect.

15. 2:58

    And I work on a project where I'm-- I have this blind faith in the harness, and it's this kind of two worlds, but they're starting to come together.

16. 3:08

    We used to have handlooms in cottages, uh, centralized mills everywhere. Uh, craftsmen were the factory workers, but the bottleneck was the weaver's hands. We're now switching to a world where engineers writing code in editors, not so much, uh, swarms across repos.

17. 3:28

    Uh, engineers are becoming factory managers, which I'm gonna talk to, and the bottleneck becomes taste. You know, that lovely word. Uh, [laughs] [REDACTED:origin] mother's hands. Yes.

18. 3:39

    So in, in context, like what does this mean? Like, are you talking absolute nonsense of people building things at absolute scale? They are. What happened was, um, very similar to the ChatGPT era, where everyone denied it at scale that they were using ChatGPT.

19. 3:55

    Everyone was in this absolute fear-mongering sort of world. But what the reality was that everyone was using it. Everyone in secret was just like, "Oh my God, what's going on?

20. 4:03

    I need to talk to it." And the same thing is happening with these autonomous agents at scale. Some organizations have openly come out with it, so for example, Anthropic, uh, with their recent work they did on building a new C compiler.

21. 4:16

    Uh, we had Spotify saying they're, they're no longer writing code by hand, supposedly. Um, Steve Yegge, which I absolutely love, uh, saying he pushes about fifty PRs a day total solo.

22. 4:28

    He calls himself a vibe maintainer. I can kind of relate to that. And OpenClaw, where we're pushing... At the peak, we were doing eight hundred commits a day. And realistically, like there's about ten to fifteen core maintainers all with day jobs.

23. 4:43

    It's kinda astronomical in terms of scale. And for me, this was March 15. Um, what was that? Like two, three weeks ago? Where I hit close to three thousand commits per day.

24. 4:56

    And if you actually chook-- look, my commits actually stop when I go to sleep. So if you, if you wanna see when I go to sleep and when I wake up and how many hours of sleep I have, you can just take a look at my commit history.

25. 5:07

    Yeah, it's astronomical. But the thing is, this is gonna become the norm everywhere else. Like this is, this is like a me telling you you need to wake up that, you know, that this scale of velocity is gonna be normal.

26. 5:17

    And trying to review PRs and go through all this nonsense may not work. But somewhere in the mix is engineering. There is a form of engineering that's gonna happen.

27. 5:28

    So we did commit maxing. You know? Let's just go out there, smash as many commits as we can. And this reminds me of Ralph looping, right? This like, this, this, this, this guy where you're like, "Hey, I'm just gonna like give you a task.

28. 5:40

    I'm gonna burn tokens for like eight to nine hours." And you're waiting, you know? Uh, you're waiting, you're hoping something happens. Maybe something happens. I don't know. Um, but what if we had a bit more of an opinionated ap- uh, approach to this?

29. 5:53

    Um, what if we call it Bart looping? I don't know. Uh, one of the other maintainers gave me this idea. Maybe we'll coin it. Do we- Do we need more than just tokens?

30. 6:02

    Uh, what does that reward mechanism look like? How do we get a bit more opinionated? Yes, let's run loops, but let's be a bit more smart about how we do this.

31. 6:12

    So, um, right about the time you saw those three thousand commits, uh, this was the day before. I was at NVIDIA with Peter, and the gentleman you see on the left is, is one of the other, uh, NVIDIA gentlemen.

32. 6:23

    And they were like, "Hey, we're building NeMo claw." I'm like, "What? What's going on?" And, uh, let's help you build it. And I was in the room. I was like, "I can't work on a laptop for like hours on end.

33. 6:33

    Can you bring me a screen?" They brought me a screen. Um, Peter didn't have a screen, so that's his laptop on the left. He asked for a screen, so they gave him an even bigger screen than mine 'cause, you know, why not?

34. 6:45

    And [chuckles] we just got to work. So he's running about maybe fifteen codec sessions, and he's got his Mac Studio at home he's VPN'd into. I'm running another like ten or fifteen.

35. 6:54

    And between, collectively between us, we're probably running with subagents included, maybe up to sixty, seventy agents. Um, but on the foreground maybe fifteen s- uh, swim lanes, if you wanna call it that.

36. 7:03

    And we're just, just going for it. Funny thing is, we're working on NeMo claw one side, but one maintainer decided, "I'm gonna move some stuff around. I'm gonna move a couple of folders around."

37. 7:13

    And that was moving entire channels. So like all our conversations with like MS Teams and Slack ended up moving to another location in the code base.

38. 7:22

    And we were like, "Oh my goodness, we're gonna have to change stuff." Um, and I found a really nice, uh, place to put my drink as well. [laughs] The NVIDIA people didn't like this.

39. 7:33

    So what ended up happening is what we call the Great Refactor, um, essentially where we were like, "Hey, we have lots of people raising PRs, and what they actually want is to build features."

40. 7:43

    The thing is, we don't wanna give everyone every single feature that they want, in which case it becomes bloat. Uh, you heard Peter say earlier on the challenge becomes who do I say no to?

41. 7:53

    It's not about saying yes. In a world where tokens are cheap, I can just say yes to absolutely everyone and merge everything in, but that's gonna turn this code base into an absolute fire dump.

42. 8:02

    So the vision was actually we need to cut this code base down. We need to, we need to rip it into pieces. And a plugin architecture somewhat made sense.

43. 8:12

    Imagine if you're OpenAI or Mistral or Anthropic, what if you owned that piece of the provider code and it was handed to you and it was separate from everything else?

44. 8:22

    So this code change that occurred was like a catalyst for us. It was two in the morning, we're tired. We thought, "Why not refactor the entire code base? Sounds like a splendid idea."

45. 8:33

    So two thousand seven hundred commits later, uh, close to a million lines of code change, uh, touching eighty-two percent of the core code base, plugins were launched. Um, the night before, I think it was like one in the morning, I'm trying to go to sleep, and the tests are not passing.

46. 8:51

    And I was like, was I Icarus and did I fly too close to the sun? Um, as we like to call it, did I, did I vibe too hard?

47. 8:57

    I, I actually generally thought I vibed too hard. But as a team we managed, we managed to bring this code base back together again. But the saving grace was these awful sort of unit tests that AI code loves to generate that actually ended up overfitting on our code.

48. 9:13

    So when we completely ripped everything out, we still had these tests that were like extremely overfitting, and as long as they would go green, we knew we were kind of somewhat close.

49. 9:22

    Um, so how do we do this, you know? Um, in my case, I call it my factory. It's many codec sessions. Everyone asks me like, "What's this magic sauce?

50. 9:34

    Like how, how do you do this? Like what's this crazy, insane thing? Like how are you guys building this?" Very simple. I have swim lanes. Um, it could be five, it could be ten, it could be twenty.

51. 9:44

    But traditionally, they kind of cut themselves up into different pieces. So like if I... Does this work, the laser? You can't really see it. But imagine you're a factory manager and you have a production line below.

52. 9:56

    Essentially, you might have a case where you have, uh, let's just say CI to one side, you might have features in one side, you might have bugs in another.

53. 10:05

    So when I'm refactoring and doing stuff, um, right now the code base is quite stable. I wanna refactor some tests. Well, that might be swim lanes one and two.

54. 10:13

    I don't need to really babysit them too much. I just tell them, "Take your time, make sure the tests pass. Just commit. Just, just push them through." Whereas with three and four, I might be looking at specific features and issues around, say, Docker or, um, one of our messaging channel- channels.

55. 10:28

    In which case I'm having a conversation with those agents. They're going off investigating, doing the work, coming back. And then maybe five is actually, um, looking at new P0s and P1s.

56. 10:38

    Um, that might be using other data. It might be using GitHub. Uh, we have agents that run inside of a Discord channel, so when we do a release, we might be like, "Hey, what's happened in the last two hours that I need to be paying attention to?"

57. 10:49

    And this will scale up and down. But what ends up becoming quite interesting is tokens are no longer the problem. Um, depends who you ask. What really ends up becoming the problem is just raw compute and my brain space in order to sort of keep an eye on all of these sessions.

58. 11:04

    So in harness we trust. What ends up happening is I don't have this really insanely complicated process. The one thing I have complicated in my life is adopting Git worktrees, and I kinda wish I hadn't.

59. 11:16

    The only reason why I say this is when you're running an extremely heavy test harness, it ended up completely nuking my machine 'cause I ended up running like every PR I touch ends up becoming a new Git worktree.

60. 11:29

    I end up with like some- close to like seventy or eighty active Git worktrees in any given day on my machine, and that's kinda hell. Um, so I had to actually build some like magic sauce around my, my codec session, so my codecs is aware of Git worktrees.

61. 11:42

    If I hit the escape key, it crashes. It will self-heal, self-recover, Git, you know, sparse stuff. But realistically, I should have adopted what Peter and other people do and just like clone the repo ten times and point ten different, you know, codec sessions to each one.

62. 11:56

    But the trick here is that like I haven't done any magical sauce. I don't use plan mode or spec mode. I have a conversation with the agent, and we work through it, and we find a way to make it work.

63. 12:06

    So realistically, it looks a little bit like this from The Matrix. Um, and people go, "Oh, Vincent, like, how do you know it's kinda working?" And this is gonna sound somewhat a little bit lunatic.

64. 12:17

    It, it-- If, if anyone's watched The Matrix and seen the scene where Neo goes over, he's like, "How do you know? How do you read the text?" And the guy's like, "Oh, you know, I've been doing this for a while, so I can see, like, woman in red dress or guy walking dog."

65. 12:29

    And you start to have this, like, relationship where you can feel the reasoning tokens. I know it sounds somewhat ludicrous, but there's times where I'm looking at the swim lane, I'm like, "This sounds off."

66. 12:40

    It doesn't sound off because of what it's doing, it sounds off because of how it's explaining itself to me. It's waffling. It's not making n- sense. It doesn't seem to know what it's doing.

67. 12:51

    And this feels a lot like how I would manage people. Um, if I had someone working for me and they started downright bullshitting, I'd be like: Wait a minute.

68. 12:59

    What's going on? So in these cases, I might just nuke the session and go, "You know what? I'm not gonna deal with this section of code. I'm gonna leave that to another maintainer," or I might come back to it four or five days later.

69. 13:08

    But that experience feels very much like intuitive. And building that intuition, I've been able to get to because of the sheer volume of token maxing I've had to go through in the previous year.

70. 13:21

    So there is engineering work. Uh, I call this the agent development environment. Um, essentially, the process goes, I have skills. I call it .skills, similar to dot, um,

71. 13:33

    .files. Both of my .skills and .files is available on GitHub. It's all open source. Go for it. Some of my skills are private, but there's skills in there for, like, writing technical documentation, for example, um, that I've co-created with other, um, developer experience and other engineers in the market.

72. 13:49

    Um, you can use a skills gym, something like a, a Gepa, which I'm also a contributor to, and... Or you could just say, "Go Codex." I've been using this skill in my last, uh, two weeks.

73. 14:00

    Go through the Codex sessions, read the logs, make improvements to the skill. Um, I would then take that skill and deploy that into my OpenClaw or take that into my, you know, personal environment, and I'll use something like Vercel skills.sh as like a mechanism to loop this.

74. 14:16

    I've added some other testing and other elements on top of this, but there's a process to how I manage and maintain my skills as an engineer. The way we manage PRs has some level of engineering work to it.

75. 14:28

    Um, there's this kind of running joke that every maintainer that joins the project decides to try and tackle, like, "Oh my God, we have six thousand PRs. How are we gonna solve it?

76. 14:34

    I'm gonna cluster everything and, like, figure this out." So the- Sixty K. How many? Sixty K. There you go. There you go. Honor. Thank you very much. So this was my flavor of, like, trying to solve this.

77. 14:44

    This is, like, a semantic graphing, uh, vector embedding on the entire GitHub stuff. This is one PR. Has seventy-three edge, uh, one hundred and six edges. What ends up happening is that everyone else has the same problem, so they decide to, to send their flavor of the PR issue.

78. 14:58

    Becomes u- utter noise. So there is even process around, like, how we even consume what we're gonna work on. We might not call it a roadmap, but we have a way of kinda de-duplicating and seeing what's out there.

79. 15:10

    This might be a signal for me to say, "Okay, if there's enough pressure coming on one issue, it must be big enough that all these other clankers decided it's a big problem.

80. 15:19

    Maybe I should go and address it." There is evals, surprisingly. Um, after all this refactoring work, we decided to make a fake Slack of sorts with both synthetic models and real models, so we can run evaluation loops to check that each of the providers and the channels work.

81. 15:37

    And this question was asked of me recently: How do you manage ten plus agents? And this is something that you're thinking. I s- asked them back, "How do you manage ten plus staff?"

82. 15:45

    And they had no answer for me. I'd worked in large organizations like airlines and other places like that, managing large AI teams. I had experience managing up to thirty, forty people plus.

83. 15:55

    So for me, it was not like a, a, a new paradigm. But I think for engineers and people working with these coding agents at scale, it's the soft skills that matter.

84. 16:05

    It's how do you ask your agent what's going on? How do you know when they're not bullshitting you, and how do you run that factory? So it's no longer about the model or the agent, it's about the process.

85. 16:15

    Uh, twenty twenty-five was about token maxing, twenty twenty-six is about not wasting them. It's about token efficiency. It's about agent in the loop. Thank you. [audience applauding] [upbeat music]