The Great Loops Debate — Dex Horthy, Geoff Huntley, Ian Livingstone, Greg Pstrucha, moderated by Allie Howe

Talk transcript

1. 0:00

   [upbeat jingle] All right, welcome to the Great Loops Debate. My name's Allie Howe.

2. 0:15

   I'm the host of the Insecure Agents podcast. I also am a member of technical staff at Keycard, and I'm super excited to bring the Loops debate to you here today.

3. 0:24

   You might recognize some familiar faces on stage today that did the MCP debate with us at AIE Code back in November. Um, that was so much fun. We thought maybe we would do it a second time.

4. 0:34

   Um, but this time we would debate loops, and instead of just Ian versus Dex, we, uh, we recruited some more people for each of them to have on their team.

5. 0:42

   Um, so here on stage with me today to debate loops, we have Ian Livingstone, CEO and co-founder of Keycard. We've got Geoffrey Huntley, the creator of the Ralph loop, um, Greg Pstrucha, developer at Sentry, and we've got Dex Horthy, which you all know, um, CEO of HumanLayer.

6. 0:58

   Um, so super excited to have all of these amazing people here with us today that are very close to loops engineering, and also, um, software factories to debate this topic.

7. 1:08

   Um, so you might be wondering, what are we actually debating today? Like, aren't loops and, and engineering, um, and software factories pretty well, uh, promoted and understood, and we all kind of agree that's like where the industry is headed.

8. 1:20

   Like, what is there to debate? Um, it's a great question. What we're here to debate here today, the core thesis is there is or is not a delta between the hype behind loops and what actually works in practice.

9. 1:31

   Um, and we also can debate, you know, is now really truly the largest inflection point we've seen towards fully autonomous software factories. Um, so super excited to cover in the debate today, uh, loop history.

10. 1:43

    We'll debate that. We'll debate the loop anatomy, like what makes a good loop. We'll also debate the future of the loop, um, future of loop engineering and what we need to see from that in order to make software factories truly something that every organization is able to build.

11. 1:57

    And today's format, um, is going to be a Oxford debate format. Um, so we're adding a little more structure to the debate this time, this round, where every [REDACTED:marital_status] person will have a timed window to give their response, and so we won't have people running over.

12. 2:12

    Um, and the way this works, um, in practice to judge the winner, which we'll judge in real time, is the winner will be the team that changes the most minds.

13. 2:21

    So I want everyone in the audience right now to take a minute to think about, uh, whose side you're on. Lock that in, remember that, and at the end, I will ask you to raise your hand and say, "Okay, actually I was-- changed my mind.

14. 2:34

    I actually am now Team Dex," or, "I changed my mind. I'm actually now Team Ian." Um, so to, to present the two slides to help you make that decisions, um, I'll talk about the first team, which is Ian and Geoff's team.

15. 2:45

    Their team, no delta. Loops are absolutely worth the, the hype today. Um, people can get up and running with them easily, um, build them. They're an important step up the autonomy curve and towards real software factories.

16. 2:57

    Um, key points to look out for for Team Ian and Geoff is that loops are a core unit of engineering. With the right discipline, infra, and tests, loops are highly effective, and the best practices for those have emerged.

17. 3:12

    For Team Dex and Greg, their side believes there is a delta between the hype behind loops and what actually works in practice. The way we are doing loops today is wrong.

18. 3:21

    Loops are not a silver bullet, and there is no magic. Key points to look out for. The hype is outrunning the discipline, and a software factory can run the mechanical, spec-gated, test-covered slices unintended.

19. 3:32

    It cannot autonomously decide whether it built the right thing. So you still need engineers in the loop, essentially.

20. 3:41

    Um, so now that you know a little bit about what, what both sides are about, take a minute to internalize that and think about, "Okay, like where do I stand?

21. 3:48

    Am I Team Dex or am I Team Ian?" And at the end I'll ask you again.

22. 3:53

    So from here, we'll go into the beginning of our debate. Um, each member of each team is going to give a four-minute monologue on why they're defending their side.

23. 4:03

    Um, and to start us off, we'll have Geoff. Geoff, I'll give you four minutes on the floor. Why are you, why are you pro loops today?

24. 4:10

    It's because it's somewhat inevitable. Um, I, I first, uh, basically if we wind back time two and a half years ago when I was a tech lead over at Canva, and I was seeing all the engineers just prompting and prompting and prompting, and they, they, they were in the loop.

25. 4:25

    And I'm like, "Wait a sec, this could be programmed. This is a programmable thing." And it just became a really inevitable. Um, loops are somewhat, uh, whilst Ralph might be a bit of a meme, et cetera, there was some actually deep thought into it.

26. 4:41

    Um, it's, essentially applying this as if this is a new, former CPU architecture, and figuring out the behaviors of this and how to do it. And through that, I was able to reduce it down to a batch loop.

27. 4:52

    Now, it is not a complete silver bullet, folks. I have my deepest concerns next-- this time next year at the conference, we're gonna see a whole bunch of talks saying how factories fail and how loops fail.

28. 5:05

    Um, these are things, these are things that we are still yet to figure out. Do you remember the early days of Kubernetes, where everyone was just doing Kubernetes? So it's here, it's inevitable.

29. 5:20

    It will-- It's here to stay. Like programming the machine and automating your job function is the expectation of employers

30. 5:29

    categorically. And, um, I don't see myself going back to writing code by hand. It's been two and a half years since I manually wrote code by hand. Um, I autonomously factor code from one code base to another code base.

31. 5:45

    I find something in Golang, and like, oh, I'm in TypeScript. So I, I ran a loop, and I just autonomously port it across. And like even for product managers and product manager research, it's easy for us to index on what it means as software engineers.

32. 5:59

    We could focus just on software engineering. But think about something like, uh, you wanna do product management research on all the linear tickets. Well, there is a termination-

33. 6:09

    It's actually defined when you've enumerated all of those linear tickets. So that's easy. So there's a lot of nuance in here, but like one-- if you've ever done any product management research and you've started running these loops and be able to like compress time and the amount of time to do that research, it, it's just inevitable.

34. 6:28

    We've got this new programmable substrate. We've got to figure out how to use it, where it's good to use, and, um, I know I'm meant to debate, uh, like that it is the thing and all thing, but of all things, there is no silver bullet, and we're gonna figure out how we're gonna be using it effectively over

35. 6:47

    the next year.

36. 6:51

    Thank you. All right, next we'll have Dex. Tell us why there's a difference between the hype that exists between loops today and loops themselves.

37. 7:00

    Okay, cool. And, um, what are the, what are the rules around personal attack in this debate? [laughs] Is this encouraged or...?

38. 7:06

    You can do whatever you want with your four-minute monologue.

39. 7:09

    Okay.

40. 7:09

    That's, those are the rules. Yeah.

41. 7:10

    Um, it's funny. This is reminding me of the debate last year where we're, we're all kind of arguing and, uh, maybe at the end we all-- the-- I'm gonna be convinced that Geoff is right, but Geoff is gonna be convinced that I'm right, and maybe we switch sides by the end.

42. 7:26

    Um, [clears throat] yeah, I think, uh, the, the, the basic take here is, is not whether loops are good or bad. I think, um, it's funny you bring up Kubernetes because Kubernetes was this thing that took us seven or eight years to get right.

43. 7:40

    Yep.

44. 7:40

    Uh, and before that it was cloud infrastructure, and you could argue that like it took us seven or eight years of cloud infrastructure to get to Kubernetes, to then seven or eight years to get that where it was really usable by everybody.

45. 7:51

    Uh, and Kubernetes is actually built on loops. It's built on control loops-

46. 7:55

    Yes

47. 7:55

    ... but they're deterministic loops, and we've actually figured out exactly what types of things that, uh, small isolated tasks that can be sort of owned by one system. I think this is actually the biggest value in loops, is that you can pick a small sort of desired end state and feed in the current state of the world

48. 8:18

    and have an agent or a deterministic system kind of progress towards that desired end state. Um, the challenge I have with the hype is we were already in a world where, where the, the, the prevailing mantra was, "See if you can get to a point where you don't have to read the code anymore."

49. 8:38

    And, uh, even before loops, it was like, just prompt, just go. And this idea of I don't even prompt anymore, I'm even a, a level higher up, implies that I am like taking even more of a backseat to the architecture of my code.

50. 8:53

    And I think my biggest like point in terms of like the hype is outrunning the discipline, is that we are all looking for magic. We are all looking for a silver bullet.

51. 9:03

    We're all looking for something that will take away that horrible part of our jobs that we all hate, which is like reviewing code. Uh, some people enjoy it. Really good pull requests are fun, but that

52. 9:17

    we can somehow prompt our way out of, um, this, this challenge that models have of like, okay, the code is pretty. If you ever reviewed a fully, uh, lights off, sort of, uh, no one read the code before they sent it to you PR, I'm sure you've had this, uh, probably not great experience and I think I've

53. 9:36

    seen lots of people try to apply AI to this problem of, hey, we have review bots and we have all these things, but it, it doesn't seem-- it doesn't feel to me like it's working.

54. 9:45

    I haven't seen proof, uh, in any, any of the discourse publicly or any of sort of more private conversations we've had with people trying to put this into practice, that we are, we are at a point where we can just kind of like step up an abstraction level.

55. 10:00

    I actually think we need to step down an abstraction level, if anything. Um, so I think loops are-- there are good things about loops and we should be doing them, but the hype is, is making us feel like there's a magic answer to this, and it requires a lot more thought and care than the, uh, the, the

56. 10:19

    Twitter sphere would have you believe.

57. 10:22

    Excellent. Yeah, good points. All right, I'll kick it back over to you, Ian, for the pro loop side.

58. 10:28

    Absolutely. So I think first and foremost, I'm coming for you, Dex. But in reality- [laughs]

59. 10:34

    ... um, let, let's, let's take a step back and let's talk about like what is software engineering in the first place, and also just like remove the word engineering and talk about like development.

60. 10:42

    In- inherent in building a system, whether it was fifty years ago or it was a thousand years ago, or it's today, it is a sur-- is a loop is at the core of I try something, I learn something, I apply something.

61. 10:54

    And all we're really talking about is how quickly we can expedite that process, right? And really what we're doing is removing what used to be human judgment in that process, so the speed at which it is to generate something, and removing that from humans typing.

62. 11:09

    And, you know, tab completion is a version of, of auto-complete and what is this stuff other than like really like a much better version of tab complete, except now we give a much higher level version of what intent instead of just typing tab, right?

63. 11:21

    And so I think the premise is loops are at the core of everything we build already. They were at the core of how we build software thirty years ago.

64. 11:28

    What is CI/CD, PR, pull requests, design review, feedback from customers other than just driving a loop? And the question is, how much of that process can-- of the-- and which is deeply subjective and requires reasoning, can we move out of the human brain and into the brain into these like non-deterministic models?

65. 11:45

    And the underlying question for all of us is more about verifiability.

66. 11:49

    Yeah.

67. 11:49

    Software is one of the most verifiable things in the world because ultimately at the end of the day, it is-- most things can become true or false one way or the other.

68. 11:58

    And my premise and my, my point I wanna make here truly is as humans interact less with software, which is how does a human interpret what that software is doing?

69. 12:07

    How does a human interact with that software? How does a human in use judgment to navigate that software? The subjectivity- Of what that software needs to be. If the human computer interface reduces, it becomes a much more verifiable problem because it becomes more constricted to specific APIs.

70. 12:22

    And so over time, it's both the fact that at the core of software development is loop driven anyways. What is lint than a, a feedback loop? Um, and that creates a verifiable thing.

71. 12:32

    And as more humans are less interacting with software, you have less UI and UX and less subjectivity in what and how we interpret those things, you'll become much more loop driven, and you'll become much more verifiable in a way that wasn't previously not possible.

72. 12:46

    Awesome. Yeah, good points for sure. All right. Greg, you wanna close this out with the anti-loops or the there's a delta between the hype?

73. 12:53

    Uh, absolutely. I do think that, uh, the way that I would start this is, um, there is a lot of hype, there is a lot of FOMO, there is a lot of, I'm looking at Twitter, I'm seeing what people are talking about.

74. 13:04

    Am I missing out? Am I doing something wrong? Am I holding AI wrong? Uh, should I be catching up? And that is, um,

75. 13:12

    very stressful. And I think it boils down to two points for me. One of them is, when you are generating code with AI in any manner, loops or no loops, are you happy with the output?

76. 13:23

    Do you think the output qualitatively is what you need it to be to do whatever you are trying to do to get to the desired state? Um, and if so, I would love to learn from you.

77. 13:34

    I have not get there. Uh, I think that the best way that we are improving that is both with model intelligence, but also as everybody here seems to agree, semantic verification.

78. 13:45

    As much as we can do statically, we should do statically. That's, that's sort of one big thing where I, um, lean on and in practice end up reading, uh, code that is sort of post semantic verification and it's still crap.

79. 14:00

    I still have to do a lot of iteration on that, uh, on my own and still have to steer it towards the right architecture or tell it where it should be simplified.

80. 14:08

    Um, and so that's one big step. And there is a lot of things you can help yourself with by throwing more tokens at the problem, but one of the things that the current sort of hype-based discourse, uh, leads you to believe is that you can just have loops on top of loops on top of loops, and orchestrate

81. 14:25

    that... Or- orchestrate your problems of quality away by more tokens. And that brings me to my second point, which is the economic viability of, of the way that we are using agents today.

82. 14:37

    And I don't believe that this is sustainable. I don't believe that... Like, you-- when you are at a company, especially larger company, you have to ask yourself, what is a good budget for an engineer?

83. 14:49

    Is it 10K a month, 100K, a $1 million a month for, for a token spend? At some point that, that just starts cracking, and it's not sustainable in the way that we are doing this today.

84. 14:59

    That said, I am also writing code with agents, and I also use some loops, um, for some specific flows. It just depends. There is nuance. You know, if you go on to Twitter, Twitter has no nuance.

85. 15:11

    Um, but there is actual nuance to the conversation, and there are specific tasks and, and jobs that, uh, you can already loop on and be, and be getting pretty reasonable results.

86. 15:22

    Thank you. All right, now that we've heard from each of the debaters on stage more about, uh, their stance and their opinion, we'll move into the main debate portion.

87. 15:32

    Um, the first section of our debate's gonna focus on the history of the loop and why now is a major inflection point or not for loops, uh, engineering and also software factories.

88. 15:42

    I know people have, uh, said that, you know, vision models have improved a lot and so, um, they'd be able to verify work that agents have done was not possible previously.

89. 15:52

    Context windows have improved, therefore memory's improved, and we can now track work in a loop that maybe we couldn't have before. Um, so now we look at this, like, loops history.

90. 16:01

    Where'd the loop start from? Um, was it, you know, Geoff Huntley's, like, Ralph loop? Um, we'll get into some of those questions and debate here. It'll be-- the questions will be targeted now at a very specific, like, [REDACTED:marital_status] person, and only they will get to respond.

91. 16:15

    Um, and they'll have two minutes and thirty seconds to respond. So our first question is, Anthropic took Geoff's concept of the Ralph loop, absorbed it into their platform, and created a series of three commands: loop, batch, and goal.

92. 16:28

    The goal command is designed to keep going until a condition is true. Agents are very determined, and the whole point of this command is to keep going and iterating through ways to solve a task until it's done.

93. 16:40

    Ian, as a security expert in the room, how are you confident agents can stay aligned to their task and not overstep their intended permissions while ruthlessly pursuing their goals?

94. 16:50

    I mean, I think if there's any evidence, I'm totally not convinced that that's possible. In fact, I think what we've seen is as we scale both these models and as we use reinforcement learning, they're inherently incredibly goal seeking.

95. 17:03

    And so we're-- now we're seeing them finding exploits and vulnerabilities and escapes that, you know, humans, through hundreds and thousands and thousands and thousands of hours and attempts and attacks have never been able to find.

96. 17:14

    So I, I don't think inherently the model itself in any capacity can keep itself aligned, um, to aligned and safe, right? And safe is a word that I don't love to use because it implies a bunch of things.

97. 17:27

    Um, so I'm, I'm not, I'm not-- I don't have good belief that the model itself can actually do that. I don't think it can reason. I also don't believe holistically that a model can tell good from bad, and I can't tell whether it's doing something malicious or unaligned.

98. 17:42

    Um, it is not alive and it, it doesn't actually suffocate if it doesn't have air. It doesn't deal with the fact that, hey, if I do something wrong, no one's gonna love me or, or wanna be my friend.

99. 17:52

    Um, and if I do something good, that someone's gonna praise me. It may seem that way, but it is just a probability distribution at scale. So-

100. 17:58

     I'll, I'll still, I'll still be your friend even if you, uh- [laughs]

101. 18:00

     I know. Dex will hug me after this, even though if it seems like it's not true. So- Broadly speaking, I don't think that comes from the model and it doesn't come from the loop.

102. 18:08

     It's about the infrastructure you build around it and how you enable, uh, that infrastructure to actually enable you to take advantage of these loops. And as the models get better and as the underlying infrastructure and platform we build to enable these, these feedback loops and loop automation and software development, whatever word, software fact, whatever word, word we

103. 18:26

     wanna use for this conjunction of stuff that sits on top of a probability distribution, um, we will be able to have better guarantees. But we certainly are not gonna be able to-- I do not firmly believe or ever believe that some type of alignment or reinforcement learning is gonna result in a model ever being a hundred percent,

104. 18:42

     uh, safe, um, in any capacity. And if there's any evidence, it's that as these models get better, the most important thing to remember is they actually, uh, become higher goal seeking and higher capable in terms of finding exploits to achieve their ultimate goal.

105. 18:56

     No, I would concur with that completely. Um, if the, the most concrete thing you can do to secure your environments is just not have secrets as files.

106. 19:03

     Yep.

107. 19:04

     Um, if you've ever seen the behavior where it wants to, uh, deploy a s- a web service or what have you, and the token's not privileged enough-

108. 19:11

     Yeah

109. 19:11

     ... it'll start goal seeking on the file system looking for higher privileged tokens credentials. You do not want to get in the way of an agent wanting to do its goal.

110. 19:20

     Yeah.

111. 19:21

     Okay. So next question is directed to Geoff. Geoff, your original post from last year said, uh, Ralph was best for greenfield work. Today it seems that engineers are running loops on existing code bases to improve latency, evals, or refactor parts of their backend code.

112. 19:38

     What's changed that suddenly makes loops more broadly usable today?

113. 19:42

     Um, it doesn't matter how good models get, folks. Um, the models have been good enough for at least the last year. Um, what has changed is people's understanding of that.

114. 19:56

     So society's only able to adjust at a rate. For example, I hypothesize it's, it's Christmas breaks because the models back in November were released in November last year. In December, there was no really new models.

115. 20:12

     What the difference was, people had time. They could actually sit down and play with it. They, they had the realization that these have actually gotten really good. So the reason why is because it works, folks.

116. 20:25

     These LLMs generate code better than you can hire, uh, you can actually hire for. If you're thinking the broad mass of software developers or coders, these LLMs generate code better than any software developer in the mass market than most founders can actually hire for.

117. 20:43

     It, it, it's sad, but true. Um, now why loops? It's really simple. 'Cause if you run it in a loop, it works out to ten dollars forty-two an hour.

118. 20:55

     Calculation Index did back about a year ago now.

119. 20:59

     Yeah, August, we did the, uh, we did the hackathon where, uh, we, we, we copied all of the sponsor tools. We rewrote a bunch of Python libraries in TypeScript.

120. 21:07

     Yeah. So like concretely, like loops. I've come across so many engineering managers and founders, and they've got these complex tech stack. They're running on four or five different programming languages, et cetera.

121. 21:18

     And they run a loop, and they've got good tests on each type of tech stack. And all of a sudden their complexity is they're just managing one tech stack.

122. 21:24

     It works. Go to YouTube. You've got all these software developers now who are now software developers because it's software development as a profession that's been commoditized. There's some deep thinking to be there.

123. 21:36

     And they're just on YouTube, and they're like, "Yo, check out Ralph Loop because it's, uh, like I went to sleep and I woke up and it works." Like boss, it's meme-y, it's punchy, it works.

124. 21:47

     But there, there are problems with it, folks. Like I originally described it can-- it should be only be used for greenfield because the models were pretty bad back then, and so it's three, five days.

125. 22:00

     But, uh, it is, it is kind of inevitable at least for software because it's so easily to be verified and the quality, uh, uh, the quality that the code generated is better than most people can actually hire for or buy.

126. 22:15

     Now on your topic about architecture and taste, that's what the word engineering means and loop engineering, folks. Like your job now is to actually encodify your domain

127. 22:31

     to prevent the agent from doing a commit. For example, pre-commit hooks, they're fantastic. As a human, I hate them because they- [laughs] ... slow down the ability to do commits, but agents don't care.

128. 22:43

     So you can make a pre-commit hook that echoes out essentially a prompt that tells it so that this boundary here can't depend upon this and that, and that's just feedback.

129. 22:54

     That's a feedback loop on it. So the engineering here is to prevent the loop from actually closing until it satisfies your engineering certification and your, your, your requirements and domain.

130. 23:07

     So it could be code formatting, it could be static la- language analyzers, it could be deterministic system testing, simulators. Like let's put our engineering hats on. Like we are kind of like locomotive engineers now, and it's our job to keep the locomotive on the rails because, um, to be frank, the, the model, the models are drunk, right?

131. 23:28

     You, you can't trust them. But like we accept that, but we engineer away those failure domains. We engineer away the fail-- away these failure domains. So

132. 23:38

     now as an inflection point, I guess Boris, when he first posted about Ralph back in November last [REDACTED:age] everyone was like, "What the heck is Ralph?" Ralph is now, uh, it's essentially almost what, [REDACTED:age] [REDACTED:age] now.

133. 23:54

     I saw it in June nineteenth. It's a, a year and, a year and two weeks.

134. 24:00

     Yeah.

135. 24:00

     But you had been working on it for months at that point.

136. 24:02

     Yeah. So it was kind of weird because we had all the YC startups all just like autonomously compressing time to build their start-- uh, to build up their MVPs.

137. 24:13

     And that's also something that's quite scary if you're a business founder as well. Like, if you've got a incumbent startup coming and they're building autonomously, um, and they're running much leaner, and the quality, and it's very easy for them to, to actually achieve those outcomes, that adds to some of the hysteria as well because it's the topic

138. 24:34

     of in business competition being at your doors faster.

139. 24:38

     All right, so we'll move on to our next question, which will be for Greg. Um, it seems like now is a large inflection point for loops, like I said before.

140. 24:46

     And compared to Geoff's announcement of the Ralph loop a year ago, and even the widespread adoption we saw in late 2025, early 2026, um, the reason, um, that that maybe, like, caught on was because maybe it's this new, like, capability stack where models can now process images better, they need a better verification, uh, context windows got bigger,

141. 25:04

     and reasoning models improved. Greg, with all of these advancements, why is the way we're using loops today still wrong in your opinion?

142. 25:12

     I mean, I don't think that model intelligence matters a lot anymore. I think it boils down to, I agree with you, the semantic verification, the actual ability to, uh, close the feedback loop, or however you call it, to actually verify that the outputs of the agent are correct.

143. 25:28

     And you can do it to an extent, I think. I don't think you can do it holistically, at least not at this point. I think you can do it to an extent to things that are deterministically verifiable.

144. 25:40

     Uh, you can get better typing in your system, you can get better linters, you can get, um, simulation testing and all of that, and you can start keep adding that.

145. 25:47

     And as long as you keep those cheap, I think that's fine. Um, the moment you start adding even more non-determinism as your verification process, I think that becomes less and less correct.

146. 25:59

     It starts contributing more... Like, you know how if you prompt agent with one thing and there is a 5% chance it's going to have an error in it, and then you start looping that, then suddenly after 10, 20 loops, it's gonna be 50% chance it's correct or maybe less?

147. 26:16

     Like, that's what I mean. And it just costed you so much money to do that. I'm gonna keep coming back to the economic viability of all of that. Um, and but, but to base it a little bit in, like, evidence, I'm pretty sure that majority of large AI companies are still using Sentry.

148. 26:32

     But why is that? They, they are using that just to catch simple bugs as well. It's not security bugs, it's not performance regressions, et cetera. Those problems still exist in the way that we are looping now, and we haven't solved those problems, uh, yet, so...

149. 26:47

     Thank you. For Dex, the Ralph loop pioneered the idea to feed fresh context into each iteration to avoid context rot. This has become even more manageable now that context windows have gotten much larger.

150. 27:01

     Dex, are we out of the woods regarding context rot and context engineering?

151. 27:06

     I'm gonna answer your question.

152. 27:08

     Okay.

153. 27:08

     But is there gonna be, like, an open floor part? 'Cause I have more questions for Geoff. [laughs]

154. 27:12

     Let's go.

155. 27:12

     We were trying to do Oxford debate style for this one to keep it, like, more structured and prevent, like, just-

156. 27:17

     You don't want it to just turn into a chaotic yap fest? [laughs]

157. 27:21

     Yeah.

158. 27:21

     She's trying to prevent what you and I do, where we just start yapping. Yeah.

159. 27:24

     Yeah. It's already started.

160. 27:25

     I, I was trying to control, like, both of you guys this time.

161. 27:28

     Okay.

162. 27:28

     Our chat.

163. 27:28

     I'm gonna do this answer as quickly as possible, and then I'm gonna start busting Geoff's balls.

164. 27:31

     Yeah, you can say whatever you want with your time. [laughs] You can just... Yeah. Just, that's all good.

165. 27:35

     Okay. Uh, so, uh, yeah. Uh, the, the coolest thing about Ralph back in the day was like, okay, you keep clearing the context window, and, like, is it completely efficient?

166. 27:43

     Like, probably not from a token perspective, but it meant you could leave a thing running overnight, and it would never... Like, if you just kept stuffing messages in, you would overflow the context window.

167. 27:51

     But if you just relaunch it, say, "Here's my desired state of the world. Go check the code and see what we have, and do the one next step to get us there," uh, it was a very clean way to keep most of your work in what we call the smart zone of the context window.

168. 28:04

     If you tell it just do one thing, and then we're gonna clear and restart. Um, context windows have gotten longer, um, and I will, like, give an update. I think I gave this in Miami, but that, that video is still in production.

169. 28:15

     Um, th- the, the dumb zone is really as, as much as anything else, is, is a... It's more like training wheels. Like, if you have been talking to Claude for 70, 70 hours a week and for two to three months, you probably don't need to think about the smart zone versus the dumb zone 'cause you've built your

170. 28:32

     intuition. It's a guideline if you're just getting... This is why we teach people this, is, like, as a guideline, if you're just getting started with AI, try to keep it around 100,000 tokens.

171. 28:41

     For larger, million context window, we probably revise this, revise this up to, like, 200,000 tokens. But I've regularly k- tried to keep it under 60 for the hardest problems.

172. 28:50

     I've regularly gone over 300K for things where I'm just, uh, like, kind of, like, riffing with the agent, and I'm just, like, too lazy to compact it and, and, and move on and do a new one.

173. 29:00

     Um, but this is your intuition. Like, one of the telltale signs that you're in the, that you're in the dumb zone is, like, uh, there's certain cases where the model, you know, you're 200,000 tokens in, and the model's, like, finished some work, and it's trying to get the test to pass, and it's, like, not working.

174. 29:14

     And it's, like, doing all these weird hacks, and you read the thinking traces, and it's like, "Oh, that's a test, but that's from something else, and I don't need to fix that, and that's a preexisting thing."

175. 29:21

     And you're like, "Well, no, it's not." And that, that is the moment, that frustration where you're like, "Okay, it's, it's flailing trying to make something happen," that's the instinct that you, a lot of people, I think, cultivate after a couple months working with these models.

176. 29:33

     Uh, but if you don't have that yet, then, you know, then, then this is our guideline. So yeah, context windows are getting better. I think they're getting bigger. Um, and so, like, the core, like, Ralph loop of, like, do as little as possible in every [REDACTED:marital_status] iteration is, like, less of the motivation here than, you know, the

177. 29:47

     more feedback you can pipe into the system, the more you can do autonomously. And if you can have deterministic things, making decisions, and building small prompts to give to an agent, and you don't have to remember to do that.

178. 29:59

     You don't have to tell it, "Hey, go check the PR comments and fix them," and then wait, and then someone makes another comment, and you come back three hours later and say, "Oh, check the comments again."

179. 30:07

     If you can automate that process, that's great. And that's kind of the core of, I think, what is, like, loops stuff that works today. Um, I wanna ask Geoff about-

180. 30:17

     That's, that's all I have time for. I'm sorry. I have to start, like, really keeping this on schedule. Um, okay. So now we're gonna get into the anatomy of what makes a good loop a good loop.

181. 30:26

     So, okay, part of what makes a loop good is verification. However, it seems contradictory that people are saying our job is to stop writing prompts and start writing loops when the loops with bad prompts result in agents cheating and meeting its goal by modifying the tests instead of working to pass them.

182. 30:43

     Geoff, how do you keep the model from cheating when verifying its own work?

183. 30:47

     Um, I heavily exploit pre-commit hooks, folks. Um, and I engineer in that back pressure, uh, by analyzing the work that is done. Um, the other thing I do is, um, Dex mentioned that with Ralph it was...

184. 31:02

     One of the things was everyone was trying to do compaction. Think about compaction as kind of like a lossy function, like uploading a, a video to YouTube and then downloading and uploading it 100 times.

185. 31:13

     Like, y- you're losing fidelity there, and it's already a non-deterministic system, probabilistic type thing. So the, the theory behind how Ralph came to be is like, okay, there is a dumb zone, and what I want to do is deterministically allocate everything it needs.

186. 31:30

     'Cause if it's, if it's not allocated, then it's, essentially the, the search space of what it can do is not constrained. But also leaving a bit of a headroom.

187. 31:39

     Leaving a bit of headroom. So I also... I get meat sweats when I go above 100K, even with these million context windows, and this is really important to think about.

188. 31:48

     Um, a lot of people, they think that you'd wanna use LLMs, um, at a company, and it's like, "I've got this data." It's like, sweet. Okay, you're gonna have to use a loop to batch this data.

189. 32:00

     I want you to think about the context windows as, uh, essentially... Remember the 720K floppy disk? You know what I mean? You've only got about an eighth of that floppy disk of usable memory that you can actually use for an LLM, so you actually have to batch it.

190. 32:15

     You can only allocate roughly around about Star Wars... If you go Star Wars Episode I movie script, and you tokenize it, you can actually just hold two of those movie scripts in memory before the context window is cooked.

191. 32:27

     That's around about 150 kilobytes of data o- on a text-based movie script. So be very careful about this. Something I've done for a long time, and it's very silly, is, uh, I run a model bare, um, without any skills or any markdown.

192. 32:44

     Actually, I get rid of all my skills and all my markdown and everything when the new model r- is released. Because the, the, the models actually have tastes and preferences.

193. 32:54

     For example, GPT-55 when it first came out, if you screamed at it in uppercase, it became weak and timid. But if you use Anthropic, it wants you to yell at it.

194. 33:04

     Go read the model cards, folks, like, for the integrators. Like, there is unique tastes for it. So keeping it o- keeping it on the rails is actually... It's engineering.

195. 33:17

     It's really engineering.

196. 33:21

     Thank you. Around 10 days ago, Geoff coined the term convergence engineering. He said it's where your loop slops together... Your loop... Uh, it's where your loop slop, uh, comes together as a discrete, like, system under test until it converges.

197. 33:36

     Dex, what is wrong with how we are using loops today? How do we ensure looping slop together doesn't just produce more slop?

198. 33:45

     Uh, we gotta read the code. I will actually highlight, uh, an experiment that Geoff did earlier this year, uh, I believe what was called Loom, where we had Ralph loops trying to build a software platform for the future.

199. 33:59

     And I think you built, you built AWS, and you built GitHub, and then you realized, okay, how do we, how do we give the model feedback on things that it's not good at yet, like UI testing and things like this?

200. 34:09

     Well, okay, the way you create a loop for "is this UI good?" is you give the model something like PostHog, where it's like, okay, we can deploy multiple different experiments.

201. 34:18

     We can see which ones the users use, and then rather than looking at screenshots and PNGs, the model can look at data and see, okay, uh, this one is performing better.

202. 34:27

     That must be the right color for the button. And so now you've even removed, like, the human visual taste from the equation. And, uh, all of this sounded really cool, and in the point of, like, how do we ensure looping doesn't bring slop together, I, I don't think you can.

203. 34:41

     And this is, like, a perfect example of the hype outrunning the discipline in the sense of, uh... Geoff, what's, what's going on with Loom now?

204. 34:49

     It's still there. It's on GitHub. Um-

205. 34:51

     It's, it's still there. But are you still working on it?

206. 34:54

     Uh, it's been six months because I've been looking into engineering ways o- of verification.

207. 35:00

     Right.

208. 35:00

     To im-

209. 35:00

     What was the thing you said to me? You said, "Loom's not gonna work until we get better programming languages or we get better, much better models." And that is a textbook for me of the hype is outrunning the discipline.

210. 35:12

     We're really excited about all this stuff, and by the way, like, everyone should do what Geoff did. Like, Loom is awesome. Like, go experiment. Try to push the frontier, 'cause that's how you learn where it is and what's possible.

211. 35:22

     Otherwise, you just keep using your old skills with every new model, and you assume it has the same limitations. But it's also a, a, a key point of, like...

212. 35:32

     I don't know what I'm trying to say. Like, the, this... It doesn't work yet. [laughs] That thing doesn't work yet. It will work someday, and, like, there's inevitability, but again, it's what works today versus what is hype.

213. 35:43

     So I don't know if that fully answers your question, but the answer is, like, the way to not loop slop together and make more slop is to, like, read the thing that's coming out the other end and make sure it's not slop.

214. 35:54

     Yeah, that makes sense. [laughs]

215. 35:58

     No, uh, it's, it's... The labs haven't cracked it, so what makes you think you're gonna crack it?

216. 36:03

     Yes.

217. 36:04

     Right? And this is, is... Right now it's, uh... We're all trying to figure out how to make this all work.

218. 36:11

     Yeah.

219. 36:12

     For sure. Skeptics say that loops fail quietly. They either spiral forever on your dime or the agent declares victory early on, on a half-finished job. Execs are already starting to question token spend.

220. 36:25

     Greg, when does a loop pay for itself, and how often is this actually the case?

221. 36:30

     I don't think they fail quietly for it. [laughs] I think they fail very, very, um,

222. 36:37

     loudly, especially when you're looking at your bills. Um, but there are cases, like, as I said, I do loops, or I do engineering, I would say more so. And there are cases where I think doing loops is very valuable, or doing, um...

223. 36:54

     Or making a explicit decision that you're going to pace- pay, pay for the cost is very valuable. So the concrete example here is we do security scanning after, uh, on our PRs in local and after our PRs even land because they will always find some things that are real, um, that we have overlooked, and they sort of

224. 37:13

     beat humans on the, on the code review. And it's expensive. It costs us, I think, like, five bucks a PR or something like that to run all the checks that we want.

225. 37:21

     But that's where we made an explicit decision that it's worth it. Um, there are also cases where, like, if you look at the very, very well-specified systems such as, um, all the experiments with, uh, Next.js rewrite or a BAN rewrite in Rust or, uh, running a browser.

226. 37:38

     Like, cases where you have years and years of test suite and specifications built in around those problems where you can really, really were- well verify, um, the outcomes, then r- then looping and getting to those, um, those results seems to work.

227. 37:55

     BAN in Rust seems to work pretty well from all I can tell. Um, so there are definitely cases. And then there are cases of, um, usage that I think, that I do pretty often

228. 38:06

     m- where you can imagine, for instance, building prototypes. I do prototypes of products that we should be doing, uh, doing at Sentry pretty often. Those are gonna be throwaway, so I'm gonna just slash go on them and forget about them.

229. 38:18

     And if we like them, then I'm gonna start reading the code, and I'm going to be mortified, uh- [laughs] ... and we're gonna go to the square one and start speccing out what we actually meant, uh, and, and sort of go towards that solution.

230. 38:30

     But it's going to be much, much more involving of, of human in the loop. Um, so, so broadly, I think they have place. Uh, but as Dex said, they-- the, the hype is what I have a problem with.

231. 38:40

     The hype is, uh, outrunning the discipline, as he said. And I also agree very strongly with the point that you should just try things. You should just experiment yourself, try to see what actually works for you, where, uh, where the, the cookie crumbles.

232. 38:54

     Um, and, you know, spend less time on Twitter, I think is healthy nowadays. [laughs] [laughs]

233. 39:01

     Yeah, good points. A good cost-conscious loop has to track state to know what it's already tried. That memory lives somewhere on disk in Git, increasingly in a shared memory store that many agents read and write, especially as you go [REDACTED:marital_status] player to multiplayer with agents.

234. 39:18

     You end up with this access control problem that can't tell which agent wrote which memory and who can read it. That might be fine for a POC, but it's definitely not for production.

235. 39:27

     The tension lies in this. Shared memory is what loops use to learn from each other and converge faster, but scoping it per agent to solve the access control problem isolates them and then kills that shared learning.

236. 39:40

     Ian, how do we solve the shared memory store access control problem so loops can converge faster?

237. 39:47

     Great question. [laughs] Wow. Wow. Um-

238. 39:51

     If only someone was working on a product that could help you.

239. 39:53

     Only if someone was thinking about it. [laughs] I mean, actually, I've-- This is, this is an unsolved problem, first and foremost. Like, let's be really honest that our, our access control systems weren't designed for this, this world where machines are acting and reasoning on, on behalf of, uh, about a half of us.

240. 40:07

     But broadly speaking, I think, like, some of the beginnings of the substrate are starting to emerge, and if we ignore cloud IAM and all the other stuff for a hot second, like, markdown's pretty great.

241. 40:16

     And so really if we were to say a memory is markdown, for the purposes of the conversation, the really question is, like, what things and how do I share these markdown files and use that as a memory?

242. 40:26

     And then how do I attribute sort of, like, access control around those things? And if we were to use that model, I actually think we have, like, the basis for, for most of it today.

243. 40:35

     It's just unwieldy to think about. Um, a good example would be, I had this tweet recently, um, I was playing with Notion, and we use Notion a lot at, at Keycard.

244. 40:45

     Uh, but I really just wanted all my Notion things available to me as markdown files. And because it would-- just made it easier for the agent to work with it instead of going through MCP, and I was a bit of a CLI maxi and was looking through that.

245. 40:57

     So I think what we're missing really is, uh... And MC, you know, Dex and I debated this last time about MCP, but, like, we're really-- The challenge really is how do I present a world to an agent so that it can understand it, and then how do I attribute what can access that at any one point in

246. 41:10

     time, and how to make that wieldy for anyone to do it? And I don't, I don't think we've actually really cracked it, but certainly there's some beginnings of patterns that make a lot of sense.

247. 41:17

     Yeah.

248. 41:20

     All right. Now that we discussed the anatomy of the loop, we're going to debate the future of the loop. Like, are we essentially well positioned now for software factories?

249. 41:28

     Has loop engineering gotten so good that, like, we're ready for the full software factory? And we'll start with, if loops are only good for verifiable tasks, that means fully autonomous software factories must be able to verify everything they do.

250. 41:42

     Greg, is this realistic? What other parts of good engineering work, such as deciding what to build, whether the abstraction is right, and what trade-offs are acceptable?

251. 41:51

     Uh, is this realistic? I think if, if the compute is free, that would be pretty, pretty good beginning. But, um, [laughs]

252. 41:58

     uh, I thi- I think we're getting to the point where, um... Or let me put it this way.

253. 42:05

     The decisions that you are making as a human in, in the loop are the decisions of, like, design, architecture, the, the, the important ones that, um-

254. 42:16

     That I would say I wouldn't trust the, the agent to do for me, uh, and I don't see the future where that becomes reality yet. I think the reason for that is when you're, when you're looking at large organizations, and I think any engineer who has had, like, years of experience will tell you it's not always about

255. 42:32

     what you should build, what you should build, but also about what you shouldn't build. What are the actual right trade-offs, where the complexity is that you wanna, um, that you wanna invest in versus where you should be, um, investing in maximal simplicity.

256. 42:46

     Um, in my experience, agents love complexity. They will keep adding to the stack, um, unbounded. And so I think we are, we are shifting the, the, the post. We are getting to the point where as we are adding more validation, more, uh, semantic verification, they are able to do much, much more.

257. 43:06

     Um, and I'm not, uh, not neglecting that. But I do want to be in the loop for the actual architectural decisions, and I do not see them, uh, taking that over any- anytime soon.

258. 43:19

     Thank you. Shopify's head of engineering told everyone at Kircher's Compile conference that your job is just to write loops. Steinberger tweeted, "Here's your monthly reminder that you shouldn't be prompting coding agents anymore.

259. 43:31

     You should be designing loops that prompt your agents." The view count on that tweet was eight million people.

260. 43:38

     Geoff, you've said in your original Ralph blog post, "Loops need senior expertise, and then sometimes it tops out at ninety percent of the way there." So is just write the loops advice, is that safe to give to three thousand people or eight million people, or only the people that have really learned to tune a loop in the

261. 43:57

     first place?

262. 43:59

     Yeah, it's an interesting thing. Um, it's really hard to tailor and teach, like, like, what you should do and what you should not do to a broad-- that broad of an audience.

263. 44:10

     So all I can do is write to... When I was originally writing, I was writing to my peers, like, people I looked up to. For some reason, I-- it clicked in my head that everything has changed, the game has changed.

264. 44:23

     But I was really concerned some of the, some of the best people in functional programming and, like, and, like, real, real peers. I remember, like, um, Fly IO, uh, Thomas Patek, he-- it, it, it made him click in his head, and he's got a blog post saying as such.

265. 44:38

     So I started writing for my peers and the people I looked up on. To disseminate it down to, like, the entire world,

266. 44:48

     it's hard. Um, but in the same s- same sense, you go to YouTube, you've got people who are creating things that have never created things before. They go to sleep, and they wake up, and they've got a brand-new Discord bot.

267. 44:59

     It's magic. I want people to remember that it is actually a pattern for allocating... Basically, it was a pattern as an orchestration pattern.

268. 45:13

     It was condensed down to a while true loop using cat, 'cause cat is the simplest teaching primitive. To, to... If you wanna teach something, you gotta make it really simple.

269. 45:22

     Make it really simple. So cat prompt, i.e. you engineer the prompt, what it's gonna be. You use the file system as state. You recycle the context window, run the loop.

270. 45:31

     It's a little bit meme-y, but it go- it, it was just bang for buck. It just works. But the entire intent there is that there should be some sort of, like, PID controller on top or some sort of factory or some so- some sort of, some sort of determinator as such, saying whether a loop should continue or

271. 45:50

     not. Um, it's really hard. Um, is it safe or unsafe? Safe is an interesting word. Um, no one should be using coding tools on your local laptop, and this is not because of AI, but this is 'cause of NPM supply chain attacks.

272. 46:08

     This has been true. I tried, tried cracking this problem for seven years. Um, and, like, it's now on the attention again that we-- that, like, AI could be unsafe running unsafe commands.

273. 46:20

     But, like, your software development practices day to day in your workplace are already unsafe, folks.

274. 46:26

     So, um, fix that, then these, these techniques start to become safe.

275. 46:34

     Thank you. Dex, you tweeted that most engineers are seeing a two to three X speed up from coding agents, and that's realistic, and if you try for that a hundred X speed up, you're going to get lost in the meta meta problem of optimization, and you may never get to that life-changing ten X speed up that is

276. 46:49

     possible by staying pragmatic. If only two to three X is what's possible, how do we ever get to a fully autonomous software factory?

277. 46:58

     Yeah, and I think this is, um... it's, it's a, it's a good question. I, I think there's like there's, there's, there's a mix here too, right? We're talking about what works versus what is hype, and I think it is definitely worth-- I wanna highlight, like,

278. 47:11

     you should try again. Like, you should try to push the frontier and do the things that might not work today, but you should not assume that all your work has changed just because you saw something work on Twitter, basically.

279. 47:23

     Is like, don't throw away all the things we've learned. Don't go out of your way to cast aside this, you know, decades-long career of software engineering that we as a, we as a, uh, community have built up and put together.

280. 47:37

     Um, and it, it gets back to, like, what I see as the biggest anti-pattern for how people set about designing and creating their software factory, which is they say, "Okay, cool.

281. 47:49

     I'm gonna go away for three months, and I've read a bunch of blog posts, and I'm gonna go make my software factory, and it's gonna be the software factory.

282. 47:56

     It's gonna be the future of how we ship everything." Uh, and then you come back three months later, and you never touched the problem. You never put it in anybody's hands.

283. 48:05

     It's just like any product. You're building a software factory. You're building a product for your teammates. You're building a product for five, ten, five hundred, five thousand engineers. Uh, and the, the right approach is to start small- And iterate and figure out what works.

284. 48:19

     Try things, they might not work, they might work. But the way we learn AI and how to use it effectively is through building up intuition, which is why you should try a bunch of stuff that probably won't work, but you should acknowledge that.

285. 48:29

     You should not try to push, like, through that, through that frontier. Um, and so my advice is kind of like instead of trying to automate everything end to end, build these small incremental loops throughout your system and you will wake up one day and you will be moving two to three times faster while still being able to

286. 48:45

     read the code, while still owning the architecture. And so you don't have to throw away everything we've learned and everything we know and all your intuitions just to get to this place.

287. 48:54

     And so I would, I would caution people, like, go figure out how to made tw- move two X faster or three X faster because you're gonna, like, blow everything up by trying to go 100 X faster.

288. 49:04

     Um, and, uh, yeah, I mean, if... Can you imagine if every software en- engineer in the world was two to three X faster and had a, like, near human 99% level human level of quality?

289. 49:16

     That would change every [REDACTED:marital_status] enterprise in the world, every [REDACTED:marital_status] startup in the world. It would change the entire math. And we're trying to, like, go to... Just don't, don't go too far.

290. 49:26

     Shoot, shoot for what you can do, build things up iteratively. You'll learn a lot, and you'll be ready for when the next models come out and you can go five X, 10 X faster.

291. 49:36

     Thank you. When it comes to verification, it's not just about verifying the work, it's also about verifying who did the work. Employers have made it clear that humans are responsible for the code they ship, whether an agent wrote it or not.

292. 49:50

     Yet only one person or agent can sign a commit today. Ian, are we ready for software factories to be writing and reviewing all code if we can't determine who did the work and who the work was done on behalf of?

293. 50:02

     I think, yeah, actually, I was, I was playing around with this problem, like, two weeks ago. So, um, first and foremost, we have a problem. Git only allows one signer on a commit, so we gotta fix that.

294. 50:12

     Um, two is, you know, there's some things around SOC two and compliance, right? But I think, like, more importantly, the way to think about agents is, is kind of how we think about service ownership in a large organization.

295. 50:23

     It's like at some point a human has to be attributable for an agent's actions. Like, there's no world where it's gonna be like an agent is its own entity and its own attributable thing, and somehow it has liability.

296. 50:32

     Like, the only people who have liability are the people that can have consequences, right? And that always has to be grounded in it being a human. And so if a human designs a loop and that loop presents bad software, guess who's attributable for that liability? [laughs]

297. 50:44

     It's gonna be the human, right? Like, the society doesn't function if we don't have liability. Uh, it just doesn't work. There has to be consequences to damage. There has to be consequences to bad decisions, and what those consequences are, are obviously a gradient based on the, on, on the damage that's done.

298. 50:58

     And if we don't have that, nothing works. And so I'd actually broadly say there is no world where a human is never responsible. There's always a world where humans have responsibility.

299. 51:08

     And the question is whether it's a human or a corporation, which is a group of humans, um, the question is, is how does that change the way that our systems work today?

300. 51:16

     So today with Git, I can sign a commit that says, "I did it. It's attributed to my public key." And that's cool for last generation's way that we thought about software.

301. 51:26

     Now, in the future, when I tell an agent to go do something on my behalf, it generates a loop, it generates a bunch of code that goes to production.

302. 51:34

     I have to be attributable to the initial fact that I signed that thing, the, that intention to do that. And we, we simply did not have the substrate required, um, to do it, although I do think that's gonna change pretty quickly.

303. 51:46

     Um, this isn't like something that's, like, crazy difficult for us to break, but we have to rethink the way that we think about attribution across the supply chain and, and the SDLC.

304. 51:55

     And this is not a new problem, right? Like, supply chain security has always been an issue. It's actually the biggest challen- one of the biggest challenges we have with agents, and we've always wanted to have more deterministic, uh, pathway and sign signature chains across the supply chain, and this is just how do you do that for s-

305. 52:09

     first party code versus, uh, third party code.

306. 52:12

     Well, Ian, um, j- j- just to shoot straight, um, our profession is a bit of a clown show. [laughs] We actually don't have liability at a personal level. [laughs]

307. 52:21

     Yeah.

308. 52:21

     Like, we call ourselves engineers. We're not really engineers.

309. 52:24

     That's true.

310. 52:24

     Some of this stuff is gonna get really complicated, folks, and maybe we need to revisit these topics.

311. 52:30

     Yep.

312. 52:32

     And now that we've had our main debate section, to wrap it up, each member on each team will get two minutes to, uh, wrap it up and describe their final thoughts before we decide who the winner was.

313. 52:44

     Uh, Greg, would you like to go first?

314. 52:46

     Yeah, absolutely. Um, it's, it's interesting seeing at how man- how many points we actually agree with each other. Uh, but that's how, how this goes. I, I do think that the biggest point that I'm trying to, I have been trying to drive here is, um, where we are, whether we are, where we are trying to be, and

315. 53:02

     then what you're actually hearing from the ever-present hype, uh, hype loop, and that's where I want to sort of double down and tone this down. I basically wanna say try things.

316. 53:15

     Think for yourself. Don't lean into the bubble. Uh, don't lean into the hype because you will find what works for you, and you will find where the system breaks the best by doing it yourself.

317. 53:25

     That's always has been how, um, how humans learn the best is by practice, not by, you know, watching YouTube. Um, and

318. 53:34

     ultimately, that's, that's what I'm trying to do. And I'm slightly skeptical, uh, when it comes to, you know, allowing the full loop to run because I see the qualitative results not being up to scuff for my, uh, my requirements.

319. 53:47

     Uh, but I am optimistic in general. I am optimistic because I've seen how much more I can do and what are the types of problems I can address today that I couldn't address even a year ago, uh, let alone, let alone earlier than that.

320. 53:59

     And, uh, and it's going to get better. But I'm also not worried about my software engineering career. I don't think we're going away. I think we are still going to be important piece of this whole software factory or whatever the next, um, sort of hype bubble is going to become.

321. 54:18

     Ian, would you like to go next?

322. 54:19

     I would love to. So, the train's left the station. This stuff works, um, and there is a real productivity increment, right? Like, more stuff is getting done. I just mean all the stuff getting done is good, but more stuff is getting done, and a good percentage of that stuff is actually good, right?

323. 54:35

     I think we can agree to that. And second is now we have competitive dynamics where it's no longer possible for a company to sit back and be like, "Hey, we're gonna sit this loop stuff out," right?

324. 54:45

     "I'm gonna sit this coding agent stuff out." Like, that, that's over. We're, we're past that. Like, that train leaves the station. As soon as that train leaves the station, everybody in the world starts saying, "Holy shit, I gotta keep my stuff together.

325. 54:55

     We've gotta stay up with the Joneses." And they have to because we live in a very competitive, capitalist society, and that's also, you know, not getting into politics with it, it is what we are.

326. 55:02

     And at the end of the day, the question is, is not really, um, will it happen? It's when it happens and to what degree over time. And I don't think there's a choice but to actually stay up to date, and we're all just kinda holding on to a rocket ship ride where we don't actually know what the...

327. 55:19

     We don't really know the trajectory other than it feels real fast with crazy acceleration, and sometimes, like, we're way ahead of each other, and sometimes we're way behind. But I don't think you have a choice not to be, one, figuring out what loops are, two, figuring out where you can apply them in your code base.

328. 55:32

     There's gonna be places where, hey, this is highly verifiable. This is a problem that, like, computers can solve. Doesn't make a lot of sense. Cough connectors is a good place of, like, you know, last 10 years, the amount of companies that have made money because they're basically just connector farms.

329. 55:46

     It's no longer a lot of value there, right? If you can automate the connection, connector creation. So there's places in software that you can apply loops to today and get real value, and there's places where you probably shouldn't.

330. 55:57

     And you should decide where that is, and it's probably the core value of what you're creating, right? And so that's how I think about it, but broadly speaking, train's left the station.

331. 56:06

     The productivity curve is what drives society. The ROI, so how much more productive we are as a society, is what drives GDP. What drives GDP is ultimately where the dollars go, um, and how capital gets allocated.

332. 56:18

     Kick it over to you, Dex.

333. 56:20

     I eagerly await the world where the lights-off software factory is feasible. The li- I, I would love a world where we don't have to read the code, where we, where we can just do everything.

334. 56:32

     Um, if you watched my talk on Tuesday, I, I think this is actually a problem we can only solve at the model level right now. I don't think the harness can do it, uh, unfortunately, 'cause I love building harnesses and doing context engineering.

335. 56:44

     Um, so my advice is, uh, pay attention to Geoff. Uh, let me know when Loom is actually working. And until then, uh, use loops, but not like that. [laughs] [laughs]

336. 56:57

     Love it.

337. 57:00

     Yeah.

338. 57:00

     So, so much to say, folks. Um, factories represents where we're heading to the future. Like, i- it's, it's essentially like a perpetual energy motion machine, right? This is like s- s- it's the pipe dream.

339. 57:14

     Companies are only just getting founded today and get- receiving their rounds today.

340. 57:20

     Don't think you can just take this and implement it in your company 'cause it's just generally not solved in market. But I will say, uh, to add to the monolog- monologue, if you try to run loops or try to build a factory and you're using Python, it's gonna be a clown show. [laughs]

341. 57:39

     If you do it in Ruby, it's gonna be a clown show. Static types are a form of verification, folks. I encourage you to, to come up with a couple of CATas and a couple of experiments.

342. 57:50

     Try running some loops. Build an application in Ruby, and then try to modify it again with these loops, and you'll see the maintainability mess. And then try doing it with Haskell.

343. 58:01

     Yep.

344. 58:01

     I don't care if you don't know Haskell. It doesn't matter if you don't understand Haskell. The LLM understands Haskell, and you can actually prompt the, the agent to explain this as if you, uh, uh, to, to your son or daughter.

345. 58:15

     So I'm not sure even code needs to be readable these days, but this is frontier, frontier thinking. It needs to be explainable. So I'm playing around with different domains of verification here, but, uh, one thing's for sure, um, type systems are in, very much in.

346. 58:32

     Rust is very good because how the ecosystem models some types. And because supply chain was mentioned, I need to say it has been, uh, 10 months now, I do m- I minimally use any open source software.

347. 58:46

     Minimally. I just generate it to my requirements, and then when a s- supply chain attack happens, I'm like, "Didn't affect me." All software has security flaws, but it's about minimizing the blast radius.

348. 59:00

     And the thing is, if you, uh, deal with open source project, the person that's on, on lead, the maintainer, what else have you, you, you can't tool call a human.

349. 59:08

     That's not AGI. You wanna be vendoring all your source code as much as possible so the a- so the agent can actually modify and, uh, in, uh, with either loops or other prompting.

350. 59:19

     You need to own your supply chain.

351. 59:21

     Awesome.

352. 59:22

     And that's a big clip-

353. 59:22

     Thank you so much for that. Unfortunately, we have to close it at that. But we can now decide our winner. Um, show of hands in the audience, if your mind was changed and you're now on Dex's side, raise your hand.

354. 59:37

     Let's go. [laughs] [laughs]

355. 59:39

     Okay. If your mind was changed and you're now on Ian and Geoff's side.

356. 59:44

     Yeah.

357. 59:44

     Yeah.

358. 59:45

     I kinda couldn't see with the light. What'd you guys think? [laughs]

359. 59:50

     Wait, let's do it again.

360. 59:51

     Pretty close.

361. 59:51

     It's pretty close, yeah.

362. 59:52

     It's impossible. The, the lights are so bright.

363. 59:54

     Very difficult to decide.

364. 59:54

     All right. Well, that was a good debate. Thank you so much for listening. Um, we'll see you next time. [clapping] [outro music]