Recursive Coding Agents

Talk transcript

1. 0:00

   Hello there. My name is Raymond Weitekamp, and today I'm gonna talk about recursive coding agents, which is this idea of applying the lessons of recursive language models, RLMs, uh, to coding agents.

2. 0:15

   This is some work that I have done both in my independent research, um, RAW.works, uh, and also more recently

3. 0:25

   in my role at OpenProse. So, to motivate this a little bit, we all want outcomes. We all want agents that are working on our behalf. We want reliable coworkers that are getting things done while we are doing something fun, while we're out on a hike, while we're cold chilling, while we're doing the do.

4. 0:45

   And my argument, and my experience is that the bottleneck to this is not intelligence. The models are intelligent enough. They know all kinds of things. They know the entire internet, but they can't reliably deliver outcomes, and so I can't trust them.

5. 1:07

   So as a very simple example, you know, one day I get almost a fully working SaaS app from a single prompt, granted a long prompt.

6. 1:16

   The next day, and I swear this actually happened,

7. 1:20

   Claude Code empties the entire contents of my Solana wallet. Oops. Okay. So that [chuckles] doesn't really instill trust. So, uh, at the bottom here, we've got this pr- this progression, okay?

8. 1:33

   And we all wanna move towards the, the one on the right where we're just sort of sitting there and meditating and, and things are manifesting. And so where does that come from?

9. 1:41

   This is from the AI Engineer Code. It's actually from the back of the T-shirt. AI Engineer Code, November twenty twenty-five. Man, I hope, I hope you were there. If you weren't, watch it on YouTube.

10. 1:55

    It was, it was amazing. So here's the thesis. The thesis is: today's agents are mismanaged geniuses. The intelligence is there, and the missing layer is how do we specify and manage and reuse and verify the work.

11. 2:10

    So this, uh, framing, this phrase, the mismanaged genius, uh, comes from Alex Zhang, Zed Li, and Omar Khattab at MIT. Um, and Alex and Omar are, uh, part of the authors of the original Recursive Language Models paper.

12. 2:25

    Uh, I've also talked a little bit about this recently on Turing Post. Um, I forgot to mention that these slides are actually a website, recursivecodingagents.com. So you can click on them, uh, by going to this website.

13. 2:40

    So everything I'm gonna show in here is, is interactive. Okay. What are recursive language models? So I like to say that in an RLM, the context itself is the object of computation.

14. 2:54

    Um, and this is essentially a marriage of tool calling and reasoning. We're gonna talk a lot more, more about that in the next slide. But the idea is that the full prompt is not a simple user query.

15. 3:08

    The full prompt is a variable. The full prompt could be a file or many files. Um, and we have this Read-evaluate-print loop REPL, um, that the agent is interacting with.

16. 3:21

    In the original paper, that's Python. And the RLM is instructed to operate symbolically on that prompt, so don't just read the whole thing into your context window. Um, explore it symbolically.

17. 3:36

    And, uh, even more, you don't even directly explore it symbolically, or maybe you do a little bit of poking around, but have, uh, other LLMs, uh, and I guess other RLMs if you allow the recursion depth to be, to be greater than one,

18. 3:53

    uh, have these, uh, other recursive, um, subagents, and, and again, we'll get, uh, a little bit into the, the weeds of the, the lingo, um, sub-RLM, sub-LLMs, uh, do this symbolic manipulation to pick apart the answer and then work our way back up to a final answer.

19. 4:15

    So it looks something like, like this in this, in this tree below.

20. 4:20

    So my take here is that RLMs are the new reasoning models, and I see this as the next paradigm of test-time compute, inference-time compute, whatever you wanna call it.

21. 4:33

    And why does it seem obvious or maybe like, "Hey, why is this even a thing?" Um, I think it's very elegant because it's a very elegant marriage of two things, reasoning and code execution.

22. 4:46

    So the code execution is reasoning. Um, and so instead of we, we had long, um, we had chain of thought as a prompting strategy that evolved into reasoning models that explicitly expressed the chain of thought as their reasoning tokens.

23. 5:02

    We already had function calling, tool calling, parallel tool calling, um, and RLMs really puts that together in a way that gets amazing results. So three very simple examples, one from the original paper, Oolong.

24. 5:16

    The RLMs can process information that is many orders of magnitude larger than their context window, tens of millions of tokens. Uh, what I showed in my own independent work was that the default RLM harness is itself a really powerful memory system.

25. 5:35

    Um, so RLM with no modifications is essentially like a top ten memory system and like, you know, up there with all the people custom-making memory systems, and there's probably billions of dollars going into that.

26. 5:48

    Um, and, uh, with a little bit of modification, you, you can get really amazing results, uh, using it as memory.

27. 5:57

    I was also able to show state-of-the-art results where the RLM framework, and specifically the DSPy

28. 6:04

    implementation of it was able to get state-of-the-art, uh, results on long reasoning tasks. Now, there's this new benchmark, LongCoT. I won't go into the details in depth, but the idea of this benchmark was that the problems are hard specifically because they require so many, um, steps of reasoning, uh, in, in the, like analysis sequence or in the

29. 6:28

    chain of thought, um, that most, uh, reasoning models, including the top ones, can't hold the thread for long enough. Um, if you allow the, uh, RLM to solve the problem, uh, using a combination of code and recursive calls to subagents, then a very small model, Qwen 3.5-9B, you could run this on

30. 6:53

    a laptop, uh, can actually beat... So Qwen 3.5-9B as an RLM can beat Opus and, um, and GPT 5.4, all the top frontier models as LLMs on these long reasoning tasks.

31. 7:11

    So they're extremely, extremely powerful. So powerful that they are [chuckles] arguably too hot to benchmark. So two examples here. On the left, a very high-profile, uh, case where, um, the Symbolica team has this RLM agent harness called Agentica.

32. 7:31

    Within hours of Arc AGI 3 being released, where the top scores of all the frontier models were around two or three percent, the, uh, Symbolica team showed 30-something percent.

33. 7:46

    This is crazy. They blew it out of the water within hours using RLMs as a framework. So much so that this, uh, very much upset the Arc Prize team.

34. 7:55

    And so, uh, they gave them what I'm interpreting as a consolation tweet, which as far as I'm reading the situation, was essentially saying, you know, "La-di-da, congratulations, um, but you didn't solve the problem the right way, and we don't like RLM harnesses, and so, uh, you can have this nice tweet, but we refuse to actually do the

35. 8:19

    full private part of the Arc AGI evaluation," uh, which to me is just insane. Uh, in my own, uh, work and, and w- on the LongCoT benchmark, um, my results, as well as Alex, uh, from MIT, the RLM first author, uh, encouraged, let's say, the, uh, the, uh, leaderboard maintainers to actually make like a separate open harness

36. 8:44

    leaderboard, uh, so that the ar- the results of the RLMs could be showcased without contaminating the original intent of the leaderboard, which was basically no tool calling is allowed.

37. 8:55

    So my take on this is I don't care. I don't care whether it's latent space or reasoning tokens or code execution. I want results, and I want AI programs that get those results.

38. 9:08

    Okay, so this can feel close to a lot of other things, and I h- I built a little rubric. There's a companion GitHub repo for this that you can go through if you wanna see.

39. 9:19

    Um, and so what do we need to be an RLM? We have an executable environment. The prompt is externalized. There's code that's actually the thing calling the model. The model is able to pick the decomposition of the problem into the subcalls or subagents, and the state itself is staying symbolic, right?

40. 9:36

    So obviously, plain LLMs and RAG and things like that don't, don't meet those. Coding agents and subagents and loops, they get close, but they're not quite there. And again, this rubric here is not to like start fights or nitpick.

41. 9:51

    It's just trying to explain like what, what's the essence of, of RLM, uh, and recursive coding agents. Another example that's close, but no cigar would be hard-coded MapReduce, and I would put, um, uh, this, this project called LambdaRLM in that category, which is essentially a way of saying, "Okay, I'm decompose the problem, uh, using lambda calculus into

42. 10:12

    a MapReduce," and then there are like LLM calls in that executing the MapReduce. But the, but the LLM is not deciding or the RLM is not deciding how to decompose the problem, and that I see as like a key element of this that makes it very agent native, you might say.

43. 10:32

    Okay, so now RLMs, what about recursive coding agents? Okay, it looks the same to me. We just swap RLMs and LLMs for agents and subagents, and don't we have the same thing?

44. 10:43

    Uh, and yeah, you do. And I think that you could take this perspective of, "Ha ha, trick question," like, RLM is a coding agent and it's already recursive, so blah-di-blah.

45. 10:58

    And that's fine, and I don't think that argument is wrong, but it doesn't really move anything forward. And so what I'm interested in is this question of: How can we apply the principles of RLMs to coding agents and make them actually useful for coding agents?

46. 11:15

    And I've been very obsessed with this problem since the October, uh, RLM blog post came out in twenty twenty-five. Okay, so I show some of my experiments on recursive coding agents.

47. 11:27

    Um, the first one i- was simply wrapping Alex's RLM package as a CLI. So the idea was, I just wanna give my coding agent an RLM as a tool call.

48. 11:42

    So let's say we go, we need to go

49. 11:45

    s-sift through a hundred million token corpus. Uh, well, now it just uses that tool, and then the RLM does the RLM thing. So that's interesting, and it can be very useful, uh, and a few other people have built, uh, things like that.

50. 12:02

    Uh, but then I thought, well, what would it really mean? What would be possible, or how could it be possible To make the coding agent fully recursive. So the coding agent harness, like, calls itself, like the exact version of itself.

51. 12:19

    Um, and how might you implement that? And that is what I called Y-Py. Uh, Y stands for the lambda calculus, uh, Y combinator, and Py, in case you haven't heard of it, is a really awesome coding agent.

52. 12:36

    It's incredibly minimal, and it's specifically designed to be extensible. So Mario, uh, wants you to write extensions for Py for new features that you would like, rather than trying to stuff your ideas into the, the main, um, agent.

53. 12:53

    So it's meant to be a very minimal core that you extend however you like. And when I originally had the, uh, recursive coding agent idea and wanted to do it with Py, I was not able to use Py extensions to achieve this goal.

54. 13:09

    Um, and so I had to fork it instead. I'm very excited to report that in anticipation of this talk, I revisited this, and now Py has evolved, um, and the Py extensions have evolved such that you can, uh, make it fully recursive with a pure extension.

55. 13:26

    So I have both the pure recursive, um, extension Py recursive, uh, package, as well as the Y-Py wrapper that is like a convenience wrapper for this. So this is very quite literally a recursive coding agent in the sense that Py calls Py, calls Py, calls Py.

56. 13:42

    You can set the depth however you want. Um, and now I wanna show a few other notable projects in this space. So obviously, there's the original, uh, implementation from Alex, RLM.

57. 13:54

    DSPy.rlm is, uh, my go-to, uh, when I'm especially doing benchmarking. Um, that's how I got all these amazing results, uh, on some of these benchmarks. Axe, I think is a very interesting one because it's incredibly agent native.

58. 14:12

    So the Axe started out as a TypeScript variation on DSPy. When RLMs came out, they obviously implemented it, uh, but they did it in this way that's, um, enables an Axe agent to, like, write a whole TypeScript interface to another Axe agent and go all the way down, uh, the, the recursive rabbit hole, um, which I think

59. 14:36

    is really cool and very interesting. Just to showcase that this, like, REPL could be anything, there's a, an example from Dan at OpenProse who made the Unix RLM. This is pure Bash, and the environment is just the Linux file system.

60. 14:52

    So that's a whole nother angle of thinking about, uh, what's possible with RLM. And then lastly, and we'll talk more about OpenProse, um, but, uh, OpenProse as a language, uh, actually enables you to convert any coding agent into an RLM, and I'll talk a little bit more about how to do that towards the end of the talk.

61. 15:12

    Uh, and then the OpenProse repo also contains a harness that executes. It's a coding agent harness that, uh, will let you use Codex, SDK, or Claude Code underneath and do an RLM-style execution of these prose programs.

62. 15:28

    So is Claude Code an RLM? This is like the, the question that keeps getting asked, and the original answer on the day, uh, release day of the blog post was, "No, no, no, it's not."

63. 15:41

    Uh, but it was literally the first question that was asked or, like, asked on the very same day, uh, to the launch tweet. Um, and it's basically saying, "Hey, this is Claude Code subagents," right?

64. 15:51

    Uh, go back to my rubric if you wanna like, dig into some of the, the nitty ditt- nitty-gritty details. Um, but arguably now it is. So, uh, o- over here on the right, we see Omar saying, "Hey, congratulations, Anthropic.

65. 16:06

    Uh, Claude Code is finally an RLM now that you have dynamic workflows." So, so what changed? And I, I think this is an interesting way of explaining, um, what's powerful about recursive coding agents and what RLMs even are, um, by using this example.

66. 16:21

    So dynamic workflows were released just a few weeks ago,

67. 16:26

    um, and they make Claude Code recursive or capable of doing these recursive, uh, workflows. Uh, I would highly encourage you to read this blog post called "A Harness for Every Task."

68. 16:37

    It shows six different workflow patterns that are very powerful. Obviously, there's many more that you can achieve. And just to, to show it, I, I wrote two, uh, workflows for Claude Code.

69. 16:50

    Uh, one that is explicitly not an RLM, so this is like a hard-coded MapReduce workflow, and one that I'm arguing is that is, uh, you could kind of think of it as like deep research over a file system.

70. 17:03

    So, you know, pick, pick a handle, um, assign that to an agent, have it go do some analysis, bring back what it finds, et cetera. Uh, again, these are in the companion repo.

71. 17:16

    Uh, and then now to OpenProse. So, so dynamic workflows are cool. They're only in Claude Code. They're also not the only way to do this. So what if you don't like Claude Code?

72. 17:26

    Or what if you do like Claude Code, but you don't want to use dynamic workflows? You want something else. So this is what OpenProse is all about. So OpenProse is technically a programming language, but it is not compiled by your computer.

73. 17:39

    It's compiled by your coding agent. Um, it's a markdown spec. It's logical English. You don't need to learn any kind of crazy syntax. And there's actually a command prose write that will get Claude Code or Codex or your favorite or Py or your favorite o- coding agent to write a .prose.md file for you.

74. 18:01

    Um, so in that way, it's similar to the, the, um, Claude Code, um, UltraCode command, where it, like decide and write the workflows for you.

75. 18:13

    And the, the Prose has the ability to turn any agent that's got a file system and subagents into an RLM. So this is an open source repo, you can check it out.

76. 18:26

    Um, and I've also written, uh, a little bit more in depth about this for Turing post in this article. And the key thing that I want to bring up with regards to the RLMs is that OpenProse can explicitly declare the subagent work.

77. 18:42

    So again, I've made two kind of demo prose.md files that are in this companion repo, if you wanna dig into the code, I'm not gonna do that here in the slides, uh, where you can break a problem up into smaller pieces that are assigned to subagents, verify the work of those subagents, um, in the, in the parent

78. 19:02

    agent session. Uh, and what's even cooler is you can actually, in OpenProse, two of the features that I added to the language are that you can add skills and tools as explicit dependencies.

79. 19:14

    So you can imagine a workflow where a certain subagent needs a very specific skill to do its role in the workflow, or, uh, must have access to a certain CLI tool, for example, or it can't run and do its job.

80. 19:31

    And so there's a way in Prose to actually wire those in as dependencies to ensure that, um, not only that the way the work is done, um, is what you want, but actually that the subagents are specifically configured with the tools and skills that they need to successfully do the work that you are declaring in the Prose

81. 19:54

    contract. Okay. Super cool. What can you actually do? So I've got two examples from Claude Dynamic Workflows, two examples from OpenProse, uh, repo scale migrations. This was kind of the launch post, uh, example refactor, like a huge thing, uh, all with a big swarm in parallel, and then merge the whole thing together.

82. 20:15

    That's super cool. Um, this idea of, like, go after a directory and then deep research or deep analyze, uh, or deep process in some way recursively, uh, inside that is, is another example, uh, I have here.

83. 20:30

    You can do audits, uh, bug sweeps. You can do adversarial things, such as having a skeptical agent or, um, you know, a red team, uh, set of agents, uh, that are going to, uh, try to, uh, improve the system adversarially or in parallel.

84. 20:51

    And then, uh, one really cool thing that you can do with OpenProse that I just added recently is kind of goes back to my very first slide, right? So like one day I get this amazing result, the next day they trade away all my, all my, uh, cryptocurrency, which is very small, thankfully.

85. 21:10

    Um, but like how do we get these things to be more reliable? And how do we get them, you know, we, we have like a golden session, and we have a great day, and now we wanna capture that and reuse it over and over again.

86. 21:23

    So I built a system where you can take a golden session for Claude Code, Codex, Pi, whatever you want, and it's a Prose program that will actually have the agent deconstruct that session and turn it into a reusable Prose workflow, um, that again can involve this idea of recursive coding agents, um, to get you to a reliable,

87. 21:45

    uh, way of getting to that golden state of performance over and over and over again.

88. 21:50

    Recursing, [chuckles] recursive coding agents for the win. Uh, I really think this is very powerful. RLMs just kind of blew my mind when they first came out. And again, uh, as you can probably see through this talk, I've been absolutely obsessed with applying the ideas of RLMs to coding agents.

89. 22:08

    The three things that I hope you'll take away from this is one, like trust is reliability. Like how can we, how can we trust something that isn't reliable? And again, my argument and this idea of the mismanaged genius is, uh, that the next step is not, um, more raw intelligence.

90. 22:26

    It's actually, uh, behavioral. It's actually orchestration. I personally believe that, um, RLMs represent this new paradigm of test-time compute, inference-time compute, uh, where tool calling and reasoning are unified, and we reason through tool calling, and we can, uh, recursively iterate, and one of those tools is to call another

91. 22:51

    agent to go do it on some other specific task or subset of the problem. Uh, and then also, I hope we settle a little bit of this drama around like, wait, like are RLMs, um, actually new?

92. 23:06

    Aren't they just coding agents? Yes, coding agents can be RLMs. They aren't automatically RLMs. Um, and so I've showed a couple of different Claude Code Dynamic Workflows that can turn Claude Code into an RLM, uh, as well as some ways of doing this with OpenProse that you can use with any coding agent.

93. 23:25

    So I see this as an incredibly powerful way of working with coding agents. I hope you will dig in more to recursivecodingagents.com. But with great power comes great responsibility.

94. 23:40

    So until next time, please recurse responsibly. Thank you very much.