RLM: Recursive Language Models for Large Codebases

Talk transcript

1. 0:00

   Hello, and welcome to this online track talk for the AI Engineer World’s Fair 2026.

2. 0:08

   Today, we're going to explore the concept of RLM, also known as Recursive Language Models, and how we can use those concepts for larger code bases. My name is Shashi.

3. 0:21

   I'm a founder of Superagentic AI. First of all, let's be clear that RLM paper has been published by MIT & Friends. As you can see, there's a full paper.

4. 0:32

   You can read about it. But the purpose of this talk is how you can use the concepts of RLM, and you can use into your own workflow to implement your own harnesses.

5. 0:43

   So first of all, what's the problem? If you're using the coding agents for smaller repos or on monorepos, they works exceptionally well. But if you have ever tried it with the monorepos with the large context, you know there's a context problem.

6. 0:58

   As the context grows, the performance degrade. And if you're working with the monorepos, this problem get worse. In this talk, we will see we selected the code base and the concept of RLMs are relevant for the larger code bases.

7. 1:15

   If you use the coding agents, then you probably saw that there are different approaches that other coding-agent harnesses have been taken to solve this problem. Most common approach is searching using the tools like Grep.

8. 1:28

   So basically, there's a file system and the coding-agent harnesses search using these tools.

9. 1:34

   The second approach, you've probably seen that the semantic search or the local search. So idea here is basically you can search through the code and curate the context. Another approach is the long context get compressed, and you can use the summarized version of the context.

10. 1:54

    And there are some memory solutions available in the market as well that you can use to persist the memory for the coding agent.

11. 2:05

    First of all, let's explore the RLM idea. The core thesis of the RLM is you need to externalize the context management into programmable execution environment, meaning you should have a separate dedicated environment, so that model can operate on that.

12. 2:23

    In this case, for example, your whole repository is treated as a data that model can operate on. Then model can write the code to inspect, slice, and compute the relevant chunks to value you can then feed into the main context window.

13. 2:42

    So basically, rather than putting everything into the model's context, create a separate dedicated environment, give them a coding agent or REPL, and then model write the code to curate the context that can be used into the main.

14. 2:59

    So it's another context management technique proved to be very effective. Could be also be used as a memory layer for your coding agents.

15. 3:10

    Let me summarize this, giving you a simple analogy. Imagine you are a lead software engineer and assigned to the new project with a huge code base.

16. 3:22

    Imagine that's monorepo. How does that lead engineer deals with the code? So rather than reading each line of code line by line, engineer probably inspect the code base, make some notes, see what are the project's dependencies, how it is structured.

17. 3:42

    If it's something else, uh, is not understood by the repository, engineer probably ask to another engineer or expert to get some ideas. And the same concepts applied in RLM.

18. 3:55

    So large project like the files and docs and texts and configs because repository has a lot of things. And the programmable REPL, it's kind of a notebook that engineer makes a note about the code base that can be used.

19. 4:08

    Researching, he may be using other techniques, or maybe he's writing some script to search something from the repo.

20. 4:16

    And then if he's stuck, then he ask another engineer or specialist where it come to the LLM query. An LLM query is basically asking another model environment to get an answer from.

21. 4:31

    And once they get answer, then the loop continues. And at the end, it returns the clean node syntheses.

22. 4:40

    So the recursion part here is engineer ask another specialist using LLM query. That can be

23. 4:48

    one question or that can be number of questions. So this is where the recursion comes in picture.

24. 4:54

    Loop is basically you have repo, uh, as your context, and then the model writes the REPL code to get some relevant context

25. 5:06

    that returns the bounded observation. And if we-- if loop needs more information, it passes through the LLM query, where it asks another language model or another system to get the response,

26. 5:21

    return the value, and continue the loop. And the loop get terminated until we get a final results.

27. 5:31

    Why are we talking about the code base and not the big context in terms of like, uh, other things, for example, books or dictionaries? Code base is different. It has directories, it has test, it has some imports, it has dependencies, it has tests, it has pictures, it has configuration files.

28. 5:50

    So the code base is not only just, um, the text, it is a structured data. And the model need to understand and reason over the text. That's why I chose this scenario to use the code bases to prove these concepts of RLM.

29. 6:09

    Now let's switch the gear and talk about our own library that we created at Superagentic AI called rlm-code. You can see rlm-code's landing page here,

30. 6:22

    where this is just a research playground where you can implement the concepts of RLM.

31. 6:29

    We have, um, documentation that you can take a look, and there's a-- the GitHub repository. It is completely open source project that you can use it

32. 6:43

    and play with it. RLM itself is a concept

33. 6:50

    and a pattern, and you can implement that concept and pattern in your own way. There are official authors also wrote some implementation in their GitHub repos. It's called RLM and RLM Minim- Minimal.

34. 7:07

    You can refer that. Implementation of RLM in DSPy.rlm. So Omar is author of RLM, and he is also author of another popular framework called DSPy. So DSPy got RLM implementation inside it.

35. 7:26

    However, you should treat they are completely different. So RLM is a pattern, and you can implement in your own ways. You can find there are various other people implemented RLM in their own way, and

36. 7:41

    in the similar way, we implemented rlm-code as our own independent harness that we will be using in this live demo.

37. 7:50

    RLM code is just a reference implementation to demonstrate how the RLM concepts works under the hood.

38. 7:59

    So we have implemented something called RLM mode. We are using RLM as it is. We are not adding anything on top of RLM's ideas and RLM's paper. We are using the same concept of recursive calls, REPL execution.

39. 8:14

    However, you can run it with a local model, you can run with-- run it with cloud-based model, you can plug into any observability framework of your choice, and that gives you, like, a lot of flexibility around RLM.

40. 8:31

    You can also plug it into the framework of your choice. For example, you can use Pathetic AI or

41. 8:39

    Google ADK or something similar framework and implement ideas of RLM over there. In order to demonstrate this, we have created a source code repository where you can try this concept by yourself using MIT's RLM paper and rlm-code.

42. 8:57

    And we will see how these things works in a, in a practice.

43. 9:03

    So basically, we will show you the loop. This, you will understand this once you-- once we see this live demo and what all these files are doing, where, where is the context is being created, where the Python REPL has written a code, and where it is passed to the llm_query, and how we get the final results.

44. 9:27

    So that will be covered as part of the live demo. So we can cover this everything here. So in nutshell, how it looks like is basically it creates the REPL, and then observation, and the final recursive language output.

45. 9:43

    So let me jump into the live demo now. Okay, let's do the live demo of these concepts of RLM and rlm-code and how it works in a, the larger code basis.

46. 9:55

    So I have a code repositories here I have checked out, and let's open it into the editor so that we can see what's inside it. So as you can see, there's a demo target, which is, um, we are using rlm-code source as a, as a demo here.

47. 10:15

    And then we have some instructions that you can follow along, um, yourself. So basically,

48. 10:23

    you have a README file that you can use to use with your local model or... So we are going to use with the Gemini. So we will try this script and see what happen.

49. 10:37

    So right now, you can see we're using the Docker as a sandbox. If you see,

50. 10:46

    the Docker container has been just started for this RLM.

51. 10:50

    And now coming back to our execution, you can see that execution had just finished.

52. 11:00

    And in this execution, what you have seen, basically in the first step, model has written the, the REPL code that you can see here, and then it's built the evidence.

53. 11:12

    And after that, it also made the calls to the llm_query

54. 11:19

    with some prompt and got the result back. And after that, it gives the, the final answer. And as you can see here, we can have all this,

55. 11:31

    all this step coming back to the, the final answer.

56. 11:37

    And here you can see the-- it made the two tool calls and how many-- the tokens is used, uh, for this model, that you can see it here.

57. 11:48

    And the good thing is that you can see all these traces in the rlm-code repositories. So for example, you can see all the runs. This is the run that we just did.

58. 12:00

    You can see all the sessions and all the observability that you can plug it into any of your observ-- favorite observability platform.

59. 12:08

    So this is the CLI path we just demonstrated, but we also have this, um,

60. 12:17

    Kind of coding agent style experimental, uh, harness where you can try the same thing. So first of all, let's connect with the Gemini model.

61. 12:29

    So you can connect with the Gemini model using the command

62. 12:33

    connect. You can, you can have a provider and the model name. Now you are connected. So you can also run

63. 12:44

    the doctor command and see if everything is okay.

64. 12:55

    Seems like doctor command found some warning, but this is related to Deep Agent ADK and other frameworks which is not relevant to this demo. And the interesting part where we will be seeing is basically you are sending the prompt.

65. 13:11

    So for what we did now, we ran the command, and we asked the question. We specify the budget so that we don't-- not spending too much, uh, on this run.

66. 13:21

    But once we do that, as you can see, you have maximum steps, recursion depth, and it completed. It's this run,

67. 13:32

    and coming back with the results. We can also see that th-this thing into the, the research lab, where we can see the spin has been completed. We can see some rewards.

68. 13:45

    We can also see the trajectory, which is important part, where we can see the all the RLM loop. For example, the, the REPL and the code and the final output.

69. 13:57

    So-- And also we can see the, the events, when it started and when it ended. So you can play around with this rlm-code terminal user interface, which is kind of harness, and you can experiment your RLM ideas in here.

70. 14:16

    So I'm going to quit this for now,

71. 14:18

    and let's switch back to the slides. In a nutshell, what we just saw, basically our context has been loaded.

72. 14:28

    We have some REPL code written to extract some snippets. We also saw the llm_query has been called to get some more context from another model. This is where the recursion comes in picture, and we got the final result.

73. 14:44

    And we got the, the traces in JSONL format that you can import it into the any of the observability platform of your choice.

74. 14:56

    We also saw these results coming from different files. You can take a look at the source code that will be available, uh, for you. Let's talk about the real thing, how AI engineer

75. 15:09

    could use these concepts in the real life.

76. 15:14

    And there are few things. For example, if you're dealing with a large source code and you want to, for example, root cause analysis or onboarding of the repositories or some unfamiliar repos.

77. 15:29

    So there are few use cases you can from here and probably try to use RLM concepts over there.

78. 15:37

    Basically, you can design your own harness, um, based on your needs, so that should capture the whole trajectory. All these things like the planning, coding, observation, subcall, budget, and the final output.

79. 15:51

    Now, coming back to the final point about RLM concepts and where it's been used.

80. 15:57

    I have recently came across a lot of the post on X saying the RLM concepts have been being used into the some of the proprietary things like the managed agent, dynamic workflows, using the RLM concepts under the hood.

81. 16:11

    So they have implemented one or more forms of RLM inside their agent harnesses. Recently, I saw that the Codex harness is writing the Python, Python code in the REPL that you can see to curate the context.

82. 16:26

    That is the one form of RLM I have seen myself. And obviously, the clouds managed agents or Gemini managed agents, they're all kind of concepts of RLM. So basically, you can get the harness in the sandbox, and then you can do the stuff.

83. 16:41

    And the recent things about the dynamic workflows where one agent, given the, given the task, you can spawn multiple agents that have their separate sandboxes. They can work together and give back the final results.

84. 16:54

    And the idea is basically generally coming from the, um, RLMs. A lot of software factories concepts are probably using the RLMs, but we are not sure yet. However, some of the Cloud Code engineers from Anthropic has accepted on X that they have used concepts of RLM.

85. 17:14

    You can use this RLM concept on your large context repository, and if you have any questions, then feel free to reach out to me.

86. 17:24

    And finally, thank you so much for listening to my talk.