It’s Tokens All The Way Down: How RLMs are Different — Kevin Madura, AlixPartners

Kevin Madura of AlixPartners starts with the two properties that define a recursive language model, or RLM. First, its input exists as a symbolic object—typically a variable in a Python REPL—rather than only as tokens placed directly in the transformer’s context window. The model can inspect, slice, search, transform, and compute over that object by writing code. 0:43

Second, code inside that environment can invoke another language model, including the same model with different parameters. The root model can isolate part of the input, delegate a focused question, receive a compact result, and repeat. A submodel can use the same method again, turning one large problem into a recursive tree of smaller computations. 1:13

The distinction from ordinary tool calling lies in where the working state lives. A conventional tool call commonly serializes arguments as JSON, invokes another program, and returns a string. An RLM keeps the source input, generated code, intermediate values, and model-call results inside one persistent computational environment. The model manipulates program state instead of repeatedly translating all state into messages.

What does that execution loop make visible? The diagram follows a large symbolic input through code execution, optional delegation, and final submission. The full input stays in the REPL while selected observations and submodel results enter the root model’s active context.

Madura traces an early precursor to work on arbitrarily long inputs with DSPy: a model could recursively summarize a long document and construct a table of contents without loading the whole source into one prompt. The lesson was not that context management had disappeared. A program could exceed one context window by deciding what to inspect and summarize at each step.

He points to long-context benchmarks including OOLONG and BrowseComp as evidence that the method can outperform other approaches on some tasks. One displayed comparison also placed an RLM above a system combining tool calls with BM25 on both performance and price. The recording does not provide the benchmark configurations or numerical values needed to generalize that chart beyond the experiments shown. 3:12

The programming model is a “relatively deterministic shell.” Developers declare the task, expected inputs, requested output types, and broad guidance. The model chooses the middle: which code to write, which evidence to inspect, whether to delegate, and how to proceed from each result. That preserves a stable interface while allowing execution to adapt to the input. 4:13

This is Madura’s “bitter lesson pilled” bet: improving models should receive more responsibility instead of having every decomposition strategy hard-coded around them. The tradeoff is direct. Less prescribed control flow can make a program concise and adaptable, but model quality—especially coding, planning, and stopping behavior—becomes part of the program’s reliability.

The motivation is context rot: performance can degrade as a context window fills, even before it reaches its advertised maximum. An RLM reduces pressure on the root model because the complete input remains a variable in the REPL. The model reads selected portions, delegates bounded analyses, and brings back only useful observations. 5:12

Madura separates the mechanisms this way:

• RAG: retrieves material and places it into the prompt, so the retrieved passages still consume context.
• Agents and tool calls: commonly exchange serialized strings, spreading logic, execution, and results across message boundaries.
• Code-oriented tool use: can run computations, but often still moves tool arguments and observations through serialized calls.
• RLMs: retain source objects and intermediate results as native REPL variables on which generated code can continue operating.

These categories can overlap. Madura notes that newer workflow systems also keep intermediate results in script variables, resembling part of the RLM approach. The useful distinction is therefore not a product label. It is whether large state remains available as programmable data and whether model calls can happen from within that persistent environment. 6:43

Recursion does not remove the cost of reading a source. It changes where the source lives and lets the root model decide which pieces deserve tokens. Storage, code execution, and submodel calls replace some direct attention over one giant prompt.

RLMs fit large or dense inputs, naturally decomposable tasks, and longer-running sessions. Madura also suggests large-output generation as an underexplored use case: a model could assemble an output incrementally in its environment instead of emitting hundreds of thousands of lines through one response. His tax-code example follows the same pattern for input: submodels explore selected regions, return promising sections, and let the root model reason across reduced evidence. 7:53

They are a poor fit when the task already fits comfortably in context, low latency matters, or the chosen model is a weak coder. Recursive model calls and code execution add runtime, cost, and more possible failure points. The method earns that overhead only when symbolic computation or decomposition addresses a real bottleneck.

A cited long-chain-of-thought evaluation increased overall accuracy from 2.6% to 45.4%, with strong gains on tasks that could be translated into code, including logic puzzles, chess, and chemistry. That result belongs to the referenced test setup; the recording does not supply enough methodological detail to treat it as an expected improvement for arbitrary workloads. 8:54

The simplest concrete example is deliberately unglamorous: sum 12 numbers scattered across 30,000 tokens. A base model must notice every number, preserve the values through attention, and perform the arithmetic correctly. An RLM can instead search the input variable with a regular expression, parse the matches, and let Python calculate the sum. A fragile language task becomes a deterministic computation selected by the model. 9:38

The causal sequence is locate candidate number strings in the external input, parse them into numeric values, compute the sum in code, and return the result. The model still chooses the extraction logic, so a bad regular expression can produce a bad answer. It no longer has to perform retrieval and arithmetic by attending to all 30,000 tokens at once.

A quick comparison with a coding agent produced a much more bloated solution, but Madura calls the comparison insufficiently investigated and potentially unfair. His production preference is narrower: use declared inputs, outputs, and a defined pipeline instead of handing a free-form prompt to a general agent and hoping its process is economical.

Implementations mentioned in the talk include DSPy, PredictRLM, Ax, and FastRLM. Their emphasis differs: some focus directly on RLM execution, while others incorporate it into a wider programming framework or knowledge-work product. Madura also points to production-trace analysis as another application—a long trace can become the symbolic object from which an RLM extracts decisions about model delegation or workload structure.

The cohort-retention walkthrough turns the mechanism into a pipeline. Three DataFrames—users, events, and subscriptions—enter the RLM. The surrounding declaration identifies what to investigate and specifies the requested output types. It does not prescribe every join, aggregation, or comparison. 12:38

The DataFrames live inside the REPL, so the model can inspect their columns, write analysis code, execute it, and use the resulting values in the next iteration. There is no need to serialize every table operation and result through repeated JSON tool calls. The interaction resembles a data scientist working in a Jupyter Notebook with persistent variables.

A large subset could be handed to a submodel for focused analysis, although the demonstrated run did not need that delegation. Recursion is a capability, not a requirement for every step. When direct code execution resolves the task efficiently, another model call would only add cost and latency.

The trace separates generated code from the model’s evolving analysis. After exploring the tables, the model formats key findings and recommendations and invokes a final submission step that returns the typed fields declared at the start. The system can impose a maximum iteration count—10, 100, or another limit—but within that budget the model decides when it has enough evidence to stop. 14:08

That stopping decision is both a benefit and a limitation. It avoids requiring developers to predict every analysis step, yet a model can stop too early, waste iterations, or mistake a suggestive relationship for a sufficient explanation. Typed outputs constrain the answer’s shape; they do not prove that the analysis is correct.

The design goal is to move upward in abstraction: define the objective and output contract, then let the model supply more of the implementation. Madura expects that balance to improve with model capability, but objectives alone do not remove the need for evaluation, execution limits, and application-specific validation.

The closing examples apply the same loop to several kinds of long input:

• Invoices and contracts: consolidate a directory of irregular documents, including a 200-page item, without manually designing every chunking and embedding strategy.
• Logs: inspect raw operational data and surface patterns through generated analysis.
• Agent traces: use Halo to study long task traces and recommend changes to the surrounding harness.
• Source code: scan an intentionally vulnerable OWASP application—described as roughly 500,000 lines—and generate a security report.

Invoice consolidation shows the practical shift most clearly. A conventional pipeline may parse files, choose chunks, create embeddings, retrieve candidate passages, and reconcile fields across documents. An RLM can inspect the document collection iteratively and build the consolidated inventory as program state. This reduces upfront context-engineering code, although correctness still depends on document parsing, model decisions, and validation of the inventory. 16:09

PredictRLM adds schemas between the root model and submodel calls. Those schemas make handoffs readable and state exactly what type of result should return. Madura expects this structure might help cheaper models, but explicitly leaves that as an experiment to run. The grounded benefit is narrower: schemas make recursive handoffs easier to inspect and maintain.

Halo moves recursion up one level. Instead of optimizing only a prompt or fixed workflow, an RLM reads the long traces produced by an agent harness and recommends how that harness should change. The traces are both the object being analyzed and evidence about where orchestration failed, making this a form of meta-optimization. 18:09

The code-security experiment pushes the idea across a whole repository. A small declaration launches analysis over approximately 500,000 lines and asks for a report. The experiment demonstrates how little calling code may be required; it does not establish exhaustive vulnerability detection. Madura’s own phrasing—getting “some type of insights out”—sets the appropriate scope. 19:08

Madura closes with the prospect that future models could be post-trained specifically to use the RLM method: inspect symbolic inputs, write effective analysis code, delegate at useful boundaries, manage budgets, and stop at the right time. Today, a framework teaches a general model how to behave this way. Native RLM awareness could make the loop more effective, but that remains the talk’s forward-looking bet. 20:08

The immediate lesson is already concrete. Tokens remain the model’s communication medium, but they no longer need to contain the entire working set at once. A REPL lets the model decide which data becomes tokens, which operations become code, and which subtasks deserve another model call. An RLM trades one enormous attention problem for a sequence of inspectable computations.