# It’s Tokens All The Way Down: How RLMs are Different — Kevin Madura, AlixPartners

Kevin Madura, AlixPartners

[Talk page](https://ai.engineer/talks/xo68uCibfm8-its-tokens-all-way-down-rlms-are) · [Video](https://www.youtube.com/watch?v=xo68uCibfm8)

## It’s Tokens All the Way Down: How RLMs Are Different

Kevin Madura explains how recursive language models move large inputs into a programmable REPL, where a model can inspect variables, write code, delegate focused work, and return only the results that matter.

### Context becomes programmable state

[0:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=12s)

Kevin Madura of AlixPartners starts with the two properties that define a **recursive language model**, or **RLM**. First, its input exists as a symbolic object—typically a variable in a Python REPL—rather than only as tokens placed directly in the transformer’s context window. The model can inspect, slice, search, transform, and compute over that object by writing code. [0:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=43s)

![Captures the second defining property: the root model can delegate focused work to another language model, including itself.](https://ai.engineer/api/talk-essay-media/7e9cdc74a7807b5b8a21ae5971d8aff8f361d2ea1c9ad4d3d2a97c83f2b6a5ee)

Captures the second defining property: the root model can delegate focused work to another language model, including itself.

Second, code inside that environment can invoke another language model, including the same model with different parameters. The root model can isolate part of the input, delegate a focused question, receive a compact result, and repeat. A submodel can use the same method again, turning one large problem into a recursive tree of smaller computations. [1:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=73s)

The distinction from ordinary tool calling lies in where the working state lives. A conventional tool call commonly serializes arguments as JSON, invokes another program, and returns a string. An RLM keeps the source input, generated code, intermediate values, and model-call results inside one persistent computational environment. The model manipulates program state instead of repeatedly translating all state into messages.

What does that execution loop make visible? The diagram follows a large symbolic input through code execution, optional delegation, and final submission. The full input stays in the REPL while selected observations and submodel results enter the root model’s active context.

The recursive language model loop
The root model explores symbolic state, executes code, optionally delegates focused work, and stops when it can submit the requested output.
- Symbolic input — A document, string, DataFrame, or other object lives as a variable in the REPL.
- Root language model — Chooses what to inspect and writes code for the next step.
- REPL execution — Generated code searches, slices, transforms, or computes over variables.
- Focused submodel call — Another model, possibly the same one, analyzes a selected subset.
- Compact result — Relevant findings return instead of the full underlying input.
- Typed final output — The loop ends when the model decides it has enough information.
- Symbolic input → Root language model (Expose variable)
- Root language model → REPL execution (Write code)
- REPL execution → Focused submodel call (Delegate selected work)
- Focused submodel call → Compact result (Return focused analysis)
- Compact result → Root language model (Continue with compact evidence)
- Root language model → Typed final output (Stop when sufficient)

### A deterministic shell around an adaptive middle

[2:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=133s)

Madura traces an early precursor to work on arbitrarily long inputs with DSPy: a model could recursively summarize a long document and construct a table of contents without loading the whole source into one prompt. The lesson was not that context management had disappeared. A program could exceed one context window by deciding what to inspect and summarize at each step.

![Illustrates Madura’s deterministic-shell model: developers define intent and interfaces while the model supplies the implementation.](https://ai.engineer/api/talk-essay-media/68149f63bce40ac899013b761bce56c6933984bf158fce4abb8416df24e806a4)

Illustrates Madura’s deterministic-shell model: developers define intent and interfaces while the model supplies the implementation.

He points to long-context benchmarks including OOLONG and BrowseComp as evidence that the method can outperform other approaches on some tasks. One displayed comparison also placed an RLM above a system combining tool calls with BM25 on both performance and price. The recording does not provide the benchmark configurations or numerical values needed to generalize that chart beyond the experiments shown. [3:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=192s)

The programming model is a “relatively deterministic shell.” Developers declare the task, expected inputs, requested output types, and broad guidance. The model chooses the middle: which code to write, which evidence to inspect, whether to delegate, and how to proceed from each result. That preserves a stable interface while allowing execution to adapt to the input. [4:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=253s)

This is Madura’s “bitter lesson pilled” bet: improving models should receive more responsibility instead of having every decomposition strategy hard-coded around them. The tradeoff is direct. Less prescribed control flow can make a program concise and adaptable, but model quality—especially coding, planning, and stopping behavior—becomes part of the program’s reliability.

### Why the root context stays smaller

[5:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=312s)

The motivation is **context rot**: performance can degrade as a context window fills, even before it reaches its advertised maximum. An RLM reduces pressure on the root model because the complete input remains a variable in the REPL. The model reads selected portions, delegates bounded analyses, and brings back only useful observations. [5:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=312s)

![Useful comparison frame distinguishing RAG, agents, tool calls, CodeAct, and RLMs.](https://ai.engineer/api/talk-essay-media/2660adafad820783163d0e4bff00e666d1fc3d94d38c02e93b057e4dc1014de4)

Useful comparison frame distinguishing RAG, agents, tool calls, CodeAct, and RLMs.

Madura separates the mechanisms this way:

- **RAG:** retrieves material and places it into the prompt, so the retrieved passages still consume context.
- **Agents and tool calls:** commonly exchange serialized strings, spreading logic, execution, and results across message boundaries.
- **Code-oriented tool use:** can run computations, but often still moves tool arguments and observations through serialized calls.
- **RLMs:** retain source objects and intermediate results as native REPL variables on which generated code can continue operating.

These categories can overlap. Madura notes that newer workflow systems also keep intermediate results in script variables, resembling part of the RLM approach. The useful distinction is therefore not a product label. It is whether large state remains available as programmable data and whether model calls can happen from within that persistent environment. [6:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=403s)

Recursion does not remove the cost of reading a source. It changes where the source lives and lets the root model decide which pieces deserve tokens. Storage, code execution, and submodel calls replace some direct attention over one giant prompt.

### Where RLMs fit—and where they do not

[7:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=473s)

RLMs fit large or dense inputs, naturally decomposable tasks, and longer-running sessions. Madura also suggests large-output generation as an underexplored use case: a model could assemble an output incrementally in its environment instead of emitting hundreds of thousands of lines through one response. His tax-code example follows the same pattern for input: submodels explore selected regions, return promising sections, and let the root model reason across reduced evidence. [7:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=473s)

![Illustrates the concrete example of summing 12 numbers buried across 30,000 tokens.](https://ai.engineer/api/talk-essay-media/990516ee1518e13e9eb53b90c5a8b660d07498d931db06983a550ceb66ef2ab3)

Illustrates the concrete example of summing 12 numbers buried across 30,000 tokens.

They are a poor fit when the task already fits comfortably in context, low latency matters, or the chosen model is a weak coder. Recursive model calls and code execution add runtime, cost, and more possible failure points. The method earns that overhead only when symbolic computation or decomposition addresses a real bottleneck.

A cited long-chain-of-thought evaluation increased overall accuracy from **2.6% to 45.4%**, with strong gains on tasks that could be translated into code, including logic puzzles, chess, and chemistry. That result belongs to the referenced test setup; the recording does not supply enough methodological detail to treat it as an expected improvement for arbitrary workloads. [8:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=534s)

The simplest concrete example is deliberately unglamorous: sum 12 numbers scattered across 30,000 tokens. A base model must notice every number, preserve the values through attention, and perform the arithmetic correctly. An RLM can instead search the input variable with a regular expression, parse the matches, and let Python calculate the sum. A fragile language task becomes a deterministic computation selected by the model. [9:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=578s)

The causal sequence is **locate** candidate number strings in the external input, **parse** them into numeric values, **compute** the sum in code, and **return** the result. The model still chooses the extraction logic, so a bad regular expression can produce a bad answer. It no longer has to perform retrieval and arithmetic by attending to all 30,000 tokens at once.

A quick comparison with a coding agent produced a much more bloated solution, but Madura calls the comparison insufficiently investigated and potentially unfair. His production preference is narrower: use declared inputs, outputs, and a defined pipeline instead of handing a free-form prompt to a general agent and hoping its process is economical.

Implementations mentioned in the talk include DSPy, PredictRLM, Ax, and FastRLM. Their emphasis differs: some focus directly on RLM execution, while others incorporate it into a wider programming framework or knowledge-work product. Madura also points to production-trace analysis as another application—a long trace can become the symbolic object from which an RLM extracts decisions about model delegation or workload structure.

### A data scientist inside the program

[12:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=758s)

The cohort-retention walkthrough turns the mechanism into a pipeline. Three DataFrames—users, events, and subscriptions—enter the RLM. The surrounding declaration identifies what to investigate and specifies the requested output types. It does not prescribe every join, aggregation, or comparison. [12:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=758s)

![Shows the final submit step, typed outputs, and the model-controlled stopping point.](https://ai.engineer/api/talk-essay-media/96296cbbf329ec745141e0c600dcb77e0e04bbed43d840d5fa426ec7d9bc72f0)

Shows the final submit step, typed outputs, and the model-controlled stopping point.

The DataFrames live inside the REPL, so the model can inspect their columns, write analysis code, execute it, and use the resulting values in the next iteration. There is no need to serialize every table operation and result through repeated JSON tool calls. The interaction resembles a data scientist working in a Jupyter Notebook with persistent variables.

A large subset could be handed to a submodel for focused analysis, although the demonstrated run did not need that delegation. Recursion is a capability, not a requirement for every step. When direct code execution resolves the task efficiently, another model call would only add cost and latency.

The trace separates generated code from the model’s evolving analysis. After exploring the tables, the model formats key findings and recommendations and invokes a final submission step that returns the typed fields declared at the start. The system can impose a maximum iteration count—10, 100, or another limit—but within that budget the model decides when it has enough evidence to stop. [14:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=848s)

That stopping decision is both a benefit and a limitation. It avoids requiring developers to predict every analysis step, yet a model can stop too early, waste iterations, or mistake a suggestive relationship for a sufficient explanation. Typed outputs constrain the answer’s shape; they do not prove that the analysis is correct.

The design goal is to move upward in abstraction: define the objective and output contract, then let the model supply more of the implementation. Madura expects that balance to improve with model capability, but objectives alone do not remove the need for evaluation, execution limits, and application-specific validation.

### From invoices to whole codebases

[15:39](https://www.youtube.com/watch?v=xo68uCibfm8&t=939s)

The closing examples apply the same loop to several kinds of long input:

- **Invoices and contracts:** consolidate a directory of irregular documents, including a 200-page item, without manually designing every chunking and embedding strategy.
- **Logs:** inspect raw operational data and surface patterns through generated analysis.
- **Agent traces:** use Halo to study long task traces and recommend changes to the surrounding harness.
- **Source code:** scan an intentionally vulnerable OWASP application—described as roughly 500,000 lines—and generate a security report.

![Captures Halo’s use of long agent traces to recommend changes to the surrounding harness.](https://ai.engineer/api/talk-essay-media/4aaf19e87787bcda3dabb74700fdcebf8c614c06d50d4894a17beb54aac74fad)

Captures Halo’s use of long agent traces to recommend changes to the surrounding harness.

Invoice consolidation shows the practical shift most clearly. A conventional pipeline may parse files, choose chunks, create embeddings, retrieve candidate passages, and reconcile fields across documents. An RLM can inspect the document collection iteratively and build the consolidated inventory as program state. This reduces upfront context-engineering code, although correctness still depends on document parsing, model decisions, and validation of the inventory. [16:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=969s)

PredictRLM adds schemas between the root model and submodel calls. Those schemas make handoffs readable and state exactly what type of result should return. Madura expects this structure might help cheaper models, but explicitly leaves that as an experiment to run. The grounded benefit is narrower: schemas make recursive handoffs easier to inspect and maintain.

Halo moves recursion up one level. Instead of optimizing only a prompt or fixed workflow, an RLM reads the long traces produced by an agent harness and recommends how that harness should change. The traces are both the object being analyzed and evidence about where orchestration failed, making this a form of meta-optimization. [18:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=1089s)

The code-security experiment pushes the idea across a whole repository. A small declaration launches analysis over approximately 500,000 lines and asks for a report. The experiment demonstrates how little calling code may be required; it does not establish exhaustive vulnerability detection. Madura’s own phrasing—getting “some type of insights out”—sets the appropriate scope. [19:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=1148s)

### When models learn the method natively

[20:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=1208s)

Madura closes with the prospect that future models could be post-trained specifically to use the RLM method: inspect symbolic inputs, write effective analysis code, delegate at useful boundaries, manage budgets, and stop at the right time. Today, a framework teaches a general model how to behave this way. Native RLM awareness could make the loop more effective, but that remains the talk’s forward-looking bet. [20:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=1208s)

![Closing frame supporting the forward-looking bet that post-trained, RLM-aware models could use the method natively.](https://ai.engineer/api/talk-essay-media/9fe47a53f304032e3655843e38d4b9a99e6c15083d560074d8e3ea279298284e)

Closing frame supporting the forward-looking bet that post-trained, RLM-aware models could use the method natively.

The immediate lesson is already concrete. Tokens remain the model’s communication medium, but they no longer need to contain the entire working set at once. A REPL lets the model decide which data becomes tokens, which operations become code, and which subtasks deserve another model call. An RLM trades one enormous attention problem for a sequence of inspectable computations.

## Key takeaways

- An RLM keeps large inputs as symbolic objects in a persistent REPL, allowing generated code to inspect and transform them without placing the entire working set in the root model’s context.
- Recursion means code can invoke another language model on a selected subset and return a compact result; every step does not need a submodel.
- The best candidates have large or dense inputs, natural decomposition, or useful deterministic computation. Small, latency-sensitive tasks may not justify the extra execution and model calls.
- The 12-numbers-in-30,000-tokens example captures the core transformation: let the model select regex and Python, then let deterministic code perform extraction and arithmetic.
- Typed inputs, outputs, iteration limits, traces, and schemas create a controlled shell, but they do not prove analytical correctness or guarantee good stopping behavior.
- The longer-term bet is that post-training models for RLM-style execution will improve decomposition, code generation, delegation, and context selection.

## Resources

- [Recursive Language Models](https://arxiv.org/abs/2512.24601v1) — The paper defining RLMs and reporting their long-context evaluations.
- [Recursive Language Models reference implementation](https://github.com/alexzhang13/rlm) — The paper authors’ implementation, with multiple REPL environments, recursive calls, logging, and examples for experimenting with the architecture.
- [Kevin Madura’s RLM implementation](https://github.com/kmad/rlm) — A compact Python implementation showing the root model, sandbox, recursive query function, execution limits, and long-document examples.
- [A Data Scientist RLM That Lives in Your Program](https://kmad.ai/A-Data-Analysis-Agent-That-Lives-in-Your-Program) — The companion walkthrough for passing native DataFrames into an RLM, declaring typed cohort-analysis outputs, and inspecting iterative execution traces.
- [DSPy workshop code](https://github.com/kmad/aie) — Examples of typed signatures, recursive summarization, document processing, and optimization that provide useful background for the deterministic-shell programming style.

## Related talks

- [Recursive Coding Agents](https://ai.engineer/talks/3hXJI2q0Jz8-recursive-coding-agents) — Extends externalized context and recursive delegation into coding-agent orchestration, memory, and long-context work.
- [RLM: Recursive Language Models for Large Codebases](https://ai.engineer/talks/8oyalrfwgjw-rlm-recursive-language-models-large-codebases) — Applies RLM execution directly to repository-scale code analysis with inspectable steps and explicit budgets.
- [The Unreasonable Effectiveness of Separating the Task from the Model](https://ai.engineer/talks/GgLQ02aO-hs-unreasonable-effectiveness-separating-task) — Develops the complementary DSPy idea of keeping typed task contracts stable while changing models, prompts, reasoning strategies, and RLM implementations underneath.

## Slides

### It's Tokens All The Way Down

Kevin Madura introduces recursive language models and how they differ.

[0:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=23s) [Title slide introducing how recursive language models are different, presented by Kevin Madura.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8fe4bcbefaacb22d7bf2c36e7b6d5a75b177839d908635df157c2f3fbf56d94.webp)

It's Tokens All The Way Down
How RLMs are Different

11:10–11:30 · TRACK 8 · CONTEXT ENGINEERING

Kevin Madura
kmad.ai · @kmad

kmad.ai

Interface text:
Recursive Language Models — Cont…
Cmpnd
The Mismanaged Geniuses Hyp…
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-1
VPN

### A Recursive Language Model

An RLM treats context as an object in an environment and can delegate work to sub-models. Those sub-models can make further calls, recursively breaking down a problem.

[0:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=35s) [A recursive language model treats context as an object, delegates to sub-models and chooses how to work; a diagram illustrates recursive calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/7dd9d752b68722c4cb39075806562841a9e6aedbc830e22b916508cc36652e6d.webp)

01 · FOUNDATIONS

A Recursive Language Model:

- treats context as an object it can manipulate in a symbolic environment
- delegates to sub-LMs as functions inside that REPL
- is bitter lesson pilled; lets the model decide how to work

¹Alex Zhang; Omar Khattab

Diagram:
RLM (root / depth=0)
query
context
Language Model
Environment E (e.g. REPL)
[illegible]
final response

RLM (depth=1)
sub-query 1
sub-context 1
Language Model
sub-response 1

RLM (depth=1)
sub-context 2
sub-query 2
Language Model
sub-response 2
Environment E (e.g. REPL)
[illegible]
…

Interface text:
Recursive Language Models — Cont…
Cmpnd
The Mismanaged Geniuses Hyp…
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-3
VPN
kmad.ai

### You Define the Task, the Model Picks the Path

The user defines the inputs, outputs and guidance in a deterministic shell. Inside it, the RLM decides how to navigate context and decompose the task; validation follows the model’s judgment-heavy work.

[4:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=240s) [Text explains model-directed navigation of context inside a REPL, alongside a diagram of a deterministic shell around a probabilistic core.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae39d7134ec88853acc9986f4a30e31cd2002b8b0b735cf36617821bca6f0bac.webp)

01 · FOUNDATIONS

You define the task, the model picks the path.

With agents and tool-calling, you are subject to limits of the context window.

With an RLM the model decides how to navigate the context — peek, grep, chunk, recurse — inside the REPL.

The deterministic shell is yours. The open-ended, judgment-heavy middle is the model's job.

DETERMINISTIC SHELL
The workflow you control

TYPED IN
 document: file
 question: str

PROBABILISTIC CORE
The open-ended, judgment-heavy steps
extract · judge · draft

validate + eval

TYPED OUT
 line_items[]
 total: float

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-6
VPN

### Context window and the smart zone

Dex Horthy’s diagram places the “smart zone” before the 40% context-used line and the faded “dumb zone” after it. Madura says RLMs can slow context rot by keeping inputs outside the main context window and deferring subtasks.

[5:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=315s) [A presentation screenshot illustrates a context window and labels a 40% threshold between the smart and dumb zones.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c81993fc422fa6c115b94c898a353d114ce81d0e8489bf27d734aecd6c2e28d.webp)

Find a way to always inject this

Read()
CLAUDE.md
AGENTS.md
constitution.md
....

System Instructions
CLAUDE.md
Claude Builtin Tools
MCP Tools
User message
Read context.md and then go do XYZ
Read()
context.md
...

This is a typescript monorepo that powers the AcmeCorp web app.

The modules are:
- module 1
- module 2

The systems fit together in these ways
...

"the smart zone"
40% context used
"the dumb zone"

AIE / CODE
PRESENTED BY
Google DeepMind

[illegible] ES NO VIBES ALLOWED: SOLVING HARD PRO[illegible]
DEX HORTHY / Founder & CEO
HumanLayer

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-7
VPN
kmad.ai

### Context as part of the environment

The slide contrasts agents, whose sub-results fill the root context window, with RLMs, which keep context in an environment variable that the root model can access through code.

[6:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=369s) [A comparison of RAG, agents, CodeAct, and RLMs highlights how agent sub-results fill the root context window.](https://assets.ai.engineer/sites/aiecode2025/sha256/305145ddca85c2c0368bd41777d19bcdb2d3580f88d9f10a5e63359b4090b492.webp)

01 · FOUNDATIONS

Meaningfully different from tool-calling; context is part of the environment.

Approach | How it treats context
RAG | Retrieves top-k passages for a single query.
Agents | Decompose task — pollute root context with every sub-result.
CodeAct | Runs code as a tool, not as the environment itself.
RLM | The code is the environment; recursion is the primitive.

The first three stuff the context window. An RLM keeps the context as a variable in the environment — the root LM never “sees” the bulk; it orchestrates via code.

Agents · decompose · delegate · merge
TASK · fanned out to sub-agents
↓ every sub-result appended to root

CONTEXT WINDOW · WHAT THE MODEL ACTUALLY SEES
task: analyze churn drivers
← sub-agent A  1,840 tok result
← sub-agent B  2,310 tok result
← sub-agent C  1,990 tok result
← sub-agent D … window saturating
window ~95% — context rot

Every sub-result flows back into root context. It fills — and rots — as the task grows.

2 / 4 · next: CodeAct
kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-8
VPN

### Where intermediate results live

The comparison highlights where intermediate results live: workflows use script variables that outlive a context window, while subagents and skills use Claude’s context and agent teams use a shared task list.

[6:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=416s) [A table compares subagents, skills, agent teams, and workflows, highlighting where intermediate results are stored.](https://assets.ai.engineer/sites/aiecode2025/sha256/d91de473a14134cd4b588fd14d45e2ba346cbc544f6290c0901417b7f661cc4c.webp)

01 · FOUNDATIONS · WHERE RESULTS LIVE

Subagents | Skills | Agent teams | Workflows

What it is
Subagents: A worker Claude spawns
Skills: Instructions Claude follows
Agent teams: A lead agent supervising peer sessions
Workflows: A script the runtime executes

Who decides what runs next
Subagents: Claude, turn by turn
Skills: Claude, following the prompt
Agent teams: The lead agent, turn by turn
Workflows: The script

Where intermediate results live
Subagents: Claude’s context window
Skills: Claude’s context window
Agent teams: A shared task list
Workflows: Script variables

What’s repeatable
Subagents: The worker definition
Skills: The instructions
Agent teams: The team definition
Workflows: The orchestration itself

Scale
Subagents: A few delegated tasks per turn
Skills: Same as subagents
Agent teams: A handful of long-running peers
Workflows: Dozens to hundreds of agents per run

Interruption
Subagents: Restarts the turn
Skills: Restarts the turn
Agent teams: Teammates keep running
Workflows: Resumable in the same session

Only workflows keep intermediate results in script variables — state that outlives any one context window. Sounds familiar!

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-9
VPN

### When to reach for it

The talk turns to when to use recursive language models.

[7:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=467s) [Section divider reading “02: When to reach for it.”](https://assets.ai.engineer/sites/aiecode2025/sha256/60014d968ee94d67ac6d566316ed25c8a0e5973e176db9a6a789a3a927428457.webp)

02
When to reach
for it.

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-10
VPN

### Rule of thumb

Madura recommends RLMs when context won't fit or work is compositional and latency can buy reliability. Otherwise, he says a single call, RAG, or a plain code tool is cheaper and just as good.

[7:52](https://www.youtube.com/watch?v=xo68uCibfm8&t=472s) [Guidance suggests using an RLM when context does not fit or work is compositional and latency is acceptable.](https://assets.ai.engineer/sites/aiecode2025/sha256/1030a1bb236f86fae81842c7a1cb0f05d294674a6e9451260a2b978eb93c4b12.webp)

02 · FIT

Rule of thumb.

Use an RLM when the context won't fit or the work is compositional — and you can spend latency to buy reliability.

Otherwise a single call, RAG, or a plain code tool is cheaper and just as good.

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-11
VPN

### LongCoT Performance

In @raw_works’ 500-question test, RLM raised accuracy from 2.6% to 45.4%, while captured cost rose from $31 to $621. Gains clustered in code-friendly tasks; both approaches scored 0/75 on MaxFlow and Hindley–Milner.

[8:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=531s) [LongCoT results compare vanilla and RLM accuracy across 500 questions: 2.6% versus 45.4%, with per-task results.](https://assets.ai.engineer/sites/aiecode2025/sha256/764429280645c0aa705bdeb5aaf0d5b86857171024b7972c8c8a5d54fa9b5217.webp)

02 · BENCHMARK · LONGCOT

LongCoT Performance

@raw_works stress-tested long-horizon reasoning using claude-sonnet-4-5 vanilla vs dspy.RLM

Across 500 questions (logic, CS, chemistry, chess, and math): RLM lifted accuracy from 2.6% → 45.4%

RLM wins where dependencies compile to code — puzzles, Hanoi, chess. It stalls on graph-shaped problems (max-flow, Hindley–Milner) and graphs.

Source: raw.works/longcot-a-benchmark-worthy-of-a-rlms-attention

500 QUESTIONS · SAME MODEL
Metric | Vanilla | RLM
Correct | 13/500 | 227/500
Accuracy | 2.6% | 45.4%
Captured cost | $31 | $621

PER-TASK — SOLVED
Task | Vanilla | RLM
Logic puzzles (Hanoi · Sudoku · Dungeon · +3) | 0/15 | 15/15
Chess | 0/100 | 85/100
BlocksWorld · Sokoban | 0/20 | 16/20
Chemistry | 13/100 | 31/100
Math | 0/95 | 6/95
MaxFlow · Hindley–Milner | 0/75 | 0/75
graph-shaped — both fail

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-13
VPN
kmad.ai

### Five questions over a 5k-row table

RLM previews the table and runs pandas in a sandbox; flat prompting truncates the table. RLM gets 3/4 answers right versus 0/4 for flat, but takes 6.6 seconds per question versus 0.9 seconds.

[9:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=576s) [Charts compare flat prompting and RLM on prompt size, accuracy, time and token usage for a table task.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f841bc501722a3e3a88146182e053be1eec487b396f101457075444cf31e17d.webp)

03 · MEASURED · FLAT VS RLM

Five questions over a 5k-row table.

Joins, group-bys, churn math. flat flattens df.to_string() into the prompt (truncated); RLM reads only a preview, then runs real pandas in the sandbox.

Complexity — peak prompt tok ↓
FLAT 3,821
RLM 1,191
3.2× smaller — frame stays out

Performance — accuracy ↑
FLAT 0%
RLM 75%
3/4 vs 0/4 — truncation loses rows

Time — s / question ↓
FLAT 0.9s
RLM 6.6s
flat 7× faster — the cost of the loop

Token usage — total tok ↓
FLAT 3,840
RLM 2,488
1.5× fewer even with the sandbox loop

The honest trade: RLM is slower per question, but the only arm that actually computes over the data — flat truncates and guesses.

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-15
VPN

### The coding-agent comparison

Claude Code matched RLM accuracy on both tasks but cost 46× and 54× more per query. The comparison uses different models; Madura cautions that the math may be unfair and needs further investigation.

[10:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=637s) [Cost and token charts compare flat prompting, RLM and a coding agent on two tasks; the slide reports tied agent and RLM accuracy.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b9066921a77bb99d628b3a8faa28ae224a87c8442333c90be5a8ca23f1afb01.webp)

03 · MEASURED · THE CODING-AGENT QUESTION

A coding agent gets it right too. For 40–115× the overhead.

Same two tasks, same grading. agent = Claude Code — its own frontier model, its own shell, not dspy.RLM at all. Not iso-model with flat/rlm: this answers “what does buying a general-purpose agent cost instead of building the scaffold,” not “which scaffold wins.”

A · cost / query — long-context agg ↓
FLAT $0.0147
RLM $0.0041
AGENT $0.190
agent 46× rlm’s cost — tied on accuracy

A · total tokens — long-context agg ↓
FLAT 19,572
RLM 3,933
AGENT 146,263
agent 37× rlm’s tokens per question

B · cost / query — workbook ↓
FLAT $0.0013
RLM $0.0035
AGENT $0.189
agent 54× rlm’s cost — tied on accuracy

B · total tokens — workbook ↓
FLAT 3,837
RLM 3,405
AGENT 144,967
agent 43× rlm’s tokens per question

Accuracy tied: agent 100% / 75% vs rlm 100% / 75% on tasks A/B — a real shell beats flat regardless of scaffold. The gap is pure overhead: Claude Code’s own system prompt, tool schema, and full-file reads dominate every call. Building the scaffold is ~50× cheaper than buying general-purpose agent overhead, per query.

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-16
VPN

### RLMs in the real world

RLM implementations range from DSPy’s core module to focused runtimes. Sam Hogan’s post describes using an RLM to generate evaluations from live data, then mirroring traffic to GLM 5.2 before switching providers.

[11:30](https://www.youtube.com/watch?v=xo68uCibfm8&t=690s) [A list of RLM implementations appears beside a social post describing an evaluation workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/feb4129dd6735939f22163fba68bd9001d773cc7a4fc5d90df7c3cd4416f2194.webp)

03 · IN PRACTICE

RLMs in the real world

dspy.RLM — the core module in DSPy. Typed signatures, pluggable sandbox.

ax (github.com/ax-llm/ax) — open source reference agent with DSPy, full workflow and orchestration.

predict-rlm (Trampoline AI) — production runtime on DSPy. Multimodal, skills, file I/O, full traces.

fast-rlm (avbiswas) — minimal; the whole thing in an afternoon.

Sam Hogan
@samhogan

Want to try GLM 5.2 in production but worried how it might change your product?

Don’t worry, we got you:

1. Install Inference Gateway (docs.inference.net)
2. Keep sending traffic to your current provider
3. Gateway automatically starts sorting through your live data using an RLM to generate evals for your app. This takes ~24 hours.
4. Gateway starts mirroring live traffic to GLM 5.2 to run evals. Traffic is only mirrored - you’re still using your old provider in prod.
5. Once evals look healthy, you get a Slack notification letting you know it’s safe to switch.
6. Switch model identifier in your code to “glm-5.2”

Congrats, you just saved 90% on your monthly token bill, and you own your LLM stack end to end.

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-18
VPN
kmad.ai

### Exploring the retention data

The RLM uses its REPL to inspect three data frames before calculating churn and cohort retention. The trace shows its reasoning alongside the Python it writes.

[13:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=801s) [A trace shows reasoning and Python code for inspecting user, event, and subscription data.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e05b9386c4244f9f603066b6054e0c6856bac9a2864a167453db8328aad308f.webp)

[Clipped] email nudges). 4) Track advanced_reports adoption and 30-day active days as leading retention KPIs by channel. 5) Review campaign promises versus product reality to reduce competitor-switching and missing-feature perception.

trajectory  5 items
0
reasoning
First I should inspect the basic structure and distributions: data shapes, date ranges, status values, acquisition channels, and cancellation reasons. That will ground the later retention/cohort analysis and help define churn correctly. Then I can compute overall churn rate and cohort retention in the next iteration.

code
import pandas as pd

print("users shape:", users.shape)
print("events shape:", events.shape)
print("subscriptions shape:", subscriptions.shape)

print("\nUsers dtypes:\n", users.dtypes)
print("\nEvents dtypes:\n", events.dtypes)
print("\nSubscriptions dtypes:\n", subscriptions.dtypes)

print("\nUsers sample:")
print(users.head(3).to_string())

print("\nEvents sample:")
print(events.head(3).to_string())

print("\nSubscriptions sample:")
print(subscriptions.head(3).to_string())

print("\nDate ranges:")
print("users signup:", users['signup_date'].min(), "to",
[clipped]

Interface text:
cmpnd
All Traces
All Optimizations
All Deployments
INBOX  1
documents -> redacted_d...  PredictRLM
PROGRAMS
No programs yet.
Account
Archived
150%

### Calculating churn and retention

The model works directly with the data frames, calculating churn and defining 30-day eligibility before comparing retention across signup cohorts and acquisition channels.

[13:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=829s) [A trace shows reasoning about 30-day retention and Python code beginning the churn analysis.](https://assets.ai.engineer/sites/aiecode2025/sha256/746bc4ad8b87f09ebb28984a827cc668130d7ea75967b7c313dcf8811e8008d2.webp)

1
reasoning
Next I should compute overall churn rate, build user-level retention labels, and examine retention by signup cohort and acquisition channel. Since signup spans Jan-May and events extend through end of June, a sensible comparable retention metric is 30-day retention/churn for users with at least 30 days of observation. I should also inspect if subscription status churn aligns with a 30-day retention framing. Then I can isolate the worst channel and see whether one cohort/channel stands out.

code
import pandas as pd
import numpy as np

# Merge core user/subscription info
df = users.merge(subscriptions, on='user_id',
                 how='inner', suffixes=('', '_sub'))

# Overall churn from subscription status
overall_churn_rate = (df['status'].eq('cancelled')).mean()
print("Overall churn rate:", overall_churn_rate)

# Observation window
max_event_date = events['timestamp'].max().normalize()
max_sub_end = subscriptions['subscription_end'].dropna().max()
analysis_end = max(max_event_date, max_sub_end)
print("Analysis end:", analysis_end)

# 30-day eligibility and retention/churn based on subscription status at/after 30 days
df['days_observed'] = (analysis_end - df['signup_date']).dt.days
df['eligible_30d'] = df['days_observed'] >= 30

eligible = df[df['eligible_30d']].copy()

Interface text:
cmpnd
All Traces
All Optimizations
All Deployments
INBOX 1
documents -> redacted_d... PredictRLM
PROGRAMS
No programs yet.
Account
Archived
platform-old.cmpnd.ai/traces/[illegible]

### Retention findings and recommendations

Among paid_campaign_x users, the trace reports 0% advanced_reports adoption in the first 7/14/30 days for churned users, vs ~17.7%/~32.3%/~55.3% for retained users. SUBMIT returns typed findings and recommendations.

[14:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=876s) [A trace shows analysis findings, recommendations, and a submission call.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca11cef88a2b2a10b9b3374ee4d6615924a3b0ec8f52e03aa224ea559deeb140.webp)

"(12.6 vs 29.0 events; 11.0 vs 25.2 active days), and within paid_campaign_x the clearest signal is failed activation: "
"churned users never adopt the sticky advanced_reports feature in the first 7/14/30 days (0% adoption vs ~17.7%/~32.3%/~55.3% for retained users), "
"with lower 30-day feature breadth and activity. Cancellation reasons are spread across competitor switching, no longer needed, support, and missing features, "
"which points to low-intent / poorly qualified campaign traffic that does not reach core product value."
)

recommendations = (
    "1) Audit or pause paid_campaign_x and tighten targeting/landing-page messaging to attract better-fit users. "
    "2) Redesign onboarding for campaign_x signups to drive first-30-day activation, especially guided paths to advanced_reports and other high-value features. "
    "3) Trigger lifecycle interventions for low-activation users in the first 7-14 days (tutorials, CSM outreach, in-app prompts, email nudges). "
    "4) Track advanced_reports adoption and 30-day active days as leading retention KPIs by channel. "
    "5) Review campaign promises versus product reality to reduce competitor-switching and missing-feature perception."
)

SUBMIT(0.2377, 'paid_campaign_x', key_finding, recommendations)

output
FINAL: {'overall_churn_rate': 0.2377, 'worst_channel': 'paid_campaign_x', 'key_finding': 'Overall churn is 23.77%, driven primarily by paid_campaign_x, the worst acquisition channel. Its users have a 44.8% cancellation

Interface text:
cmpnd
All Traces
All Optimizations
All Deployments
INBOX 1
documents -> redacted_d...  PredictRLM
PROGRAMS
No programs yet.
Account
Archived
Browser address: platform-old.cmpnd.ai/traces/[illegible]
150%
VPN

### dspy.RLM: exploring a dataframe

The model analyzes three DataFrames with 16MB / 250k events using previews in the prompt. In the shown step, it reads a KeyError and patches a merge before checking feature adoption by signup month.

[15:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=933s) [A slide lists the analysis stages and shows code correcting a missing signup-month field.](https://assets.ai.engineer/sites/aiecode2025/sha256/96cb26df4b1f8df4f13c78a13528f5f3a889a83cfa662ff76e4959a41d4990dc.webp)

03 · DSPY.RLM

dspy.RLM — exploring a dataframe

Three DataFrames, 16MB / 250k events. Only previews hit the prompt — the model writes and runs the analysis in the sandbox.

Explore
shapes, dtypes, channel & plan counts

Define churn
cancelled / total; cohort by signup month

Segment by channel
paid_campaign_x worst, every cohort

Dig into behavior
retained vs churned feature usage

Find the signal
advanced_reports adoption

Verify & submit
sanity checks, then
SUBMIT(...)

→ step 7 / 8

REASONING
turn 6 / 7

KeyError: signup_month missing from the verification table. Model reads the traceback and patches the merge.

fa = feature_adoption.merge(
    base[["user_id", "signup_month"]], on="user_id", how="left"
)

print(pcx.groupby("signup_month")["used_advanced_reports"].mean())
print(
    pcx.groupby(["signup_month", "used_advanced_reports"])["is_churned"]
    .agg(["mean", "count"])
)

kmad.ai

Interface text:
Recursive Language Models — [illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-20
VPN

### Invoice to inventory

The example turns two invoice PDFs into a typed Excel workbook, with line items on vendor sheets and a reconciled summary.

[15:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=942s) [Two invoice PDFs are converted into a spreadsheet of line items.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a1456c4fdcc64d9aae3716c8231cb697ab116ec8cee7ce3d006809213b43235.webp)

03 · PREDICT-RLM · LIVE RUN

Real world use case:
Invoice to inventory

2 INVOICE PDFS IN

Invoice 1: INVOICE; Acme Corporation; [illegible]
Invoice 2: INVOICE; [illegible]

ONE .XLSX OUT

Description | Quan... | Unit Pri... | Amou...
Software license - Enterprise (annual) | 1 | 12000 | 12000
Implementation consulting (40 hrs) | 40 | 175 | 7000
Data migration service | 1 | 3500 | 3500
User training sessions (8 hrs) | 8 | 200 | 1600
Priority support add-on (annual) | 1 | 2400 | 2400
Custom integration development | 1 | 4500 | 4500
Discount (10%) | 1 | -3100 | -3100

Two PDFs → one typed .xlsx — line items lifted straight off the page, one sheet per vendor plus a reconciled Summary. predict-rlm/examples/invoice_processing

kmad.ai

Interface text:
Recursive Language Models — Cont...; Cmpnd; The Mismanaged Geniuses Hyp...
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-22
VPN

### Focus on abstractions

The invoice run surveys two files, extracts pages in parallel with gpt-5.1, and reconciles a total before saving a workbook. It reports 5 REPL iterations, 60s, and $0.12 ($0.06/page).

[16:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=1015s) [Code and a three-stage workflow show invoice survey, extraction and verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/b8948aff8d88e1ff13e620c916b1d74d07a6969ec95fc08f11fb9e8c00da76b0.webp)

03 · PREDICT-RLM · THE RUN

Focus on abstractions instead of context engineering.

from predict_rlm import PredictRLM, File
from predict_rlm.skills import pdf, spreadsheet

rlm = PredictRLM(ProcessInvoices,
                 skills=[pdf, spreadsheet],
                 sub_lm="openai/gpt-5.1")
await rlm.acall(invoices=[File("acme.pdf"),
                          File("globaltech.pdf")])

One DSPy signature → typed InvoiceExtractionResult + an .xlsx workbook. No agent scaffold. Orchestrator gpt-5.4 drives the REPL; gpt-5.1 reads the page images.

01 Survey  8.0s
pymupdf.open(p)  # 2 files · 2 pages
# vendors: Acme Corp, GlobalTech

02 Render + extract  15.8s · 2 sub-calls
await asyncio.gather(*[
  predict("page: dspy.Image -> invoice", page=img)
  for img in pages])  # parallel, gpt-5.1

03 Self-verify  12.4s
# GlobalTech: subtotal+tax != total
# 33817.9 vs 30717.9 → reconciled
wb.save("invoice_extraction.xlsx")

5 REPL iterations · 60s · $0.12 ($0.06/page)

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hyp[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-23
VPN

### HALO engine architecture

HALO’s root model explores OpenTelemetry traces with tools, delegates work to bounded subagents, and compacts findings into persistent context items. A per-depth semaphore limits parallel subagents to four.

[18:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=1109s) [Diagram of a root language model querying traces, spawning bounded subagents, and storing compacted context items.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2ca30ce0db6eee736bb5f951215be6e5a61977ae51e8585945e091b85bd5bc8.webp)

03 · IN THE REAL WORLD · HALO

RLM (Root LLM)
depth = 0

invoke tool
get_dataset_overview
query_traces
count_traces
view_trace
search_trace
get_context_item
synthesis
run_code (sandboxed)

OTel Trace Store

invoke tool
call_subagent
Compaction model
context-item store

Subagent 1
depth = 1
Subagent 2
depth = 1
Subagent 3
depth = 1
Subagent 4
depth = 1

call_subagent
Grandchild Subagent
depth = 2 (terminal)
no call_subagent tool

per-depth semaphore (max_parallel_subagents = 4)

HALO Engine — bounded recursion, per-depth parallelism, persistent context items.

HALO (context-labs/halo) — a recursive-LM engine over OpenTelemetry traces: the root LM navigates the trace store through tools, spawns bounded per-depth-parallel subagents, and compacts findings into persistent context items.

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-26
VPN

### What it caught, lesson by lesson

The audit caught 3 of 10 lessons, partially caught 3, and missed 4. Misses included authentication bypass, billing concurrency abuse, race conditions, and vulnerable dependencies.

[19:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=1196s) [Table of ten security lessons: three caught, three partially caught, and four missed.](https://assets.ai.engineer/sites/aiecode2025/sha256/b09a270f7ba8805caa9997b7f1eb56b0e23dfb31c5057932b9882f6d21a2106a.webp)

03 · DSPY.RLM · SECURITY AUDIT

What it caught,
lesson by lesson.

LESSON | TOPIC | STATUS | NOTES
#1 | Event Injection | Partial | Caught S3 command injection; missed node-serialize code injection
#2 | Broken Authentication | Missed | JWT bypass & open billing API not documented
#3 | Sensitive Info Disclosure | Caught | Admin receipt access via S3
#4 | Insecure Cloud Config | Caught | S3 public write & command injection
#5 | Broken Access Control | Partial | Caught IDOR/privilege escalation; missed payment bypass
#6 | Denial of Service | Missed | Billing concurrency abuse not mentioned
#7 | Over-Privileged Functions | Caught | Comprehensive IAM policy violations
#8 | Logic Vulnerabilities | Missed | Race condition/TOCTOU not documented
#9 | Vulnerable Dependencies | Missed | node-serialize, node-jose, shell-quote not mentioned
#10 | Unhandled Exceptions | Partial | Generic error disclosure noted; specific examples missed

3 caught · 3 partial · 4 missed — every miss was runtime behavior static reading can't see.

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-28
VPN

### Imagine RLM-native models

Madura imagines models post-trained to understand and use the RLM methodology natively, which he calls the biggest promise he sees.

[20:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=1201s) [Closing slide asks viewers to imagine models post-trained to be RLM-native.](https://assets.ai.engineer/sites/aiecode2025/sha256/2252efb118ca180206915f6af4b5b5bae594e06218125c83e8963f9d5fae4966.webp)

WHAT'S NEXT?

Imagine models post-trained
to be RLM-native...

kmad.ai · @kmad ·
RLM paper: arxiv 2512.24601

kmad.ai

Interface text:
Recursive Language Models — Cont[illegible]
Cmpnd
The Mismanaged Geniuses Hy[illegible]
file:///Users/kmad/dev/rlm-preso/presentation/rlm-context-as-environment.html#slide-29
VPN

## Full transcript

[0:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=12s) Yep. Awesome. Thanks everyone for being

[0:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=15s) here. My name is Kevin Madura. I'm from

[0:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=17s) a company called Alex Partners. We're

[0:19](https://www.youtube.com/watch?v=xo68uCibfm8&t=19s) we're a consulting firm. Um I'm here to

[0:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=21s) talk to you today about RLMs. Just

[0:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=23s) curious, show of hands, who here is

[0:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=26s) familiar with RLMs? So we know how much

[0:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=29s) time to spend on it. Okay. So not many.

[0:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=31s) All right. Well, that's good. So we'll

[0:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=33s) start with what an RLM is and and why

[0:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=35s) it's different. So RLM is recursive

[0:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=38s) language model. And really the the key

[0:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=40s) difference here is that it treats the

[0:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=43s) context as an object that it can

[0:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=46s) interact with symbolically in its

[0:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=47s) environment. So it differs from a tool

[0:50](https://www.youtube.com/watch?v=xo68uCibfm8&t=50s) call in the sense that typically when

[0:52](https://www.youtube.com/watch?v=xo68uCibfm8&t=52s) you do a tool call it's JSON or some

[0:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=55s) type of string that's being sent being

[0:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=58s) interpreted elsewhere maybe by some

[1:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=60s) other program and that's that's

[1:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=62s) returning effectively as a string. The

[1:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=64s) key difference here is that it's

[1:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=66s) interacting with a symbolic environment.

[1:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=68s) So typically that's a ripple Python

[1:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=70s) ripple. Um so that's key difference

[1:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=73s) number one. Key difference number two is

[1:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=75s) that it's it has the ability to delegate

[1:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=78s) to another LM often to itself. You can

[1:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=80s) specify whether it's the same model or a

[1:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=83s) different model, but fundamentally

[1:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=84s) because it lives in this environment,

[1:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=86s) you can offload or or make a sub call to

[1:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=89s) another LM with particular parameters

[1:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=91s) that also lives in that ripple

[1:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=93s) environment. And so you get this ability

[1:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=96s) to recursively decompose problems and

[1:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=100s) apply and have the LLM basically decide

[1:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=102s) how to apply certain logic or certain

[1:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=105s) interpretations or write its own code to

[1:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=108s) solve those problems and then that

[1:50](https://www.youtube.com/watch?v=xo68uCibfm8&t=110s) recurses down. So the subLM can do the

[1:52](https://www.youtube.com/watch?v=xo68uCibfm8&t=112s) same sort of thing in terms of

[1:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=113s) understanding and interpreting what it

[1:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=115s) thinks it needs to do. And I added this

[1:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=118s) last one here. It's largely bitter

[2:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=120s) lesson pilled in my opinion, right? is

[2:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=123s) and and shared by Alex and the rest of

[2:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=125s) the the creators of it. But as models

[2:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=128s) get better, you should be able to defer

[2:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=130s) more and more to the model for it to

[2:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=133s) kind of figure out on its own what it

[2:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=135s) needs to do.

[2:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=137s) So the I don't know if this is the

[2:19](https://www.youtube.com/watch?v=xo68uCibfm8&t=139s) actual kind of starting point for RLMs.

[2:22](https://www.youtube.com/watch?v=xo68uCibfm8&t=142s) This is one that that I consider to be

[2:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=144s) one of the the first kind of inklings of

[2:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=146s) it. This is a tweet from Omar who is

[2:28](https://www.youtube.com/watch?v=xo68uCibfm8&t=148s) Alex's adviser um for for RLMs. And this

[2:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=152s) was a concept that he had come up with

[2:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=154s) where it was basically an ability to use

[2:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=157s) DSPI and some other techniques to take

[2:39](https://www.youtube.com/watch?v=xo68uCibfm8&t=159s) in arbitrary

[2:41](https://www.youtube.com/watch?v=xo68uCibfm8&t=161s) um arbitrary length inputs. And

[2:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=163s) basically the use case here would be

[2:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=165s) summarizing an arbitrarily long document

[2:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=167s) and coming up with a table of contents

[2:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=169s) and some summary of of that content. But

[2:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=173s) at least to me, this is kind of the

[2:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=174s) first inkling of okay, context windows

[2:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=176s) might not be something you need to

[2:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=179s) deliberately manage. Although there's of

[3:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=180s) course benefits to doing so, uh there

[3:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=182s) there could be ways to um to exceed the

[3:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=185s) the the context windows using some of

[3:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=187s) these clever techniques.

[3:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=190s) Uh and so if you read the paper and some

[3:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=192s) of the blog posts that are out there

[3:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=193s) from Alex and Omar, I mean it it it has

[3:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=196s) demonstrabably better performance on

[3:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=198s) some of these long context tasks. So,

[3:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=201s) ulong is one benchmark where um the the

[3:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=205s) intent of the benchmark it's is to

[3:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=206s) measure model performance on answering

[3:28](https://www.youtube.com/watch?v=xo68uCibfm8&t=208s) questions about excessively long

[3:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=209s) context. This another one browse comp

[3:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=212s) where it needs to iterate through a

[3:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=214s) large a large body and corpus of text

[3:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=216s) and answer particular questions about

[3:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=218s) it. You can see the blue line at the top

[3:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=220s) there is the RLM. It's very good

[3:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=222s) performance as compared to some of these

[3:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=224s) other models. And even on on the price

[3:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=226s) curve, the the purple is actually just

[3:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=229s) using tool calling um with GBT5 calling

[3:52](https://www.youtube.com/watch?v=xo68uCibfm8&t=232s) a BM25

[3:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=234s) tool. And that's actually even more

[3:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=236s) expensive for worse performance than

[3:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=238s) than an RLM. So it's worth reading into

[4:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=241s) if you're interested in in some of the

[4:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=242s) benchmarks and how RLMs perform. Um but

[4:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=245s) fundamentally uh an RLM again takes in

[4:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=249s) your input and you're kind of deferring

[4:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=250s) to the model about how to decompose the

[4:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=253s) process what code it needs to write and

[4:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=256s) it is very tightly integrated with the

[4:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=258s) ripple itself. So it it by itself

[4:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=260s) defines what it needs to do. And so I

[4:22](https://www.youtube.com/watch?v=xo68uCibfm8&t=262s) kind of had this mental model in terms

[4:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=264s) of and I'm very DSP pipel if if you

[4:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=267s) couldn't tell by now basically a student

[4:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=269s) of of Omar and the rest of the group

[4:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=271s) there where you have this relatively

[4:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=273s) deterministic shell of what you want to

[4:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=276s) do like what what is your intent what is

[4:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=278s) your actual task that you're trying to

[4:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=280s) accomplish. You define that in terms of

[4:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=282s) your inputs and your outputs and some

[4:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=284s) type of guidance or prompt or what have

[4:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=287s) you to the model to say this is

[4:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=289s) generally what I want to achieve. go off

[4:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=291s) and do it. Here's the things that you

[4:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=293s) can expect as your input. Here's what I

[4:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=294s) want out of it. Go figure out the rest.

[4:57](https://www.youtube.com/watch?v=xo68uCibfm8&t=297s) And so this applies for using something

[4:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=299s) like DSPI, but I think it applies to

[5:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=300s) RLMs as well because you don't have to

[5:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=303s) worry as much now about how the actual

[5:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=307s) implementation works in the middle. You

[5:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=310s) can just have some guarantees about the

[5:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=311s) inputs and the outputs and you can let

[5:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=312s) the model figure out the rest of of that

[5:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=315s) part of it.

[5:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=317s) So, a lot of this comes down to if if

[5:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=318s) you were at um I think it was code in

[5:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=321s) November in uh in New York City, Dex had

[5:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=324s) this great talk about just broader

[5:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=325s) context engineering and he he coined

[5:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=327s) something like the dumb zone which it's

[5:30](https://www.youtube.com/watch?v=xo68uCibfm8&t=330s) kind of grayed out at the bottom there.

[5:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=331s) But the point is that we all know that

[5:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=333s) there's context rot, right? Once you

[5:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=335s) fill up the context window to a certain

[5:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=337s) degree, performance starts to degrade.

[5:39](https://www.youtube.com/watch?v=xo68uCibfm8&t=339s) And so RLMs somewhat get around this

[5:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=342s) problem because the context itself

[5:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=346s) doesn't fill up as quickly because

[5:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=347s) you're deferring a lot of the subtasks

[5:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=349s) to the subm models and it's the full

[5:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=353s) kind of context and the inputs aren't

[5:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=355s) exposed to the context window itself. It

[5:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=358s) lives as a variable in the ripple and so

[6:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=360s) the main LM can choose how to um how to

[6:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=364s) access that. It can offload some of

[6:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=366s) these subtasks to to sublim. And really

[6:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=369s) the only context that it gets back are

[6:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=371s) the things that actually matter. So in

[6:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=373s) terms of how it's meaningfully

[6:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=375s) different, rag of course you kind of

[6:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=376s) just stuff the context window. You want

[6:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=378s) it to limit there. Agents are largely

[6:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=381s) just bringing strings back and you don't

[6:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=384s) have this tight coupling between the

[6:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=386s) logic, the execution and the results.

[6:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=389s) And so you still run into the same sort

[6:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=391s) of problem there. Same thing with tool

[6:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=393s) calling and and codec. And then RLMs as

[6:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=397s) I mentioned you're act the the LLM is

[6:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=400s) actually just interacting with the

[6:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=402s) context the results as variables in the

[6:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=404s) ripple so that it can do additional

[6:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=406s) computation on versus it trying to

[6:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=409s) attend to all these different tokens in

[6:50](https://www.youtube.com/watch?v=xo68uCibfm8&t=410s) this in its context window. It's it's a

[6:52](https://www.youtube.com/watch?v=xo68uCibfm8&t=412s) meaningfully different way of of the LLM

[6:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=414s) interacting with the uh the actual

[6:57](https://www.youtube.com/watch?v=xo68uCibfm8&t=417s) content itself.

[6:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=419s) And so people always say, okay, what's

[7:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=420s) the difference between that and and

[7:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=422s) encoding agents? Um,

[7:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=425s) in my mind, the largest difference is

[7:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=427s) that the way that tool calls are tool

[7:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=429s) calls calls are done is passing strings

[7:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=432s) back and forth. Um but you can see with

[7:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=434s) the release recently of workflows that

[7:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=438s) uh Anthropic is doing something fairly

[7:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=440s) similar and they um at the CIS

[7:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=443s) conference I think it was Tar or someone

[7:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=445s) similar um mentioned the RLM paper as a

[7:30](https://www.youtube.com/watch?v=xo68uCibfm8&t=450s) key driver of workflows and how they're

[7:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=453s) how they've implemented it. And you can

[7:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=455s) see here the intermediate results for

[7:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=458s) workflows live in script variables i.e.

[7:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=462s) a variable in the context. So, it's

[7:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=464s) driving some of these these

[7:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=466s) breakthroughs and some of these

[7:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=467s) techniques from the from the labs as

[7:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=468s) well.

[7:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=473s) I'll skip through this a bit just

[7:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=474s) because I have about 10 minutes left,

[7:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=476s) but generally speaking, when you want to

[7:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=478s) use it, it's obviously for large or

[8:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=480s) dense input context.

[8:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=482s) An underexplored area is outputs as

[8:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=485s) well. So if you have a some type of task

[8:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=489s) where you need to generate hundreds of

[8:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=491s) thousands of lines or whatever it might

[8:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=492s) be, RLMs I think would be a good

[8:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=494s) candidate for that as well. Obviously

[8:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=497s) tasks that are imunable to some type of

[8:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=498s) decomposition. So if you want to look

[8:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=501s) through the entire I don't know the

[8:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=503s) whole tax code as an example and try and

[8:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=505s) find loopholes or something. You can't

[8:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=507s) obviously put all of that into context

[8:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=509s) at once. You could use an LLM to crunch

[8:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=512s) through all of that and iteratively

[8:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=514s) explore and use sub agents to explore

[8:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=516s) interesting areas of this of the tax

[8:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=518s) code. Bring back those sections and then

[8:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=520s) reason over that and then just generally

[8:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=522s) for for longer horizon sessions. And

[8:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=525s) when you want to skip it, of course, it'

[8:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=526s) be something that fits in context. You

[8:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=528s) want something that's low latency or the

[8:50](https://www.youtube.com/watch?v=xo68uCibfm8&t=530s) the model itself is as strong of a

[8:52](https://www.youtube.com/watch?v=xo68uCibfm8&t=532s) coder.

[8:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=534s) And uh our friend Raymond here did some

[8:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=536s) great perfor performance testing on the

[8:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=538s) long chain of thought benchmark. Um I'll

[9:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=541s) leave this link as a as a leave behind

[9:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=543s) after, but just to give you a sense of

[9:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=544s) how well it performs on some of these

[9:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=546s) tasks. It's a meaningful jump overall

[9:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=549s) from from 2.6 to 45.4%

[9:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=552s) um accuracy on many of these tasks. And

[9:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=554s) you can see it performs really well on

[9:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=556s) things that are amanable to code. So

[9:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=558s) logic puzzles and and chess and

[9:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=560s) chemistry and things like that where it

[9:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=563s) can dynamically write code bring in only

[9:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=566s) the relevant part of the context compute

[9:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=569s) that and then return the result where

[9:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=571s) the main model is really just harvesting

[9:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=572s) the results from from the subLM and try

[9:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=574s) instead of trying to do that by itself.

[9:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=578s) Um I put together a few just super

[9:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=580s) simple examples. I mean, these are kind

[9:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=582s) of they're somewhat unfair, I suppose,

[9:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=584s) to to the base model, but it it makes

[9:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=586s) the point that there are certain tasks

[9:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=588s) that base models just aren't really fit

[9:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=591s) to do themselves because they because

[9:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=593s) they have to attend all these to

[9:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=594s) different tokens at once in the context

[9:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=596s) window where you need or want to use

[9:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=599s) some type of coding approach to that. So

[10:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=602s) in this random example, summing 12

[10:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=604s) numbers that are buried across 30,000

[10:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=605s) tokens, the LLM trying to figure all

[10:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=609s) that out by itself and give you the

[10:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=610s) answer isn't always going to work as

[10:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=612s) well as something that you can write reg

[10:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=614s) x4 or something similar. And then the

[10:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=617s) same same sort of thing particularly for

[10:19](https://www.youtube.com/watch?v=xo68uCibfm8&t=619s) data frames and we'll walk through a

[10:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=620s) brief example here where because the LLM

[10:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=623s) can

[10:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=625s) interact with the data frame within the

[10:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=627s) ripple. It just has a much better

[10:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=629s) understanding of the content and can

[10:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=631s) iterate through that much more quickly

[10:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=632s) than having to pass tool calls back and

[10:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=635s) forth in terms of like JSON strings and

[10:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=637s) and that sort of thing.

[10:39](https://www.youtube.com/watch?v=xo68uCibfm8&t=639s) Um, and then I threw this in there in

[10:41](https://www.youtube.com/watch?v=xo68uCibfm8&t=641s) terms of running the same experiments

[10:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=643s) with a coding agent. Now, I didn't look

[10:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=645s) into this too deeply. There's probably

[10:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=647s) some unfair math going on here, but you

[10:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=649s) can see that it was totally bloated in

[10:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=651s) terms of the way that cloud code tried

[10:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=653s) to um tried to solve these tasks. So,

[10:57](https://www.youtube.com/watch?v=xo68uCibfm8&t=657s) there's more work to be done there, of

[10:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=659s) course, in terms of like running

[11:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=661s) experiments to compare base models

[11:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=663s) versus RLMs versus something like a

[11:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=666s) coding agent. But there's just for

[11:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=667s) certain tasks for like production

[11:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=669s) workloads. My sense is you probably

[11:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=671s) don't want to just do cloud-p

[11:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=674s) your prompt and like hope for a good

[11:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=676s) result. Like you want more of a

[11:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=677s) structured approach to your inputs, your

[11:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=680s) outputs and you want a defined pipeline

[11:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=681s) for doing so which reduces your cost, it

[11:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=684s) reduces your complexity, reduces your

[11:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=685s) bloat, all that sort of thing. Um where

[11:28](https://www.youtube.com/watch?v=xo68uCibfm8&t=688s) RLMs can can shine.

[11:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=692s) So in the real world there are a bunch

[11:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=694s) of different open source libraries that

[11:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=696s) implement RLMs at some level. Some of

[11:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=698s) them are more RLM focused uh like a

[11:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=702s) predict RLM would be a good example of

[11:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=704s) that versus others are kind of just

[11:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=706s) integrating it into the broader approach

[11:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=708s) or the broader framework. DSPI obviously

[11:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=711s) uh there's axe which is really

[11:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=713s) interesting work uh that's being done

[11:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=715s) there. Predict RLM is more focused on

[11:57](https://www.youtube.com/watch?v=xo68uCibfm8&t=717s) like knowledge work. So it works with

[11:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=719s) spreadsheets and PDFs and that sort of

[12:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=721s) thing and then fast RLM. And then

[12:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=724s) there's a tweet yesterday from this guy

[12:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=726s) Sam Hogan

[12:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=727s) um where who runs inference.net. He's

[12:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=729s) using an RLM to basically run and

[12:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=733s) extract um insights from your particular

[12:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=736s) um production workload traces so that

[12:19](https://www.youtube.com/watch?v=xo68uCibfm8&t=739s) they can see what makes sense to defer

[12:22](https://www.youtube.com/watch?v=xo68uCibfm8&t=742s) off to something like a GLM 5.2 too and

[12:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=745s) do that iteratively and automatically as

[12:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=746s) your traffic goes through. So point

[12:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=749s) being, you don't have to worry about

[12:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=752s) context engineering. You can kind of

[12:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=753s) just throw the RLM at it and have it

[12:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=755s) figure it out. Um, I only have five

[12:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=758s) minutes left, so we won't go through

[12:39](https://www.youtube.com/watch?v=xo68uCibfm8&t=759s) this whole example and I'll I'll skip to

[12:41](https://www.youtube.com/watch?v=xo68uCibfm8&t=761s) some of the traces because that's

[12:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=762s) probably the most interesting. Um, but

[12:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=764s) this is all you would really need to do

[12:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=766s) in terms of a simple, in this case it's

[12:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=769s) like a a cohort retention analysis,

[12:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=771s) something that you might give to a data

[12:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=773s) scientist. But this concept of applying

[12:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=776s) an RLM to a complex data structure like

[12:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=779s) a data frame becomes very easy to do.

[13:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=782s) This is all the code you need to do it

[13:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=784s) where I'm feeding in three different

[13:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=786s) data frames. I'm saying these are the

[13:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=788s) sorts of things you need to look for.

[13:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=789s) These are the output types that I want.

[13:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=791s) and then just let the RLM go on it. And

[13:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=795s) I'll I'll show you some of the traces.

[13:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=798s) Um, and so it has its own ripple where

[13:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=801s) it can interact with those data frames.

[13:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=803s) And you can see it reasoning through.

[13:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=805s) Okay, first I need to do this. It's

[13:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=806s) writing the code. And because it's li

[13:28](https://www.youtube.com/watch?v=xo68uCibfm8&t=808s) it's living in the ripple with the dataf

[13:30](https://www.youtube.com/watch?v=xo68uCibfm8&t=810s) frame, you don't have this additional

[13:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=812s) bloat of the tool calls back and forth.

[13:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=814s) It's actually interacting directly with

[13:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=816s) the dataf frame as if it was typing in

[13:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=818s) its own Jupyter notebook. And there

[13:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=820s) there's significant advantages for for

[13:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=823s) doing so. And so you can see the sorts

[13:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=825s) of outputs that it gets as a result. And

[13:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=828s) it by itself will iterate. And in this

[13:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=831s) case it didn't, but it has the option to

[13:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=833s) defer to subLM to do okay. And now I

[13:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=836s) have this big whatever this big subset

[13:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=838s) of the data sublm go off and do this

[14:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=840s) analysis give me the result and it can

[14:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=842s) do that iteratively over time. Uh but

[14:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=845s) the point is that the LM is directly

[14:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=849s) interacting with the data frame in its

[14:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=850s) ripple

[14:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=852s) um and kind of iterating through the

[14:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=854s) results. And so this u this platform

[14:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=857s) compound is RLM and DSPI native. So it

[14:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=861s) gives you this really nice breakdown of

[14:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=863s) the reasoning. It separates out the code

[14:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=865s) that's being generated and ultimately

[14:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=867s) you can see

[14:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=869s) uh the final output which is here where

[14:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=873s) it's formatting. Okay, here are the key

[14:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=875s) findings that I have. Here are the

[14:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=876s) recommendations. And then you have this

[14:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=878s) final submit which is the final answer

[14:41](https://www.youtube.com/watch?v=xo68uCibfm8&t=881s) that gives you the the typed um outputs

[14:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=883s) that you had defined up front. And the

[14:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=886s) key thing here is that the LLM itself is

[14:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=888s) deciding when to stop. So you have this

[14:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=891s) you have a variable of max iteration. So

[14:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=893s) you can just you can decide whether you

[14:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=895s) want it to have a maximum of 10 or 100

[14:57](https://www.youtube.com/watch?v=xo68uCibfm8&t=897s) or whatever it is. But it will by itself

[15:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=900s) explore the data, understand what needs

[15:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=902s) to happen and then when it it itself is

[15:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=905s) comfortable, it can run submit and give

[15:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=907s) you the final output. Again, being

[15:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=910s) bitter or less impilled, this will get

[15:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=911s) better over time. You can kind of just

[15:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=913s) defer everything and it will figure out

[15:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=915s) what to do. And so the hope would be you

[15:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=918s) don't have to I mean we're already you

[15:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=920s) know whatever this is 20 lines of code

[15:22](https://www.youtube.com/watch?v=xo68uCibfm8&t=922s) or something. Um, but you can you can

[15:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=925s) see a world where you can continue to go

[15:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=927s) up levels of abstraction. As long as you

[15:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=929s) can define what your objective is and

[15:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=931s) what you want it to do, the the model

[15:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=933s) will kind of figure out the rest. Uh, so

[15:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=936s) we just walk through a bunch of this,

[15:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=937s) but um these are the different steps

[15:39](https://www.youtube.com/watch?v=xo68uCibfm8&t=939s) that it took in this example in the code

[15:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=942s) that it wrote. Um, and then I'll just

[15:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=945s) breeze through a few real world case

[15:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=947s) studies and where it's actually being

[15:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=948s) used. So I mentioned predict rm before.

[15:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=951s) So the company trampoline AI I think it

[15:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=954s) is they're doing really interesting work

[15:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=956s) in applying RLMs u like I mentioned

[15:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=958s) before for different pieces of knowledge

[16:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=961s) work. So natively interacting with PDFs

[16:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=964s) and and spreadsheets and that sort of

[16:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=966s) thing. So in this relatively simple

[16:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=968s) example okay I have a bunch of I have a

[16:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=971s) directory of invoices that I need to

[16:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=973s) create one consolidated inventory out

[16:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=975s) of.

[16:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=977s) As we all know invoices can be

[16:19](https://www.youtube.com/watch?v=xo68uCibfm8&t=979s) complicated. that can be very long, that

[16:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=981s) can kind of be all over the place. To do

[16:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=984s) that today without RLMs or this sort of

[16:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=987s) like framework gets very complicated

[16:30](https://www.youtube.com/watch?v=xo68uCibfm8&t=990s) very quickly. I have a lot of battle

[16:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=992s) scars to to prove it. Um, but with

[16:34](https://www.youtube.com/watch?v=xo68uCibfm8&t=994s) something like an RLM, you you don't

[16:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=995s) need to worry as much about, okay, if I

[16:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=997s) have a 200page invoice or contract or

[16:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=1000s) whatever it is, you can let the RLM just

[16:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=1003s) churn through all of that and give you

[16:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=1004s) the result instead of having to worry

[16:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=1006s) about chunking and embedding maybe and

[16:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=1009s) doing all these different strategies to

[16:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=1011s) try and get around the context window

[16:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=1013s) management that we've all had to to do

[16:55](https://www.youtube.com/watch?v=xo68uCibfm8&t=1015s) previously.

[16:57](https://www.youtube.com/watch?v=xo68uCibfm8&t=1017s) Um, so it allows you, the point there is

[16:59](https://www.youtube.com/watch?v=xo68uCibfm8&t=1019s) that you can focus on the abstractions

[17:02](https://www.youtube.com/watch?v=xo68uCibfm8&t=1022s) and what you actually want to do instead

[17:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=1023s) of the context engineering itself, which

[17:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=1025s) I think is a really helpful um, helpful

[17:08](https://www.youtube.com/watch?v=xo68uCibfm8&t=1028s) output of all of this. And an

[17:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=1030s) interesting tidbit for all the DSPI fans

[17:12](https://www.youtube.com/watch?v=xo68uCibfm8&t=1032s) in the room, predict RLM uses DSPI to

[17:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=1036s) determine the schemas between the main

[17:19](https://www.youtube.com/watch?v=xo68uCibfm8&t=1039s) LM and the subLM calls, which I

[17:22](https://www.youtube.com/watch?v=xo68uCibfm8&t=1042s) personally think is is a nice feature

[17:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=1044s) because you have a lot more readability

[17:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=1046s) and maintainability. So you understand

[17:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=1049s) exactly what the model is trying to

[17:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=1051s) achieve and the model can be much more

[17:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=1052s) precise and prescriptive about the types

[17:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=1055s) of data that it's looking for from the

[17:37](https://www.youtube.com/watch?v=xo68uCibfm8&t=1057s) subLM. And I would want to do some

[17:40](https://www.youtube.com/watch?v=xo68uCibfm8&t=1060s) experiments to test this out, but I

[17:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=1062s) would think that this would improve

[17:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=1063s) performance for cheaper models like a a

[17:46](https://www.youtube.com/watch?v=xo68uCibfm8&t=1066s) Quinn or some of the other ones because

[17:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=1068s) you're specifying the inputs and outputs

[17:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=1069s) and you're enforcing those types coming

[17:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=1071s) back. And so you get all the benefits of

[17:54](https://www.youtube.com/watch?v=xo68uCibfm8&t=1074s) the RLM being able to churn through all

[17:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=1076s) this information, but you have a lot

[17:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=1078s) more of the structure in between where

[18:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=1081s) when it's handing off to a sublm, it it

[18:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=1083s) enforces some of the uh some of those

[18:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=1085s) schemas.

[18:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=1087s) This is an example from um an AWS

[18:10](https://www.youtube.com/watch?v=xo68uCibfm8&t=1090s) engineer from a couple days ago. We were

[18:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=1093s) just kind of playing around with it, but

[18:14](https://www.youtube.com/watch?v=xo68uCibfm8&t=1094s) I just thought it was a nice example of

[18:17](https://www.youtube.com/watch?v=xo68uCibfm8&t=1097s) you can kind of just throw arbitrary

[18:20](https://www.youtube.com/watch?v=xo68uCibfm8&t=1100s) data at RLM. In this case, it was a

[18:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=1103s) bunch of log data um to surface some

[18:26](https://www.youtube.com/watch?v=xo68uCibfm8&t=1106s) interesting uh results and he he found

[18:28](https://www.youtube.com/watch?v=xo68uCibfm8&t=1108s) it useful. Um there's a a project called

[18:32](https://www.youtube.com/watch?v=xo68uCibfm8&t=1112s) Halo which uses an RLM to look at traces

[18:36](https://www.youtube.com/watch?v=xo68uCibfm8&t=1116s) of um of different t uh agent tasks.

[18:41](https://www.youtube.com/watch?v=xo68uCibfm8&t=1121s) And basically the promise of Halo is

[18:43](https://www.youtube.com/watch?v=xo68uCibfm8&t=1123s) that instead of optimizing a particular

[18:45](https://www.youtube.com/watch?v=xo68uCibfm8&t=1125s) like workflow or DSPI uh or or other

[18:49](https://www.youtube.com/watch?v=xo68uCibfm8&t=1129s) framework

[18:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=1131s) uh like structure itself, it's it's

[18:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=1133s) actually iterating on the harness. So

[18:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=1136s) it's like a meta abstraction almost or

[18:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=1138s) meta optimization of the harness itself

[19:01](https://www.youtube.com/watch?v=xo68uCibfm8&t=1141s) and it uses an RLM because as we all

[19:03](https://www.youtube.com/watch?v=xo68uCibfm8&t=1143s) know tracing can get very long and and

[19:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=1145s) complicated.

[19:07](https://www.youtube.com/watch?v=xo68uCibfm8&t=1147s) So the RLM can not only take in all that

[19:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=1149s) context but also leverage the um the

[19:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=1153s) structure of those traces to to

[19:15](https://www.youtube.com/watch?v=xo68uCibfm8&t=1155s) recommend a better um a better harness.

[19:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=1158s) And then this last one uh this is all

[19:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=1161s) the code you need. I ran this little

[19:23](https://www.youtube.com/watch?v=xo68uCibfm8&t=1163s) experiment. There's a an intentionally

[19:25](https://www.youtube.com/watch?v=xo68uCibfm8&t=1165s) vulnerable application called uh it's

[19:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=1167s) from OASP, but basically there's a it's

[19:30](https://www.youtube.com/watch?v=xo68uCibfm8&t=1170s) a web app with a bunch of

[19:31](https://www.youtube.com/watch?v=xo68uCibfm8&t=1171s) vulnerabilities in it. This is all the

[19:33](https://www.youtube.com/watch?v=xo68uCibfm8&t=1173s) code you need on the right hand side to

[19:35](https://www.youtube.com/watch?v=xo68uCibfm8&t=1175s) run basically an agent to run through

[19:38](https://www.youtube.com/watch?v=xo68uCibfm8&t=1178s) whatever it is 500,000 lines of code to

[19:41](https://www.youtube.com/watch?v=xo68uCibfm8&t=1181s) generate some type of security report.

[19:42](https://www.youtube.com/watch?v=xo68uCibfm8&t=1182s) That's just an arbitrary example, but

[19:44](https://www.youtube.com/watch?v=xo68uCibfm8&t=1184s) the point is you don't need a lot of

[19:47](https://www.youtube.com/watch?v=xo68uCibfm8&t=1187s) context engineering. You don't need a

[19:48](https://www.youtube.com/watch?v=xo68uCibfm8&t=1188s) lot of structure around it to achieve

[19:51](https://www.youtube.com/watch?v=xo68uCibfm8&t=1191s) what you want to do. And so you can feed

[19:53](https://www.youtube.com/watch?v=xo68uCibfm8&t=1193s) in an arbitrary uh size codebase into

[19:56](https://www.youtube.com/watch?v=xo68uCibfm8&t=1196s) this and get some type of insights out.

[19:58](https://www.youtube.com/watch?v=xo68uCibfm8&t=1198s) So you can imagine that being applied to

[20:00](https://www.youtube.com/watch?v=xo68uCibfm8&t=1200s) other areas as well. Um so I know I I

[20:04](https://www.youtube.com/watch?v=xo68uCibfm8&t=1204s) rushed through everything a little bit

[20:05](https://www.youtube.com/watch?v=xo68uCibfm8&t=1205s) but I'm happy to answer questions

[20:06](https://www.youtube.com/watch?v=xo68uCibfm8&t=1206s) afterwards. Uh the I'll leave you leave

[20:09](https://www.youtube.com/watch?v=xo68uCibfm8&t=1209s) you with this. The biggest promise I see

[20:11](https://www.youtube.com/watch?v=xo68uCibfm8&t=1211s) here is just imagine a world where the

[20:13](https://www.youtube.com/watch?v=xo68uCibfm8&t=1213s) models are actually post-trained and

[20:16](https://www.youtube.com/watch?v=xo68uCibfm8&t=1216s) actually like RLM aware. I think things

[20:18](https://www.youtube.com/watch?v=xo68uCibfm8&t=1218s) will get pretty crazy pretty quick when

[20:21](https://www.youtube.com/watch?v=xo68uCibfm8&t=1221s) they actually know how to use and kind

[20:24](https://www.youtube.com/watch?v=xo68uCibfm8&t=1224s) of take advantage of the RLM methodology

[20:27](https://www.youtube.com/watch?v=xo68uCibfm8&t=1227s) natively. So, thank you so much for your

[20:29](https://www.youtube.com/watch?v=xo68uCibfm8&t=1229s) time.
