Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse

Give a coding agent enough tokens and it can keep producing possible changes. Someone still has to decide which change goes into production. Raymond Weitekamp of OpenProse asks how much of an agent’s work has that second shape. If the options already exist, what should a model return?

A decision model ranks or selects among the options it receives. Its output is a judgment. Application code decides what to do with it. The term System One borrows the distinction in Thinking, Fast and Slow between quick, intuitive judgments and slower, deliberate problem solving. Zero-shot classifier is another name used here. The interesting engineering question is whether a series of fast judgments can accomplish a task that seems to require deliberate reasoning.

A recursive language model (RLM) gives a root model the ability to call a child RLM, which can in turn call another. The program caps the depth. In Weitekamp’s example, a branch that reaches depth two ends with an ordinary LLM call. A recursive decision model (RDM) applies the same rule to selection. One of the choices at a node can be to ask a narrower question. At maximum depth, a single decision call settles it. Recursive Decision Models

The position of the first call changes which work the application pays for.

• Decision as a tool. A coding agent can use a classifier when compacting context, checking whether a proposed tool call may execute, or routing a request to a model. The surrounding agent is already running when it asks for the judgment.
• Decision at the root. The application starts with the classifier. Calling a language model or delegating to an agent is one of its choices. A branch that can finish with decisions alone can go all the way to an action without generating text.

What can a Doom agent learn from the map that it can’t see from the player’s perspective? One group of decisions handles immediate choices such as holding position, walking, or dodging. The program combines the answers into the next player action and repeats as the game changes. A second group uses a bird’s-eye view of the map to choose where to explore. Coupling the two gives the player somewhere to go and a way to react to what happens on the way there. Both jobs use decision calls.

Doom illustrates how to compose decisions. Weitekamp adds a further capability for recursion and uses three rules to guide the design.

• Keep context outside the model. Code holds the full game state and cuts a different slice for each question. The movement controller gets its local view. The exploration controller gets its map view.
• Let a decision request another decision. Descending into a subproblem is itself a choice the model can make. The program enforces the depth limit.
• Make generation optional. First get a task to work with decision calls alone. The same architecture should also be able to call an LLM or agent when a task needs one.

On an idealized Galton board, a falling ball has a 50–50 chance of going left or right at each pin. After many drops, the piles form a bell shape. Now give each ball a number and let the pins read it. The first pin sends lower numbers one way and higher numbers the other. The next pin divides its own range again. Successive choices sort the balls into numbered buckets.

What happens if a pin can answer two questions at once? The three-dimensional board sorts by color along one axis and number along the other. A cross-section along each axis lets you check each judgment separately. With perfect classification, a ball lands in the bucket for both its number and its color. One classifier call could assign that pair directly. The board earns its keep as a picture of how an early choice restricts the choices farther down.

A product catalog gives the hierarchy a practical job. A search example uses 12,000 Shopify product categories and a query such as laptop backpack. Presenting every category at once risks overloading the context window. Give the classifier the query and the children of the current parent category, and its ranking can tell the program which part of the catalog to open for the next call.

The original example used beam search, keeping several promising paths alive. Code can also make the search greedy and follow only the highest-ranked child. That saves exploration at the price of committing to an early classification. Keeping the top three, one alternative Weitekamp gives, allows a lower-ranked branch to reach a more plausible category farther down. The classifier supplies the rankings. The programmer chooses how many paths to pursue.

For an image, which pixels should the next call see? In a beetle search, a positive judgment prompts the program to zoom in and ask again about a smaller box. Another implementation recursively halves the image and classifies the resulting blocks. In the first, a hit directs the crop. In the second, the subdivision rule determines which regions the classifier can inspect. A capable vision model might answer these particular questions in one pass. Recursion gives the program control over the area each judgment concerns.

Which part of a bank of similar production lines needs repair? In Weitekamp’s Factorio factory, the RDM picks the group with a problem, narrows it to one line, then identifies an inserter on that line for replacement. The character runs over to do the job and moves on to the next one. Each selection changes what the next call has to inspect. By the time the character moves, it has a particular part to attend to, with no generative model in the loop.

The regular layout helps. The factory is effectively a matrix of repeated components, so each question has a manageable set of answers. The character also runs the long way around to the next repair. Weitekamp considers adding a pathfinding component. The diagnosis gave the character a destination and left the quality of its route to another part of the program.

Where do the options come from? A script, the application, or another agent must enumerate what the classifier can choose. Games can expose their state and available moves programmatically. The talk’s slide website offers a smaller example. Its document object model (DOM) exposes three interactions on the page in question. You can go up, go down, or start a video. A desktop has many more visible controls and keyboard shortcuts, but wherever code can enumerate them, selection becomes a plausible way to operate the interface.

An outline walk can descend from a document title to a heading, subsection, paragraph, sentence, and word. The document supplies the possible paths. What changes when a model can introduce a new option? In the call trees from Weitekamp’s smart-home demo, squares mark generative calls. A request can narrow down to an existing action or reach a generator that can change the choices available. The branches take different shapes depending on what the request needs.

Weitekamp expanded an earlier smart-home demo with more rooms and appliances. A bedtime request implies several changes, including locking the door, leaving a lamp on, and switching other devices off. A request to watch a film is less explicit still. The house needs a way to associate the user’s phrase with a combination of appliance commands.

The shared interface is a map. An entry has a name, a short description, and a pointer to behavior. TV on can point to a function that sets a Boolean. Another entry can point to a whole loop. The decision model judges which named entry fits the request. Code invokes the function behind that entry, so a classifier can select a complicated operation as readily as it selects a switch.

The LLM gets a structured way to add or edit entries. If the user’s movie routine means turning the TV on and the lights off, the LLM can record that combination as movie time. On a subsequent request, the decision model has a single option pointing to both actions. The learned preference lives in the map. The RDM implementation can also represent entries in several dimensions, including low-dimensional embeddings whose dimensions carry semantic meaning.

Fast, inexpensive decision calls make speculation affordable. Several models can read the same state and suggest actions concurrently. Let each one act as soon as it finishes, though, and completion order starts deciding what happens. A fast model can change the state while a slower model is still evaluating the old state. Its answer may call for the opposite change.

Which proposal should be able to change the state? For speech arriving in real time, different channels can already favor an action before the sentence finishes. An arbiter considers the evidence available across channels at that moment and decides whether it has enough confidence to let an action through. Weitekamp compares this to concurrent reads with a lock on writes. The arbiter adds a judgment about whether a proposed write is justified at all.

The first RDM implementation was a module in a programming framework that already supported System One API calls. Its conventions gave Weitekamp a way to check that the modules composed as intended. He also built a standalone Rust package for applications that need a fast, portable implementation. RDM repository

Operating System One is a play on learning to operate System One models, with a physical goal behind it. Mousing left Weitekamp with shoulder tension. Graduate-school deadlines brought repetitive strain and a collection of vertical mice, trackballs, and ergonomic keyboards. He wanted to keep doing computer work while spending less of the day holding the same posture. Voice seemed within reach.

An earlier experiment turned the laptop itself into a drum kit. The machine had a built-in accelerometer. Hit one of four locations on its case and a classifier distinguished the impact well enough to trigger the corresponding drum. Tilting the laptop controlled an effect. In another mapping, each hit triggered a side-chain compressor, briefly ducking the sound. The same hardware could take a discrete input like a strike or a continuous input like tilt.

Weitekamp learned programming and machine learning through the Princeton Laptop Orchestra. Some pieces made an instrument of the trackpad or keyboard. Others synchronized clocks over Wi-Fi so players’ machines could exchange precisely timed messages. Small models such as k-nearest neighbors and single-layer neural networks could learn mappings on the fly, classify gestures, or interpolate between controls. A performer could hear what latency cost. The sound had to arrive in time with the gesture.

Earshot gives a voice interface a task you can watch and hear. Spoken numbers select desktop workspaces one through eight. Initially, the system transcribes the audio, identifies the number, and switches to the corresponding workspace. You hear the count and then see the desktop catch up. Could the system recognize enough of a familiar sound to switch sooner? This experiment was a precursor to oSo.

How can the same spoken number operate the desktop and teach a faster response? Two paths listen at once. While the transcription path supplies an action and a teaching signal, a much smaller supervised model takes coarse frequency bins from short slices of the same audio. Weitekamp gives 40, 80, and 100 milliseconds as example slice lengths. The small model starts without a learned association between those features and a workspace. Another layer judges when its prediction is trustworthy enough to follow. That lets training begin well before the learner gets to move the desktop.

The counting in Einstein on the Beach makes the handoff perceptible. One through four recur more often, so the learner gets more examples of those sounds. Their workspace switches start keeping up with the count while later numbers still lag. By the end of the demonstration Weitekamp reports about 45 milliseconds for the learned response, with an occasional blip. That timing belongs to the repeated-number task, where the fast path only has to distinguish a handful of familiar sounds.

How fast can a recognizer respond when the input is a drawing? Hunch plays Pictionary by classifying a user’s doodles locally with an embedding model running through WebGPU. Weitekamp points to a classification time of roughly 150 milliseconds in that demo. Here the input is a sketch and the immediate output is a guess about what the user is drawing.

oSo, short for Operating System One, was a preview at the time of the talk. Its starting question is what it would take to work on a computer without touching it, and eventually without needing to see it. Voice is the first input. In the proposed design, an RDM can handle a familiar command directly or hand a job to a coding agent or recursive language model when the job needs generation.

There are two things for such a system to learn. One is what a particular user wants, as with the household’s movie routine. The other is which model can reliably recognize that intent. oSo aims to move a recurring decision to the smallest, fastest available model once its past performance gives the arbiter reason to trust it. Earshot supplies the working example of that second kind of learning. The desktop starts responding sooner to the numbers it has heard most often.

Weitekamp reads engineers’ return to the terminal as a willingness to leave behind some of the constraints of conventional applications. For someone whose shoulder hurts after a day of mousing, typing commands still leaves a physical problem to solve. He invited viewers to join the oSo waitlist for an experiment with a different kind of control. A word could operate the computer as directly as a tap on a laptop once played a drum. oSo project