Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci

How do you turn a small language model that barely plays tic-tac-toe against a random opponent into a strong player? Giving it examples is one starting point. Giving it a game engine, opportunities to explore, and feedback on its decisions opens another route. Stefano Fiorucci, who develops Haystack at deepset and experiments with small models outside that work, uses this problem to explore reinforcement learning environments.

An environment gives an agent somewhere to use tools, execute code, and attempt tasks that take multiple steps. Its responses make the consequences of those actions available for evaluation and learning. Fiorucci places this work in a growing ecosystem of environment startups working with major AI labs, and cites DeepSeek and MiniMax reports describing training across thousands of environments. Tic-tac-toe makes the engineering questions small enough to inspect: what can the model do, what changes when it acts, and what should count as success?

The basic reinforcement learning loop has two participants: an agent and an environment. The agent observes the current state and chooses an action. The environment changes in response and returns a numerical reward indicating how good or bad the outcome is.

The objective is to maximize cumulative reward. That requires balancing exploration, trying actions that might reveal better strategies, with exploitation, choosing actions already known to work. The resulting sequence of states, actions, and rewards is a trajectory, also called a rollout. Here, those terms mean a complete episode—for example, one whole game, rather than one move.

A language model takes a prompt and generates a completion. Fiorucci introduces its conventional training recipe in three stages:

PPO, or proximal policy optimization, is one technique used in the third stage. The distinguishing feature of SFT is statistical imitation: given prompt–response pairs, the model learns to produce responses resembling those examples.

The motivation for reasoning-oriented RL comes from the limits of that recipe. Fiorucci invokes Ilya Sutskever’s NeurIPS 2024 discussion of diminishing returns from pretraining, then points to OpenAI o1. OpenAI described improvements from both more reinforcement learning during training and more time spent thinking at inference. The slide illustrates those two compute axes with AIME accuracy plots, although the release disclosed relatively little about the training procedure.

DeepSeek-R1 offered a more concrete route. Reasoning traces can improve answers, but producing enough curated reasoning examples for SFT is expensive. DeepSeek used reinforcement learning with verifiable rewards, together with GRPO, group relative policy optimization, which offers a lighter training setup than PPO.

In the simplest verifiable-reward task, a model generates a reasoning trace and an answer. A checker compares the answer with known ground truth, then supplies a reward for training. The same principle extends to a won game or a successful tool call: an automatically checkable outcome can become a learning signal. SFT draws behavior toward the examples supplied; RL samples alternative trajectories from the model’s existing capabilities and reinforces those that score better. That creates room to discover strategies beyond the demonstrations.

The language model now occupies the agent’s place in the loop. Its environment includes the dataset, interaction harness, and scoring rules—everything needed to run and judge the task. Once the model can call a weather API or use a terminal, that environment becomes a dynamic software system. In tic-tac-toe, the model’s action is a text response specifying a move. The engine constructs prompts, tracks the board, generates opponent moves, and decides when the game ends. A simple reward might assign +1 to a win and 0 to a loss. Through repeated play, the model can discover useful strategies without a human-written example for every position. Fiorucci invokes Andrej Karpathy’s description of environments as opportunities to take actions and see outcomes, extending learning beyond expert imitation.

Verifiers, from Prime Intellect, packages these environments as installable, distributable Python packages. It supplies base classes for single-turn, multi-turn, and tool interactions, plus abstractions for response parsing and reward functions. The same environment can serve evaluation and training.

Model serving sits behind an OpenAI-compatible endpoint, allowing the environment to work with OpenAI, OpenRouter, or a local model served through vLLM. Verifiers handles asynchronous interactions and parallel trajectories. The workflow presented also includes a trainer and integrations with Prime RL, Tinker, and SkyRL, so the environment author can concentrate on task behavior and rewards.

The first example is Reverse Text. Its load_environment entry point assembles a dataset, parser, reward rubric, and SingleTurnEnv. The default dataset contains 1,000 paragraphs in a prompt column. Mapping creates a question containing the original paragraph and an answer containing its reversal.

An XMLParser extracts the response from the requested reversed-text tags. A reward function then compares that text with the reference answer. Although the spoken explanation calls the score a longest-common-subsequence ratio, the companion implementation uses difflib.SequenceMatcher(...).ratio(). The core transformation and comparison can be expressed directly in Python:

from difflib import SequenceMatcher


def prepare_example(row: dict[str, str]) -> dict[str, str]:
    text = row["prompt"]
    return {"question": text, "answer": text[::-1]}


def score_reversal(parsed_answer: str, answer: str) -> float:
    return SequenceMatcher(None, parsed_answer, answer).ratio()


example = prepare_example({"prompt": "Hello"})
score = score_reversal("olleH", example["answer"])

Parsing keeps output extraction separate from scoring. A rubric collects weighted reward functions; the environment bundles that rubric with the dataset and interaction setup.

Evaluation makes the assembled environment concrete:

1. Invoke load_environment and select five dataset examples.
2. Sample three rollouts for each example, producing fifteen conversations. Repeated questions can yield different completions because model sampling is stochastic.
3. Send the system prompt and question to the model.
4. Parse the response, calculate its reward, and save the result.
5. Aggregate summary statistics and the reward distribution.

Training uses the same interaction and scoring machinery, then adds parameter updates. This shared path is what makes an evaluation environment useful as a training artifact.

The DoubleCheck environment adds another exchange. The model answers a math question; the environment replies, “Are you sure?” The rollout continues so that the model can reconsider. MultiTurnEnv introduces a state dictionary for information maintained throughout the episode. A setup_state method can initialize it, although this particular example does not need that hook.

env_response returns a list of messages. Here the reply is fixed, but a richer environment can generate its response from the current state. A method marked with @vf.stop, the decorator shown in the companion code, is checked on each turn. When it returns true, the rollout ends. Underneath, the model and environment alternate messages and update shared state until a stopping condition is satisfied; the completed trajectory can then be evaluated.

MultiTurnEnv supplies the core single-agent rollout loop for the environment types presented. ToolEnv adds Python functions as callable tools. A model response can request a tool, the environment executes it, and the returned result becomes input for the next model response. This continues until the model responds without tool calls. Fiorucci points to wiki-search as a more realistic example of that pattern.

Other abstractions extend the same foundation:

• MCP environments connect to Model Context Protocol servers and expose their tools.
• StatefulToolEnv keeps per-rollout resources such as a database connection or session ID.
• Recursive language model environments let a model decompose and recursively inspect input through a REPL. The strategy addresses context that need not fit into one model prompt; it does not make a single context window infinite.
• Third-party integrations connect Verifiers with other environment libraries.

Each abstraction changes what the agent can interact with while retaining the need to track an episode and judge its outcome.

The Environments Hub adds a distribution layer: a community space for sharing environments, tightly integrated with Verifiers. This matters because an environment tied to one training stack is difficult to reuse elsewhere. As a market for closed environments develops, open packages provide another route to the tasks and feedback that open models need for training. They also make other people’s experiments available to inspect and extend.

Tic-tac-toe has a small state space and a deterministic solution, yet small language models can struggle with it. Its multi-turn dynamics also make it awkward to capture fully in a static dataset. The practical starting point is a minimal environment that can run an evaluation before any training complexity is added.

The first version gives the model a narrow contract: play as X, go first, and return a cell number from 0 through 8 inside <move> tags. The opponent chooses random legal moves. load_environment creates examples containing the initial user message, and setup_state initializes per-game information such as the board and winner.

env_response implements the game’s transition:

1. Parse the model’s latest move and check whether it is legal.
2. End the game as a loss if the move is invalid.
3. Apply a valid move, generate the opponent’s random move when play can continue, and check for a win or draw.
4. If the game remains active, return the current board and ask for the next move.

The rubric combines a win reward with weight 1 and an XML-format reward with weight 0.2. This separates two requirements: learning useful game behavior and producing a response the engine can interpret.

The next version varies whether the model starts, then introduces an optimal opponent using minimax. Against optimal play, a draw is the best achievable result. But an opponent that is perfect from the beginning can deprive a weak model of wins and useful learning signals. The environment therefore lets the opponent sometimes choose a random move instead of an optimal one.

min_random_move_prob and max_random_move_prob define the range of opponent randomness across games:

Rollouts derived from the same dataset example share the same random-move probability. They therefore face the same difficulty setting even when the sampled model behavior differs.

The response contract also grows to include a thinking trace before the move. A regular-expression reward checks for <think> tags as well as the move format. Invalid outputs need a gentler treatment too: malformed responses and occupied-cell choices were common enough that immediate losses could cut off useful experience. The revised rule lets play continue with a flat −0.1 penalty and an eight-turn cap in the walkthrough.

Group-based RL needs reward differences to reflect the model’s decisions. If two rollouts start from the same example but receive different opponent luck, their scores become harder to compare. Fiorucci assigns an example seed that controls the starting player, then derives each turn’s seed from the example seed and the board state. Within that example, reaching the same board position produces the same opponent response. The model can still explore different moves, but a repeated position no longer receives an unrelated random continuation.

There is a second source of noise across batches. Batch size here counts the games used before a weight update. When opponent difficulty varies, a small batch can accidentally contain mostly easy opponents or mostly hard ones. The average reward then fluctuates with the sampled curriculum. Stratified sampling forces each batch to contain a balanced spread of difficulties across the configured range. Seeding controls comparisons within a rollout group; stratification controls the difficulty mix across an update. The repository contains the detailed sampling implementation.

Baseline evaluation compares GPT-5 Mini with Liquid AI’s LFM2-2.6B, the small open model identified in Fiorucci’s companion repository. GPT-5 Mini follows the format well and plays competently, though imperfectly. LFM2 struggles with both formatting and legal moves: it sometimes wins against a random opponent but rarely survives an optimal one. These are sampled evaluations, so some variation is expected.

LFM2 is nevertheless a promising starting point: it is small, fast, and already instruction-tuned. The proposed division of labor is to use SFT for the output contract and valid-move syntax, then RL for stronger strategic behavior. An environment that already generates and scores trajectories can also supply the synthetic demonstrations for that warm-up.

Fiorucci generates 200 examples with GPT-5 Mini and filters out losing games before SFT. The teacher’s reliable formatting makes it useful even though its game strategy is imperfect. Prime RL runs the supervised fine-tuning stage; Fiorucci reports that it takes a few minutes on his 96 GB GPU, with smaller hardware also possible. Afterward, formatting is almost perfect, invalid moves are less frequent, and game performance improves, but substantial strategic weaknesses remain.

Applied to tic-tac-toe, the group-relative training loop is straightforward:

1. Sample several complete games from the same initial board.
2. Score each rollout with deterministic win, format, and invalid-move reward functions.
3. Calculate the group’s average reward and compare each rollout with that baseline to obtain its advantage.
4. Update the model to favor trajectories that performed better than the group baseline.

The comparison is relative to other attempts at the same starting task. That is why shared opponent difficulty and controlled environment randomness matter: they make the baseline more informative about the model’s play.

The experiment uses CISPO, introduced in MiniMax-M1, which clips importance-sampling weights. Fiorucci presents it as an improvement over GRPO for this training setup. Training runs through the Verifiers RL simple trainer, with one GPU for inference and another for updates. This is the historical vf.RLTrainer workflow: the companion training chapter now labels it legacy, pins verifiers[rl]==0.1.9.post3 for reproduction, and recommends Prime RL for new projects.

The first RL run samples opponent random-move probabilities from 20% to 70%. It excludes both purely random and perfectly optimal opponents, creating opportunities to learn attack and defense. num_groups configures the stratified sampling. The remaining trainer settings must balance stable learning with GPU utilization and memory limits; the repository supplies the detailed memory guidance.

Fiorucci observed unstable training and model collapse with batch sizes below 256 games in this environment. His explanation is that too few games per update expose the model to too narrow a set of matches and opponent types, allowing a locally successful but poor strategy to dominate the update. Batch size therefore affects the quality of the learning signal as well as resource use.

The first training run looks healthy: win reward and total reward rise, formatting remains near perfect, and the invalid-move penalty converges toward zero. A separate evaluation is still needed to find out whether those trends translate into stronger play.

After the first RL run, Fiorucci reports an 85% draw rate against an optimal opponent in a 100-game evaluation, with one rollout per example and opponent random-move probability set to zero. The model also dominates random opponents, and invalid moves fall to near zero. A draw is the strongest possible outcome against optimal play, so the remaining losses identify room for improvement.

Inspecting the rollouts reveals a recurring weakness: fork traps. An opponent creates two winning paths, and the model can block only one. The final move may look sensible in isolation, but the decisive mistake occurred earlier, when the model allowed the fork to form. The three-board sequence makes that delayed consequence visible. Improving further requires changing the strategies explored during training, rather than merely polishing the last response.

For the next run, Fiorucci uses larger GPUs to iterate faster, while noting that they are probably unnecessary. He raises opponent strength by narrowing the random-move probability to 0%–25%. Training exclusively against perfect opponents had produced an unwanted result: an overly defensive model that failed to exploit mistakes when tested against random players. Keeping some imperfect play in the curriculum preserves opportunities to learn how to win.

A stronger opponent alone does not guarantee escape from an established strategy. After several unsuccessful attempts to improve, Fiorucci adjusts sampling temperature to encourage exploration. Higher temperature increases behavioral variety, but too much can cause incoherent output. In this run, win reward and total reward initially fall. Fiorucci interprets that dip as exploration of new, initially worse strategies; the rewards later recover and reach new highs. Formatting and invalid-move rewards also dip briefly but remain near their best values overall.

After evaluation, Fiorucci calls the result a tic-tac-toe master and tries a game against it. He deliberately makes an imperfect move, then has to block the model repeatedly and eventually loses. The interaction tests a practical question that reward plots cannot answer by themselves: will the model exploit a human opponent’s mistake?

The final comparison returns to GPT-5 Mini, the teacher that supplied the SFT examples. Fiorucci reports similar performance against random opponents and better performance from the trained LFM2 against an optimal opponent. The spoken comparison gives no exact final percentages. Its significance is task-specific: the demonstrations established useful behavior, and subsequent environment-driven training improved the student beyond its teacher on this game.

The successful runs followed several failures. Larger batches appeared to learn slowly but offered more stable updates. Small batches drawn from diverse games and opponent strengths could reinforce suboptimal strategies and lead to collapse. The relevant unit is the collection of experiences behind each update, not simply how quickly the next point appears on a chart.

A more subtle failure came from an earlier minimax implementation that Fiorucci had delegated to Claude. When several moves had the same optimal score, it always chose the first free position among them. Evaluation scores looked excellent, yet the trained model was ineffective when Fiorucci played against it. The opponent was optimal in score but narrow in behavior, and the model had memorized that particular player. An optimal opponent can still contain a training bias.

Starting-model choice introduces another tradeoff. A model already trained for reasoning may generate long thinking traces. With a tight completion limit and limited GPU resources, many early completions can be truncated, wasting budget and potentially damaging useful behavior during training. An instruct model can instead acquire reasoning tailored to the task. Very small models may still lack enough capacity, so evaluate the base model and inspect several completions for promising behavior before committing to a run.

Continue inspecting rollouts as training progresses, and try the actual task after programmatic evaluation. Aggregate rewards reveal trends; individual trajectories reveal which strategies produced them. At the beginning of a run, watch for out-of-memory failures and instability. Once the run is healthy, give it time. Constantly watching slow progress can invite premature stopping or unnecessary parameter changes. As Fiorucci puts it: “start training and go for a walk.”

SFT and reinforcement learning play complementary roles in this experiment. Demonstrations teach an initial response contract; interaction supplies a space to practice, and rewards guide which behaviors to retain. Fiorucci’s broader proposal is to use a clear reward signal to train a small specialist that can outperform a larger closed model on a particular task. He also suggests a cost advantage, though this experiment supplies no comparable total-cost measurement.

The LLM RL Environments Lil Course contains the environment and training walkthroughs, while the Environments Hub offers other tasks to explore. A concrete next project is to train a small model on two or three tools you use frequently and test whether it can outperform a larger model on that specific workflow. Fiorucci recommends the wiki-search example in the Prime RL repository as a starting point. The next environment can be a small, useful task whose actions and outcomes are clear enough to check.