Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci

Hello everybody, and welcome to Let LLMs Wander: Engineering Reinforcement Learning Environments. A few words about me. I am Stefano Fiorucci, AI and software engineer. By day, I work on AI orchestration at Deepset, where I develop Haystack, an open source LLM framework.

By night, I love tinkering with small language models, fine-tuning, and reinforcement learning.

Today, I'm going to talk about reinforcement learning environments for language models evaluation and training. This has been a hot topic over the past year, and I find it fascinating for several reasons.

These environments let models learn by interacting, exploring, and improving from feedback. They are natural gyms for LLM agents that can use tools, run code, and solve multi-step tasks. In addition, startups building RL environments are getting major funding and working directly with big AI labs.

Recent technical reports by DeepSeek and MiniMax show that they are effectively using thousands of reinforcement learning environments to improve model performance on challenging tasks and scale intelligence.

Don't worry if you know nothing about RL environments. I'll cover that soon.

Here is the agenda for the talk. We'll first review classic reinforcement learning concept and see how they map to the language models domain. We'll then introduce Verifiers, an open source library to build environments as software artifacts, and explore some common patterns to implement them.

Finally, I'll walk you through an experiment where we take a small model that can barely play tic-tac-toe against a random player and transform it into a master using a reinforcement learning environment.

Let's start. First, a quick refresher on reinforcement learning.

In reinforcement learning, there are two main characters, the agent and the environment. The environment is the world the agent interacts with. At each step, the agent sees the current state of the world and takes an action.

The state of the environment then changes in response to that action. The agent also receives a reward from the environment, number indicating how good or bad the state is.

The agent's goal is to maximize its cumulative reward over time, and to do this, it has to balance exploration, so trying new actions to discover better strategies, and exploitation, using actions known to work.

By interacting with the environment, the agent learns from experience and improves its behavior.

A trajectory or rollout is the sequence of states, actions, and rewards that the agent goes through while interacting with the environment. It's a record of the experience. In this presentation, I'll use trajectory to mean a complete episode, like one entire game.

Let's take a look at LLMs training. Language model is a statistical model that given some text, the prompt returns a text completion.

The standard training recipe is divided into three phases.

First, pre-training on a massive amount of internet text. Here, the model learns to create text completions. The base model is knowledgeable, but can't follow instructions and is hardly usable in applications.

During supervised fine-tuning on conversational examples, the model is trained to follow instruction and can learn new tasks.

In the third step, reinforcement learning is often used with techniques like proximal policy optimization to align the model with human preferences.

It's worth showing an example of supervised fine-tuning data, as later we'll frequently compare supervised fine-tuning learning with new reinforcement learning approaches. As you can see, we have pairs of prompt and responses, and during this phase, the model learns by statistical imitation.

It's essentially trying to mimic the examples provided.

You might remember Ilyas Tatkiver talk at NeurIPS twenty twenty-four. He pointed out that the LLM training paradigm we just saw is starting showing its limits. In particular, pre-training no longer seems to be enough to keep improving model quality at the same rate.

We needed a new way to scale. Then OpenAI published its o1 models series.

In their release blog post, they mentioned the reinforcement learning training to make models use chain-of-thought effectively. They also underlined that the performance of o1 consistently improved with more reinforcement learning, train-time compute, and with more time spent thinking, test-time compute.

Unfortunately, they did not share many details on how this model was actually trained.

The release of DeepSeek-R1 shed some light on how you can possibly achieve- Those results. First, they recognized that reasoning and chain-of-thought can improve the performance of models. But teaching this behavior to models using supervised fine-tuning requires curated data that is too expensive to produce at scale.

They used reinforcement learning with verifiable rewards, which we'll see in a moment.

And DeepSeek also used GRPO, a new reinforcement learning algorithm that offers a simple, lighter setup compared to techniques like PPO.

So what is reinforcement learning with verifiable rewards?

In this paradigm, the model is asked the question and generates both a reasoning trace and an answer. The answer is then checked against a known correct answer. The reward is used for reinforcement learning training.

The underlying idea is more general. Any task where the outcome can be verified automatically, like a correct answer, a won game, a successful tool call, can serve as a training signal.

And this is fundamentally different from supervised fine-tuning. In SFT, the model learns from curated examples, and its completions tend to stay close to the distribution of those examples. In reinforcement learning with verifiable rewards, the model explores different trajectories from its pretraining and learns to favor the ones that

maximize rewards. And this is exciting because the model is no longer limited by the quality of human examples. Through trial and error, it can discover more efficient reasoning strategies.

We can finally map the classic reinforcement learning concepts to LLMs. The language model acts as the agent.

The environment for any task included data, harnesses, and scoring rules, everything needed to check and possibly train the model on the task.

From a software perspective, this marks a shift from supervised fine-tuning to reinforcement learning with verifiable rewards. While SFT mainly relies on conversational datasets, this new paradigm usually requires an environment, a dynamic system that the model can interact with.

The definition of an agent is also expanding. Language models can now be given tools from a weather API to a terminal, and this makes environments for training and evaluation more complex and critical.

To make this more concrete, consider teaching a model to play tic-tac-toe. The agent is the language model. Its action is generating a text response with a specific move.

The environment acts as the game engine. It handles prompting the model, tracking the board status, generating the opponent's moves, and deciding when the game is over. The reward is the signal from the environment.

For example, plus one for a win and zero for a loss. This reward guides the model to find winning strategies through trial and error.

This setup allows the agent to discover strategies that maximize its score without needing preexisting human examples.

You have probably understood that I am enthusiastic about this topic, but let's also use Andrej Karpathy's words to describe the environments. "They give the LLM an opportunity to actually interact, take actions, see outcomes.

This means you can hope to do a lot better than statistical expert imitation."

Now, let's see how to build these environments.

To build the environments as software artifacts, we can use Verifiers, an open source library by Prime Intellect.

Verifiers provides modular components to create reinforcement learning environments for LLM agents. This can be used for both evaluation and training.

Environments are Python packages that can be easily installed and distributed. The library provides base classes for several setups: single-turn environments with just one interaction between the model and the end, multi-turn environments, tool environments where the model is equipped with tools, and several others.

It also includes abstractions for parsing model responses and defining reward functions.

Verifier abstracts model serving. It accepts an OpenAI-compatible API endpoint, so you can plug in OpenAI, open router, or local models via vLLM. It handles async interaction and parallel trajectories, so you focus on the environment logic.

For training, Verifiers come with its own trainer. It integrates with other frameworks such as Prime RL, Tinker, and Sky RL. In short, Verifiers lets us focus on the task and the rewards rather than the infrastructure.

Let's start with a single-turn environment. Reverse Text is a simple environment to evaluate or train language models on their ability to reverse a string of text.

What's going on here? load_environment is the entry point for every Verifiers environment. It contains all the setup logic. First, a dataset is loaded and mapped. The default dataset contains a thousand text paragraphs stored in the prompt column.

During the mapping step, we transform the original dataset. Question is the text paragraph, while answer is the reversing text.

Next, XML parser is initialized. It extracts the text inside the reversing text, text as specified in the system prompt. We then define the reward function. This compares the model's output to the ground truth and returns a longest common subsequence ratio.

Finally, we bundle this into a rubric, a collection of weighted rewards, and initialize the single term env.

But how does this come to life? Let me show you an evaluation run.

Here is what happens under the hood. load_environment is invoked. Five examples are taken from the dataset, and each example is used three times for three rollouts. Each rollout gets the same question, but may produce different completions due to model randomness.

We have fifteen rollouts in total. For each rollout, a conversation is prepared with the system prompt and the question. The conversation is sent to the model. The model generates a response.

The response is parsed, extracting the answer. The reward is computed, and results are saved. At the end of the evaluation, you get summary statistics and additional info about reward distribution.

Training follows the same core mechanism with the additional step of updating model parameters. We'll look at that in more detail later.

Let's look at the different example from Verifiers, the double-check environment. Here, the model answers a math question, and the environment then asks, "Are you sure?" This is a multi-term environment, similar in spirit to tic-tac-toe, in which each trajectory involves multiple interaction between the model and the environment.

Let's look at what multi-turn env introduces. State makes its first appearance here. It's a dictionary that tracks information during a rollout. We can set the initial state through the setup state method not used in this example.

And response. Instead of the interaction ending after one turn, the environment can reply with a list of messages. Here, it just says, "Are you sure?" In more complex environments, the response can be dynamically generated based on the state.

The the_if_stop decorator marks a method as a stopping condition. This method runs at every turn of the agent environment interaction. Once it returns true, the rollout terminates. Under the hood, Verifiers runs a loop in which the model and the environment take turns exchanging messages, updating shared state until a

stopping condition is met, and the full trajectory can be evaluated.

Another interesting type of environment is the tool environment. All environment types in Verifiers are built on multi-turn env, which implements the core single agent rollout loop. Tool env adds tool calling to this foundation.

As you can see, tools are defined as Python function. During rollouts, the model can call tools, receive results, and continue reasoning until it produces a response without tool calls.

Each turn consists of a model response followed by the environment's tool execution.

For a more realistic example, I recommend checking out the wiki-search environment.

Beyond the fundamental environments I just showed, Verifiers provides more abstraction to build environments. The NCP environment automatically connects to model context protocol servers to expose their tools.

Stateful tool env is for tools that need per rollout persisted state, like database connection or session ID.

There is also a class implementing recursive language models, a novel idea you might have heard of. It's an inference strategy where language models can decompose and recursively interact with input context or unbounded length through REPL environments.

Verifiers also plays well with others, integrating with several third-party environment libraries.

Verifiers is tightly integrated with the Environments Hub, a community space for sharing these RL environments. Verifiers and the Environments Hub are different phases of the same intent. They aim to fight environment fragmentation.

Too often, environments are locked into specific training stack, making them difficult to reuse. And as a market for closed source environments emerges, these open initiatives ensure we have a robust alternative.

We don't want open source models to lag behind just because they lack the right playground training. Plus, beyond the serious side, it's just fun to explore the hub and see what people are building.

Now, let's move on. Tic-tac-toe. We'll use verifiers to create a tic-tac-toe environment for training and evaluating language models on this game. Now, why tic-tac-toe?

It's a simple game, but requires multi-turn interaction, and capturing its dynamics with a static dataset is challenging. Despite its small state space and deterministic solution, small language models often struggle with it.

Let's see if reinforcement learning can help bridge that gap.

It's best to start with a simple version, run evaluation to verify it works, and then iterate. To start, we make a few assumptions. The model always plays as X and goes first.

It must output a number between zero and eight inside move tags, and the opponent just plays randomly.

In load environment, we create the dataset containing the initial user message that starts each game.

For each rollout, setup state populates the state dictionary with information used and updating during the game, such as the board and the winner.

Env response contains the core game logic. It parses the model last move,

check if it's valid. Invalid moves are an immediate loss for now, and applies it to the board. Then it applies a random opponent move and checks for a win or draw.

If the game isn't finished, it returns a user message with the current board state and asks for the next move. We use two reward function: win reward function with weight one and format reward function with weight zero point two, which rewards the model for respecting the XML format.

We can now make our environment more flexible, realistic, and suitable for both evaluation and training. I made some of these improvements gradually while it evolves during training. First, we want the model to sometimes play first and sometime play second.

Let's now address opponent skill. Always playing against a random opponent isn't realistic. So we introduce an optimal opponent using the minimax algorithm. Against this opponent, a draw is the best achievable outcome.

However, for training, we want the opponent's skill to be controllable. If the opponent is too perfect too early, the model might never see a win and fail to learn.

We can do so by introducing a probability for the opponent to choose a random move instead of the optimal one. In load environment, we introduce min random move prob and max random move prob varying from zero to one.

If we set both to zero, all games will be against an optimal opponent. If we set both to one, all games will be against a random opponent. Using these parameters allows us to control the opponent's skill across all games.

For different rollouts originating from the same dataset example, the opponent will always have the same probability of choosing random moves, ensuring fair comparison.

Now, about reasoning. It's common to ask models to produce a thinking trace before the final answer. It can improve performance at inference time, but it's also instrumental to make models better during training.

We define a new format reward function using a regular expression to also check the presence of think tags.

Let's now cover invalid moves. When experimenting with small open models, I observed many of them. Sometimes the output format was incorrect. Other times, the chosen cell was occupied. Ending the game immediately is harsh.

It might stop smaller models from getting a useful learning signal. Instead, we now let the game continue and apply a flat minus zero point one penalty, capping the turns at eight.

Let's discuss reducing noise in group-based reinforcement learning. In GRPO learning, we compare several rollouts from the same starting point to see which ones to reinforce based on rewards. For this to work, differences in rewards should come from how the model plays, not from environment randomness.

And how can we reduce noise in this setup?

First, we set an example seed for each example in the dataset. To select the starting player. Then for each turn, we derive a specific turn seed based on the example seed and board state.

This guarantees that if two rollouts reach the same board position, the opponent will always respond the same way.

Last point, reducing noise across batches. For our training, the batch size is the number of games taken into consideration before the models were, are updated.

In our setup, the opponent's skill varies across the dataset according to mean random move prob and max random move prob. If we train with a small batch size and the random move probability is not fixed, we might sample a batch in which many opponents are hard or many are easy.

This causes the average reward to fluctuate a lot, making training unstable. To fight this, I added the stratified sampling. This forces every batch to contain a perfectly balanced mix of opponent difficult spanning the chosen range.

I know this slide is dense, but you can find our code and more details in the GitHub repository.

Time to evaluate existing models. We choose GPT-5 Mini and LFM 2 by Liquid AI, a small fast open model. Using verifiers evaluating models just requires a few commands. Allowing for some statistical variability, GPT-5 Mini is excellent at following format and is a good tic-tac-toe player, but not perfect.

The small open model by Liquid AI struggles to follow format and to make valid moves. It's a weak tic-tac-toe player, sometimes winning against a random opponent, but rarely surviving against an optimal one.

There is a significant gap. We decide to train LFM 2 for some reasons.

It's a good model for its size, and it's an instruct model, ideal for transforming it into a reasoning model. How can we improve it? We saw that this model struggles to follow format and often provides invalid moves.

We can use supervised fine-tuning for a warm-up phase where we teach the model the format and valid move syntax. We can then use reinforcement learning to build deeper capabilities.

The first step is generating synthetic data for supervised fine-tuning.

Once you have the good environment, generating data requires a single command.

Here we use GPT-5 Mini since it followed format perfectly, and we don't need many examples. We generate two hundred and filter out losing games to avoid backing in suboptimal strategies.

With the synthetic data at hand, we can clearly spin up a supervised fine-tuning run using Prime RL. In this example, I am using a ninety-six gigabytes GPU, but you can use a smaller one.

Training requires only a few minutes. Time to evaluate our fine-tuned model. Compared to the original model, it learned format almost perfectly and reduced the number of invalid moves.

It also improved the game performance, but there is still significant work to do.

Before jumping into RL training, let's do a quick recap of group relative policy optimization applied to tic-tac-toe.

Rollouts. Starting from the same initial board, the model plays several games via LLM sampling.

Each rollout is evaluated using deterministic reward function, in our case, win format and invalid moves rewards.

An average score is calculated across the group of rollouts, and each rollout is then compared against this average advantage computation.

The model is updated to favor trajectories that did better than the group baseline.

We'll use CISPO, which is an improvement over GRPO.

For reinforcement learning training, I used Verifiers RL simple trainer. Here we use a GPU for inference and a GPU for training. Let's comment some parameters in the training configuration.

In this training run, random move probability is ranging from twenty to seventy percent. No purely random players and no optimal players. It's a good playground to get signal and learn both attack and defense.

The num groups parameter is used to set up stratified sampling. When it comes to trainer arguments, we want our model to learn stably while fully utilizing our GPUs without crashing.

For tips on how to use the GPU without going out of memory, I recommend checking out the GitHub repo. Here, I want to stress that reinforcement learning training is sensitive to either parameters and can be unstable I learned the hard way that batch size is a key parameter.

In this environment, I observed unstable training and model collapse and experimented with values lower than two hundred fifty-six. The explanation is intuitive. Batch size is the number of games used to update the model's weights.

If this number is low, this means learning to play from a very small number of matches and opponent types at once, and this likely leads to suboptimal strategies.

Let's take a look at training plots. Win reward function and the total reward constantly improved.

Format reward function was already near perfect and did not change significantly. Invalid move penalty function started well and converged to zero towards the end of training. It seems a good training run, but let's run proper evaluation.

Impressive. Thanks to reinforcement learning, our model has become a very competent tic-tac-toe player. It dominates random players and draws eighty-five percent of the time against an optimal opponent. Invalid moves have dropped to near zero.

These results are already satisfying, and one can say there's not much more we can learn from this example.

I am a perfectionist, and I'd like a perfect AI player. Inspecting the rollouts, I found some recurrent failure modes. In particular, our model sometimes falls into four traps. This example shows the end of the game.

Our model is not playing badly here, but it had already lost the game by allowing the opponent to have two winning paths.

Is it possible to use our RL environment to push our model further toward perfection?

Let's try. We have to make some changes. In this run, I used bigger GPUs just to experiment quickly. It's probably not required.

First, let's discuss opponent skill. We increase opponent skill by setting the probability of a random move to range from zero percent to twenty-five percent. I also tried making the model play ag- against perfect opponents only, but it didn't work.

The model became overly defensive and failed to exploit errors when testing against random players.

But how to make our model explore beyond learned strategies? I made several experiment where this model failed to improve and forget the suboptimal strategies. We want the model to experiment with new approaches, and temperature is the right parameter to tweak.

But this is a bit risky. If the temperature is too high, the model can start generating gibberish. Let's train.

Things get really interesting. There was a significant initial drop in win reward function and total reward. I interpret this as an exploratory phase where the model tried new and random strategies, which hurt performance at first.

But over time, it recovered and improved to new highs. Also, format reward function and invalid move penalty function had an initial drop, but overall always stayed around their maximum values.

Let's move to proper evaluation. Oh, we finally got tic-tac-toe master, but why not playing a game against it?

Okay, let's make, uh, not a perfect move.

Hmm, we now need to block the model.

Hmm, let's block it again. Oh, no, we lost.

It could be interesting now to compare our model performance with GPT-5-mini, the teacher model we used to generate our synthetic data.

Hmm, against a random opponent, performance is very similar.

Let's see against an optimal opponent. Oh, in this case, our model is superior. This is a very nice achievement.

To get these results, I went through several failed experiments, and I'd like to share the findings with you. First of all, batch size. If this value is large, yes, your model apparently learns slowly, but in exchange you get stable training.

If the batch size is small and your environment produces diverse matches and opponent skill, the model will learn from a small number of games at once. This can reinforce suboptimal strategies, and you may observe unstable training or model collapse.

Second lesson, watch for hidden biases in environments. Let me explain. In a previous experiment, I used a different minimax algorithm for the optimal opponent. I thought this was an implementation detail and let Claude handle it.

I got great benchmark results, but then playing against the model, I realized it was too ruthless. Looking better at minimax, these are the bias. If multiple moves have the same optimal scores, the first free position was always selected.

I basically was training my model against a specific type of optimal player. Over many games, the model simply memorized it.

Now about model choice. You can start from a model which is already trained for reasoning, but they tend to output long thinking traces. If you have limited GPU resources and time, you may end up truncating most of the longer completion at the beginning to fit the short limits.

This means wasted budget and also the risk of damaging the model's intelligence. It might make more sense to start from an instruct model and transform it to-- into a reasoning model for your task.

Another point about models, it could be hard to push very small models to competency. This of course depends on the task. What I recommend, evaluate the base model in your environment, look at a few completions, choose a model that shows promising behaviors, even if the numbers are not satisfying yet.

In general, it is always a good idea to inspect some rollouts to see the model evolve. Also, after training, do not stop at programmatic evaluation. Try the model in the real task.

The final recommendation, it's natural to watch your logs and plots when you start training to identify early out-of-memory errors or instability. It's difficult for me too, but once training begins well, I suggest stopping staring at plots for a while.

Reinforcement learning is slow and takes time to see progress. If you continually monitor it, you risk the temptation to stop it and tweak something prematurely, while a slowly progressing run can be surprisingly well given enough time.

So start training and go for a walk.

During this presentation, we mapped the reinforcement learning concept to the language models domain. Then I introduced Verifiers, an open source library to build environments as software artifacts. Finally, I walked you through my experiments where I took a small model and turned it into a tic-tac-toe master using supervised fine-tuning and reinforcement learning with verifiable

rewards. We did not just show the model how to play. We gave it a space to play and guided it through rewards.

Nowadays, reinforcement learning complements supervised fine-tuning in language models post-training. You can do this at home too. If you can define a clear reward signal, you can build an environment and train a small specialized model to beat a large closed model on a specific task at a fraction of the cost.

I want to leave you with a few ideas and resources on this topic. All what I shared today can be found in my free LLM RL environment link course, where I go deep at explaining the detail.

Take a look and give a star. To figure out what others are building, explore the Environments Hub. And what to build next? Something I'm very excited about is this.

Train a small language models on two, three tools you often use and try to outperform a large model in that specific task. I recommend the wiki-search example in the prime-rl repository as a starting point.

Thank you.