AI Engineer Code 2025
RL Environments at Scale
Read the talk
RL Environments at Scale
A reusable environment connects evaluation, synthetic data, and model training. Will Brown’s WikiSearch example shows how that abstraction can make research part of everyday AI engineering.
From a talk by Will Brown
Before you start: Basic familiarity with language models, evaluation, and Python will help; no experience implementing reinforcement learning is assumed.
How do you scale the ability to do research?
Running thousands of parallel rollouts and sandboxes on hundreds of GPUs is one kind of scaling problem. Another is increasing data, compute, parameters, or inference time to improve model performance. But neither fully explains where better algorithms, useful tricks, and new research ideas come from. That raises a different question: how do you scale the community capable of producing those improvements?
Ideas develop through a cycle of applications and experiments. Someone builds an application, encounters a problem, and develops an idea; the application then becomes a testbed for trying it. Sharing the result gives the next person a starting point. Reusable tools and accumulated knowledge reduce the work each researcher must repeat, making more ambitious experiments accessible.
This is the talent bottleneck that Will Brown wants to address. Competing to hire existing researchers is one response. Expanding the pool of people who can do research is another—and it depends on making the practice easier to enter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An open stack for research as a practice
Prime Intellect combines a research lab, a compute provider, a platform business, and an open-source ecosystem. These roles support a common goal: give engineers a research toolkit they can use while improving applications, systems, and products, without first joining a large lab, acquiring a massive cluster, or completing a PhD. Research becomes part of the engineering workflow rather than a separate institutional activity.
The useful analogy with open-source software is broader than releasing model checkpoints. Linux, Node, and Apache illustrate how ecosystems accumulate abstractions, tools, and working practices. Brown applies that analogy to research as a practice: better tooling makes iteration faster, reusable foundations lower barriers, and those improvements allow people to build increasingly complex systems.
Prime Intellect calls the supporting infrastructure the open superintelligence stack. Its layers include compute, orchestration, training and evaluation libraries, and services for code execution, evaluation, inference, and fine-tuning. Together, these provide the machinery for running research, not just accessing a finished model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train the model inside the product
Giving product builders the ability to train models creates another way to improve the user experience. Customization might affect one component, one model, or a larger part of the product. It need not be the first step, but having the option expands what engineers can change when improvements around an existing API are insufficient.
Cursor’s Composer and OpenAI’s Codex are Brown’s examples of products whose experience depends directly on a model trained for that product. In his framing, the product increasingly becomes the model: using the model and using the product become closely connected experiences. The mechanism is to take a harness representing the product and train the model inside it. That harness becomes part of an RL environment.
An environment is a harness together with tasks and rewards. The abstraction applies beyond reinforcement learning:
| Use | Role of the environment |
|---|---|
| Evaluation | Run tasks through the harness and measure the results. |
| Synthetic data | Generate examples that can feed supervised fine-tuning, or SFT, and distillation. |
| Reinforcement learning | Train the model using rewards from its attempts. |
| Deployed agents | Handle incoming user tasks and monitor outcomes. |
The tasks may come from an offline dataset or a live stream of product requests. Keeping the harness, tasks, and rewards together gives these activities a common experimental foundation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small project that can grow into research
Brown calls environments the “web apps of AI research.” They can begin as simple, self-contained projects and grow to represent a full product. As complexity increases, the builder encounters new questions about systems scaling, hyperparameters, and algorithms. Those questions arrive through a concrete task, without requiring the builder to implement an entire training infrastructure first.
The distinction between an agent harness and an environment is the requirement to define tasks and rewards. A harness alone lets you try an agent and decide whether its behavior seems promising. An environment also requires you to specify what it should do and how its attempts will be assessed. That supports experiments comparing models and hyperparameters, and provides a path toward RL, distillation, or fine-tuning when those become useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Share tasks, then build on common components
The Environments Hub provides a place to create, discover, and share RL environments and evaluations. Brown reports hundreds of builders and environments. Contributions include reimplementations of papers, benchmarks adapted with new data or examples for RL, games, and original tasks. Packaging the task creates a starting point for experimentation even when the builder does not yet know how to improve the model.
This changes the data problem. Brown attributes some of the difficulty of adopting SFT to the work of collecting examples of the desired solutions. An environment can instead define situations the model will encounter and a way to measure its answers. If you can assess generated answers, you need not supply every solution upfront. The environment can become an engine for producing data as it runs. Brown connects this approach to Verifiers, the library he had released nine months before the talk.
Verifiers supplies components that can be composed into evaluations, question-answering tasks, games, math problems, tool use, sandbox execution, and agent frameworks, including CLI coding agents. The resulting environments are designed to support RL training. Covering that range requires extensibility: the toolkit cannot anticipate every application, and different applications introduce different patterns and special cases.
The hierarchy separates application-specific behavior from shared foundations. Brown’s coding example runs through Kleinbench → Harbor → CLI agent → multi-turn environment → environment. Other branches include TextArena and Wordle, search through MCP, and a Python REPL inside a sandbox. Each branch helps identify what belongs in the base environment and what should be added by a more specialized layer.
Editor’s note: the benchmark name is likely cline-bench, whose announcement connects it to Harbor and Prime Intellect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
WikiSearch: from a Python package to training
WikiSearch makes the abstraction concrete. An agent receives tools for searching Wikipedia pages and finding answers. Its Hub entry is a Python project with dependencies and versions, alongside facilities for uploading evaluation results. The Hub therefore serves as both code management and a package registry for environments.
The environment’s tools are asynchronous Python functions. A dataset supplies tasks, and a rubric manages the components of the reward. The rubric can also record metrics that contribute zero reward, preserving visibility into behavior without making every observation an optimization target.
For a search rollout, the following Python function expresses that separation. It takes an answer score supplied by the task’s evaluator and records the number of tool calls without rewarding that count:
python
def score_search_rollout(
answer_score: float,
tool_calls: int,
) -> tuple[float, dict[str, float]]:
metrics = {
"answer_score": answer_score,
"tool_calls": float(tool_calls),
}
weights = {
"answer_score": 1.0,
"tool_calls": 0.0,
}
reward = sum(
weights[name] * value
for name, value in metrics.items()
)
return reward, metrics
Here, changing tool_calls changes an observable metric but leaves the reward unchanged. The environment still has to define how it obtains answer_score.
Training adds a configuration for Prime RL, Prime Intellect’s stack for large-scale asynchronous RL. Its configuration exposes choices that bring the builder closer to the training algorithm while keeping the setup relatively self-contained. Brown describes the defaults as intended to be sensible starting points for many users. The environment definition and training configuration remain separate pieces of the experiment.
In the workflow Brown demonstrates, the final step is a command that identifies the environment. If it is available on the Hub, the trainer automatically installs it and starts the run. The progression is therefore straightforward:
- Package the tools, task dataset, and reward rubric as an environment.
- Select the training configuration.
- Launch the run with the environment identified in the command.
The automatic installation is part of the workflow described in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The reward curve starts another round of experimentation
A reward curve that rises quickly is a successful example, not a guaranteed outcome. The work continues through iteration on the environment, rewards, data, tasks, and training parameters. Looking at the data and reconsidering what the reward measures are part of making the task work in practice. Brown emphasizes that this process can improve both small and larger models.
For WikiSearch, Brown reports the following change:
| Model and stage | Reported WikiSearch score |
|---|---|
| Qwen3-4B, before training | About 55% |
| Qwen3-4B, after training | 89% |
Brown describes the trained result as on par with GPT-4.1 and GPT-5 Mini in this WikiSearch setting. These are reported task-specific results; the talk does not specify the evaluation split, judge, or matched inference budgets for the comparison.
Customization can serve different product goals: a faster model, a cheaper model, or better performance on a task where the strongest available models remain insufficient. A small model improved for one application is especially useful when speed or cost matters. Building an environment preserves the option to pursue those improvements without requiring customization from the outset.
If a product already needs evaluations, expressing them as environments also supports prompt tuning, model selection, and investigating how the system behaves with many users in parallel. The design work forces a useful set of decisions: what is the agent, what is the product, what does the harness permit, and what outcome should be optimized?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stress-testing the stack with INTELLECT-3
The larger validation effort was INTELLECT-3, still forthcoming at the time of the talk. Brown describes INTELLECT-3 as a 100B+ parameter model trained on about 500 GPUs, with an end-to-end post-training process covering SFT and RL. This work exercised the full Prime RL stack at a much larger scale; Prime RL also supports SFT for users who need it.
The training effort also provides a way to test techniques from research papers, determine which work in practice, and incorporate useful implementations into the library. Keeping Prime RL and Verifiers open lets downstream users consume those practices without reproducing all the implementation work themselves. The intended result is a shorter path from an idea in the literature to an experiment that improves a model or product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Community feedback and a managed research workflow
A broader community exposes problems that a single research team may miss. Prime Intellect collects feedback about what builders want, what works, and what remains painful. Its programs include sponsoring small tasks, a research residency involving graduate students around the world, and manually reviewing a subset of Hub environments in the repository Brown calls Prime Environments. Brown reports hundreds of submissions to this review process. Those implementations reveal rough edges and force improvements in the tools.
The lessons also feed into Lab, the platform Brown describes as forthcoming. Its proposed workflow lets users browse environments and run evaluations, inference, and fine-tuning through one interface. The operational burden includes getting Torch, Flash Attention, and vLLM versions to work together. Prime Intellect offers to manage that infrastructure: users can inspect the open code without having to operate every component themselves.
The environment remains the entry point. Evaluation begins with an environment; synthetic data for SFT begins with an environment; RL begins with an environment. A managed platform handles execution around that shared definition, allowing the builder to concentrate on the task, the agent’s available behavior, and the measurement of success.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Different deployment goals, deeper understanding
Environment building can support several destinations as demand for model customization grows:
- Fine-tuning through model labs. Brown anticipates labs offering services that let customers adapt their models to specific needs.
- Specialized on-premises models. A team may care most about the smallest model it can run locally at the lowest latency, optimized for one task.
- Research for its own sake. Experiments can advance shared understanding without an immediate product objective.
These goals differ, but each benefits from being able to define a setting, try changes, and examine what happens.
That ability also changes the relationship engineers have with models. Treating a model as a black box limits how much its behavior can teach you. Inspecting it, modifying it, and sometimes breaking it provides a more direct understanding of how it works and how its capabilities developed. Shared experimentation makes it easier to discuss what is being built—and to prepare for where the models may go next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Library for building LLM training environments and evaluations, integrated with Prime Intellect's training tools.
Training framework with SFT, asynchronous RL, evaluation and examples ranging from small tasks to large models.
Community-contributed RL environments and evaluations with tooling for creating and publishing environment packages.
Further reading
- Introducing cline-benchArticle
An initiative to turn real open-source engineering tasks into reproducible coding evaluations and RL environments.
Updates since the talk
Current walkthrough for training a Qwen3 model to answer trivia using Wikipedia search and reading tools.
Release announcement describing INTELLECT-3's model, post-training recipe, environments and GPU infrastructure.
Current Lab guide covering baseline evaluation, training configuration, monitoring and evaluation of trained adapters.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] Today, we're talking about RL environments and how to scale them,
- 0:25
but the title's a little bit of a red herring. We'll talk a bit about the engineering pieces and, like, running these with thousands of parallel rollouts and sandboxes on hundreds of GPUs, but I'm mostly gonna focus on a different notion of scale.
- 0:37
Uh, and m- but what I mean by scaling here is we-- there's a, a number of different ways we talk about scaling in the context of AI and research.
- 0:46
We know about scaling laws, and we talk about how much data you need, compute, and parameters, and that if you pour in more data and compute and parameters or inference time, all of these things make models smarter or more performant.
- 0:58
But there's also a fuzzier side of scaling, which is sometimes referred to as unhobbling or algorithmic tricks or talent. But where does this come from? It's not just pouring in resources, but it's something that is more intangible, harder to put a finger on.
- 1:13
But really it comes from a community of people, a company, an organization, universities, the world, the internet, talking about ideas and sharing them and working on different applications, having these applications inspire ideas, using these ideas as testbeds for different techniques, and building on top of these to increase the accessibility for other people in the future to not
- 1:35
have to reinvent the wheel and to be able to build from, uh, what has been done by those before them to, uh, do more effective research and accelerate the pace of innovation.
- 1:46
And so why do we have this talent bottleneck? There's a big issue that we hear all about with AI labs trying to, like, find more talent, and salaries are going through the roof, and everyone wants to hire the best and brightest AI researchers.
- 1:58
But one other approach besides trying to just pay the most is increase the pool. Uh, and so how do we increase the pool of AI researchers? How do we make doing AI research more accessible?
- 2:09
And I wanna talk a bit about who we are at Prime Intellect. If you haven't heard of us, we are a mu- a bunch of things. We're a research lab, we are a compute provider, we're a platform company, and we are an open-source ecosystem.
- 2:19
We do a lot of things, and they all fit together in a way that I'm gonna try to explain in this talk. But we see these as all different pieces of how we can build a business around doing exactly this, which is increasing the accessibility of AI research and making doing research more of a toolkit available to
- 2:35
people at organizations around the world without needing to be inside of a large lab or without needing to spend crazy amounts on massive clusters or go do a PhD.
- 2:43
We think that there's versions of doing AI research that really should be part of the bread-and-butter workflows of AI engineers around the world as we build applications and try to improve our systems and models and products.
- 2:57
And I think, I think people are kind of iffy about in terms of AI is whether open-source models are gonna work. And in my mind, that's not quite the right analogy to draw.
- 3:06
And so when we're comparing, like, AI to traditional software, there's lots of, like, great examples of open-source software ecosystems that have thri- been thriving in the past, things like Linux and Node and Apache.
- 3:17
But in my mind, the analogy in AI is not models as kind of these fixed checkpoints, but it's about research as a practice and research as a set of ideas, and it's one that's more intangible.
- 3:28
But there's a lot of parallels in terms of the goals of the best practices of growing a research ecosystem as well as a software ecosystem, where you want to, uh, compound abstractions and best practices and have better tooling and iteration efficiency and have these gains over time allow, uh, more advanced, powerful, complex things to be built by,
- 3:48
uh, decreasing barriers to entry for any given application and allowing this to become more accessible. And so one thing that we-- a term we'll use to describe some of what we're building at Prime Intellect is we like this phrase called the open superintelligence stack.
- 4:01
One, 'cause it's a fun acronym, but also I think the idea of the stack of, of all the pieces of the puzzle to build the engine to go do research.
- 4:09
Uh, there's a lot of layers to it. You need compute, uh, you need orchestration, you need libraries for doing, uh, training and evaluation, and you need platforms to support things like code execution and evals and inference and fine-tuning, and we're doing all of these things.
- 4:23
Uh, but really the goal of this is to give people the tools to be able to go train models. We want people, more people in the world, and we think...
- 4:30
I'll explain why in a bit. There's a lot of reasons why, uh, the best products are going to be the ones that are not just kind of taking the thing out of a, a box of an API and putting a thin wrapper on it.
- 4:42
There's ways you can kind of improve around APIs, but I think in many cases people are realizing that winning products are going to be the kinds of things that whether it's the, a part of the model or a part of the stack, a part of the product or the whole thing, the ability to do research and have
- 4:56
at least the option of deciding where in your product you might want to customize a model or improve a model gives you a lot more flexibility to really, uh, make the best user experience.
- 5:07
Um, and so we have heard the phrase in the past that the model is the product, and I think we're starting to see now this change a little bit to a lot of winning applications have the product kind of be the model.
- 5:19
And I think the two notable examples of this that I'm big fans of and heavy users of are Cursor's new Composer model as well as, uh, OpenAI's Codex. And I think these are both, both good examples of models that really are...
- 5:31
where the product kind of is the model very directly, where the, the model was trained to be the model for that product, and the experience of using the model is the experience of using the product.
- 5:41
And the way that this is done is by taking a harness that represents the product and training the model in the harness in essentially an environment, an RL environment.
- 5:49
And environments really are just a harness with a collection of tasks and rewards, but they also have many other parallels throughout the ecosystem. Environments are not just for RL.
- 6:00
Environments are also essentially the same thing as evals.
- 6:02
Environments can also be engines for synthetic data, which then you can use for SFT or distillation. You can do RL in them directly, but also the agents we're actually deploying and monitoring out in the world, these are environments.
- 6:13
The product of these things, the tasks, the harness, and the rewards, whether this is a dataset offline or the stream of user tasks coming into a product, is an environment.
- 6:23
And so this as an abstraction I think is a very useful way of framing what it might look like to start having, uh, research become more of a, a practice that is adopted more broadly beyond just large AI labs.
- 6:36
And I also think that there's a sense in which they're a really accessible entry point. Uh, and so I like the analogy of environments as kind of like the web apps of AI research, and what I mean by this is that they're very simple, they're self-contained.
- 6:47
They can ... They start simple, but they can also get quite complex. They can get very elaborate representing the full complexity of a large product. They're also pedagogical in nature in that you can start simple and as you build complexity, you start bumping into these walls where you have to start learning new concepts, understanding more about scaling
- 7:03
the system side, understanding more about the hyperparameters and the algorithms. And they kind of open this door where you can, by playing around with them, start enter- entering into a world of research without needing to kind of build a whole training infrastructure system from scratch.
- 7:17
Um, and they also require experimentation. And so I think the key differen- diff- uh, differentiation between just an agent harness and an agent environment is that the environment forces you to also have your tasks and your rewards predefined to be able to do this experimentation.
- 7:32
It's a proper eval. And what this means is that you can't just vibe check it. You can't just, like, build it and test it out a bit and say, "Hey, it's good.
- 7:41
We're gonna ship it." It forces you to say, "Okay, let's think about this a little more scientifically. Let's do some experiments. Let's try out different models, try different hyperparameters."
- 7:50
Uh, and it also gets you to the point where you can start doing more advanced research in terms of RL training or distillation or fine-tuning. And, uh, so to really facilitate this, we wanted to make the environment as an entry point much more accessible.
- 8:04
A few months back, we launched what we called the Environments Hub, which is a open source community platform for creating, discovering, and sharing RL environments and evals. And so far we've had a lot of fun kind of seeing everyone build here.
- 8:16
We've had hundreds of builders and environments come create either their own ideas or re-implement papers. Uh, there's a bunch of examples here I can show you, but really it's just a bunch of people who have wanted to do research and found this as an entry point to start digging a little deeper, whether this was investigating some benchmark
- 8:33
and figuring out how to re-implement it or modify it to be appropriate for an RL context in terms of, like, new data or new examples, or whether this is some game that they'd been thinking about or some other task.
- 8:44
But having this as an abstraction for encapsulating the, the thing you want the model to do is a way of allowing yourself to start experimenting with ways of improving it without needing to have the answers.
- 8:54
So I think people talk a lot about how fine-tuning never really took off in the SFT regime, and I think a big part of this is that getting data was really hard of the actual, like, solutions.
- 9:04
I think having labeled examples of what you want the model to do is a very difficult thing to ask someone to go create. But if you can just think about the s- the settings it might be in without having the answers up front, if you can measure the answers, now you kind of can start creating data on
- 9:19
the fly. And this engine is really what the environment is about unlocking. Um, and so actually nine months ago, I was right here in this room. I had just released a library called Verifiers, which I'm still working on today.
- 9:32
Um, it's come a long way, but it's a toolkit for building these things, and it's been a lot of fun over this past year just playing with it and extending it to support more features and kinds of environments.
- 9:43
But the idea with Verifiers is to give people a toolkit that is a, a, essentially a bunch of components that you can mix and match and compose to do things like from simple evals or QA or games to things like tool use or using sandboxes or agent frameworks or, uh, c- uh, like CLI coding agents or math
- 9:59
problems. There's all sorts of things you might want models to do or agents to do, and it's a toolkit for building environments that is then, uh, ready to be, uh, automatically trained with reinforcement learning.
- 10:10
And the way we thought about this design, it's been a lot of fun and also a big challenge to think like, "Okay, how do you make a toolkit for this stuff that actually covers all the bases?"
- 10:19
And I think there's a lot of different approaches I've seen people go about, and I, I think they all make sense depending on what sorts of things you're wanting to work on.
- 10:27
But we took a very kind of a general approach where we tried to say, "We are not going to know all the answers right away. There are going to be lots of patterns.
- 10:35
There's going to be lots of special cases. There's going to be hierarchies of complexity. There's going to be patterns." And we really wanted to prioritize extensibility, so we think about these things hierarchically, where let's say you wanna do a co- a coding agent environment for Kleinbench.
- 10:49
Uh, this, which is an instance of the Harbor framework, which is a example of a CLI agent, which is a multi-turn environment, which is an environment. Uh, similar for Textrina and Wordle or for search with MCP or for giving a model a Python REPL in a sandbox.
- 11:03
And so thinking of these things hierarchically allows us to kind of really de- de- determine, like, what are the foundational pieces, what is generic across all environments, and then how do you build up the stack towards applications?
- 11:14
And so for one ex- like example of this that I'll kind of walk through the whole process end to end, uh, we, we call this one WikiSearch, but it's basically a simple search setting where we give an agent the ability to, uh, call some tools to search over Wikipedia pages and find some answers.
- 11:28
And so here is the Environments Hub page. So the Environments Hub is a kind of full stack, uh, code management package registry. So every environment is a Python project where you can have dependencies and versions and uploading your evals and whatnot.
- 11:41
Um, but the environments are very simple. They start simple. They can get really complicated, but this one's pretty simple, where we just kind of define our tools as async Python functions.
- 11:50
We have our dataset, and we have what we call a rubric. And so a rubric is the abstraction for managing the different pieces of your rewards, where you can kind of compose different things.
- 11:59
You can also have metrics that are just a zero award but are for im- uh, observability of what's going on. And then the other piece of doing training will be a config.
- 12:08
And so the config here is for our Prime RL trainer, which is our kind of large scale training stack, which has been our, uh, culmination of all the best practices from the research literature for large scale asynchronous RL training.
- 12:19
Um, but the config files are intended to expose kind of the pieces that people need to think about. In ways that are starting to get you more into the algorithm but are also still designed to be pretty high level, pretty self-contained, and with, with defaults that we think are going to be sensible for a lot of people.
- 12:34
And so running this is just kind of running a command line with, uh, you specify the environment, and if it's in the Environment Hub, it'll automatically install it and start your training run.
- 12:43
And then you can, if you're lucky, see your reward curve just shoot right up. Um, and sometimes it doesn't g- go this nicely, but the process of doing this is iterating on your environment, on your rewards, and your data and your tasks to understand what makes this task holistically actually tangible in practice.
- 13:01
How do you tune the parameters? How do you look at your data? How do you define your rewards? Uh, and if you do this right, you can get really good improvements, especially from really small models, but also from much larger, larger models.
- 13:10
And so in this example, for the, the WikiSearch one, we started with a, a Qwen-34B model, which was about fifty-five percent, and after training, it was at eighty-nine percent on par with, uh, much larger models like GPT-4.1, as well as reasoning models like, uh, GPT-5 Mini.
- 13:24
And so I think this practice of taking small models and being able to make them much better is a big win for a lot of applications where you either you want a really fast model, you want a really cheap model, you want a really, really powerful model because the best models out there just aren't quite good enough.
- 13:38
These are all the different things you can do with model customization. And this practice of doing, of creating environments isn't only for customization, but it gives you this option.
- 13:46
And so if you need to do evals anyways, it's useful to think of them as environments because the environment opens a lot of doors for whether this is prompt tuning or whether it's model selection or whether it's just getting a better sense of how your system could work at scale with many, many users in parallel.
- 14:04
It's a design process that really forces you to kind of pin down what is the thing I care about. What is my agent, what is my product, what is my harness, what am I optimizing for?
- 14:14
Um, and so to kind of fully stress test this, we've been training a large model which will be out into the world quite soon called Intellect-3 with our full Prime RL stack.
- 14:22
And this has been us really kind of validating the efficiency and performance at a very large scale. So this is a hundred B+ model trained on five hundred GPUs, where we've kind of done the end-to-end, uh, post train of SFT and RL, which the Prime RL stack also has SFT if people wanna do that.
- 14:37
But it's also been about just understanding all the best practices. We love reading papers, and we try to kind of try out all the tricks and see which ones work and see which ones don't, and then distill this into a library with Prime RL that can then be kind of consumed by the end user without needing to
- 14:52
do all of this, uh, implementation themselves. And so for us, it being open is very important. So Prime RL is on GitHub. You can go find it, verifiers is on GitHub if you wanna check it out.
- 15:02
And for us, this is really about opening the door for more people to start learning about these things and for incorporating it into their workflows for optimizing their models and their products.
- 15:12
Um, and the only way to do this that we've... What we see as the best way to do this is through growing community. And so for us, it's been really important to really think about getting good feedback loops from the people who are building with this and understanding what they want, understanding what's going well, understanding what's painful,
- 15:29
and addressing those problems. And so we've done a number of community programs in terms of sponsoring different kind of small tasks to, uh, a research residency program with, uh, grad students around the world, uh, and collecting like a, a smaller subset of the Environments Hub ones where we'll actually review them manually.
- 15:43
And so this repo here, the Prime Environments repo, is the ones where we are doing these directly, where we're kind of offering to look over someone's kind of example implementation.
- 15:52
And so we've had hundreds of these come in, and there will be hundreds more, and, uh, it's been a great learning process because it's forced us to fix a lot of things.
- 15:59
We kind of understand the rough edges, we understand what we need to add, and we're kind of then distilling all of these learnings into what will be our kind of upcoming, uh, platform product, which we're calling Lab.
- 16:11
And the idea of Lab is to give people an interface, a platform where they can browse environments, they can run their evals, they can do their inference, they can do their fine-tuning, and they can have research be more accessible in a way that it hasn't been historically.
- 16:25
Because I think a lot of people find infrastructure very painful. They find dealing with Torch versions painful, Flash Attention, vLLM, and getting all these things to work. We are happy to do that, but we understand that a lot of people may not want to.
- 16:40
Um, and so the idea with this is that if you wanna go read the code, you can go read the code, but you don't have to run it. We can run it for you.
- 16:47
Um, and so this has been our version, which will be kind of out into the world in the near future, of trying to allow people to really focus on the environment where the entry point to Lab will be the environment.
- 16:59
If you wanna do synthetic data in SFT, build, let's build an environment. If you wanna do your evals, you build that as an environment. If you wanna do RL, you build an environment.
- 17:06
And I think building an environment is the kind of thing that
- 17:11
I imagine a lot p- more people are going to want to be doing as we start really seeing where models are headed. In some cases, this will be we're gonna use fine-tuning services from the labs because they're gonna offer this because people want it.
- 17:24
In some cases, this will be we really care about the smallest model we can run on-prem at the lowest latency, and we're really just gonna optimize for our one thing.
- 17:32
Or it could just be research for the sake of research and advancing our kind of collective understanding of how this stuff all works. And I think that's really our goal, is to have a world where there's gonna be a lot of AI and where we can all kind of talk about it and understand it and look at
- 17:46
it and poke at it and tweak it and have a better sense of what we're actually building, because I think there's a lot of times when it feels like we're just kind of the model is a black box.
- 17:55
And digging into the research and going under the hood and changing things and breaking things tells you a lot about how these models work. It tells you a lot about understanding where they came from, where they could be going, where they might be headed, and preparing for that future.
- 18:10
Thanks. [upbeat music]