AI Engineer World's Fair 2026
Data and Environment Curation for Post-training LLMs
Read the talk
Curating Data That Makes Agents More Reliable
A wrong tool call can interrupt an otherwise capable agent. Better post-training starts with testing which questions, reasoning traces and environments actually improve the model.
From a talk by Mahesh Sathiamoorthy
Before you start: Familiarity with language-model prompting and fine-tuning will help; SFT, rollouts and the role of RL environments are explained as they appear.
Connect data curation to model improvement
How do you know whether the data you curate will actually improve a model? Mahesh Sathiamoorthy introduces this problem through Bespoke Labs, where he identifies himself as co-founder and CEO after working as a researcher and engineer at Google DeepMind. The team’s work progressed from Curator, a tool for synthetic supervised fine-tuning data, to Bespoke Stratos after DeepSeek arrived, and then to OpenThoughts. Contributions to Terminal-Bench, RL environments and enterprise custom models extended that work from reasoning answers to agent behavior.
Data curation needs feedback from training. Data producers and researchers often occupy separate roles, but a dataset that looks useful to its creator may not move the model’s metrics. Sathiamoorthy advocates doing both: curate examples, train on them, and use the resulting measurements to decide what to curate next. That connection is the organizing principle behind the recipes that follow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From knowing answers to completing work
The shift from knowledge benchmarks to agent benchmarks changes what training data must support. The slide contrasts MMLU’s tests of knowledge across subjects with SWE-bench’s tests of software work; Terminal-Bench likewise asks what an agent can accomplish. Answering a question correctly is only part of the requirement when a model must act autonomously.
Longer autonomy exposes the next constraint: reliability. An agent may call the wrong tool or make another mistake that interrupts its work. Hours, days and weeks of autonomy are goals here, not reported achievements. Reaching those horizons requires reducing the failures that accumulate during execution.
There are several places to intervene:
- Prompts: improve the instructions guiding the agent.
- Harness and tools: change the execution machinery and available actions.
- Model training: use post-training to improve the behavior the model brings to the task; stronger pretraining can also contribute.
Reinforcement learning is one popular post-training method, which makes the environments in which agents practice an essential part of the reliability problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Environments are another form of training data
Sathiamoorthy uses data broadly enough to include RL environments. Supervised fine-tuning, or SFT, consumes demonstrations; RL environments provide settings in which a model can generate experience. The training input has changed shape, but someone still has to construct it and establish its quality.
His diagnosis is that high-quality data and environments are often the bottleneck, especially for enterprises. Compute, capable starting models and training infrastructure are comparatively established, with providers such as Fireworks and Tinker supplying parts of the infrastructure. Frontier labs face the same demand for useful environments even when their training systems are already in place.
The payoff is broader than benchmark capability. Post-training can also target latency, cost and throughput. The examples move through reasoning-data curation, agent trajectories and environments, an enterprise deployment, and finally the tooling needed to run this process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a reasoning-data recipe through ablations
OpenThoughts began with a gap: after DeepSeek’s arrival, the open community lacked comparable access to high-quality reasoning data. Bespoke Stratos expanded into a collaboration with researchers at Stanford, UC Berkeley, UW and other institutions. The resulting OpenThoughts paper, published at ICLR 2026, documents a recipe for constructing that data.
Sathiamoorthy reports that increasing dataset size under the OpenThoughts recipe improved evaluated performance, including on AIME and LiveCodeBench. This is the relevant scaling test: does producing more data with the same recipe continue to help? He also cites Microsoft attention and John Schulman’s report of internal use at Thinking Machines as signs of adoption, separate from the experimental evidence.
The pipeline contains several decisions that must be tested rather than assumed:
- Collect source questions. Start with existing prompt-response datasets and extract their prompts.
- Choose the mixture. For a target such as 10,000 samples, decide how many questions to draw from each source.
- Select questions. Test LLM judgments of quality and difficulty, then mix and filter the questions.
- Generate reasoning answers. Use teacher models, such as DeepSeek, Qwen-based models or Gemini, to produce responses.
- Test answer selection and multiplicity. Compare filtering answers with retaining them, and generating one answer per question with generating several.
- Run staged ablations. Measure the effect of choices at each stage and carry useful choices forward into the next experiment.
The final recipe is the result of these comparisons, not simply the pipeline diagram.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More answers can matter as much as more questions
One productive choice was to sample multiple answers to the same question. The comparison is between spending a generation budget on more distinct questions answered once and answering a smaller set of questions repeatedly—for example, sixteen times each. Sixteen is a tested choice, not a universally optimal setting.
A small Python helper makes that allocation explicit. Here, 100 questions answered sixteen times and 1,600 questions answered once each create 1,600 generation requests. The sample_index distinguishes requests for the same prompt; the teacher’s generation settings determine whether the resulting traces actually differ.
python
from collections.abc import Iterable
def generation_requests(
questions: Iterable[str], samples_per_question: int
) -> list[dict[str, str | int]]:
if samples_per_question < 1:
raise ValueError("samples_per_question must be positive")
return [
{"prompt": question, "sample_index": sample_index}
for question in questions
for sample_index in range(samples_per_question)
]
questions = [f"Solve for x: x + {n} = {2 * n}." for n in range(1, 1601)]
repeated = generation_requests(questions[:100], 16)
broad = generation_requests(questions, 1)
assert len(repeated) == len(broad) == 1600
Matching request counts isolates this allocation choice, although it does not guarantee equal token usage or training compute.
Why might repetition help? Fine-tuning includes the reasoning traces, not just final answers. Sathiamoorthy’s hypothesis is that several traces expose different ways to reason through the same problem. He also reports that stronger models were not invariably better teachers: teacher capability and the usefulness of its demonstrations to a student are different quantities.
Other results were similarly dependent on the recipe. Synthetic question generation and answering worked in the reported experiments, while answer filtering did not work well. The slide also lists question filtering as successful and increased source diversity as unsuccessful. These are experimental observations, not general prohibitions: the original paper explicitly notes that its unfiltered answer baseline contained twice as many examples as the filtered runs and was not compute-controlled. Filtering quality cannot be cleanly separated from that difference in training volume.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extend the recipe to trajectories and environments
OpenThoughts-Agent extends the same experimental approach to agent trajectories and RL environments. The object being curated is now a task and its execution, rather than a reasoning answer alone. The scaling question remains: as the dataset grows, does the recipe continue to improve the trained agent?
Sources can include material such as Stack Exchange. From there, the decisions concern task mixtures, filtering, rollout generation and teacher selection. A rollout is the agent’s sequence of interactions while attempting a task, so choosing a teacher also means choosing the behavior the student will observe.
Several findings echo the reasoning-data work:
- Teacher selection: stronger models were not necessarily better teachers. Sathiamoorthy tentatively recalls some Qwen models outperforming some Claude models as teachers, without identifying an exact comparison.
- Repeated sampling: multiple samples helped again.
- Synthetic task changes: rewriting and task augmentation did not work as well as expected.
The recurrence of these findings makes them useful experiments to run, rather than settings to adopt without measurement.
Sathiamoorthy reports that SFT supplied substantial gains, while compute-intensive RL contributed smaller final improvements in this work. His enterprise implication is practical: SFT can already work well for many applications, so the decision to add RL should depend on the remaining capability gap and the cost of closing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Teach the form without memorizing the numbers
The enterprise example is Intuit’s Credit Karma, where the application explains why a credit card has been recommended. A prompted model can write the explanation, but compliance creates a second requirement. Long lists of rules increased latency without making the responses consistently compliant. Sathiamoorthy presents the engagement as a production post-training example.
Simply fine-tuning on existing examples introduced another failure mode: imbalanced values. If many examples contain 0% APR, the model can learn that recurring number and reproduce it where it does not belong. Fluent explanations are insufficient when a financial attribute is wrong.
The intervention changed the training representation. Instead of relying only on plain-language prompt-response pairs, the team added tags intended to make the model focus on the form of the explanation rather than particular numeric values. The talk does not specify the tag syntax or a runtime substitution procedure; the demonstrated principle is to change what the training examples encourage the model to learn.
Sathiamoorthy reports improved compliance, latency and throughput for the Credit Karma engagement, without numerical measurements or evaluation conditions. He also emphasizes ownership: a specialized model gives the enterprise more control over cost and reduces the pressure to revise its system whenever a frontier model changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn prompts and logs into training examples
Curator provides tooling for the generation step. Inputs may come from a Hugging Face dataset containing prompts or from collected application logs. Generating responses to those inputs produces examples that can be used for fine-tuning. This makes the earlier curation decisions operational: the source prompts are inputs to a repeatable data-generation process.
Curator was used for the original OpenThoughts curation, and the talk describes integrations with Tinker and Fireworks. Those integrations belong to the later tooling: the repository dates them to March and June 2026, respectively, rather than the initial 2025 OpenThoughts release.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect environment quality, execution and optimization
The closing architecture broadens the data pipeline into a stack for building environments and improving agents. Start with the environment layer: construct tasks, measure their quality and track their versions. Versioning matters because the environment is itself part of the training input, not merely infrastructure surrounding it.
Below that layer, compute and orchestration start sandboxes and generate rollouts. Long-horizon execution may require checkpointing, snapshots and rollback so that work can be preserved or an earlier state revisited. Above the environment layer sits agent optimization, including SFT and RL.
Optimization can also change prompts. GEPA uses LLM reflection to propose prompt improvements; Sathiamoorthy includes system-prompt and harness updates in this broader optimization layer. The closing stack therefore connects three responsibilities:
| Layer | Responsibility |
|---|---|
| RL environments | Build tasks, measure quality, track versions |
| Compute and orchestration | Run sandboxes and rollouts; preserve execution state |
| Agent optimization | Improve models through SFT/RL and prompts through reflection |
The environment supplies the work, orchestration makes that work executable, and optimization uses the resulting experience to improve the agent. This is the architecture Sathiamoorthy proposes for both curating RL environments and post-training agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Research on reasoning-data selection, teacher generation, filtering and dataset scaling.
The initial agent-data release explains separate pipelines for SFT trajectories and RL environments.
The June 2026 report covers task mixtures, rollout generation and scaling agent training data.
Python tooling and examples for synthetic-data generation, curation and fine-tuning integrations.
A prompt optimizer that reflects on execution traces and tests candidate improvements.
Further reading
An Intuit and Bespoke collaborator explains product-scoped tags, controlled synthetic data and validation for credit-card recommendation explanations.
Installation instructions and examples for reflective optimization of prompts and other text artifacts.
Read the complete timestamped transcript
- 0:00
[on-hold jingle] Hey, everyone. Um, today I'll be talking about data c-- and, uh, environment curation for, uh, post-training LLMs.
- 0:21
And I am Mahesh Sathiamoorthy. Um, I'm co-founder and CEO at Bespoke Labs, and previously I was a researcher and, uh, engineer at, uh, Google DeepMind. So very briefly, I will tell you a little bit about, uh, Bespoke, and, uh, after that, the talk will be mostly around, uh, open source work we have done.
- 0:41
So Bespoke is an applied data research lab with a mission to help enterprises and frontier labs access high-quality data and RL environments for their post-training needs. So very briefly, what we do and what we have done is that last year we put out something called Curator, which is a tool for curating, uh, synthetic data for post-training with
- 1:05
basically SFT. And right after that, actually DeepSeek landed, and we started an effort to curate reasoning data, and that's how we started something called Bespoke Stratos, which eventually formed some-- uh, into the project called OpenThoughts, which some of you hopefully know about.
- 1:22
And we have also been core contributors to Terminal-Bench. Um, you know, these days we, we do a lot of research and build and ship RL environments, so I was actually looking forward to the previous talk, uh, from Nick, who's also, you know, uh, doing something similar.
- 1:40
Uh, and, and the other thing we do is we do a lot of post-training and help enterprises, uh, to get their own custom models, right? That's the name of-- That's how we ended up with the Bespoke, uh, ta-title for the company.
- 1:56
Uh, the, the other thing I want to kind of mention is this, there is this, you know, you know, in, in our industry, there are a lot of people who create data, uh, create RL environments, and then there are the, uh, researchers who consume this.
- 2:09
But I feel like there is this slight mismatch, and it's kind of beneficial for someone to kind of go do both at the same time. And in fact, uh, as you're curating data, you want to put yourself in the shoes of the researcher to see what, what does it take to, you know, uh, actually move the metrics
- 2:28
on the models. So that's one of the motivations of how we kind of think about... The other thing I want to kind of talk about is, you know, how, you know, uh, AI has evolved, right?
- 2:40
So early on, we used to think about and evaluate models on what they know. Um, for example, this is a-- this was a very popular benchmark on, uh, testing LLMs on various kinds of STEM, humanities, and all that knowledge.
- 2:56
And these days, we have all these benchmarks that test, uh, how, how, how agents are able to do things. We have moved on from knowing to doing, right? So that's the idea of agents, obviously.
- 3:08
And the-- One of the key principles or one of the key things about agents is that they are autonomous. And there are, as I was saying, there are many benchmarks, including, uh, SWE-bench, Terminal-Bench, and so on.
- 3:21
But ultimately, for many people, what they care about is, are these agents autonomous for long durations of time? Uh, Nick had, uh-- Sorry, uh, Ross had a great talk on long horizon, right?
- 3:33
So that's the goal, is eventually we make these agents autonomous for maybe few hours or a, you know, few days or a few weeks. And what is it that's blocking the, uh, autonomy of agents?
- 3:48
It's basically reliability, right? So at some point, something falls apart. Like, either they call the wrong tool, or they made a mistake and, you know, what, whatnot, right? And what, what's one lever to improve reliability?
- 4:02
There, there are, of course, many. Um, obviously, you can prompt your way to improving the agent's, uh, reliability, or you can, uh, update the harness, you know, the tools and whatnot.
- 4:14
But post-training is a very powerful tool to improve reliability or, or maybe even pre-train good models, right? So if you think of frontier labs, this is one of their, uh, primary mechanisms of improving agents over to, to, uh, get [clears throat] better, uh, capabilities in,
- 4:33
um, va-various domains or, you know, uh, for, for better, uh, benchmark numbers or better, uh, um, autonomy for long-longer and longer durations. And for post-training, one of the popular techniques, as you know, is reinforcement learning, and that's kind of, um, something, you know, a lot of you are, um, excited about is the, uh, you know, notion of
- 4:58
RL environments. But ultimately, for post-training, be it SFT or, or, uh, reinforcement learning, data is the bottleneck, right? So when, when I talk about data, R-RLMs are also something I'm calling it as data.
- 5:11
It's just the data is now in a very different shape. Um, again, here, you know, compute is kind of well-defined. Models, you know, uh, good sort of models exist and the, uh, infrastructure to post-train, for example, uh, there, there are various providers like Fireworks, Tinker or, uh, SLIM-VIL and whatnot.
- 5:36
So RL, all of those are somewhat well-defined. Most of the places where people struggle, at-- especially enterprises, is that they don't have access to good quality data and RLMs.
- 5:47
And this obviously also applies to frontier labs, where they have all this infra set up and they are, you know, needing good quality RLMs, right? Uh, beyond-- So that, that's one of the-- This is kind of how we are thinking about at why to invest time in, you know, doing data research and RL and research.
- 6:06
And as a side note, one of the, um, other, uh, the-- there are many other benefits of post-training. For example, you can reduce latency or improve cost, throughput and whatnot, and I'll give one concrete example of a post-training work we did, uh, with one of the enterprises.
- 6:25
So in this talk, I will mostly, uh, cover some of the work we have done in the open source, uh, community. So we did some work on curating reasoning data for reasoning models and for, uh, you know, curating trajectories and, uh, en-environments for agents.
- 6:43
And recently, we had an engagement with post-training, which, uh, I'll, I'll very briefly talk about, and some tools on data curation.
- 6:51
So OpenThoughts, um, is, uh, reasoning dataset as well as a paper, right? So we, we started this effort last year. As I was saying, this, uh... We, we-- After DeepSeek came out, we realized that there is a lack of very high-quality reasoning data in the, uh, community.
- 7:12
Obviously, the labs have access to good data, but outside we didn't have access to data, right? So we, we at Bespoke started this effort called Bespoke Stratos, and then we realized that this is actually quite useful.
- 7:25
So we joined, um, together with various folks in, uh, Stanford, UC Berkeley, UW and so on to create this consortium called OpenThoughts, and we did a lot of work on basically identifying the curation recipe, and we also published this as a paper in ICLR of this year, and this is the main figure of the paper.
- 7:46
So what it shows is, like, we, we figured out a curation recipe, and it shows the scaling law, right? So again, this is last year when Amy, uh, and, and, and, uh, LiveCodeBench and these, these were some of the popular benchmarks.
- 8:01
What we showed is that with this recipe, if you keep, you know, scaling up the dataset size, the, the, you know, the, the... It's a scalable recipe, right? The, the metrics also improve.
- 8:12
Um, it's actually very widely used as well. For example, this is, um, uh, Microsoft CSO tweeting about the work, and this Ale-Alex is my, uh, co-founder. He's, uh, chief scientist and also a professor at UC Berkeley.
- 8:27
And this is John Schulman talking about OpenThoughts that he has-- he and his, uh, colleagues have been using it internally at, uh, Thinking Machines, right? And some of their blog posts also reference this.
- 8:39
So I'll ve-- uh, talk about how we did the curation for OpenThoughts. Um, th-this is the pipeline that we used. So you start with curate-- You start with a bunch of source questions, right?
- 8:52
So there are various datasets out there that have the, uh, prompt response, and we ch-choose with the prompts. We start with the prompts. These are various, uh, sources we have.
- 9:03
And then, uh, if you look at the paper... So if you look at this graph, for any given data point, say if there are ten thousand samples that you want, the question is then how do you choose, uh, the questions from all these different dataset so that you have ten thousand, uh, for the data point?
- 9:24
So the, the-- then there is the aspect around how do you mix these questions. So, uh, you can use various methods. So the paper talks about, for example, using LLMs to check for whether this is a good, good question, hardness of a question, and so on.
- 9:40
And then you want to filter questions, uh, and generate the answers. Again, this is all, like, driven by LLMs, right? So this is the curation recipe we did for, uh, creating this reasoning dataset.
- 9:52
And j-- The, the answer generation is using teacher models, so you can take other reasoning data, uh, uh, reasoning models such as DeepSeek or Qwen-based models, uh, or even Gemini and whatnot.
- 10:03
And then you can also filter the answers once you have the answers for, uh, these questions. Um, and, and then you can also, you know, given a question, generate multiple answers or a single answer.
- 10:16
So these are various knobs in the curation recipe, and the systematic way of doing this is, like, you run ablations and figure out which, uh, you know, in each of these stages what works, and you kind of proceed to the next.
- 10:31
So after doing all of this, you get the final recipe, right? So this, uh, you can read this paper. It has lots and lots of, uh, you know, information about how we did the curation.
- 10:42
But here are some of the learnings that, you know, some of them are quite, uh, counterintuitive, and some of this was also covered in last year's, uh, AI, uh, AI engineer conference.
- 10:54
For example, sampling, um, multiple answers per question works pretty well. This is something that, uh, we... It's, it's kind of counterintuitive. So as an example, something else we could have done is we could have had more, much, many more questions and then just answered them e-exactly once versus taking one question and answering them sixteen times.
- 11:18
The-- I think the, the reasoning is probably that it gives like a variety of how reasoning is done. So the, the-- During fine-tuning, we also use the, the reasoning traces, right?
- 11:29
So I think the diversity helps there. And the other thing we saw is, like, the stronger teachers are not always the best, uh, uh... St-stronger models are not always the better teachers.
- 11:42
And there were a few other counterintuitive aspects around like, you know, uh, synthetic question generation or qu-question answering working, whereas answer, answer filtering and other aspects not working very well.
- 11:54
And after the OpenThoughts work, which was around, uh, data curation for reasoning models such as, you know, uh, DeepSeek kind of models, we moved on to OpenThoughts Agents, which is, um, very similar, but how do you curate these, the, the data and RL environments for, uh, training agents now, right?
- 12:13
Not reasoning models. We, we have a very similar figure here. Again, we want to establish scaling loss. Um, so as you increase the dataset size, we want to make sure that the curation recipe actually works.
- 12:25
Uh, and again, I'm, I'm not going to go into details here, but very similarly, there are various ways of choosing different sources, for example, Stack Exchange and, and whatnot.
- 12:38
How do you mix the tasks? How do you filter generating the rollouts, fil-- uh, choosing the teacher, and so on? And again, th-these are some of the lessons, learnings.
- 12:48
Um, a-as an example, e-ev-even here we saw that stronger models are not necessarily the, uh, best teachers, right? So we found out some, some, some of the, I think, uh, um, Qwen models were better than, for example, um, um, um, um, Cl-Claude models, I think.
- 13:09
And sampling multiple answers, again, helped in this case. Synthetic rewriting and task augmentation, um, is something we thought will work, but it didn't very work-- uh, work very well.
- 13:21
And the other thing is, like, in, in this whole process of building this OpenThoughts agent, SFT still contributed a lot to the gains. Um, RL was kind of, you know, it's very compute-intensive and for, for the last few per-few percentages, it really helped.
- 13:38
Uh, but, but, you know, in many of the situations, for example, in enterprises, SFT actually works pretty well, right?
- 13:46
And here is one concrete example I wanted to share on, um, uh, actually deploying something to production, right, by post-training. So we have seen a lot of people talk about post-training, but in enterprise settings we haven't seen a lot of successes, at least I haven't seen, uh, that.
- 14:04
Here is a very concrete example of, uh, with Intuit, there is, uh, this app called Credit Karma, which if you install, there is a page wh-- uh, place where you can, uh...
- 14:13
The, the, the, the app gives you a reasoning as to why a credit card has been recommended, and this you can prompt a model to do this. But one of the reasons-- one of the places where it fails is that the, you know...
- 14:27
It, it-- it's not always compliant, so you have to have a long list of rules to make sure the responses are compliant, and that actually blows up the latency.
- 14:37
So answer here is, like, you want to curate data and post-train, right? Seems kind of straightforward, but one of the things that we ran into is, um, the dataset can be quite impa-- imbalanced and lo-lots and lots of places, for example, you will have zero percent APR, and the model after fi-fine-tuning can kind of hallucinate the, these,
- 14:59
uh, numbers. So this, again, kind of ties back to what Ross talked about some time back with respect to the tags, and we kind-- uh, we, we created this specific, uh, curation recipe where instead of just having these, uh, question-- the, the prompt response pairs in plain language, we added these, uh, tags, which helped the model to
- 15:23
focus on, you know, uh, the, the kind of form rather than the specific numbers itself, and that gave a big boost. And, uh, we, we saw that, um, the, the overall, the compliance metrics improved, the latency improved, the throughput improved, and eventually, you know, they, they are able to own the model, right?
- 15:42
As frontier models improve, they don't need to kind of go and, um, um, uh, update it. And also, as we see now, the, uh, [clears throat] frontier models are also getting more and more expensive and, you know, th-this kind of give, gives them a very good way for owning the model and also, um, lowering the costs.
- 16:04
I think with that, I want to briefly touch upon, um, uh, you know, Curator, the tooling that we had built last year, um, which is for curating reasoning data.
- 16:16
So, um, what it does is you can basically, uh, you know, um, specify the... You, you can either go with, say, a Hugging Face dataset where you have various prompts or, uh, in many situations you may have collected logs and you want to, uh, get the responses and fine-tune a model.
- 16:36
So this Curator kind of makes it pretty easy to do that. And it comes with the integration with, you know, uh, Tinker and Fireworks, and this, this is again, the tool that we used, um, originally for curating OpenThoughts.
- 16:52
And here is a very, very detailed diagram of what we are building today, but, uh, this again connects back to, um, what Ross was talking about, where he was talking about algorithms, uh, environments, and compute, right?
- 17:06
So it, it feels like, you know, we are kind of converging on something very similar. So if you think about, uh, the, the stack that is needed to, say, not just curate these RL environments, but to post-train models, one of the things you need is obviously handle on, like, how do you build these RL environments?
- 17:27
How do you measure the quality? How do you track the different versions and so on? So that's one of the layers. And below that you want various infrastructure to, uh, um, sandbo-- u-use sandboxes, right?
- 17:39
So spin up the rollouts-- to, to spin up the sandboxes to generate rollouts. And especially if you have long-horizon rollouts, then maybe at some point you need to do a checkpointing, and then you need to be able to snapshot or roll back to something else, right?
- 17:53
So that's the other, uh, the, the lower level, uh, you know, compute and orchestration. And at the top, I have been giving examples on post-training. So there is all this, uh, layer around, like how do you do SFT, how do you do RL, and so on.
- 18:07
But there is also this method called JEPA, which is around, uh, which is on prompt optimization. I don't know if you, if you guys have heard of it, but you can use LLMs itself to, uh, to, to kind of optimize the prompts based on reflection.
- 18:22
Um, so that also works pretty well for updating the system prompts and also the harnesses. So this is kind of, I feel like, you know, the, the new architecture or the new reference, uh, stack for how, how at least we are building and how many others are building, um, the, the stack on how to build the RLMs
- 18:42
and then also post-train agents. I think with that, uh, I'll, uh, end the talk and, uh, you know, happy to take questions offline. [audience applauding] [upbeat music]