AI Engineer World's Fair 2026
The Messy Reality of Scale: Synthetic Data and Pre-Training
Read the talk
The Messy Reality of Scale: Synthetic Data and Pre-Training
Scaling a coding model means expanding what its data can teach while checking that thousands of GPUs still compute what the training recipe intends.
From a talk by Marah Abdin and Robert McHardy
Before you start: Familiarity with language-model pre-training, gradients, and the basic idea of distributed training will help with the numerical failure examples.
What has to change when a training recipe scales?
What has to change when a model needs more training than its original data and implementation can support? Poolside’s successive Laguna releases make that question concrete. With Laguna M.1 and Laguna XS.2, the company expanded from enterprise releases to publicly downloadable weights. The generation suffixes reflect changes to the recipe, not just differences in model size. Marah Abdin, who leads synthetic data, begins with the data changes; Robert McHardy follows with the distributed training failures that helped motivate the next generation. Their technical report documents the released models.
The data work proceeds along three complementary paths:
- AutoMixer: Run cheaper sweeps over clusters of datasets before committing to expensive training experiments.
- High-recall web sampling: Reconsider which web material enters the corpus, broadening the supply of useful organic data.
- Synthetic generation: Produce additional training material through several kinds of transformation and generation pipelines.
These changes address different parts of the same problem: finding useful material, deciding how much of it to train on, and making more of its teaching signal available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expose what organic data leaves implicit
Synthetic data complements organic data by making its implicit teaching signals explicit. A document can contain useful reasoning without presenting the rationale, planning, or structure in a form that is easy to learn. Generation can expose those features, present them in another form, and fill gaps in the original material. Abdin treats this as a way to regularize both the presentation of tokens and what those tokens teach, rather than as a replacement for organic data.
Synthetic data accounted for 13% of XS.2’s pre-training mix, before post-training. Abdin reports that the continually growing synthetic corpus had reached six trillion tokens by the time of the talk. The mix fraction and the corpus size describe different things: the former is a share of training data, while the latter is the pool of generated material available to draw from.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More training makes repetition a bottleneck
The initial data strategy emphasized quality over quantity. At a smaller training budget, that was a reasonable starting point. As models and training budgets grew, however, the same high-quality material appeared too often. Repeated exposure began saturating the model earlier than the team wanted. The shortage was not simply a shortage of tokens; it was a shortage of useful, distinct material within those tokens.
Rephrasing provides a relatively inexpensive response, also explored in work such as BeyondWeb. The ablation shown in the recording compares repeated seed documents, in orange, with a mixture that replaces some repeated tokens with multimode rewrites, in green. Abdin emphasizes the consistent qualitative benefit of reducing repetition, while warning against reading too much into the particular ablation numbers. The comparison concerns replacing some repeated exposure, not discarding the original seeds.
Poolside used a generic multimode rephrasing pipeline, then added two specialized paths: one transforms raw code into code plus text, and another targets STEM documents. Cheap rephrasing remains heavily dependent on the seed: the generator has to get much of the substance from its input. Specializing the transformation gives the team more control over how that substance becomes training material, particularly for STEM content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the task fit the generator
A modular pipeline makes these transformations easier to vary. Abdin describes seeds as primary inputs, metadata as secondary inputs, a generator function, and supplementary functions such as filters and validators. The companion report organizes the framework into six component types: inputs, metadata, generator, filtering, validation, and pre- or post-processing. The generator itself can be a prompted LLM or an agent with tools; the surrounding components determine what it receives and which outputs survive.
This framework spans very different cost profiles. Seed-heavy transformations such as rephrasing can use smaller models and scale cheaply. More elaborate workflows spend additional orchestration on outputs valuable enough to justify it, such as educational material. That extra structure also offers a way around the limits of a teacher model: when the full task is too difficult, reduce the difficulty of each generation step. Abdin’s rule of thumb is that an overchallenged model falls back on its biases and loses both correctness and diversity. Decomposition helps preserve those properties while building a more ambitious result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From rewriting to iterative generation
The first reusable shape is form rewriting: retain source material while changing how it is expressed. The next is multistage construction, where successive steps build and aggregate a larger output. Abdin illustrates the difference with a novel. Generating chapters one at a time is already a decomposition, but the workflow can first establish the material those chapters will depend on:
- Generate the setting.
- Establish character names and styles.
- Develop the plot and its twists.
- Use that groundwork to generate chapters sequentially.
Each chapter then has more structure to work from than a request to continue an otherwise unplanned story.
Two further shapes change either the representation or the interaction:
- Cross-domain porting: Translate code between languages, or turn math problems into code. Poolside used the latter to move existing mathematical material into a coding form.
- Multi-turn roleplay: Let generation evolve through interaction. Examples include two agents holding a conversation, or a judge and an evolver alternating for
Kiterations in a task-evolution pipeline.
The iterative pattern lets later generations respond to earlier outputs instead of following only a fixed sequence of transformations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hive configures who generates what next
Hive turns this modular view into configurable generation infrastructure. A queue contains agents with specified prompts, parameters, models, inputs, and outputs. Configuration also controls when an agent enters or exits the queue, how many agents enter, and how frequently they appear. That separates an agent’s individual generation task from the scheduling of the overall workflow.
Orchestrators sit between agents. They can change the instructions passed to the next agent, select which agent runs next, or skip an agent altogether. This hierarchy constrains generation while leaving room for the workflow to respond creatively to intermediate results. A supervisor operates above the orchestrators with a global view, checking the decisions made at that level. The result is a configurable control structure for both staged and iterative generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check the training system’s invariants
Better data only helps if the training implementation is correct. McHardy treats data quality, architecture, distributed training, and implementation correctness as one system: a serious defect in any of them can prevent a good model from emerging. At billions to hundreds of billions of parameters across thousands of GPUs, assumptions need explicit checks.
One such check is replica-weight hashing. In distributed data parallel training, corresponding model replicas should hold identical weights at synchronized check points. Poolside periodically hashes those weights and compares the results. Matching hashes let training continue; a mismatch crashes the run because an invariant has been violated.
The comparison step can be expressed as a small Python guard. Its input is the set of hashes already collected from corresponding replicas at the same check point:
python
from collections.abc import Mapping
def check_replica_hashes(hashes: Mapping[int, str]) -> None:
if not hashes:
raise ValueError("No replica hashes were collected")
reference_rank = min(hashes)
reference_hash = hashes[reference_rank]
mismatched_ranks = [
rank for rank, digest in hashes.items()
if digest != reference_hash
]
if mismatched_ranks:
raise RuntimeError(
f"Replica weights differ from rank {reference_rank}: "
f"{sorted(mismatched_ranks)}"
)
This guard checks agreement among the supplied hashes. It does not establish that every operation producing those weights was correct.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The same run, with one broken GPU
The first failure example shows two runs with the same model configuration, training data, and training implementation. Their curves nevertheless look very different: the purple run has bumps and spikes in its loss, accompanied by much larger gradient norms. Those symptoms might initially look like a problem with the training recipe.
The difference was a broken GPU. It silently corrupted data in one run, producing the abnormal training behavior. In this instance, the corruption also broke agreement among replicas, so the weight-hash check could detect it. The invariant converted a hardware fault that might otherwise masquerade as an optimization problem into a reason to stop training.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A local precision choice stops the whole model learning
The next failure appeared later in Laguna M.1 training. McHardy reports that the initial run stopped converging at around 50,000 steps. Its loss flattened while activations immediately before the LM head, or unembedding, kept growing. Because the unembedding used tensor parallelism, it required an accumulation operation across the partitioned computation.
That accumulation used BF16 by default. As activation magnitudes grew, the available numerical precision became insufficient for an accurate accumulation. The resulting error occurred at a particularly consequential location: its effects propagated backward from the output head through the model trunk. A locally inadequate numerical operation could therefore impair learning throughout the model.
The team resumed from the affected checkpoint after moving that accumulation to FP32. Convergence returned, and gradient norms began decreasing instead of following their earlier increasing trend. The intervention was specific: increase precision for the problematic accumulation, not move the entire training run to FP32. It also illustrates why an implementation that behaves well early in training may fail after activation scales change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the revised recipe into Laguna S
XS.2 incorporated the lessons from M.1 across the whole training system: greater data diversity, less repetition, numerical fixes, more observability and checks, and improvements to training and architecture. The open-weight XS.2 model focuses on agentic coding. McHardy describes XS.2 as a 33-billion-parameter model, making it a useful test bed but not the final test of whether the revised recipe would hold at greater scale.
Laguna S supplied that larger test and was still a nonpublic preview in the recording. McHardy reports 118 billion total parameters, eight billion active parameters, and training on 30 trillion tokens using 4,000 GPUs. This run tested whether the data and architecture improvements transferred, and whether numerical problems would return under a larger workload.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Matching weights can hide incorrect computation
Another failure appeared after the team added FP8 training based on DeepGEMM kernels. McHardy explicitly distinguishes this from a failure inherently caused by scale: it was a race condition. Illegal memory accesses and NaN gradients provided visible symptoms, and debugging eventually traced them to the kernels.
There was also a quieter symptom: McHardy reports that about 0.5% of the gradient was silently corrupted, effectively replaced with random values, in the observed case. At the time of the talk, the team had made a fix available in a public pull request, but it had not yet been merged into DeepGEMM.
This exposed a blind spot in replica hashing. Normal data parallel training does not redundantly execute the same forward and backward computation with both identical weights and identical data: replicas process different data. Agreement among their weights therefore cannot establish that those computations were themselves correct.
| Check | What must match | What it establishes |
|---|---|---|
| Replica-weight comparison | Weights at corresponding check points | Replicas agree on model state |
| Redundant computation check | Weights and input data | Repeated forward/backward results can be compared |
The second check requires deliberately creating the redundancy that normal training lacks. McHardy describes work on such a checker, at least for a dry run; it was a planned extension, not an already demonstrated safeguard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the results through the coding objective
The final comparisons concern base-model evaluations, with post-training still underway. They are signals about the pre-training recipe, not a guarantee of the finished model’s behavior. McHardy cautions that the results will not necessarily transfer one-to-one after post-training.
On coding evaluations including MultiPL-E, LiveCodeBench, and BigCodeBench, McHardy reports Laguna S ahead of XS.2 and the much larger M.1, as well as GLM-4.5-Air, Nemotron 3 Super, and DeepSeek V4 Flash Max in the displayed comparison. He identifies GLM-4.5-Air as an older comparator and the latter two as recent models, with DeepSeek also larger.
The results were not uniformly leading. McHardy describes Laguna S as competitive but not best on BIG-Bench Hard, and close to competitors on EvalPlus. On SWE-bench Agentless Multilingual, which Poolside used as a proxy for agentic performance during pre-training, he reports Laguna S ahead of every other model tested in the comparison. That distinction matters for a team optimizing specifically for agentic coding.
MMLU-Pro exposes the other side of that specialization. McHardy reports that Nemotron and DeepSeek performed substantially better than Laguna S on MMLU-Pro. He attributes the gap mainly to data and treats broader knowledge coverage as a lower priority than coding, rather than claiming the model is strongest across all tasks.
The larger result is that the combined recipe held at the tested scale: changes to data, numerics, architecture, and checking all carried into the next model. Poolside intended to continue scaling from there. The recording closes with a commitment to release Laguna S as freely downloadable open weights relatively soon, without a firm date.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Poolside's account of model architecture, pre-training data, synthetic generation, Hive, AutoMixer, and post-training.
Downloadable model weights and instructions for running Poolside's XS.2 coding model.
Research on synthetic pre-training data, including source selection, rephrasing, and generator choices.
DeepSeek's GPU kernel library, including FP8 matrix multiplication implementations.
Updates since the talk
Current model specifications, weight links, usage examples, and evaluation methodology for Laguna S 2.1 and XS 2.1.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everybody. Thanks for coming to our talk.
- 0:15
Uh, my name is Marah Abdin. Uh, I'm, uh, I'm from Poolside. I'm the synthetic lead for, uh, our, uh, data team. And, uh, today, my colleague Robert and I will be talking a little bit about, um, some of the challenges that we've seen as we scale our models, uh, over here.
- 0:33
Uh, particularly if you haven't heard, we've switched from recently from, uh, releasing our models towards enterprise to also releasing towards, uh, you know, uh, everybody. Um, we've actually put out two open weight models, uh, that are on Hugging Face a few weeks ago, uh, with Laguna M and Laguna XS.
- 0:50
We also put out a tech report, which has a ton of detail if you're interested. Um, as you can see here by the Laguna M.1 and XS.2, uh, this, uh, this is, this actually is because, uh, we have switched out quite a b- few things, um, between those two models.
- 1:06
And so, um, big flavor of this, uh, talk is going to be about kind of how did we transition from, uh, one to two. And in fact, we've continued to do so, and, uh, now we actually have a newer version and, uh, Robert will give a sneak peek about, uh, soon-to-be-released model. [lip smack]
- 1:21
Okay. So I will be particularly talking about synthetic data, uh, part of things. So, um, there's three things that we did on the data side to, uh, kind of resolve some of the issues that we've seen with scale.
- 1:33
One is that we implemented an auto mixer that, um, basically just gives us a chance to do a cheaper sweep on clusters of our sets before moving on to more expensive, uh, experiments.
- 1:44
And then we improved, we just rethought our, uh, sampling of web data for high recall.
- 1:50
Um, and then the third one is that we relied a lot more on synthetic data, uh, in a f- in a few forms, which I will go into. Okay.
- 1:57
So before we kind of going into what does that mean and what have we done, et cetera, um, one, uh, why would we kind of... It's, it's, it's sometimes it's fair at least to ask why synthetic data.
- 2:06
And, uh, the thing is that, [lip smack] uh, at least at Poolside, we don't see it as a way to replace organic data. I don't see it, so in the current state of the world at least.
- 2:15
But, uh, it is a way to kind of complement it. And the thing is that organic data has a lot in it that is basically kind of implicitly hidden.
- 2:23
A lot of things that could teach the model are not very presented in the most optimal way sometimes. And so synthetic data gives us, uh, a track to extract some of those features and then project them on some new planes.
- 2:34
And this is how we get to expose, um, implicit rationale, implicit planning, implicit structure, and a way for us to fill gaps and regularize not only how we present the tokens, but also how we are teaching the model.
- 2:46
Um, for XS.2 in particular, we settled on thirteen percent of the mix. This is only pre-training stages before post-training. Um, and since then, we've just been continuously generating more data in a bunch of directions.
- 2:57
Now we have a six trillion token, uh, corpus that's continuously growing. Okay.
- 3:03
Um, so yeah. So, uh, uh, w- kind of what I, uh, I, I just said is that we saw some, some, I guess, uh, limitations f- switching from Laguna M to, uh, point one to, uh, or point two models.
- 3:16
And so one of those things is that we basically be- it started on data. This is really not a, a crazy kind of problem. We tr- very intuitively started from a place on a smaller scale where we were, um, we were basically focusing on quality versus quantity, um, maybe a little too much because eventually when we started
- 3:35
scaling our models, we had to scale our training budget. And with that, uh, came some limitations because we started hitting repetition, uh, like non-optimal repetition on some of our higher quality data, which saturated the model a little too early.
- 3:49
So, uh, we-- one of the ways that we've, particularly for this reg- like a token uniqueness problem, uh, we relied on, which is a very common form of synthetic data, uh, rephrasing, uh, which, you know, you just, uh, might heard it in like Beyond Web, for example.
- 4:05
It's become pretty trendy these days. Um, and you can see here, uh, that, uh, uh, you know, this is a, take these oblation. This is an oblation result, so take the numbers with a grain of salt.
- 4:15
But what persists pretty consistently is the diff between using the orange would be just the seeds with repetition, and then the green would be replacing some of those repeated tokens with, um, higher, like with a multi-mode reverse, or at least, uh, yeah, all of them are at least reducing the repetition.
- 4:31
Um, and so, uh, for rephrasing in particular, we did do the, you know, what everyone's doing with the, you know, generic kind of multi-mode, very scalable pipeline, but we also took it a step far and, uh, we did two other specialized pipelines, one from to go from raw code to code and text and one to go specifically
- 4:49
for STEM data, just because this is a very cheap, scalable pipeline. So it kind of, you kind of have to rely very heavily on the seed, and we push a little further on that for the STEM documents.
- 5:00
Okay. So, uh, uh, okay. So, uh, if you kind of think of everything as a kind of in a, uh, modular way, uh, you can think of every synthetic data pipeline is composed of, uh, the same six components.
- 5:14
And so you have your seeds, your primary inputs, your metadata, your secondary inputs, your generator function, which can be an agent with tools or an LLM, you know, with some prompt funk- with some prompt template, and then some supplementary functions like filters and validators and so on.
- 5:26
And really, you can compose just about all pipelines from like very simple to very ex- to very expensive pipelines like this. And on, on that note, and kind of, uh, we've, we have covered quite a bit of a wide scope o- on the, the axis of complexity, and you kind of can think of it if at one
- 5:40
end you have like the cheap, scalable pipelines that have used smaller models and can get away with it be- because they're seed heavy, um, examples of phrasing. And then on the other end, you have more complex pipelines with, uh, a little more orchestration in the workflows.
- 5:54
Um, this is reserved when the s- like we're building on something that's worth it- Educational data, uh, but really it's, uh, this is how we're not blocked or limited by whatever teacher model can do, and this is how, um, we can be ambitious in our synthetic data.
- 6:09
Uh, because the rule of thumb is if a task is too hard for your model, then your model will start to fall on its bias, lose correctness, lose diversity.
- 6:17
So break down the task, make it simpler. Um, and, uh, uh, yeah. Uh, I will give some examples of kind of shapes, uh, rather than just like, uh, something more concrete about how do we use this, uh, this modularity.
- 6:32
And one shape is the form rewriting, it's just rephrasing. We already talked about this. Multi-stage, uh, pipelines and multi-stage, uh, workflows. Basically, this is, uh, what I also just said.
- 6:41
You take a step and you break it down into multiple steps. Uh, you can, uh, aggregate, uh, the process and it- it slowly build up the, the generation. Um, an example of this where if you wanted to generate a novel, for example, it's, uh, you could generate one chapter at a time, but you could also, you know,
- 6:58
take it a little slowly. One first like generate, you know, uh, I don't know, the setting, the character names, the, the character styles, the plot, um, you know, some twists.
- 7:07
And then, and then from there go into generating chapters one by one. You will absolutely get a better novel. Um, and okay. Third is cross-domain porting, which really is just like trans-- like moving from one mode to another.
- 7:20
Uh, example would be chang- like translating code. Another example would be, uh, something we did, which is take our math problems and convert them to code. The last one is, um, multi-turn roleplay.
- 7:29
And by that, all I mean is that instead of having kind of a very one, like very senior or, or linear, uh, or even non-linear kind of, uh, view of things, you have more of an iteration.
- 7:38
This encapsulates pretty much everything. Uh, and by that, uh, uh, like an example of that would be multi-turn chats when you have two agents talking to each other, or a task evolution pipeline where, you know, you have a judge and an evolver going back and forth for some K amount of time.
- 7:52
Um, and so on. Okay. So lastly, uh, forwarding off to Robert, I do wanna kind of just mention that we, because of the modularity of the way we think about this, we can implement infrastructure that's pretty configurable.
- 8:04
Um, so this is how we present Hive. Hive is basically a way for us to easily, um, that, uh, to easily construct, uh, generations where now you have, uh, a queue of agents that you define.
- 8:16
Each one has, you know, its prompt, its parameters, its model, et cetera, inputs, outputs. But it also has when, uh, you can configure when it enters the queue, when it exits, how many, how frequently they come in.
- 8:25
And then we have orchestration in the middle between agents. And with this orchestrators are really useful because they give you more flexibility and kind of presenting a hierarchy between the LLMs that we're generating is very, uh, th- th- it's, this is how you police them basically, but also give them some, some form of creativity and dynamically change
- 8:41
the instructions for the next agent, or choose which agent goes next, which agent skipped, and so on. Um, and lastly, you have the supervisor, which basically someone who polices the orchestrator and has more of a global view.
- 8:53
Um, cool. Okay. That's, uh, that's it for me and Sandiga. Hope you learned something interesting. Handing over to Robert for, uh, pre-training scale.
- 9:03
All right. Thank you, Marah. Um, 'cause I will talk a little bit more on the, um, actual pre-training side rather than just data. Uh, I liked in the previous talk, um, the, the speaker made a point that we should treat different data mixes, uh, holistically, different training stages.
- 9:19
I want to make the same point that we should treat, uh, data and implementation of your training code base, uh, correctness of it and so on, also holistically. Uh, if you've got data that sucks, you can't train a good model.
- 9:30
If you've got a training code base that sucks, you also can't. Um, so, uh, I specifically focus on architecture work on, and distributed training and so on. Um, and the way we, we look at things in my team is, uh, we don't trust anything.
- 9:43
There are so many things that can go wrong when you scale models to billions of parameters, to hundreds of billions of parameters, um, train on thousands of GPUs and so on.
- 9:52
And I want to show you some of the learnings that we got from training Laguna M.1. Um, and yeah, some of the surprising things that happen at scale. Um, so one thing we do is we've got these, um, model replica hashtags.
- 10:04
So essentially when we train a model, we've got multiple replicas of the same model, right? Um, distributed data parallel. And, uh, we know there's an invariant. The weights should always be the same across all of these replicas.
- 10:14
That's something you can verify, right? We-- You can calculate a hash over the weights, and you know that should always be the same across all replicas. So we do that in training and periodically compare them.
- 10:22
Um, if all of these hashes are identical, then we know we can continue training. If they're not identical, we know something has gone seriously wrong, uh, because that should never happen, and we crash the training.
- 10:31
And I will give you some examples now of things that we've not shared before publicly like this. Um, so I hope they're interesting. Um, so first example here of, uh, shit that happens at scale are broken GPUs.
- 10:43
On the left-hand side, we've got two loss curves. Uh, and on the right-hand side, the corresponding, uh, gradient norms, um, that we observe during training. And you can see that these loss curves look quite different, right?
- 10:52
Like the purple one has got quite some bumps, uh, looks a bit spiky. The gradient norms are huge for that run. And there's actually no difference, uh, in terms of model configuration, training data, training implementation, um, between these runs.
- 11:05
They're exactly the same run. Just in one of them, we got unlucky and we had a broken GPU included. That broken GPU caused silent data corruption and, um, therefore made the training behave the way it did.
- 11:16
Uh, and that is one of those cases that you can catch with these hashtags because you know this computation should be the same across all replicas, but it wasn't.
- 11:24
Um, which brings me to the next, uh, instance of shit that happens at scale. Uh, in this case, exploding gradients. Um, again, we're looking at two different loss curves and the corresponding gradient norm curves.
- 11:36
Um, the purple run is our initial training run for Laguna M.1. We're a bit further into training here, around fifty thousand steps or so. And you can see it stops converging, right?
- 11:43
Like it just flattens out. And the reason here was that during training, the, uh, activations grew and grew, um, right before the, um, LM head, the un-embedding. And, um, we have to perform some sort of accumulation here, uh, because we use Tensor Parallel for the, uh, un-embedding.
- 12:00
And that accumulation, um, was performed in BF16 by default. And because of the growing scale that we observed in the activations, um, there wasn't enough, um, numerical precision available anymore, um, to do this accurately, and hence the model just couldn't learn anymore.
- 12:15
And this is also very, um, dramatic point for this to happen because it-- from there on it really, like, back propagates into the full model trunk. Um, the orange curve is essentially just adding a fix on that.
- 12:25
So we took the checkpoint from the purple curve. We moved that accumulation into FP32, and then from there on, the model started converting again. Um, the gradient norms, you can see, actually started decreasing.
- 12:35
Um, before then we had an increasing trend and, um, this is also something you can only observe at scale, and that will break your model if you, if you're not careful about it.
- 12:44
So as Marah said, we took all of these insights on data, on, um, numerics and so on, and we turned them from M.1 into XS.2. That's why we say it's a newer generation model.
- 12:53
Um, this included, uh, increasing diversity, reducing data repetitions, um, all of these numerical things I just mentioned, and just adding more observability and checks on that side, as well as generally optimizing the training and the architecture.
- 13:06
And XS.2, if you look at it, it's open weights, right? So you can download, uh, can download and use it for free. Um, it's one of the most competitive models for its size and for coding specifically.
- 13:15
That's what we focus on, agentic coding. Um, so we're pretty happy with it. However, you can say that, um, the model with thirty-three billion parameters is pretty small. Like, you wouldn't probably observe any issues anyways training it.
- 13:27
So what was important to us was to scale this, right? And this is where Laguna S comes in. This model is not public yet, so this is a preview.
- 13:34
Um, as I said, we treated XS as a test bed in a sense and, uh, with Laguna S we scaled this to a model that's a hundred and eighteen billion, uh, total parameters and eight B active parameters.
- 13:45
Again, we trained it on thirty trillion tokens on four thousand GPUs. So the scale was sufficient to not only test whether, uh, all the improvements we made on the data and architecture side ha- hold, but also if any of these numerical issues come up again.
- 13:59
And, um, of course, something happened. Um, in this case, doesn't actually have anything to do with scale, so it was just unfortunate. Um, in this case, we had a race condition because we added FP8 training, um, based on DeepGem FP8 kernels that are also, like, open source.
- 14:13
Um, we noticed this because we had illegal memory accesses as well as NaNs in the gradients, uh, which after a while of debugging we traced back to those kernels.
- 14:21
Um, there's also an unobservable, unobservable effect, um, that you wouldn't know about if you don't know that there's an issue. Uh, in our case, we noticed about point five percent of the gradient gets silently corrupted, essentially replaced by random values.
- 14:33
Um, we do have a fix available that's, uh, in a PR right now. Uh, it's not been merged into DeepGem yet, but it's public, uh, on that QR code if anyone's interested.
- 14:41
Um, and it's also an interesting point because it's a blind jack in-- uh, blind spot in the hashtags. In real training runs, you don't have any redundancy where you have the same model weights and the same data, so you can never check if forward and backward actually behave the same across different model replicas.
- 14:54
So you can also never check if there's a race condition in that. Uh, that's something that we're working on right now, uh, to essentially have a hash checker, um, that can also do that as a, at least as a dry run.
- 15:04
Um, and I want to end on, um, some early results from, uh, this new model, um, and demonstrating how it performs against some open weight models and also against our previous models.
- 15:15
So first I want to caveat this with these are base model evals, right? They are partly indicative of how the final model will look, but also not perfectly, right?
- 15:23
Um, there's still post-training happening. Uh, not all of these will translate one-to-one to the final model. Um, but if we look at them, uh, specifically on the coding part of, of the evals, so for instance, MultipleE, LargeCodeBench, BigCodeBench, uh, Laguna S is not only stronger than XS.2, which is our previous, uh, smaller model that performed very well,
- 15:42
but also then the much larger M.1. Um, and it's also mu- much better than GLM four point five-R, which is admittedly a bit older. Nemotron 3 Super, which is quite recent, and then DeepSeek V four Flash Max, which is quite recent and a fair bit larger.
- 15:55
Um, we can see it's competitive on BigBench-Hard, for instance. It doesn't achieve the, the top eval results compared to these models, but it's quite close. Um, we also see it's quite close on Eval Plus.
- 16:06
And, um, quite importantly for us, it does very well on Speed Bench Agentless Multilingual, um, which we use to sort of proxy agentic performance during pre-training. Um, and in that case, it performs much better than all the other m-- uh, all other models we tested here.
- 16:21
Um, I also want to point out that, of course, there are-- like, it's not the strongest model in the world, right? Like for instance, MMLU Pro, a knowledge benchmark, is something we don't care about that much compared to coding because we want to build the strongest agentic coding models.
- 16:34
Um, so here, like compared to Nemotron and DeepSeek, uh, we have to say that they perform much better and this mainly comes down due to data, right? It's a data gap, um, that we could plug if we wanted to.
- 16:44
Um, but I think the, the point is all of the things we've, we've found before were included in the recipe. The recipe held, it scaled, and we will continue scaling it from here.
- 16:53
So this model will also be available sometime in the future, relatively soon. Um, again, open weights, so all of you can download it and use it for free. And, um, with that, I want to thank everyone for attending our talk.
- 17:06
Um, I added also a link to our careers page and our Twitter page if you want to, uh, check it out. And yeah, thank you very much. [applause] [outro jingle]