AI Engineer Europe 2026
How We Built Zeta2: Training an Edit Prediction Model in Production — Ben Kunkle, Zed
Read the talk
How Zed trains Zeta 2 to predict your next edit
Ben Kunkle follows Zed’s training pipeline from production snapshots and repaired teacher predictions to cheaper data filtering and live editor experiments.
From a talk by Ben Kunkle
An edit prediction on every keystroke
An edit predictor looks at the code around your cursor and predicts the next change you will make. That is the task behind Zeta 2, the model whose training process Ben Kunkle, edit predictions lead at Zed, walks through here. Its immediate task is to propose an edit within a supplied region of code.
The editor can supply more than the current text. Recent edits reveal what you have been doing; the cursor identifies where you are working; nearby type and variable definitions help explain the code. Diagnostics and errors provide additional clues. But the prediction runs on every keystroke, so it must return quickly enough to remain useful while you type. That combination of rich context, a narrow task and frequent invocation motivates a small, specialized, fine-tuned model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn production snapshots into checked teaching targets
Training starts with opt-in production snapshots. These preserve the context available when a prediction was requested, including the related definitions collected by the editor. Zed turns those snapshots into training examples through distillation: give a frontier model the captured input and ask it to produce an edit prediction. The frontier model acts as the teacher; Zeta 2 learns to produce the resulting target.
A capable teacher does not automatically produce consistent training data. Across many requests, its answers vary, and some fail the requirements of the edit-prediction task. The team refines the teacher prompt to obtain more useful predictions, then checks the generated output instead of assuming that every answer is suitable for training.
The checking and repair sequence addresses concrete failures:
- Run static heuristics to detect predictions that undo what the developer just typed.
- Check whether a prediction violates the editable-region boundary.
- Send a failing prediction to another frontier model with the specific failure described, asking for a corrected prediction.
This repair step turns a flawed teacher answer into a usable candidate target. After repair, the teacher’s edit becomes the expected output for the student.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cache the teaching work, vary the student input
The teacher and repair work is reusable across experiments. Zed caches those results before applying experiment-specific formatting. One experiment might include diagnostics; another might omit them or change how much edit history the student receives. The desired output format can also vary. These choices change how an example is presented for training without requiring the teacher to generate its answer again. Each experiment then trains its student and runs a final set of offline evaluations.
Change the input, keep the teaching signal
A small constructed record shows two experiments reusing the same checked-and-repaired teacher edit.
One cached training target
REUSEDtotal = price * quantityProduced by the teacher; checked and repaired before the experiments split.
Without diagnostics
Student input
{
"code": "total = price * quant"
}BASE INPUT · code around the cursor
Target same cached teacher edit
With diagnostics
Student input
{
"code": "total = price * quant",
"diagnostics": [
"quant is not defined"
]
}ADDED INPUT · diagnostic message
Target same cached teacher edit
The pipeline uses JSONL: each line contains one large JSON object representing an example. Successive stages add fields or rearrange existing ones, so the record can carry information through the whole process. The files become large, but the representation stays straightforward. Kunkle reports training sets of roughly 100,000 examples at the upper end, with smaller experiments using 10,000–50,000 examples.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use later code as evidence of intent
The developer eventually writes the code they want. That observation motivates an experiment with what Kunkle calls settled data: wait until editing stops in the region supplied to the model, then snapshot the resulting code. Because Zed is the editor, it can observe both the original prediction context and the later state of that same region.
The later state is not automatically a clean answer to the original prediction task. The developer may change direction, or an agent may rewrite the region entirely. A prediction that was reasonable when requested can look wrong against code produced after that change of intent. Training directly on every eventual result would mix useful examples with outcomes that were not predictable from the original input.
One proposed filter generates ten teacher predictions from the same original context, then compares them with the settled region using a measure such as Levenshtein distance. If a candidate is close, that provides evidence that the later code was reachable from the earlier context. But sampling makes the cost multiply: generating ten teacher predictions for each of 100,000 examples requires one million frontier-model requests. Kunkle describes that approach as prohibitively expensive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find useful near misses with the student
The student trained on the original teacher predictions offers a cheaper way to perform this filtering. Kunkle reports that its edit-prediction quality is approaching the teacher’s. For this filter, he describes sampling a student checkpoint about 50 times per example at much lower cost than repeatedly calling the frontier teacher. The comparison remains the same: look for a generated prediction close to the settled region.
The distances help distinguish different kinds of training material:
- Far from the settled code: the mismatch suggests noise or a change of intent.
- Very close: the continuation may already be obvious to the model. Kunkle’s example is an addition function where the second operand is easy to anticipate.
- Almost right: the intermediate cases can expose something useful to learn, such as an unfamiliar function introduced after the student’s training-data cutoff.
The interesting examples are therefore not simply those the model already reproduces most easily. Near misses can identify gaps between what the student knows and the code developers now want to write.
Even after filtering, Zed generally does not use the raw settled region as the training target, because it remains noisy. Instead, the team can train on the generated prediction closest to that region. The later code helps select a target without being treated as an unquestioned answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate more than one reasonable next edit
Offline evaluation uses a held-out test set, keeping the examples used to assess a model separate from those used to train it. That separation lets the team compare experiments without testing them on their own training material.
Zed’s deltaChrF compares character n-grams of different lengths. The original chrF paper provides background on character n-gram scoring, rather than a definition of Zed’s custom metric. Alongside similarity, the team tracks reversal ratio: predictions that undo exactly what the developer just typed. This measures a specific unwanted behavior that a general similarity score may not adequately expose.
Kunkle also points to production kept rate as another signal. For offline evaluation, there is a further complication: a context often permits several reasonable next edits. The team generally evaluates each held-out context against three distinct frontier-model predictions. A student answer close to one of those references can be useful even if it differs from the others.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy experiments and measure what survives
Offline scores do not necessarily correlate with what developers want in their editor. Zed therefore evaluates models through deployed experiments as well. The experiment page shows two live variants: one receives 15% of production traffic, and the other receives the remainder.
After deployment, the team can adjust an experiment’s traffic allocation, including from 15% to 20%, or promote it to the live model. A separate dashboard tracks acceptance rate and latency. Kunkle identifies Zeta 2 among the experiments managed through this process.
The team is also exploring signals that look beyond immediate acceptance:
- Kept rate compares the text after a prediction with the later settled region to assess how many characters were retained.
- Diagnostic-error counts compare the number of errors before and after a prediction.
These ask different questions. A suggestion can be accepted, subsequently edited, and affect diagnostics in ways that acceptance alone does not capture. Retention and error changes give the team additional ways to judge an edit’s usefulness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The stopping rule is ten seconds of inactivity
The closing audience question returns to a practical detail: how does the editor decide that a region has settled? A Git commit might indicate that the developer is satisfied with a block of code. Does Zed use that kind of signal?
The implementation Kunkle describes does not inspect Git commits. Zed snapshots the region after ten seconds without edits to that area. This is a rough collection heuristic, not a declaration that the developer has finished or approved the code. If they keep editing the same location without a ten-second pause, the region is not captured until such a pause occurs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The edit-prediction model's public weights and prompt format; the current card identifies a different checkpoint label from the talk's experiment screen.
The editor behind the production context collection and edit-prediction workflow; current source continues to evolve.
Further reading
- How We Developed Zeta2Article
Kunkle and Syvokon's companion account of teacher training, reversals, base-model selection and rollout.
Current instructions for trying edit predictions, including settings that have evolved since the recording.
Background on the character n-gram metric family, not the definition of Zed's custom deltaChrF score.
Read the complete timestamped transcript
- 0:00
[on hold music] I'm Ben Kunkle.
- 0:15
I'm the edit predictions lead at Zed. Um, we recently announced our models Zeta 2, and this is how we trained it. I'm gonna go through a lot. This is obviously a pretty short talk, so I'm gonna try and leave enough time for questions at the end.
- 0:27
But it's... If you're not familiar with training models, uh, it's gonna be a bit of a whirlwind tour. So if you're not familiar with edit prediction, it's essentially giving the model a region of code around the cursor, asking them to predict the next edit that you're gonna make.
- 0:41
We give it various data in, such as your recent edits, your cursor position, the type definitions and variable definitions or, of things around your cursor, as well as diagnostics, errors, et cetera.
- 0:52
It also needs to be very fast 'cause it runs on every keystroke, and so it's ideal for a small specialized model, fine-tuned. It can do this task and this task only.
- 1:02
Um, so that's what we've, we've done. So the pipeline, in essence, is taking these opt-in production data. Uh, this works really well 'cause it's snapshots, so all of that data that we have collected, related, like related types and definitions and et cetera, all of that gets captured, and then we're able to turn that into training data.
- 1:23
In order to do that, we use a process called distillation, where we take a frontier model, we give it all of that input, and we say, "What prediction would you make?"
- 1:31
This is a pretty difficult process as it turns out. Even though the frontier models are pretty smart, if you ask them 100,000 times, they're gonna give you 100,001 answers, right?
- 1:41
And so there's a bunch of problems there that we've had to like finely tune the prompt that we're giving that frontier model in order to get good things out.
- 1:51
One of the things we've done to try and get better, um, predictions to train off of is we run some offline or static evaluations. So, uh, we have some heuristics for, you know, is it just undoing what you just typed?
- 2:06
Is it ignoring that editable region boundary that we've given it, et cetera. And if it does, then we send it to another frontier model with a similar prompt like, "Hey, it failed in this way.
- 2:18
Can you fix it?" And so that, we call that the repair step, and then once we've repaired the bad predictions, then we can essentially turn what the teacher made into the expected output of the student model, or Zeta 2.
- 2:36
Um, up until that point is reusable across experiments. So this is stuff that we can cache. We can train multiple experiments on top of that by turning what the, uh, frontier model predicted into the format that we want the experiment to output.
- 2:53
And so that's the next piece, is this prompt formatting. This is experiment specific, i.e., are we including diagnostics this time? Are we not? How much of the edit history are we including?
- 3:04
Those are the kinds of experiments we're running. And so we'll turn what the teacher gave us into the prompt to, uh, distill and train our student model, and then we'll f- we'll do our, uh, final set of offline evaluations.
- 3:21
Um, the nice part about this whole process, we've designed it in such a way that it's all JSONL or a single line has a giant JSON object. These files get huge.
- 3:31
Um, but each stage just adds some more fields to it or moves some fields around, so it's a very like, uh, fluid and dynamic process. Um, we're generally doing 100,000 examples to train a model.
- 3:46
Like that's our peak. For the smaller experiments, we'll cut it down lower to 10 to 50K range.
- 3:53
Um, one interesting thing that we're trying right now is to use what we call settled data, which is the idea that eventually the user writes the answer, right? When you request a prediction as you're typing, eventually you're going to like write the code in the way that you wanted it.
- 4:11
And so we can wait, given that we're the editor, we can just wait until you stop editing that editable region that we gave the model, snapshot it and save it, um, and then use that to inform our training.
- 4:25
Uh, this is actually, this is very noisy because by waiting on the edit region to settle, you could change your mind. You could have an agent come in and rewrite it completely.
- 4:37
It could be completely different from what it looked like when the prediction was made. So what was maybe a reasonable prediction, it no longer looks reasonable. So we need some way to filter that out.
- 4:49
One way that we can do that is by having, generating 10 of the teacher predictions and seeing if any of them are close using like a Levenshtein s- distance type of thing.
- 5:01
See if any of those are close to the settled state, and if they are, we know a couple of things. We know it's predictable and that it's not noisy and, you know, completely different than what the input was, right?
- 5:14
'Cause we're giving the same input that we gave for the original prediction for this new prediction. Um, that turns out to be quite expensive. For 100,000 examples, you're then doing, you know, a million frontier model requests.
- 5:29
That is prohibitively expensive. Fortunately, given that we've now trained models using the original teacher predictions, uh, our student models, or Zeta 2, is approaching the teacher in terms of quality of prediction.
- 5:43
So instead of running the teacher, we can run our student checkpoint or something 50 times. That costs us basically nothing, and we can s- do the same process, see if any of them are close to the settled region using Levenshtein or something similar.
- 5:59
Um- This gives ideal training examples, right? 'Cause there's a, by looking at the range of distance to the settled state, there's a region that are super far away, we can be confident that that's just noise.
- 6:14
There's a con- there's a region that's super close, that's like, it's super obvious what you're gonna be doing, right? You, you typed function add a+. It's obviously b, right?
- 6:27
Um, but then there's this interesting section in the middle where it's almost. That's like the ideal, what we want in our training examples. For example, the stuff that's past the training data cutoff of our student model.
- 6:42
So new functions, et cetera, that it's never seen before that you actually wanted, and that's gonna show up in these new training examples that we can then, um, train off of.
- 6:52
We generally don't train off of the actual settled state, just because it's still noisy, but we can train off of, you know, what was closest to the settled state.
- 7:03
So to run those offline evals, we're running on a held-out test set, um, just making sure we're not training the model on the same stuff we're testing it on.
- 7:13
Um, delta chrF is our Levenshtein. Essentially it does a, like, N-gram comparison of various sizes of N. Um, and then we're tracking this reversal ratio, reversals being it's undoing exactly what you just typed.
- 7:28
Um, then we can also look at kept rate in production. Uh, we're gen- when we're evaling, we're generally running against three teacher predictions, 'cause a lot of these have no one right answer.
- 7:40
And so by generating three distinct answers that were all generated by a frontier model, we can be pretty sure that if it's close to one of those, it's a pretty good prediction.
- 7:54
So for our experiments, um, this is the training and production part of it. Uh, those evals that we have, um, don't necessarily correlate to what users actually want in their editor.
- 8:08
And so we have this page set up of our experiments. These are the two that are live right now. You can see over here, we've got this one being sampled at 15%, and that's gonna get the rest of production traffic.
- 8:22
And so we have a dashboard that I can't show you of the acceptance rate, latency, all of that kinda stuff for these experiments. But this is a page that we created so that we can, you know, once we've deployed it, set it to 15% of traffic, set it to 20, make it our, our live running model.
- 8:40
Um, so this v0211 Seed-Coder, this is what we released as Zeta 2, uh, last week.
- 8:48
Um, and so yeah, like I said, we have these dashboards for, um, the acceptance rate. Uh, we're trying new diagnostics right now, which is kept rate and diagnostic error counts, essentially comparing, for kept rate, comparing, um, what was the original text after the prediction, and then that settled state, and see how many characters between the prediction and
- 9:14
the settled state were kept. Um, for diagnostic error counts, it's pretty much exactly what you'd think. We snapshot how many errors there are before the prediction, how many there are after, and then we're then trying to use that to judge the quality of the model.
- 9:27
So that's it. I... There was a lot. Happy to answer, uh, questions. I think we have five or eight minutes left, so yeah.
- 9:37
Uh, you said it was very noisy to determine, like, the settled state. Are there any signals that you can share that you use, like a, like Git commit, for example, or-
- 9:43
Sorry, what was the algorithm?
- 9:45
Uh, so like you said that, uh, determining the settled state, like when the user has, is satisfied with that block-
- 9:51
Mm-hmm
- 9:51
... of code, for example, is very noisy. Are there any particular signals that you can use already that are useful, like for example, they Git committed something?
- 9:59
Sure, yeah.
- 10:00
Do you, do you look at any of that stuff?
- 10:00
Um, so we don't look at the Git commit. We could, but, uh, right now we just do, like, you stop editing that area for 10 seconds.
- 10:09
Gotcha.
- 10:09
And that, that serves as a rough enough heuristic that-
- 10:12
Okay
- 10:12
... um, so it's only in the cases where you are, like, consistently editing that location for longer, um, without pausing for 10 seconds that we wouldn't snapshot it.
- 10:23
Gotcha.
- 10:24
Um, yeah. Any other questions? All right. I guess you guys get your time back. [laughs] Thank you for coming. [audience applauding] [outro music]