Model-Maxxing: RFT, DPO, SFT (Fine-tuning with OpenAI) — Ilan Bigio, OpenAI

What do SFT, DPO, and RFT actually do—and will fine-tuning solve the problems that prompting has left behind? Those are the opening questions of Ilan Bigio’s workshop. Working on OpenAI’s developer-experience team, Bigio tests early API features and builds demonstrations around them. Here, the experiments are partly prepared in advance: training jobs take too long to make watching them finish a useful live demonstration.

An LLM application has three places to improve: the input, including instructions and context; the model, including model choice and weights; and the surrounding system, including tools and orchestration. Fine-tuning addresses the middle component through continued training for a particular domain or behavior. It does not replace the other two.

Bigio compares prompting to hand tools and fine-tuning to a CNC machine. Prompting has a low entry cost, supports quick adjustments, and is sufficient for many jobs. Fine-tuning requires clean data, preparation, and longer iteration cycles. Once a training pipeline exists, however, more of the improvement process can be automated. It is a specialized investment, worthwhile when the task benefits from repeatable learned behavior or from capabilities that prompting alone cannot elicit.

Supervised fine-tuning (SFT) teaches imitation. Direct preference optimization (DPO) teaches a preference between alternatives. Reinforcement fine-tuning (RFT) rewards generated answers and reinforces behavior that produces better results. Bigio connects the last approach to the reasoning-model training behind o1, o3, and o4-mini.

SFT’s mapping is learned rather than a literal lookup table. DPO’s difference-vector analogy is useful because it directs attention to what distinguishes the two answers. RFT lets the model generate its own reasoning and answers, then uses the grader’s assessment to reinforce successful behavior.

• SFT: constrain the output. Classification, formatting, structured extraction, and distillation into a smaller model are natural fits. The desired response is usually concrete enough to demonstrate directly.
• DPO: express a preference. Tone and style can be difficult to score in isolation but easy to compare side by side. A/B choices provide a direct signal about which response a user prefers.
• RFT: solve a gradable problem. Medical, legal, and coding tasks can be hard to solve while having comparatively straightforward verification. Another candidate is an LLM judge trained against trusted grading results.

The distinction is the supervision available, not merely the subject matter of the application.

For substantial new information, Bigio favors retrieval: RAG or an agent searching relevant sources. He describes the hosted training methods as adapting a subset of weights through low-rank adaptation, using a small pair of matrices to represent an update instead of readjusting every base weight. The useful distinction is between changing behavior and supplying reference information. Preserving the base weights does not, by itself, guarantee that every existing behavior remains unchanged.

The first customer problem makes the tradeoff concrete. An assistant needed to turn user commands into actions under a strict latency constraint. GPT-3.5 was fast but insufficiently accurate; GPT-4 was accurate enough but too slow. There was no substantial labeled dataset ready for training. The customer also wanted 120 functions—far beyond the five to ten that Bigio recalls researchers initially suggesting—and supporting training with that many functions required platform work.

The first data pipeline started from the function schemas and worked backward:

1. Enumerate valid invocations. A Python script generated combinations of parameters. A light-setting function with an enum produced calls for off, on, and red. A song-search function introduced multiple parameters and combinations.
2. Control combinatorial growth. When the number of combinations became too large, sample from them. For open-ended values, ask a model for plausible examples, such as an artist name.
3. Generate the missing command. Give GPT-4 a function call and ask for a user command that would lead to it.
4. Reverse the pair for training. The generated command becomes the input; the known function call becomes the target output.

This creates broad coverage of the action space, but it does not establish that the generated commands resemble what users will actually say.

The second pipeline started with unlabeled real user inputs. Since GPT-4 already performed adequately, the team used it to produce target function calls, then filtered the results through additional stages. These labels were still synthetic, but the inputs came from the deployment distribution. Combining this dataset with the schema-generated pairs supplied both action coverage and more realistic language. The final task remained tightly constrained: choose one function and populate its arguments, rather than produce an open-ended assistant personality.

Existing high-quality labels would have made the project simpler. Bigio sketches an ideal handoff of a thousand labeled examples instead of the synthetic-data work. His broader SFT guidance is to look for an initial signal with roughly 50–100 examples and expand toward several hundred when the task warrants it. Those are starting heuristics, not the size of this customer dataset. He does not remember the mixture ratio, but recalls that ablations favored using both sources together.

For the customer case, Bigio recalls generating roughly 20–200 examples per function depending on its permutations, with total dataset estimates reaching the low-to-mid thousands. Both function selection and argument population improved. Bigio recalls the tuned function caller reaching within 2% of GPT-4’s performance at a small fraction of its latency. He supplies no exact latency measurement or detailed benchmark specification for that comparison.

The team also tried removing functions and parameter details from context to save latency. That made the task harder to learn and hurt results. Bigio treats cutting input context for latency as a late-stage optimization, after more promising options have been exhausted. Fine-tuning did not make the function definitions dispensable.

Adding functions required more training in this setup. Continuing from an already fine-tuned model was possible, but adapting it with only a few new examples was a plausible next experiment rather than a demonstrated maintenance strategy. The team could remove much of the prose instruction and retain a simple classification request, yet still needed the functions, parameter types, and other schema details. The historical context window was small enough that moving from a 4K model to a 16K fine-tuning setup mattered considerably.

Bigio points to Fine tuning for function calling as the companion notebook for the approach. It provides a place to explore the workflow beyond the customer story.

The banking demonstration starts with a record containing text such as “I'm still waiting on my card” and a numeric label that maps into a list of intent names. The classifier must distinguish labels such as card arrival, card activation, and card not working. Bigio gives it a deliberately minimal instruction and the label list. That makes the task easy to state, but the neighboring categories still require judgment.

SFT can learn a constrained task from demonstrations even when the prompt is sparse, though Bigio recommends retaining a decent instruction. DPO and RFT need good prompts because the model must already understand what it is supposed to attempt. The notebook uses Chat Completions messages: instructions followed by the user’s text. A request about locating a card sometimes receives the expected arrival label and sometimes an inappropriate alternative.

Whether to fine-tune is an evaluation question. The notebook’s evaluator accepts a model, prompt, sample set, and worker count. It limits concurrent requests with a semaphore and scores each prediction by exact equality with the correct label. The following Python expresses that same evaluation structure:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def classify_intent(model, prompt, text):
    response = await client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": text},
        ],
    )
    return response.choices[0].message.content or ""

async def evaluate(model, prompt, samples, label_names, workers=8):
    if not samples:
        raise ValueError("Evaluation requires at least one sample")
    semaphore = asyncio.Semaphore(workers)

    async def score(sample):
        async with semaphore:
            prediction = await classify_intent(
                model, prompt, sample["text"]
            )
        expected = label_names[sample["label"]]
        return prediction == expected

    scores = await asyncio.gather(*(score(sample) for sample in samples))
    return sum(scores) / len(scores)

The semaphore bounds concurrency; choosing its limit still depends on the available API capacity. Exact matching also makes output discipline part of the measured task.

On 150 banking test examples with the workshop’s minimal prompt, Bigio reports 75% accuracy for GPT-4o mini and 83% for GPT-4o. These establish the notebook’s baseline before training, rather than the ceiling obtainable through prompt engineering.

The training file uses JSONL: one JSON object per line. Each record contains a messages array with instructions, user input, and the desired assistant answer. In this single-answer demonstration, the final assistant message is the training target. The same pattern can represent a function-call target rather than ordinary text.

The job-creation procedure has a few explicit dependencies:

1. Serialize the examples. Write training and validation records in the expected message format.
2. Upload the files. Use client.files.create with purpose="fine-tune"; the job references the returned file IDs.
3. Select the base snapshot. The demonstration requires the full model snapshot identifier rather than an alias.
4. Create the job. Supply the training file, validation file, and training settings such as the epoch count.
5. Wait for completion and retrieve the model ID. Check that the job succeeded, then use its resulting identifier for evaluation and inference.

The API examples and support boundaries here follow the recording’s workflow; model availability and platform capabilities are version-dependent. A validation file provides diagnostic measurements, not an automatic guarantee against overfitting. Bigio had already completed the jobs rather than waiting through training live.

He trained GPT-4o and GPT-4o mini with both 150 and 500 examples, then compared the results with base and reasoning models. The displayed comparison motivates his enthusiasm for task specialization: the stronger fine-tuned runs outperform the lightly prompted baselines, and he highlights GPT-4o mini trained on 500 examples. He also acknowledges that more prompt engineering might have improved the comparison models. The result shows the convenience of applying available labeled data, not that prompting had been exhausted.

The audience first asks whether training stops automatically when it overfits. In the workflow Bigio describes, it does not: inspect the loss diagrams, stop an unhelpful run, and adjust the next attempt. He is tentative about whether a latest snapshot remains usable after cancellation. The decision to train is similarly operational: continue prompt iteration while evaluations improve, and start fine-tuning when the remaining gap and the available clean data justify the investment.

Model support is something to check in the model documentation, not infer from a release date. The banking experiment did not include a hyperparameter search. Learning rates and other settings can matter, but Bigio offers no universal tuning recipe; their effects need task-specific experiments.

Changing the prompt after training creates a distribution shift. Bigio recommends using the training prompt at inference, retraining when it changes, or deliberately including multiple prompts in the training set. Then an audience member notices something more consequential: the GPT-4o run trained on 150 examples appears to have regressed. Bigio cannot explain it from the available view and will not expose the internal organization’s job details on screen. The anomaly remains unresolved.

An older job provides a view of the monitoring interface, including a validation-loss curve. It illustrates where to investigate, not the cause of the banking regression. SFT can also regress on tasks absent from its training data, making coverage and diversity important even when the target task improves. Bigio’s closing heuristic is to start around a hundred examples and expand toward five hundred or more, while keeping the task constrained and measuring the behavior that must be preserved.

Humor offers an intuitive DPO task: it is hard to assign a reliable absolute funniness score, but often easier to prefer one joke over another. Bigio generated candidates with GPT-4.5, then supplemented mediocre validation with manual filtering based on whether a joke made him laugh—or at least exhale more noticeably. The resulting dataset was only as good as those judgments.

He again worked backward from the desired output. For each selected joke, he extracted its topic, then asked GPT-4o to generate a corny joke about that same topic. This yielded a prompt, a preferred answer, and a non-preferred answer. Holding the topic constant helps isolate the preference. Pairing an amusing cat joke with a dull joke about something unrelated could accidentally teach a preference for cats instead of humor.

One output was the familiar story of two hunters: one collapses, the other calls emergency services, and an instruction to make sure the friend is dead is followed by a gunshot and a request for the next step. The room recognizes the joke. It demonstrates eliciting a joke Bigio finds funny, not creating original comedy. That narrower result still matters to him: getting a model to produce a good joke on demand had been difficult.

The training record contains the user’s topic request, the preferred output, and the non-preferred output. Bigio expects this preference signal to transfer more flexibly than SFT: a model trained on joke comparisons might become funnier without turning every requested speech into a joke. Evaluation still needs people or a useful natural preference signal. DPO can also follow SFT, adding a preference adjustment after learning the basic response behavior.

RFT changes where the successful output comes from. The model receives an input, generates reasoning and an answer, and has that answer graded. Training reinforces behavior associated with higher scores. Bigio presents this as a route to very strong task-specific performance, but its requirements are demanding: a useful reward must distinguish genuine progress from noise.

A practical test of an unambiguous rubric is whether competent people applying it independently reach the same answer. Incorrect labels can become amplified training signals, and the base model must already be capable of reaching the solution sometimes. RFT is teaching it how to arrive there more reliably, not supplying an entire missing body of knowledge. Bigio suggests that even 40–80 examples can be useful when they are exceptionally clean and informative; the quality of that signal is the essential condition.

The Python grader runs in a sandbox without internet access and with a limited set of preloaded libraries. Multi-graders can nest other graders and combine their results with a specified function. The flexibility lies in defining success precisely enough that maximizing the score improves the actual task.

Bigio’s email goal is to decide which messages deserve his attention before he has to read them. He built a labeling interface and labeled about six hundred emails with actions: glance, ignore, archive receipts, store for later, respond within a time frame, or take action within a time frame. He did not end up using the topic labels.

Each input used an XML-like wrapper containing sender, subject, date, and a plain-text body. Training records included messages plus reference-answer fields available to the grader. The grader itself was simple: compare the generated action with the reference action using string equality.

The result was disappointing because the labels encoded unstable personal judgments. Bigio doubts he would assign exactly the same actions if he labeled the emails again. He suspects a small number of labeling mistakes, and his weak prompt left important preferences unstated. The run used 150 examples, but he sees ambiguity and low signal as the central problem, rather than the count alone.

His analogy is a person locked in a room, allowed to take notes, but never properly told what the task owner wants. The model receives feedback without enough explanation to infer the rule. String equality is mechanically clear while the meaning of the target label is not. A small action set also permits occasional correct guesses, weakening the connection between reward and useful reasoning.

The chemistry example has a more verifiable target: given a chemical representation, predict the number of hydrogen-bond donors and acceptors. The response contains two numeric fields, donors and acceptors. A multi-grader awards half the total score for each correct count, giving partial credit when only one field is right.

The scoring rule can be expressed directly in Python:

def grade_counts(prediction, reference):
    donor_score = float(prediction["donors"] == reference["donors"])
    acceptor_score = float(
        prediction["acceptors"] == reference["acceptors"]
    )
    return 0.5 * donor_score + 0.5 * acceptor_score

The output schema constrains the response shape; the grader supplies correctness feedback for its two parts. This is a richer signal than requiring both fields to be correct before awarding any credit.

Bigio reports o4-mini’s displayed chemistry performance rising from roughly 65% toward 80%; he does not establish whether those percentages measure joint correctness or the half-credit aggregate score. He also notes that the task still requires chemistry knowledge and that he is not a domain expert. A well-defined answer makes this a better RFT candidate than his subjective email labels, but the base model’s knowledge still constrains what it can learn to solve.

The available training families impose another boundary: the workshop’s SFT and DPO offerings target non-reasoning models, while RFT targets reasoning models, so the demonstrated products cannot simply be chained across those families. RFT also evaluates a single turn. It cannot yet train the long trajectory of an agent calling tools, observing results, and continuing toward a final outcome.

The extended discussion first distinguishes learning a reasoning trace from learning through a reasoning outcome. SFT can teach a model to imitate a demonstrated thought process, including its surface structure, whether or not that process is useful to the model. RFT instead scores answers reached through the model’s own generated reasoning. Successful outcomes provide the feedback, although a correct result alone cannot certify every intermediate thought.

An audience member asks about grading generated Triton code by checking whether it runs. Bigio sketches a Python grader executing generated text with eval or exec, but explicitly warns against treating that hack as a safe pattern. Execution would also depend on the sandbox’s libraries and lack of internet access; parsing as valid Python is not the same as successfully running a Triton kernel. Asked whether reward components should sum to one or one hundred, he gives a qualified answer without resolving the effect of scale. The workshop supplies no reward-normalization rule to rely on.

For agents, the missing piece is the interaction loop. A model can emit a function call, but the described RFT rollout has no mechanism to receive its result and continue. End-to-end training would need that repeated model–tool exchange. Within a single turn, however, a more informative grader may help: the audience proposes SMILES validity and chemistry-specific checks. Additional checks are valuable when they produce a more accurate definition of success, because training maximizes what the grader rewards.

Bigio does not disclose the exact RFT algorithm. He describes it as closely related to the approach used for OpenAI’s reasoning models, while noting that proprietary examples and limited access to high-quality personal datasets constrain what he can demonstrate. His adoption advice remains grounded in an existing system: use prompting while it meets requirements, then consider training when evaluations expose a remaining gap and the data is ready. The upfront cost can be justified for demanding specialist tasks.

Asked what happens during hosted inference, Bigio describes a new LoRA component used alongside unchanged base weights. The service loads that adapted component when serving the fine-tuned model. This is his account of the hosted system, not a procedure for exporting the weights or running them independently.

The DPO discussion returns to the design of negative examples. If the desired improvement is mediocre-to-excellent, an obviously terrible negative may teach the wrong contrast. Pairs should resemble the behavior the model actually produces and isolate the change the user wants. Bigio regards real A/B preferences as a more direct signal than synthetic negatives whose quality depends on the dataset creator’s intuition. He considered having the audience label pairs through a website, but the resulting training job could not have finished during the session.

A request for output containing exactly 30% emojis exposes the difference between product support and theoretical possibility. Bigio thinks DPO might approximate such a preference but gives no exact guarantee. RFT on non-reasoning models is unavailable because the platform does not support it, not because the discussion establishes that reinforcement learning is impossible for those models. Custom GPTs, he adds, use configured prompts rather than these fine-tuning methods.

Directly rewarding the contents of a chain of thought introduces a different risk. Bigio invokes OpenAI research on models learning to hide undesirable behavior when their thoughts are penalized. The relevant concern is monitorability: optimization can suppress the visible evidence without reliably eliminating the underlying behavior. This is the issue examined through reward hacking in agentic coding environments in Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. Bigio favors placing pressure on outcomes rather than policing thought content; that should not be read as a guarantee that an unpenalized chain of thought is fully faithful.

Model choice should follow the required behavior. If an open model meets the task, Bigio sees no reason not to use it. The practical comparison is the work of tuning and operating that model versus simply prompting a capable hosted model. His sequence is to first make the product work with a sufficiently strong model, then optimize size, cost, and efficiency. When asked why o3 can score better yet hallucinate more than o1 in a cited system-card comparison, he offers no causal explanation and says the behavior is being investigated.

The same sequence helps answer where an RFT reward should come from. If a working application already has a meaningful evaluation, it already has at least part of the definition of success. A validated LLM judge can become a reward function, and existing evaluation examples can become training material. Bigio presents that as one path rather than a mandatory architecture. He credits Teo’s Exploring Model Graders for Reinforcement Fine-Tuning as a companion treatment of the grader-design work.

The modality question adds image inputs to the discussion. Bigio mentions image fine-tuning and describes text outputs, such as bounding-box information, as a use case. For personal and domain knowledge, however, he again starts with context, RAG, and search. Fine-tuning can memorize some information, but he frames its stronger role as teaching procedures and formatting rather than maintaining a large, correct, referenceable knowledge store.

A retrieval component provides a useful middle ground. With hypothetical document embeddings, a model transforms a user query into text resembling the documents the system wants to retrieve, then that text is embedded for search. Bigio suggests fine-tuning the transformation on actual knowledge-base outputs so the generated text better matches the corpus. The generated passage helps find information; it is not itself authoritative evidence. For a reasoning model, another option is to provide search tools. Searching and then interpreting results requires multiple steps, so the workshop’s single-turn RFT interface cannot train that whole interaction end to end.

The email failure was a sincere attempt, not a deliberately bad example. Reconsidering it, Bigio describes a subtler problem: two apparently similar emails can receive different actions because of a rare, unstated preference. Instead of forcing training to discover that preference from a noisy label, perhaps a reasoning model could explain the difference and write it down.

The proposed retrieval step is modest. Use ordinary embeddings to find emails similar in content but carrying different labels. Then ask a reasoning model what consequential difference could explain those labels. The embeddings do not need to encode Bigio’s preferences; their job is to find candidate contrasts. The reasoning model’s proposed principles would become additional prompt context.

The live notebook starts from the personal email dataset and a generalized eval_model helper accepting a run function, an evaluation function, a dataset, and a worker count. The baseline results are tightly clustered and weak: GPT-4o mini is reported at 58%, while the other spoken scores fall between 51% and 57%. Bigio interprets the pattern as evidence that the prompt exposes little usable preference signal.

He proposes a text-based prompt-optimization loop:

1. Split a mini-batch into training and test subsets.
2. Run the classifier on the training subset.
3. Give the predictions and expected actions to a reasoning model.
4. Ask for generalizable insights and append them to the instructions.
5. Evaluate the revised prompt on the test subset, then continue experimenting.

Here, prompt tuning means editing an explicit notes section, not learning a soft prompt or updating model weights. The procedure is improvised, and whether it helps remains to be measured.

The implementation begins by splitting samples that contain action and text, then retaining each original sample alongside its model output. Bigio initially tries a mini-batch of twenty and reduces it to ten while the calls remain serial. Notebook top-level await makes the experiment convenient, though parallel execution would shorten the wait. Inspection reveals blank email inputs, introducing a data problem before the optimization logic is even complete.

Next comes extract_insights, an o3 call over the batch of results. Its job is to identify what went wrong and return notes that apply to emails outside the current batch. Bigio asks for deduplication and explicitly discourages overfitting, retaining a plain string rather than a JSON array. These are proposed instructions for the analysis step; asking for generalization does not establish that the resulting notes generalize.

The first output is too long, so Bigio uses it as a negative example while revising the extraction prompt. He also questions whether he has reused the same examples. The demonstration is still assembling the experiment: a forward pass produces records, an analysis call proposes notes, and those notes must reach the next classifier invocation before any improvement can be attributed to them.

A small evaluation prints 0.6, after which Bigio adds training and test results to the output to inspect possible overfitting. He also proposes experimenting with both subsets, so the evolving live run cannot support a clean held-out improvement claim. He constructs the next model invocation from the original instructions plus initially empty notes. Then the audience catches a crucial omission: the prompt is not actually being updated. Assigning the extracted notes fixes that missing connection.

Printing the notes makes the evolving state visible. Within each mini-batch, the implementation replaces them; Bigio proposes later appending notes across batches or aggregating them in a tree. Another audience observation catches a return inside the loop that ends execution prematurely. These are ordinary program-state errors with direct consequences for the experiment: extracting a useful insight accomplishes nothing if it never reaches a subsequent call, and a stopped loop cannot demonstrate iterative improvement.

The final result is explicit: the numbers are not moving significantly, and Bigio stops the experiment. The notebook displays scores and generated notes, but it has not demonstrated a successful optimization method. Asked for dependable prompt-engineering advice, he returns to clear instructions without contradictions and useful few-shot examples. The proposed loop remains a skeleton to investigate, not a successful recipe established by this run.

The final explanation returns to what the notes were supposed to contain: explicit observations about unintuitive classifications, derived from successful and failed predictions paired with expected actions. A package-delivery email exposes the difficulty. Bigio considers glancing at it, then notices that he had labeled it ignore, while also discussing why archive might be wrong. The dataset supplies an action without explaining the preference behind it.

That ambiguity weakened RFT, but a reasoning model might still help inspect it. Give the model enough successes and failures, and ask it to propose explicit preferences or techniques for the next classification attempt. An audience member extends the idea: perhaps the analysis could identify missing inputs, such as when focused work happens and interruptions matter.

Bigio agrees that discovering missing information is useful whether or not fine-tuning follows. An error-analysis pass can show where labels need explanations, where context is absent, or where the task owner’s own policy is inconsistent. The workshop ends with two failed experiments acknowledged openly. Their practical value is in making the next problem clearer: before optimizing a model against a signal, establish that the signal expresses a decision the model has enough information to make.