Post-training for your own task: Comparing SFT, distillation, and RL

Post-training adapts a completed language model to your task. There are three basic methods. SFT teaches the model format and behavior from sample answers. Distillation transfers capabilities from a larger model, making the small model more capable. RL makes existing capabilities more reliable. Most good recipes combine all three.

Anyone wanting to build their own model quickly hears three abbreviations: SFT, Distillation, RL. They sound like alternatives to choose from. In fact, they solve different problems. Anyone who chooses the wrong method spends money on computing time and measures no progress afterward.

This article explains what each method changes in the model, how to recognize which one solves your problem, and what it costs. It is based on the technical reports of Qwen, DeepSeek, and Ai2 as well as on studies and research reports, all read in the original on October 5, 2026. Whether training is worthwhile for your task at all is covered in a dedicated article: When it's worth fine-tuning a model.

Terminology: The base model is the model before your training. SFT (Supervised Fine-Tuning) is training on given sample answers. In distillation, a small model, the student, learns from a large one, the teacher. RL (Reinforcement Learning) means: The model tries out answers and learns from their evaluation. pass@1 is the success rate on the first attempt, pass@k is the proportion of tasks solved at least once in k attempts. A GPU hour is one hour of computing time on a graphics card.

Summary in Brief

• The three methods solve three different problems. SFT teaches format and behavior. Distillation expands what the model can do in the first place. RL makes what it can already do more reliable.

• RL raises the floor, distillation raises the ceiling. In the Qwen3 report, RL lifted an 8B model on the first attempt from 55.0 to 67.6 percent. With 64 attempts, it remained exactly at 90.0. Distillation raised both values, to 74.4 and 93.3 percent, with the latter being exactly one more solved task.

• For small models, distillation is often cheaper and better than RL. In the same comparison, RL required 17,920 GPU hours, distillation 1,800. DeepSeek comes to the same conclusion: a distilled 32B model achieved 72.6 percent, while the same model with RL achieved 47.0.

• Training requires fewer data than expected. The s1 model was trained with 1,000 examples in 26 minutes on 16 GPUs. Combined with a runtime trick, it then outperformed o1-preview in competitive mathematics.

• Two open recipes follow the same order. Tülu 3 and Olmo 3 train first with SFT, then with preference training, then with RL on verifiable rewards. Qwen3 distills its small models from the large ones instead.

• RL forgets less than SFT. A study from MIT shows: with the same performance on the new task, a model retains more of its old capabilities after RL.

• The convenient path via OpenAI is closing. OpenAI is shutting down its fine-tuning platform and is no longer accepting new users. This affects SFT, preference training, and Reinforcement Fine-Tuning.

Table of Contents

1. What Post-Training Is

2. The Three Methods at a Glance

3. SFT: The Model Learns from Sample Answers

4. Distillation: A Large Model Teaches a Small One Something

5. RL: The Model Learns from Evaluations

6. Floor and Ceiling: What RL and Distillation Change in the Model

7. Which Method Fits Which Problem?

8. The Order of the Open Recipes

9. What the Methods Cost

10. Where the Methods Meet Limits

11. How We Go Through This with You

12. Frequently Asked Questions

What Post-Training Is

A language model is created in two phases. In pre-training, it reads huge amounts of text and learns to predict the next word. After that, it can understand language, but it does not follow instructions. In post-training, it learns to answer a question helpfully, adhere to a format, and solve a task reliably.

Pre-training requires huge computing clusters and is therefore reserved for large labs. Post-training, on the other hand, is the part you can do yourself. You take an open model that already follows instructions and continue training it on your task. The result is a small model that does exactly one thing very well, and whose weights belong to you.

OpenAI demonstrated how much leverage this has as early as 2022. The 1.3-billion InstructGPT model, post-trained with SFT and RL, provided answers that human evaluators preferred over those from the 175-billion GPT-3 model. The model was a hundred times smaller.

Source: Ouyang et al. 2022, Abstract.

The Three Methods at a Glance

The difference lies in where the learning signal comes from. In SFT, it is a completed sample answer. In distillation, it is a stronger model. In RL, it is an evaluation of its own attempts.

A grader is a test that evaluates an answer. This can be a program that compares a calculation result, checks a JSON schema, or runs unit tests. It can also be another model or a subject matter expert.

SFT: The Model Learns from Sample Answers

SFT is the simplest method. You show the model a question and the answer you want. The model adjusts its weights so that this answer becomes more likely. In this way, it adopts the structure, length, and tone of the examples.

It is amazing how little data is sufficient. The research team behind s1 trained a Qwen2.5-32B-Instruct with only 1,000 carefully selected math problems including their solutions. The training took 26 minutes on 16 H100 GPUs. The solutions were written by a Gemini model. Strictly speaking, s1 is therefore already a case of off-policy distillation, as described in the next section. Combined with a runtime trick that lets the model think longer, the result outperformed OpenAI's o1-preview in competitive mathematics by up to 27 percent.

Source: Muennighoff et al., EMNLP 2025, Abstract, Introduction, and Section 2. The 27 percent applies with "Budget Forcing," not for training alone.

SFT has two weaknesses. The first: the model learns to mimic the examples and generalizes worse to cases that look different. A study that specifically compared both methods summarizes it as follows: SFT tends to memorize training data, while RL generalizes. However, the same study also emphasizes that SFT remains indispensable before RL because it stabilizes the output format.

The second weakness is forgetting. A model that is heavily trained on a new task often loses capabilities it had before. How this behaves in SFT compared to RL is explained in the section on RL.

Source: Chu et al., ICML 2025, Abstract.

Distillation: A Large Model Teaches a Small One Something

In distillation, a large model generates training data for a small one. The idea is old. Hinton, Vinyals, and Dean described in 2015 how the knowledge of a large model can be transferred to a small one that is cheaper to operate.

There are two variations, and the difference is important.

Off-policy distillation is essentially SFT with a teacher as the author. The teacher answers thousands of questions, and the student learns from these answers. This is simple and works via any API that delivers text. The disadvantage: the student only sees perfect solutions. If it makes its own mistake in use, it has never learned to recover from it.

On-policy distillation flips this around. The student writes an answer itself, and the teacher evaluates every single word in it. The student therefore learns from its own mistakes, with feedback for each token instead of a single grade at the end. A team from Google DeepMind introduced the method in 2024. It requires the teacher's probabilities, the logprobs, for each of the student's texts. Closed APIs do not provide them this way. Practically, this means the teacher must be an open model that you run yourself, ideally with the same tokenizer as the student.

Source: Hinton et al. 2015, Abstract. Agarwal et al., ICLR 2024, Abstract.

How powerful distillation can be is demonstrated by DeepSeek. The team compared two ways to train a Qwen-32B for mathematics. Method one: RL directly on the model over 10,000 training steps. Method two: distillation from the large DeepSeek-R1. On the AIME 2024 mathematics exam, the RL model achieved 47.0 percent, while the distilled model achieved 72.6 percent.

The authors' conclusion is clear. Distilling strong models into smaller ones delivers outstanding results. Training small models with large RL costs enormous computing power and may not even reach the level of distillation.

Source: DeepSeek-AI, DeepSeek-R1, arXiv version v1, Section 4.1 and Table 6. Published in Nature 645 (2025).

Before choosing a teacher, check the terms of use. Anthropic, OpenAI, and Google contractually prohibit using their outputs to train competing models. We have evaluated which teachers are permitted individually: Which model can you use as a teacher?

Source: Terms of use of the three providers, evaluated with quotes in the linked article.

RL: The Model Learns from Evaluations

In RL, there is no sample answer. The model writes several of its own answers to the same question. A grader evaluates them. Training makes the highly-rated answers more likely and the poorly-rated ones less likely.

Three variations are common in practice:

• RLHF (RL from Human Feedback): Humans compare answers, a reward model is trained from this, and this reward model serves as the grader. This is how InstructGPT was built.

• DPO (Direct Preference Optimization): You provide pairs of better and worse answers. The model learns directly from them, without its own trials and without a separate reward model. Strictly speaking, this is no longer RL, but it serves the same purpose.

• RLVR (RL with Verifiable Rewards): The reward comes from a verifiable rule. Tülu 3 describes it like this: the model is only rewarded if its answer is demonstrably correct. For extraction, classification, or code, this is the natural choice because right and wrong can be determined automatically.

Source: Ouyang et al. 2022, Fig. 2. Rafailov et al. 2023, Abstract. Lambert et al., Tülu 3, Section 6.

A widely used RL method for open models is called GRPO. It comes from the DeepSeekMath paper. GRPO lets the model generate a group of answers and compares each one to the group average. This eliminates the need for a second model that would otherwise have to estimate the value of each answer, saving memory and computing time.

RL has an advantage that is highly valuable in practice: it forgets less. A team from MIT compared SFT and RL on new tasks, with the same performance on the new task. After RL, the model retained its earlier capabilities significantly better. Their explanation: RL favors solutions that are close to the original model. SFT can drift arbitrarily far from it.

Source: Shao et al. 2024, Section 4.1. Shenfeld et al., "RL's Razor", Abstract.

Floor and Ceiling: What RL and Distillation Change in the Model

This is the most important section of this article. Imagine a model trying every hard task 64 times. The ceiling is the percentage of tasks it solves at least once in these 64 attempts. It shows what the model is fundamentally capable of. The floor is the success rate on the first attempt. It shows how reliably it retrieves that capability.

RL raises the floor. It barely moves the ceiling. A team from Tsinghua University measured this across math, code, and image tasks across multiple model families. Models after RL win on the first attempt. However, with a very high number of attempts, the untrained base model reaches the higher value. According to this study, RL in its current form does not bring new reasoning paths. It ensures that the model more often chooses the correct answer, which it already knows, first.

Distillation raises the ceiling. The same study notes: unlike RL, distillation brings new solution patterns from the teacher and thereby actually expands what the model can do.

Source: Yue et al., NeurIPS 2025, Abstract and Section 4.2.

The Qwen3 report shows the same pattern in a direct comparison. Both methods start from the same 8B checkpoint, which has already gone through an initial off-policy distillation. The comparison is therefore RL versus a second, on-policy distillation.

RL raised the floor by 12.6 points, while the ceiling remained unchanged. Distillation raised the floor by 19.4 points, with a tenth of the computing time. Caution is advised with the ceiling: AIME'24 has 30 tasks, so the 3.3-point increase represents exactly one additional solved task. The more robust support for the ceiling effect is the Tsinghua study above.

Source: Qwen Team, Qwen3 Technical Report, Section 4.7 and Table 21.

Which Method Fits Which Problem?

A simple diagnosis follows from floor and ceiling. Let the base model solve your hardest cases many times and count how often it succeeds. This costs hours, not weeks, and it decides the choice of method before you spend money on training. How to set up the test is explained in the article When it's worth fine-tuning a model.

For the third row, there are two methods to choose from. RL does not need a teacher, but it needs a grader. On-policy distillation needs an open teacher that you run yourself, but according to the Qwen3 report, it is much cheaper for small models. If you have an allowed teacher that solves the task better than your student, distillation is usually the first attempt. If your student is already as good as any available teacher, RL remains.

The Order of the Open Recipes

Open labs publish their training recipes in full. This shows how the methods interact.

Tülu 3 from Ai2 trains Llama 3.1 in three stages: SFT, then DPO, then RLVR. According to the report, the result outperformed the Instruct versions of Llama 3.1 and Qwen 2.5, as well as GPT-4o-mini and Claude 3.5 Haiku. Olmo 3 follows the same three-step process. The logic behind it: SFT provides the format, DPO the style, and RLVR the accuracy on verifiable tasks.

Qwen3 takes a different path for the small models. The large flagships go through four training stages. The small models, however, are distilled from the large ones instead. According to the report, this delivers better results on the first attempt and with 64 attempts, with a tenth of the GPU hours of the four-stage process.

Source: Lambert et al., Tülu 3, Abstract. Ai2, Olmo 3 announcement from 11/20/2025. Qwen Team, Qwen3 Technical Report, Section 4.

For a corporate project with a small open model, this yields a practical sequence:

1. Measure baseline. Base model and a strong frontier model on your test set, with a fair prompt.

2. SFT or off-policy distillation on a few hundred to a few thousand examples. This brings the format and most of the improvement.

3. On-policy distillation or RL if the floor is still too low afterward.

4. Re-evaluate, also on general capabilities and safety, not just on the target task.

What the Methods Cost

Computing time differs by orders of magnitude. The cheapest is SFT on a few examples, the most expensive is RL on a model that barely masters the task yet.

The Euro column is an order of magnitude, not a precise calculation. It assumes the cheapest European H100 list price of €2.80 per GPU hour from our comparison GPU prices in Europe. The Qwen3 report does not name the GPU type. Furthermore, the Qwen runs are research runs on broad mathematics and code datasets. A project on a single, narrowly defined corporate task requires far less.

Source: Muennighoff et al., Section 3. Qwen Team, Table 21.

The larger cost block is often not the training, but the data. In distillation, you pay for every answer from the teacher. In RL, you pay for the grader, and if subject matter experts evaluate, you pay for their time. Thinking Machines compared on-policy distillation with classic SFT on teacher answers. Depending on whether the teacher costs were already paid, the method saved nine-fold, about 18-fold, or about 30-fold.

Source: Thinking Machines Lab, "On-Policy Distillation", 10/27/2025, section on computational efficiency. Blog post, not peer-reviewed.

Where you can train in Europe and what options exist besides your own GPUs is covered in LLM Fine-Tuning in Europe.

Where the Methods Meet Limits

Each method has a typical point where projects fail. Knowing them beforehand helps in planning.

Training on proprietary documents destroys behavior. Thinking Machines trained a Qwen3-8B on internal documents so that it would know their content. Knowledge increased from 18 to 43 percent. At the same time, the ability to follow instructions fell from 85 to 45 percent. By including general chat data in the training and a subsequent on-policy distillation, the model achieved 41 percent knowledge and 83 percent instruction adherence. The lesson: after any training on specialized texts, general behavior must be measured and usually repaired.

Source: Thinking Machines Lab, section "Distillation for personalization", table on Mid-Training.

RL is only as good as its grader. A grader that lets incorrect answers pass teaches the model to exploit exactly that loophole. Therefore, build the grader like a test set: from real cases, checked by experts, and tested against examples before training. How to do this is described in the Evals Guide.

Distillation is legally restricted. The best teacher is often contractually blocked. Open models with a free license are the alternative. Whether one of them solves your task better than your student is only revealed by a measurement on your test set.

The managed path is closing. OpenAI is winding down its fine-tuning platform. New users no longer get access, existing trained models will continue to run until their base model is deprecated. Reinforcement Fine-Tuning was only available there for o4-mini anyway. Mistral has also discontinued its fine-tuning. Anyone starting a post-training project today almost always ends up with an open model.

Source: OpenAI, Reinforcement Fine-Tuning Guide and Supervised Fine-Tuning Guide, accessed 10/05/2026. Mistral, Fine-Tuning documentation, accessed 10/05/2026.

Personal data in training requires a legal basis. This applies to SFT examples just as much as to RL prompts. What needs to be considered under GDPR and the AI Act is explained in Training AI models on proprietary data.

How We Go Through This with You

The diagnosis from this article can be carried out by your team itself. The method is rarely the difficult part. What is difficult is a test set that you trust afterward, and a grader that reliably distinguishes right from wrong.

That is exactly where we come in. We build a test set with you from your real cases, using your subject matter experts as graders. Then we measure the floor and ceiling of your base model and choose the method afterward, not before.

Training and operation take place on dedicated GPUs in the EU, without US hyperscalers in the loop. Weights, eval set, and documentation belong to you, with export possible at any time. No model goes into production without an eval gate, and every rollout runs as an A/B test with rollback. The data processing agreement is signed before the first data record is transferred.

The first step is a conversation about your task: 30 minutes, engineering is present, no sales pitch. If it turns out that a prompt solves your problem and no training is needed, we will tell you. Schedule an appointment.

Frequently Asked Questions

What is the difference between Fine-Tuning and Post-Training?

Post-training is the umbrella term for everything that happens to a model after pre-training. In everyday language, fine-tuning usually refers to SFT, i.e., training on sample answers. Distillation and RL also belong to post-training. Anyone who says "fine-tuning" therefore often only means a part of what is possible.

Do I need RL if SFT already delivers good results?

Not necessarily. Measure the success rate on the first attempt and over many attempts after SFT. If both are close and high enough, RL brings little benefit. If the first attempt is significantly below the value over many attempts, RL can bridge this gap. It makes what the model can already do more reliable.

Is distillation the same as SFT?

Off-policy distillation is technically SFT, only the sample answers come from a teacher model instead of humans. On-policy distillation is different. Here, the student writes the answer itself, and the teacher evaluates each token. This is similar to RL, with much finer feedback than a single score at the end.

How many training examples do I need?

For SFT, a few hundred to a few thousand good examples are often enough. The s1 model got by with 1,000, and OpenAI recommends 50 as a starting point in its fine-tuning guide. Quality matters more than quantity. For RL, you do not need sample answers, but enough tasks with a grader that evaluates reliably.

Which model is suitable as a teacher?

One that solves your task significantly better than your student, and whose terms of use permit distillation. The major US providers prohibit training competing models with their outputs. Open models with a free license like Apache 2.0 are the safe choice if they are powerful enough.

Does my model lose general capabilities through training?

This can happen and is called forgetting. It is more pronounced with SFT than with RL, as a study from MIT shows. Countermeasures include general data in the training mix and a brief on-policy distillation at the end. After each run, measure general capabilities and safety as well, not just the target task.

Sources

• Qwen Team, "Qwen3 Technical Report" (arXiv:2505.09388): Strong-to-Weak distillation and the direct comparison with RL in Table 21

• DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (Nature 645, 2025): Distillation versus RL on a 32B model

• Yue et al., "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" (NeurIPS 2025, Oral): pass@k, floor and ceiling

• Lambert et al., "Tulu 3: Pushing Frontiers in Open Language Model Post-Training" (arXiv:2411.15124): the recipe SFT, DPO, RLVR

• Muennighoff et al., "s1: Simple test-time scaling" (EMNLP 2025): SFT on 1,000 teacher solutions, i.e., off-policy distillation

• Shenfeld, Pari, Agrawal, "RL's Razor: Why Online Reinforcement Learning Forgets Less" (arXiv:2509.04259): forgetting in SFT and RL

• Chu et al., "SFT Memorizes, RL Generalizes" (ICML 2025): generalization and the role of SFT before RL

• Agarwal et al., "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes" (ICLR 2024): On-policy distillation

• Hinton, Vinyals, Dean, "Distilling the Knowledge in a Neural Network" (arXiv:1503.02531): origins of distillation

• Ouyang et al., "Training language models to follow instructions with human feedback" (arXiv:2203.02155): InstructGPT and RLHF

• Rafailov et al., "Direct Preference Optimization" (arXiv:2305.18290): preference training without an RL loop

• Shao et al., "DeepSeekMath" (arXiv:2402.03300): introduction of GRPO

• Thinking Machines Lab, "On-Policy Distillation" (10/27/2025): cost comparison and personalization

• Ai2, Olmo 3 (11/20/2025): three-stage post-training recipe

• OpenAI, Reinforcement Fine-Tuning Guide and Supervised Fine-Tuning Guide: models, minimum examples, and platform deprecation. Also, the Mistral Fine-Tuning documentation with the deprecation notice

• The Evals Guide: building test sets and graders

As of October 5, 2026 · kontinent.ai. All percentage values are from the publications cited and apply to the models and tasks tested there, predominantly mathematics and code. The Euro values are orders of magnitude based on public list prices. Provider sites were accessed on October 5, 2026, and are subject to change.