The R1 training recipe

DeepSeek-R1 published the recipe for a reasoning model. The steps are more involved than the core algorithm, and it is worth knowing their order. The most important result is that large-scale RL with checkable rewards can produce reasoning habits that nobody demonstrated. Supervised data then polishes the result, rather than being the only source of the skill.

The steps

The R1 report describes this path. The details here are simplified.

1. R1-Zero. Start from a strong base model, not a chat model. Run large-scale RL with verifiable rewards and a format reward. Long reasoning, self-reflection and "aha" style revisions appear without supervised demonstrations. The text can be hard to read, with mixed languages and messy structure.
2. Cold-start SFT. Fine-tune the base model on a smaller set of readable, structured reasoning traces. This fixes presentation.
3. Reasoning RL again. Resume verifiable-reward RL from the cold-started model.
4. Rejection-sampled SFT. Collect good traces from the RL model, and mix them with ordinary data such as writing, FAQs and safety examples.
5. Final RL. Add broader rewards, including helpfulness and harmlessness preferences.
6. Distillation. Fine-tune smaller models on the R1 traces, so they inherit the reasoning patterns without the full RL cost.

Step 1 is the surprising part. Step 2 exists only to make the output readable.

The transferable pattern

You do not need the exact graph to use the ideas. The pattern is:

• Verifiable RL for the reasoning skill.
• A little supervised data for style and readability.
• Distillation for serving at lower cost.

For a team, that usually means running RL on your own unit tests or answer checks, then distilling into a smaller model.

Why distill

Serving a very large reasoning model is expensive, and running RL on one is costly too. A smaller student model, trained on the big model's traces, recovers much of the behavior at lower latency. For many products, distillation is the only way to meet the cost target.

Evaluate the student on the same verifiers as the teacher. Do not assume that fluent reasoning means the accuracy carried over.

Common mistakes

• Copying the full compute budget. The ideas transfer at small scale. Start with a 1B to 8B model and a verifiable task.
• Assuming the students are as accurate. Measure accuracy after distillation.
• Skipping the readability step. Raw RL traces can be correct and still unreadable to users.

Check yourself

1. What problem does the cold-start step solve? Answer: R1-Zero's traces can mix languages and read poorly, so supervised readable traces fix presentation.
2. Why distill instead of serving the big model? Answer: serving the large reasoner is expensive, and distillation gives smaller models much of the behavior at lower cost.

Key takeaways

• R1-Zero shows that verifiable RL alone can produce reasoning behavior.
• Cold-start SFT makes that reasoning readable, and more RL and mixed SFT follow.
• Distillation is how reasoning reaches products at a reasonable cost.
• Check the students against the same verifiers as the teacher.

Further learning

The R1 paper is the primary source:

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningGeneral reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) and chain-of-thought prompting, have achieved conside…arXiv.org

Next, see how reasoning models use the per-step signal in process vs outcome rewards.