Kinds of fine-tuning

Fine-tuning is not one recipe. The data you have, and the failure you are fixing, pick the method. All of them change weights. They differ in what "good" looks like in the dataset.

The four methods

Supervised fine-tuning (SFT). Input to ideal-output pairs. The default. Teach format, style and task completion.

Preference tuning. Pairs or rankings of better vs worse responses. DPO (direct preference optimization) pushes the model toward the preferred answer without a full RL loop. Classic RLHF trains a reward model, then optimizes with RL. Use when "good" is comparative, such as tone, and you can label preferences reliably. See RLHF and preferences.

Distillation. Teacher generations become student SFT data. Keep the teacher's prompts and tools in the pipeline so the student learns the product behavior, not a random chat style.

Reinforcement fine-tuning / verifiable rewards. When a checker exists (unit tests, exact match, a schema), optimize against it. High signal for code and structured tasks. Needs a robust grader.

The big idea: LoRA

Full fine-tuning updates every weight. For product work that is rarely worth it. LoRA (low-rank adaptation) freezes the base weights W and learns a thin update W + B·A with rank r much smaller than the matrix size. You train far fewer parameters. Adapters are small files you can swap or merge.

QLoRA keeps the base in 4-bit precision while training LoRA, so a large model fits on fewer GPUs.

One matrix, d = k = 4096, rank r = 8:
  full update  = 4096 × 4096 ≈ 16.8 million parameters
  LoRA A + B   = 2 × 4096 × 8 = 65,536 parameters   (~0.4%)

That is why LoRA won for product teams: the adapter is small, training is cheaper, and you can keep several adapters on one base. The mechanics live in SFT and PEFT. The picture above is the practitioner minimum.

The analogy is a sticky note on a textbook. You do not reprint the book. You add a thin overlay. The analogy breaks when the base model is retired: the sticky note does not automatically stick to the next edition. You re-train.

Which method for which failure

Common mistakes

• SFT on wiki pages to "add knowledge." You get unciteable mush. Retrieve the wiki.
• Preference pairs that are only "A is slightly nicer." The difference must be the behavior you care about. Prefer a bad production reply vs an edited good one.
• Distilling a mediocre teacher. Gold outputs should come from your best system: frontier model, best prompt, then a human edit.

Check yourself

1. You have 2,000 schema-valid invoice extractions from a large teacher, and an 8B student that scores 71% on a held-out set. Which method? Distillation as SFT. The teacher outputs are the labels.
2. Why does LoRA with rank 8 update about 0.4% of a 4096-by-4096 matrix? The full matrix is 16.8 million weights. A and B together are 2 × 4096 × 8 = 65,536.

Key takeaways

• SFT is the default: input to ideal output.
• Preference methods need comparative labels, not random niceness.
• Distillation copies a teacher's product behavior into a cheaper student.
• LoRA trains a thin adapter. QLoRA keeps the base in 4-bit so it fits.

Further learning

Next, collect the data in fine-tuning data. For LoRA intuition from its inventor:

SFT and PEFT