Guide

Scale and post-training

Instruction tuning and loss masking

Supervised fine-tuning teaches a base model to follow instructions. Gradients should flow through the assistant tokens, not the user's prompt, and train and serve must share one chat template.

149 of 187 · 3 min read

A base model predicts the next token on internet text. It completes documents. It does not reliably follow instructions, refuse harmful requests, or speak in a consistent assistant voice. Supervised fine-tuning (SFT), often called instruction tuning, teaches that behavior from labeled examples.

This sits in the middle of the post-training stack: after pretraining, before preference optimization (RLHF/DPO). Product-facing choices about when to fine-tune versus retrieve live in AI fine-tuning. Here the mechanism is the point.

Concept

Loss masking means the training loss is computed only on some tokens. For SFT, you typically mask the prompt so gradients flow only through the assistant's reply.

The big idea

An SFT example is a prompt (system plus user turns) and a target completion (the assistant reply). Training uses the same next-token cross-entropy as pretraining, with one critical difference: mask the prompt tokens. Otherwise the model wastes capacity memorizing user text and can get worse at following new instructions.

Chat templates serialize multi-turn dialogue into a flat token stream with special markers such as <|im_start|> or [INST]. Every model family has a tokenizer chat template. Mismatch between train and serve templates is a classic "it got dumber in production" bug. Always apply the same template the hub model card specifies.

A single training row, after templating, might look like:

<|im_start|>system
You are a support agent for Acme Billing. Be concise. Never invent invoice IDs.
<|im_end|>
<|im_start|>user
Why was I charged twice in March?
<|im_end|>
<|im_start|>assistant
I can help investigate a double charge. Please share the invoice IDs or the last four digits of the payment method and the approximate dates. I will not guess invoice numbers.
<|im_end|>

Loss is computed only on the assistant span. If you train on the whole string, the model also learns to imitate the user's wording and the system prompt verbatim. Packing several short conversations into one context, without crossing loss masks, is a common throughput trick.

Walking through data quality

The LIMA result (Less Is More for Alignment) pushed a useful slogan: a few thousand carefully curated instruction examples can outperform noisier sets that are orders of magnitude larger, once the base model is strong. Labs may still use millions of examples. The broader lesson holds: garbage demos teach garbage habits.

Quality in practice:

  • Correctness for factual and task prompts, plus honest "I don't know" where appropriate.
  • Diversity of tasks and phrasings so the model does not overfit a template.
  • Style matched to the product (terse API assistant versus tutoring voice).
  • No train/eval leakage from popular benchmarks if you care about honest scores.

Synthetic data and distillation are now standard: sample from a stronger model, filter with rubrics or unit tests, then SFT the student. Distillation also compresses reasoning traces into smaller models. Filtering beats dumping raw teacher output.

Fine-tuning on unfiltered scraped "instruction" dumps often teaches sycophancy, confident hallucination and weird refusal patterns. Prefer small clean sets or aggressively filtered synthetic data.

For tiny datasets (under 1k examples), prefer more diverse rewrites of the same intents over a third epoch of identical rows. For large synthetic sets, one epoch is often enough. Watch held-out loss.

Common mistakes

  • Training on prompt tokens. The model memorizes user text instead of learning to answer.
  • A one-character template mismatch between train and serve. Token IDs change and instruction following collapses.
  • Dumping unfiltered teacher output. Filter with rubrics or tests first.

Check yourself

  1. Why mask the prompt during SFT? Answer: So gradients update the model on the assistant reply, not on copying the user's words.
  2. You fine-tuned with Llama's chat template and served with a raw concatenation of roles. What happens? Answer: Special-token IDs no longer match training, so the model often ignores instructions or emits raw markers.

Key takeaways

  • SFT teaches assistant behavior with next-token loss on labeled replies.
  • Mask the prompt. Score only the completion.
  • Train and serve must use the same chat template.
  • Data quality dominates quantity. LIMA-style curation and filtered synthetic data beat raw dumps.

Further learning

Next: LoRA. Product choices live in AI fine-tuning.

Supervised fine-tuning and PEFT

Instruction tuning, chat formats, data quality, LoRA/QLoRA math, and the tooling that turns a base model into a useful assistant.

Pretraining a language model

Data, objective and compute: how a base model is trained on trillions of tokens.

RLHF and preference optimization

How language models learn from human and AI preferences. Supervised fine-tuning, a reward model, a KL-leashed RL step, and DPO as a single-loss alternative. Start here for the map.