Supervised fine-tuning and PEFT

A base model predicts the next token on internet text. It completes documents. It does not reliably follow instructions or speak in a consistent assistant voice. Supervised fine-tuning (SFT) teaches that behavior from labeled examples. Parameter-efficient fine-tuning (PEFT) methods like LoRA make the same job cheap enough to run on one GPU.

This is the practical middle of the post-training stack: after pretraining, before preference optimization (RLHF/DPO). Product-facing choices (when to fine-tune versus RAG) live in AI fine-tuning. Here the focus is mechanisms and memory.

Concepts in this section

Read them in order. Each one builds on the one before.

Working in practice

Most companies never full-fine-tune a 70B model. They QLoRA on proprietary tickets, SOPs and tone guides, evaluate on a fixed rubric, and export adapters.

Ship checklist: identical train/serve template, held-out golden prompts, regression on a general smoke set, adapter version pinned next to the base revision.

Interviews: explain LoRA shapes, why you mask prompt tokens, the LIMA intuition that quality beats volume, and QLoRA's 4-bit plus adapter split.

Your SFT and RLHF FLOPs are tiny versus pretrain. Pick the best base rather than the bigger number on the badge. See scaling laws.

Typical LoRA knobs: learning rate around 1e-4 to 2e-4, rank 8–64, one to three epochs on solid data. Overfitting shows up as memorized demo phrasing and a falling held-out rubric. For tiny datasets, rewrite the same intents rather than repeating identical rows. At serve time, merge the adapter for simple latency, or keep it separate when each tenant needs its own tune.

Common mistakes

• Template mismatch between train and serve. The model looks "dumber" because the special-token IDs changed.
• Unfiltered instruction dumps. They teach sycophancy and confident hallucination.
• Full fine-tuning as the first experiment. Start with LoRA. Raise rank before you spend a cluster.

Key takeaways

• SFT teaches assistant behavior with masked loss on completions and consistent chat templates.
• Data quality dominates. Curation and filtered synthetic data beat raw dumps.
• LoRA trains BA of rank r beside frozen W — orders of magnitude fewer trainable parameters.
• QLoRA keeps the base in 4-bit NF4 so large models fine-tune on one GPU.
• Default to PEFT for product work. Use full fine-tuning when you need a large shift and have the budget.

Further learning

Shaw Talebi's QLoRA talk builds the memory picture and walks Python code for single-GPU fine-tuning:

LoRA: Low-Rank Adaptation of Large Language ModelsAn important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which re…arXiv.org QLoRA: Efficient Finetuning of Quantized LLMsWe present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates…arXiv.org Hugging Face TRL documentationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co