LoRA

LoRA (Low-Rank Adaptation) is the usual way to fine-tune a large model without touching every weight. The base matrix W stays frozen. The update is the product of two thin matrices. Only those train.

The big idea

For a frozen weight W with shape d_out × d_in, LoRA adds:

W' = W + (α / r) × B A
  A: r × d_in     (often init Gaussian)
  B: d_out × r    (often init zero so initial ΔW = 0)

Only A and B train. Parameter count per adapted matrix: r × (d_in + d_out) instead of d_in × d_out. α (lora_alpha) scales the update. Many configs set α ≈ 2r.

B starts at zero so the first forward pass matches the base model. That is why a well-init LoRA does not wreck the checkpoint on step 0.

Walking through a 4096×4096 matrix

Take W as 4096 × 4096 (~16.8 million parameters). Full fine-tuning trains all 16.8M for that matrix. LoRA with r = 16 trains 16 × (4096 + 4096) = 131,072 parameters — about 128× fewer for that layer.

Apply LoRA to all attention and MLP projections across a 32-layer 7B model and trainable params often land in the tens of millions (roughly 0.1–1% of 7B), while the 7B base stays frozen.

Llama-class 7B full fine-tuning trains about 7B params. LoRA r=16 on all attention projections might train about 20–40M params depending on width and which modules you hook — roughly 200× fewer. Adam state shrinks from on the order of 112 GB toward adapter-only memory plus the frozen base. That is why laptop and Colab fine-tunes exist at all.

Where to apply adapters: the original paper targeted attention q_proj and v_proj only. Modern practice often targets all linear layers (q, k, v, o and MLP) for better quality at modest extra cost. Dropout on LoRA layers can regularize small datasets.

Merge adapters into W for deployment if you do not need to hot-swap, or keep them separate for multi-adapter serving. See multi-tenant serving.

from peft import LoraConfig

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)
print(lora.r, lora.lora_alpha)

Output:

16 32

Common mistakes

• Init B with noise. Initial ΔW should be zero so you start at the base model.
• Raising rank after the run already overfit. Clean the data and cut epochs before you jump to full fine-tuning.
• Merging and then serving a different template or quantization path without a prompt check. Test a few prompts after every export.

Check yourself

1. W is 4096×4096. LoRA rank 16. How many trainable parameters is that for this matrix? Answer: 16 × (4096 + 4096) = 131,072.
2. Why initialize B to zero? Answer: So BA starts at 0 and the first step matches the frozen base.

Key takeaways

• LoRA trains BA of rank r beside frozen W.
• Parameter count is r (d_in + d_out) per matrix, often 100×+ fewer than full fine-tuning.
• Target all linear layers unless you have a reason not to. Rank 8–64 is typical.
• Merge for simple serving, or keep adapters separate to hot-swap.

Further learning

Next: QLoRA.

LoRA: Low-Rank Adaptation of Large Language ModelsAn important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which re…arXiv.org