Guide

Serving systems

LoRA adapters

LoRA fine-tunes a model by training a small add-on beside frozen weights. The add-on is tens of megabytes, so one base model can carry thousands of them.

109 of 138 · 2 min read

LoRA (Low-Rank Adaptation) changes a model without retraining all of it. The base weight matrix W stays frozen. The training learns a small update, ΔW = B A, made of two thin matrices. The rank r is the inner size of that update, and it is much smaller than the hidden size. A LoRA for a 7B to 70B base is usually tens of megabytes.

Concept

A hidden size is the width of the model's internal vectors. Rank r is the inner dimension of B and A. A smaller rank means a smaller add-on. For how adapters are trained, see PEFT and SFT.

The big idea

Think of a textbook with a sticky note for each student. The book is the base model: large, expensive, and one copy. Each note is an adapter: small, cheap, and specific to one student.

The picture breaks in one place. The note has to be read on every page, which means the adapter does work on every forward pass. It is not a free label. The arithmetic below shows how small that work is in practice.

Walking through the sizes

The forward pass for a LoRA layer is:

output = x W + x B A

Here x W is the frozen base, and x B A is the small add-on. Now compare sizes, using the example from inference math:

Base, 8B model in FP16:   about 16 GB of weights
One LoRA, rank 16:        about 40 MB (attention and MLP projections)
Add-on as share of base:  40 MB / 16 GB = 0.25%
Adapters per 16 GB base:  16 GB / 40 MB = about 400 (storage only)

So each tenant costs about a quarter of a percent of a base copy. Serving 400 tenants with 400 full fine-tuned copies would need 400 times the memory.

Common mistakes

  • Merging each adapter into the base for serving. That means a full copy per tenant and multi-minute reloads when you switch.
  • Assuming an adapter works on any base. An adapter is tied to the base checkpoint it was trained on. A base upgrade means retraining or remapping.
  • Mixing ranks without a plan. If one tenant uses rank 64 and the rest use rank 8, the server must be sized for rank 64.

Check yourself

  1. Why does serving 400 LoRA tenants cost far less than serving 400 full fine-tuned copies? Answer: all tenants share one frozen base matrix. Only the small B and A matrices differ per tenant.
  2. Can you serve an adapter trained on base v1 with base v2 unchanged? Answer: not reliably. Adapters are tied to their base checkpoint, so they must be rebuilt or remapped.

Key takeaways

  • A LoRA freezes the base weight W and learns a low-rank update B A.
  • An adapter is tiny compared with the base, often under 1% of it.
  • Many adapters can share one base, as long as the server can apply each one per request.
  • Adapters are tied to their base checkpoint.

Further learning

Next, read multi-LoRA batching to see how one server applies many adapters at once. The vLLM LoRA documentation covers the engine side.

Multi-LoRA and multi-model serving

Serving many fine-tuned adapters and models on shared GPUs efficiently.

Supervised fine-tuning and PEFT

Instruction tuning, chat formats, data quality, LoRA/QLoRA math, and the tooling that turns a base model into a useful assistant.

Fine-tuning vs prompting vs RAG

When to change weights instead of the prompt or the retrieved context, and how to decide with evals.