Multi-LoRA and multi-model serving

Product teams fine-tune faster than platform teams can buy GPUs. A support copilot for each enterprise customer, a tone adapter for each brand, a small head for each workflow: soon you have thousands of LoRA adapters on one base model. Each one is too small to justify its own GPU. Multi-tenant serving keeps one copy of the base weights in GPU memory and serves many adapters or models from it. Utilization stays high, and no tenant becomes a noisy neighbor.

Before you start, read LoRA adapters.

Concepts in this section

Why naive approaches fail

• One GPU per adapter. Financially out of reach at thousands of tenants.
• Merging each adapter into the base and reloading. Multi-minute cold starts are unusable for interactive products.
• One full replica per adapter. Duplicates the base N times, and memory runs out quickly.
• Serving one adapter at a time. Simple, but latency suffers when tenants are mixed.

Model multiplexing

Not everything is a LoRA. You may serve several full models, such as an 8B chat model, a 7B code model, an embedding model and a reranker, on one node. Three patterns:

• Always-on colocated models. If they fit together with their KV caches, run them side by side. This is the simplest option, but watch total memory.
• Time-share swapping. Unload model A to load model B when traffic shifts. Loading a 70B model from disk can take tens of seconds to minutes. Only use this for batch windows or rare models.
• Pool per model size. Small models go on partitioned or cheaper GPUs. Large models go on high-memory nodes. A gateway routes by model name.

Cold-load time has several parts: download, deserialize, copy to GPU memory, capture CUDA graphs, and the first-batch compilation. Local caches, pre-captured graphs and a minimum of one standby replica help.

Lifecycle of an adapter

1. Train on a fixed base revision. Record the base model digest, rank, target modules and eval scores.
2. Register it in a control plane, with the artifact location and metadata. Reject adapters that fail schema or eval checks.
3. Serve it behind a gateway alias, such as customer-x-support.
4. Warm it on first login or at deploy time.
5. Retire it from the GPU and eventually from NVMe, and keep an archived copy for audit.

Without a registry, teams copy adapter folders onto servers by hand. After a base upgrade, they then serve the wrong weights.

Pricing a fine-tuning platform

If customers train adapters on your platform and you serve them:

• Charge for adapter storage separately from inference. Storage is cheap. Inference is not.
• Reflect hot versus cold. Price cold adapters with a latency difference, or sell a reserved SKU that pins adapters and KV headroom.
• Oversell with a published cold-start p95. You can sell more adapters than fit hot, as long as customers know the paging cost.

A rough model: cost per request ≈ (GPU seconds × price per GPU second) / batch efficiency. Batch efficiency falls when adapters thrash or KV fragments.

Common mistakes

• Treating a fleet of LoRAs as a fleet of GPUs. The win comes from sharing the base.
• Swapping full models on interactive paths. Keep them warm instead.
• Judging the setup by tokens per second alone. Track adapter hit rate, cold-load p95 and per-tenant latency.

Key takeaways

• Thousands of LoRAs can share one base. The base stays loaded and adapters page through a GPU cache.
• Batching across adapters in one forward pass is what keeps utilization high.
• Full-model swaps are for batch work or scale-to-zero tiers, not interactive traffic.
• Enforce per-tenant limits, or noisy neighbors will set your p99.

Further learning

Read the S-LoRA paper and the Punica paper for the batched-adapter design. The vLLM LoRA documentation covers the engine settings.