Multi-LoRA batching and adapter paging

Serving one GPU copy per adapter wastes memory. Serving adapters one at a time hurts latency. Multi-LoRA batching puts requests for different adapters into the same forward pass, while the base weights stay on the GPU once. Adapters that are not used often wait in host memory and are loaded when a request needs them.

The big idea

Think of a print shop with one large press and many stencils. The press prints every job in one pass, and each job uses its own stencil. The stencils that are rarely used sit in a cabinet and get loaded when needed.

The picture breaks in two places. Applying a stencil per job is extra work, which needs special kernels. And a cold stencil takes time to load onto the press, which is the source of cold-start delay.

Walking through a batch

Take four requests in one decode step, for adapters T1, T2, T1, T3.

Base matmul:      x W for all 4 rows, computed once
LoRA term, T1:    rows 1 and 3 use B1 A1
LoRA term, T2:    row 2 uses B2 A2
LoRA term, T3:    row 4 uses B3 A3
Output per row:   x W + (that row's adapter term)

The base matrix is read once for the whole batch. Only the small adapter term changes by row. Punica and S-LoRA use custom kernels, such as SGMV (segmented gather matrix-vector), to do the per-row adapter step without running a separate matrix multiply for each adapter.

Adapter paging

Treat GPU adapter slots as a cache, usually least recently used (LRU) or frequency-aware:

1. A request arrives for adapter T.
2. If T is on the GPU, schedule it into the batch right away.
3. If not, evict a cold adapter, copy T from CPU memory or NVMe to the GPU, then schedule it.
4. Cap concurrent loads so PCIe transfers do not stall the decode loop.

Pin the adapters of your most important tenants so they are never evicted. Keep adapters on local NVMe, which loads faster than remote object storage, and prefetch when a session starts.

Packing math

For an 8B FP16 base (about 16 GB), suppose 40 GB of the GPU is left after overhead, and 30 GB is reserved for KV cache. The rest is for hot adapters:

Adapter room:        10 GB
One hot adapter:     40 MB
Hot adapters:        10 GB / 40 MB = about 250

Common mistakes

• Mixing bases. Every adapter in a batch must target the same base checkpoint and compatible modules. Mixing bases is multi-model serving, not multi-LoRA.
• Sizing for the largest rank. Set the max rank to the largest rank you load. Unused headroom is wasted memory.
• Judging by tokens per second alone. Track adapter cache hit rate and cold-load p95 too.

Check yourself

1. A batch has requests for adapters A, B and A. How many times is the base matrix read? Answer: once. Only the small per-adapter term differs by row.
2. Your adapter hit rate falls from 92% to 60% during a campaign. What is the first fix? Answer: a larger hot set, or a warmup job that prefetches the campaign adapters. More base replicas alone do not fix it.

Key takeaways

• One forward pass can serve requests for many adapters, while the base weights stay loaded once.
• Custom kernels apply each row's adapter without a separate matmul per adapter.
• Cold adapters page in from host memory. Size the hot set from traffic skew.
• Watch cache hit rate and cold-load latency, not only throughput.

Further learning

Read the S-LoRA paper for the paging design and the Punica paper for the batched kernels. Next, see GPU sharing.