Serving systems
GPU sharing with MIG, MPS and time-slicing
MIG, MPS and time-slicing let several workloads share one GPU. Each trades isolation against density, and each fits different workloads.
A big GPU is often too much for one small model, such as a 1 to 3B classifier or an embedding model. Three mechanisms let several workloads share one GPU. They differ in how strongly they isolate workloads, how densely they pack them, and how much latency they add under load.
The big idea
Think of a building. MIG is a building with walls: each floor is split off with its own fixed-size utilities. MPS is several tenants on one floor with no walls. They share the space at once, so one heavy tenant can crowd the others. Time-slicing is a shared room where tenants take turns, so each waits its turn.
The picture breaks in two ways. MIG walls come in fixed sizes that you cannot reshape at will. Time-slicing turns look fair on paper, but a waiting tenant feels every spike.
The three mechanisms
| Mechanism | What it does | Good for | Watch out for |
|---|---|---|---|
| MIG (Multi-Instance GPU) | Hardware-partitions a GPU, such as an A100 or H100, into isolated slices | Hard isolation and predictable performance | Slice sizes are fixed, and not all SKUs and modes behave the same |
| MPS (Multi-Process Service) | Lets several CUDA processes share the GPU concurrently | Several small processes | Weaker isolation: one heavy process can still slow the rest |
| Time-slicing (Kubernetes device plugin) | Oversubscribes a GPU to many pods | Dev and batch density | Latency spikes under contention |
Choosing one
Work down the list:
- Two different trusted workloads need hard isolation? Use MIG to split the GPU.
- Several small processes you trust can share one GPU? Use MPS to run them concurrently.
- Dev or batch jobs where latency does not matter? Use time-slicing to pack them densely.
- A latency-sensitive LLM decode service? Use one engine per GPU, or one per MIG instance, with multi-LoRA inside the engine. Do not put the main chat service on time-sliced GPUs.
Warning
Time-slicing is a common cause of unexplained latency spikes. A chat service that looks healthy at low load can miss its time-to-first-token target as soon as a neighbor starts a batch job.
Common mistakes
- Using time-slicing for the main chat SLO. Contention turns into latency spikes.
- Assuming MPS isolates workloads. It shares the GPU, so a heavy process can still hurt the others.
- Assuming MIG slices are flexible. The slice sizes are fixed, so plan around them.
Check yourself
- Which mechanism gives hard isolation between two workloads? Answer: MIG, because it splits the GPU into separate hardware partitions.
- Why not run a chat LLM on time-sliced GPUs? Answer: under contention, turns are shared, which adds latency spikes that break the latency target.
Key takeaways
- MIG gives hard isolation with fixed slices. MPS gives concurrency with weaker isolation. Time-slicing gives density with latency spikes.
- Pack small models with MIG, MPS or time-slicing, depending on how much you trust the neighbors.
- Keep latency-critical LLM decode on one engine per GPU, or per MIG slice.
Further learning
Next, read tenant isolation for limits that protect tenants who share a GPU, whatever the sharing mechanism.
Related
Multi-LoRA and multi-model serving
Serving many fine-tuned adapters and models on shared GPUs efficiently.
Tenant isolation and noisy neighbors
When tenants share GPUs, one heavy customer can push everyone else's latency up. Admission caps, fair scheduling, priority classes, separate pools and per-request limits keep tenants apart.
Serving engines: vLLM, SGLang and TensorRT-LLM
How modern engines are built, how they differ, and which knobs move TTFT, TPOT and tokens per GPU-second.