Security, cost and GPUs
Gang scheduling and Slurm vs Kubernetes
Why a multi-GPU job must start all of its workers together, how a default scheduler can deadlock it, and how Kueue, Volcano and Slurm solve the problem differently.
A training job often needs 32 GPUs at the same time, because every worker waits on the others. Gang scheduling starts a group of workers together, or starts none of them. Without it, a job can take part of its GPUs and wait forever for the rest.
Concept
A gang is a group of pods that only makes sense if they all run. A scheduler decides which node each pod goes to. The default Kubernetes scheduler places pods one at a time, so it does not know that 32 pods belong together.
The big idea
Picture a group booking at a restaurant. The party of 32 needs 32 seats. If the restaurant seats 24 of them and holds those seats while the rest wait, the 24 sit idle and block other guests. Gang scheduling means the party sits down only when all 32 seats are free.
Walking through the deadlock
Here is the failure with a default scheduler.
| Step | What happens | GPUs held |
|---|---|---|
| 1 | Job A asks for 32 GPUs and the scheduler places pods one by one | 0 |
| 2 | 24 workers land on GPUs | 24 |
| 3 | The cluster has no room for the last 8, so those workers stay Pending | 24 |
| 4 | Job A cannot run, and the 24 held GPUs cannot serve any other job | 24 wasted |
With a gang scheduler, steps 2 and 3 never happen. The job waits with zero GPUs held. It starts when all 32 fit.
Kueue and Volcano extend Kubernetes with queues, quotas and gang semantics. Slurm, the batch system many HPC groups use, works differently. It hands out a job's nodes as a unit with exclusive allocation.
Slurm vs Kubernetes
| Choose Slurm when | Choose Kubernetes when |
|---|---|
| The organization is HPC-native | The same platform serves online inference |
| Jobs are batch-shaped and large | You already run GitOps and cloud IAM on Kubernetes |
| Fair-share queues per account matter most | You are willing to invest in Kueue or Volcano |
| Exclusive node allocation is the norm | You want one control plane for many kinds of work |
Trade-off
Slurm vs Kubernetes with Kueue or Volcano. Many organizations run both: Slurm for giant pretraining runs, and Kubernetes for serving and smaller jobs. They must share storage and identity, or the two pools drift apart.
Common mistakes
- Assuming the default scheduler handles gangs. It does not. Add Kueue, Volcano or Slurm.
- Running multi-GPU jobs without queues and quotas. One team can take every GPU.
- Running two schedulers with separate storage and identity. Data and access then drift apart.
Check yourself
- What does gang scheduling prevent? Partial starts, where some workers hold GPUs while they wait for the rest of the job.
- When is Slurm the better fit? When the organization is HPC-native and runs large batch jobs with fair-share queues.
Key takeaways
- A multi-GPU training job needs all its workers at once.
- Default Kubernetes places pods one at a time, so it can deadlock a job.
- Kueue, Volcano and Slurm provide gang semantics. Pick based on your existing platform.
Further learning
Next, see what happens when a GPU fails mid-run in GPU failures and checkpoints.
Related
GPU infrastructure
Provisioning and scheduling GPUs for training and inference, covering hardware and fabrics, Kubernetes device plugins, gang scheduling, failures and checkpoints. Start here for the map of GPU pages.
Kubernetes core concepts
The control loop at the heart of Kubernetes, the objects you use every day, and how a request reaches a pod. Start here for the map of Kubernetes pages.
Distributed training
How to train models too big for one GPU, using data, sharding, tensor, pipeline and context parallelism. Covers the memory math that forces the choice and how to judge the result.