GPUs on Kubernetes

Kubernetes knows nothing about GPUs until a device plugin tells it. Once it does, a pod asks for whole GPUs the same way it asks for CPU and memory. This page is the path from a bare GPU node to a pod that uses one.

The big idea

The stack has four layers, and each one depends on the one below it:

1. NVIDIA GPU Operator (or your distro's equivalent). It installs drivers, the container toolkit, the device plugin and GPU metrics exporters.
2. Node pools labeled and tainted, such as nvidia.com/gpu=true:NoSchedule.
3. Pods that tolerate the taint and request nvidia.com/gpu: N.
4. Sharing, with MIG or time-slicing, if one GPU is too big for one replica.

Walking through a pod

Here is a pod that asks for one H100 on a matching node.

apiVersion: v1
kind: Pod
metadata:
  name: infer-worker
spec:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
  containers:
    - name: server
      image: registry.example.com/infer:1.4
      resources:
        limits:
          nvidia.com/gpu: 1

Apply it with kubectl apply -f infer-worker.yaml. On a cluster with a matching H100 node, the output is pod/infer-worker created. Without a matching node, the pod sits in Pending, which is the first thing to check.

The GPU count is an integer. The scheduler counts whole GPUs, and it does not know about partial memory. That is why sharing needs MIG or time-slicing.

Sharing one GPU

Common mistakes

• Leaving the driver unpinned. Pin driver versions. A surprise upgrade mid-training is an outage.
• Using time-slicing for latency-sensitive serving. Neighbors cause tail latency.
• Forgetting the taint. CPU pods then land on GPU nodes and waste them.
• Copying a density setting from a blog. Test p99 under your own load first.

Check yourself

1. Why does the pod request nvidia.com/gpu: 1 rather than a memory amount? The device plugin advertises GPUs as whole integers, so the scheduler counts whole GPUs.
2. What does MIG give you that time-slicing does not? Hardware-isolated memory for each slice. Time-slicing shares the GPU in software with weaker isolation.

Key takeaways

• A device plugin, usually installed by the GPU Operator, makes GPUs schedulable.
• Taints keep ordinary pods off GPU nodes. Tolerations let GPU pods on.
• MIG gives isolation and time-slicing gives flexibility. Pick based on the latency and tenancy needs.

Further learning

Next, see why a plain scheduler can deadlock a large training job in gang scheduling. The NVIDIA GPU Operator docs are the reference for installing the stack.