Security, cost and GPUs
GPU failures and checkpoints
How to detect failing GPUs with XID errors and DCGM, drain and replace bad nodes, and use checkpoints so a card failure costs minutes of training, not days.
GPUs fail more often than the CPU nodes next to them. A good GPU platform treats that as routine. It detects a bad card, drains the node, replaces the hardware, and loses only the work since the last checkpoint.
Concept
An XID error is an NVIDIA driver error code, logged in the kernel's message buffer. Some XIDs are informational. Others mean the GPU should be fenced off. ECC is the memory error-correction system. A correctable error is fixed silently. An uncorrectable error means the data is bad. DCGM (Data Center GPU Manager) is NVIDIA's monitoring tool for GPU health.
The big idea
The routine is detect, cordon, replace, reschedule. The health check notices the fault. The node is cordoned so no new pods land on it. The hardware is replaced, and the work moves to healthy nodes.
Think of a delivery fleet. A van with a warning light comes off the route, goes to the shop, and returns. The route keeps running without it. The analogy breaks because a GPU can be "fixed" by a reboot that hides the real fault. A reboot does not clear the cause.
Walking through a training failure
A training run checkpoints its state to object storage periodically. A checkpoint saves model weights and optimizer state, so training can resume from there.
| Event | What the platform does |
|---|---|
| A GPU logs an uncorrectable ECC error | DCGM alert fires on the node |
| Health check fails | The node is cordoned, and no new pods land on it |
| The job's worker on that node stops | The job restarts from the last checkpoint on healthy nodes |
| The bad node drains | The card is logged and sent for replacement |
The checkpoint interval sets the worst case. A crash just before the next checkpoint loses everything since the last one. Checkpoint often enough that a single failure loses minutes, not days. Inference is easier: stateless replicas behind a load balancer mean a bad node is just one fewer replica.
Warning
"Reboot cleared it" is not a fix. Log the XID, mark the card for RMA if it keeps failing, and keep a spare pool if your SLA depends on capacity.
What to export
- GPU utilization and memory used, from DCGM.
- XID counts and temperatures.
- GPU-seconds per job, for chargeback and FinOps.
- NCCL timeouts and training step time, from the job itself.
A cluster that is fully allocated but shows 15% GPU utilization has a data or application problem. The fix belongs to the platform team too, because the queue may be provisioned for work that is not reaching the GPUs.
Common mistakes
- Treating a reboot as a fix. The fault comes back.
- Checkpointing rarely. One failure then costs the whole run.
- Watching nodes but not jobs. NCCL timeouts show problems the node metrics miss.
- Keeping inference state on the node. Stateless replicas recover on their own.
Check yourself
- A node reports an uncorrectable memory error. What is the usual response? Drain the node, because uncorrectable errors mean the data on that GPU cannot be trusted.
- Why are stateless inference replicas easier to run on flaky GPUs? Losing one replica only removes capacity. The load balancer sends traffic to the others.
Key takeaways
- GPU failures are routine. Detect them with DCGM and XID monitoring.
- Cordon and drain bad nodes, then replace the hardware.
- Checkpoint often, so one failure loses minutes of training.
- Export job-level metrics as well as node-level ones.
Further learning
Next, see how GPU-seconds feed into cost reporting in cloud cost and FinOps.
Related
GPU infrastructure
Provisioning and scheduling GPUs for training and inference, covering hardware and fabrics, Kubernetes device plugins, gang scheduling, failures and checkpoints. Start here for the map of GPU pages.
Observability: logs and traces
Structured logs, distributed tracing and OpenTelemetry, so you can ask new questions of a running system instead of guessing which dashboard to open.
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.