GPUs are the most expensive compute most teams will ever operate — and the easiest to waste. Fragmentation strands half-used cards, over-commitment strands queues, and failures strand users. This guide is the operational playbook: scheduling, utilization, bin-packing, and failover for fleets that must actually stay busy.
The scheduling model
GPU orchestration starts with making GPUs a schedulable resource — in Kubernetes via a device plugin exposing nvidia.com/gpu, in job systems via affinity/anti-affinity rules. The subtlety: GPUs are multi-dimensional. A job consumes not just a card but GPU memory, which is the scarce dimension in practice:
Node: 8 × 80GB cards
Job A needs 48GB → packs onto a card
Job B needs 40GB → packs onto the SAME card (fits: 48+40 ≤ 80? no — 88 > 80)
Memory-aware bin-packing is the difference between 45% and 75% fleet utilization. Default schedulers count GPUs; good schedulers count gigabytes.
Utilization: the 60–80% band
The operating band that balances throughput and resilience:
| Utilization | Reality |
|---|---|
| < 50% | Paying for idle silicon; check scheduling and demand |
| 60–80% | Sweet spot: busy but with failover headroom |
| > 80% | Over-committed: queues grow, failover cannot land |
The metrics to watch: average GPU utilization per node, fragmentation ratio (usable-but-unallocated memory), queue wait time, and preemption rate.
Fragmentation: the silent killer
Fragmentation is the accumulation of partial-GPU gaps that no job can fill. Causes: small jobs packing unevenly, jobs with rigid memory requests, and node drains leaving odd remnants. Mitigations:
- Fractional scheduling — allow jobs to request memory slices with strict bin-packing.
- Consolidation passes — periodically migrate small jobs to re-pack nodes (drain-and-refill during low traffic).
- Job shape awareness — reserve whole cards for large jobs, slices for small ones, by policy rather than luck.
Failover patterns
GPUs fail — drivers crash, cards error, nodes die mid-job. The patterns that keep workloads alive:
1. Job-level retry with checkpointing. Long jobs must checkpoint (every N steps); on node failure, restart from the checkpoint. Without checkpoints, a 3-day training run is a 3-day loss.
2. Capacity headroom. Keep 10–20% of the fleet schedulable as a failover pool. The 60–80% utilization band above is precisely this: the headroom is the failover.
3. Spot/preemptible tier for batch. If the provider offers preemptible GPUs, batch workloads (training, eval, data prep) run there at 60–80% discount — with checkpointing making preemption a non-event.
4. Health-based draining. Node health checks (ECC errors, temperature, driver state) drive automatic draining before a failure becomes an outage.
The fleet dashboard
Operate from five numbers:
utilization % (fleet-wide, per node)
fragmentation % (stranded memory)
queue wait time (jobs waiting for GPUs)
checkpoint age (how recent is the safest restart point)
failover success % (restarts that landed within budget)
The observability for AI systems guide covers the tracing half; these five are the fleet half.
Conclusion
GPU orchestration is scheduling, utilization, and failover — in that order. Make memory the scheduling dimension, run the fleet at 60–80% with failover headroom, checkpoint everything long-running, and drain failing nodes automatically. The serverless alternative shows when to skip owning this problem entirely.