GPUs are the most expensive compute most teams will ever operate — and the easiest to waste. Fragmentation strands half-used cards, over-commitment strands queues, and failures strand users. This guide is the operational playbook: scheduling, utilization, bin-packing, and failover for fleets that must actually stay busy.
The scheduling model
GPU orchestration starts with making GPUs a schedulable resource — in Kubernetes via a device plugin exposing nvidia.com/gpu, in job systems via affinity/anti-affinity rules. The subtlety: GPUs are multi-dimensional. A job consumes not just a card but GPU memory, which is the scarce dimension in practice:
Node: 8 × 80GB cards
Job A needs 48GB → packs onto a card
Job B needs 40GB → packs onto the SAME card (fits: 48+40 ≤ 80? no — 88 > 80)
Memory-aware bin-packing is the difference between 45% and 75% fleet utilization. Default schedulers count GPUs; good schedulers count gigabytes.
Utilization: the 60–80% band
The operating band that balances throughput and resilience:
| Utilization | Reality |
|---|---|
| < 50% | Paying for idle silicon; check scheduling and demand |
| 60–80% | Sweet spot: busy but with failover headroom |
| > 80% | Over-committed: queues grow, failover cannot land |
The metrics to watch: average GPU utilization per node, fragmentation ratio (usable-but-unallocated memory), queue wait time, and preemption rate.
Fragmentation: the silent killer
Fragmentation is the accumulation of partial-GPU gaps that no job can fill. Causes: small jobs packing unevenly, jobs with rigid memory requests, and node drains leaving odd remnants. Mitigations:
- Fractional scheduling — allow jobs to request memory slices with strict bin-packing.
- Consolidation passes — periodically migrate small jobs to re-pack nodes (drain-and-refill during low traffic).
- Job shape awareness — reserve whole cards for large jobs, slices for small ones, by policy rather than luck.
Failover patterns
GPUs fail — drivers crash, cards error, nodes die mid-job. The patterns that keep workloads alive:
1. Job-level retry with checkpointing. Long jobs must checkpoint (every N steps); on node failure, restart from the checkpoint. Without checkpoints, a 3-day training run is a 3-day loss.
2. Capacity headroom. Keep 10–20% of the fleet schedulable as a failover pool. The 60–80% utilization band above is precisely this: the headroom is the failover.
3. Spot/preemptible tier for batch. If the provider offers preemptible GPUs, batch workloads (training, eval, data prep) run there at 60–80% discount — with checkpointing making preemption a non-event.
4. Health-based draining. Node health checks (ECC errors, temperature, driver state) drive automatic draining before a failure becomes an outage.
The fleet dashboard
Operate from five numbers:
utilization % (fleet-wide, per node)
fragmentation % (stranded memory)
queue wait time (jobs waiting for GPUs)
checkpoint age (how recent is the safest restart point)
failover success % (restarts that landed within budget)
The observability for AI systems guide covers the tracing half; these five are the fleet half.
Rightsizing the hardware decision
Orchestration starts before the fleet exists — with choosing what to buy. The decisions that shape everything downstream:
- Memory is the planning unit, not card count. The practical scarce resource is GPU memory. Buy cards whose memory sizes match your job envelope: several 80GB cards versus many 40GB cards changes bin-packing dramatically. Model the job mix against each option before you commit.
- Generations differ more than you think. A newer generation often delivers 2x the memory-bandwidth per dollar. But mixed generations in one fleet create uneven scheduling; prefer homogeneous pools per node group.
- Training vs serving have opposite shapes. Training wants large contiguous jobs and deep checkpointing; serving wants many small replicas and fast startup. Split the fleet by workload type so policies (checkpoint frequency, drain strategy) can differ.
- Don't ignore the CPU and network. GPUs starve without enough PCIe/NVLink bandwidth, CPU to feed data, and fabric for collective communication. A "GPU" problem is frequently a networking problem.
Cost and budgeting the fleet
GPU fleets make fixed cost a real line item, so budgeting discipline matters:
- Charge-back by job. Tag every job with owner, team, and workload so utilization and cost attach to a P&L. Unowned GPU time is how budgets quietly double.
- The idle tax. Track "allocated but under-used" hours — jobs that reserve 8 cards but average 30% utilization. This is invisible in utilization averages and is the single biggest silent cost.
- Preemptible tier economics. Run batch and eval on discounted preemptible capacity when checkpointing makes interruption cheap; keep interactive serving on on-demand or reserved cards. The blend typically cuts blended fleet cost 30–50% (serverless guide shows the alternative of renting none of it).
- Review at a fixed cadence. A monthly fleet review — utilization, fragmentation, idle tax, charge-back — converts orchestration from a firefight into a managed asset.
The multi-tenant reality
As soon as multiple teams share a fleet, you need more than a scheduler:
- Quotas and priorities. Team quotas with guaranteed minimums and a burst tier prevent one team's spike from starving another. Priorities decide who preempts whom during contention.
- Namespace isolation. Separate node pools or network policies so one team's workload cannot see another's data.
- Fair scheduling. A FIFO queue with no fair-share logic lets a giant job block everyone. Use a fair-share scheduler or per-queue weights.
- Reporting to the fleet owner. Each tenant gets a utilization and cost report; the fleet owner gets a cross-tenant view to tune the whole.
Key takeaways
- Make GPU memory the scheduling dimension; it is the scarce resource in practice.
- Run the fleet at 60–80% utilization — busy, with 10–20% headroom as the failover pool.
- Attack fragmentation with fractional scheduling, consolidation passes, and job-shape awareness.
- Tag jobs by owner and team, use preemptible capacity for batch, and review the fleet monthly. Checkpoint everything long-running.
Measuring success
The fleet is healthy when utilization stays in the 60–80% band, fragmentation stays below ~15%, queue wait time holds within budget, checkpoint age stays small, and failover success stays high. Track those five numbers on one dashboard and review them weekly — a fleet that cannot report these is a fleet being operated by vibes.
Conclusion
Conclusion
GPU orchestration is scheduling, utilization, and failover — in that order. Make memory the scheduling dimension, run the fleet at 60–80% with failover headroom, checkpoint everything long-running, and drain failing nodes automatically. The serverless alternative shows when to skip owning this problem entirely.