The network inside a GPU cluster
Why AI training clusters need a different network from ordinary servers, and the questions to answer before buying one.
When a model trains across many GPUs, those GPUs exchange gradients constantly. If the network is slow or drops packets, expensive GPUs sit idle waiting for it. In a training cluster, the network is part of the computer.
Three networks, not one
Most GPU clusters separate traffic into:
- Compute (back-end) fabric — GPU-to-GPU traffic during training. Highest bandwidth, lowest latency, usually non-blocking (1:1).
- Storage network — loading datasets and writing checkpoints.
- Front-end and management — user access, orchestration, monitoring and out-of-band management.
Keeping them apart stops a large checkpoint write from slowing training traffic.
Lossless transport
GPU traffic typically uses RDMA (remote direct memory access), which performs badly when packets are dropped. The two common approaches are InfiniBand, which is lossless by design, and RoCE (RDMA over Converged Ethernet), which needs careful configuration of priority flow control and congestion notification on every switch.
Questions to answer first
- Training or inference? Inference clusters rarely need a lossless back-end fabric; training clusters almost always do.
- How many GPUs now, and in two years? This sets the switch radix and whether you need two or three tiers.
- InfiniBand or Ethernet? Consider your team's skills and existing tooling, not only benchmark numbers.
- Can the data leave the building? For private or air-gapped deployments, plan model and package mirrors inside your environment from day one.