GPU Cluster Architecture: 12 Critical Network Decisions
GPU cluster architecture is becoming one of the defining design decisions in modern AI data centers. As accelerator counts grow from dozens to thousands, the network can no longer be treated as supporting equipment around the compute. It determines how effectively GPUs share data, how quickly distributed jobs finish, how much infrastructure is utilized, and how much useful AI output a facility produces from every megawatt.
The challenge is that one network cannot efficiently solve every communication problem inside an AI factory. Modern systems increasingly use separate interconnect domains for separate jobs. Scale-up networking connects accelerators that must behave almost like one computer. Scale-out networking connects servers and racks across the facility. NVIDIA also describes a scale-in domain for storage, security, access, observability, and other services surrounding accelerated compute.
GPU Cluster Architecture Executive Summary
- Scale-up provides extremely high-bandwidth, low-latency communication inside a tightly coupled accelerator domain.
- Scale-out connects GPU systems across racks and pods using Ethernet or InfiniBand.
- Scale-in handles storage, security, application access, management, telemetry, and north-south traffic.
- Performance depends on assigning every traffic class to the fabric designed for its latency, bandwidth, and resilience requirements.
Why GPU Clusters Need Multiple Networks
AI training and inference create very different communication patterns. Accelerators may exchange tensors, gradients, parameters, activations, or expert-routing data at extreme speed. These operations are often synchronized, meaning one slow transfer can delay many GPUs.
Meanwhile, servers must access storage, receive requests, load datasets, write checkpoints, interact with orchestration platforms, and enforce security controls. Treating all those flows as one generic networking problem creates compromises. A scale-up interconnect does not require the same reach and interoperability as a storage or application network. A conventional data center network may offer excellent compatibility while lacking the latency and congestion behavior required by synchronized accelerators.
Scale-Up Makes GPUs Behave Like One System
Scale-up networking connects accelerators within a closely coupled compute domain. Its purpose is to distribute a workload across many GPUs without allowing data movement to consume an unacceptable share of execution time.
NVIDIA NVLink illustrates this approach. Sixth-generation NVLink provides up to 3.6 terabytes per second of bidirectional bandwidth per GPU and supports low-latency, all-to-all communication within rack-scale systems. NVIDIA says an NVL72 rack provides up to 260TB/s of aggregate bandwidth.
Rack-Scale Networking Is Part Of Compute
NVIDIA increasingly treats the rack rather than the server as the basic unit of AI compute. NVL72 systems combine accelerators, CPUs, switches, and networking into one platform. The scale-up fabric gives GPUs substantially more communication bandwidth than a standard server NIC normally provides.
This is commercially important. An AI factory creates value when accelerators complete useful work, not simply when they are installed. Communication overhead that lowers utilization strands part of the capital investment and allocated power. Scale-up networking is therefore part of the computing system itself.
Scale-Out Connects The AI Factory
Scale-up domains cannot expand indefinitely. Multiple racks, pods, or systems must eventually connect across the data center through a scale-out fabric. Ethernet and InfiniBand are the principal technologies used for this layer.
Scale-out networks require high bandwidth and low latency, but they must also support more endpoints and longer distances. Synchronized AI workloads can produce sudden traffic bursts. Static multipath routing may place several large flows on one path while other links remain underused, creating stragglers that delay an entire distributed job.
AI Ethernet Requires More Than Faster Ports
Moving from 400G to 800G Ethernet does not automatically solve congestion. NVIDIA Spectrum-X combines adaptive routing, congestion control, telemetry, and plane load balancing to improve delivered bandwidth for AI traffic.
A fabric can contain enormous theoretical capacity and still perform poorly when traffic repeatedly collides on a subset of links. Infrastructure teams should evaluate latency variation, routing behavior, congestion recovery, and collective-communication performance alongside port speed. Our guide to 800G Ethernet in AI data centers examines those planning issues in more detail.
Scale-Out Topology Affects GPU Economics
Adding switching tiers increases endpoint capacity but also introduces switches, optics, cables, latency, power consumption, and potential congestion points. Spectrum-X Multiplane divides host bandwidth across independent two-tier planes. NVIDIA says the design can support more than 128,000 endpoints while retaining a shallow topology.
Scale-In Supports The Infrastructure Around GPUs
NVIDIA uses scale-in to describe services surrounding accelerated compute rather than communication between GPUs. The domain includes user access, external storage, security, provisioning, data movement, orchestration, observability, and north-south traffic.
NVIDIA’s scale-in implementation uses BlueField-4 DPUs, DOCA software, and Spectrum-X Ethernet. BlueField-4 combines a Grace CPU, dedicated acceleration engines, and high-speed networking to process infrastructure services separately from host compute.
Offload matters because CPU cycles consumed by networking, encryption, storage, policy enforcement, and telemetry are unavailable to applications. At AI-factory scale, infrastructure processing can become a bottleneck of its own.
Storage Belongs In GPU Cluster Architecture
Training systems ingest datasets and write checkpoints for recovery. Inference systems access model weights, embeddings, contextual data, logs, and key-value caches. If storage cannot sustain the required throughput, accelerators become idle even when compute and GPU networking appear healthy.
Operators should design GPU, storage, and application-access networks together. Storage paths must be tested under realistic concurrent load rather than validated independently. The goal is to prevent checkpointing, data loading, or cache access from becoming the slowest stage of the workload.
Resilience Requires Detailed Telemetry
A failed switch or link may not cause a complete outage. It may reduce bandwidth, raise latency, or force traffic onto inefficient paths while the cluster continues operating. That degraded state can quietly extend job completion times and increase cost.
AI networking must expose enough telemetry to distinguish compute problems from fabric problems quickly. Teams should monitor link health, queue depth, congestion events, route changes, collective-operation timing, and workload completion. Resilience plans should include redundant paths, controlled firmware updates, component replacement, and procedures for restoring full performance after failures.
Power And Cooling Include The Network
Large AI fabrics contain thousands of switches, NICs, DPUs, optical modules, and cables. These devices consume power and produce heat while competing with accelerators for a fixed facility budget. Operators should evaluate useful bandwidth per watt rather than port speed alone.
Optics become more significant as links progress through 800G toward 1.6T. Linear pluggable optics, silicon photonics, and co-packaged optics aim to increase bandwidth while controlling electrical and thermal costs. Network plans should therefore be coordinated with rack-power distribution and liquid-cooling architecture.
12 GPU Cluster Architecture Decisions
- Define how many accelerators require tightly coupled scale-up connectivity.
- Identify workloads dominated by collective GPU operations.
- Calculate the number of racks and endpoints the scale-out fabric must support.
- Set acceptable oversubscription, latency, and jitter targets.
- Test congestion control and traffic balancing under synchronized workloads.
- Design storage and checkpoint paths alongside GPU networking.
- Determine which services should be offloaded from host CPUs.
- Specify tenant isolation, security, and policy-enforcement requirements.
- Confirm telemetry can diagnose degraded performance rapidly.
- Include switches, NICs, DPUs, and optics in power budgets.
- Map failure domains and recovery procedures across every fabric.
- Preserve a migration path toward 1.6T and future interconnects.
Map The Compute Domain
Document model parallelism, expert routing, collectives, and distributed inference patterns. Use that evidence to size the scale-up domain and define latency requirements.
Validate Performance Under Load
Define The Operations Domain
Assign responsibility for telemetry, storage access, security offload, firmware, and recovery. Establish thresholds that reveal when network degradation is reducing GPU utilization.
Future Outlook
GPU cluster architecture will become more specialized as AI systems grow. Scale-up domains will connect more accelerators. Scale-out fabrics will adopt 1.6T interfaces and more sophisticated congestion control. Infrastructure processors will handle more networking, storage, security, and management work.
The boundaries will also evolve. NVIDIA describes scale-across networking for distributed AI factories, while context-memory systems create another domain for shared inference data. This specialization reflects the central challenge: no single interconnect can optimize every movement of data across an AI factory.
Frequently Asked Questions
What Is Scale-Up Networking?
Scale-up networking connects accelerators inside a tightly coupled domain using high-bandwidth, low-latency links so they can operate as one larger computing system.
What Is Scale-Out Networking?
Scale-out networking connects GPU servers, racks, and pods across a data center. Ethernet and InfiniBand commonly serve this layer.
What Does Scale-In Mean?
Scale-in describes the storage, security, access, provisioning, data movement, and observability services surrounding accelerated compute.
Why Does Networking Affect AI Performance?
Distributed workloads depend on frequent communication. Congestion, latency, or uneven bandwidth leaves GPUs waiting for data, lowering utilization and extending completion times.
Conclusion
GPU cluster architecture is no longer a matter of choosing one fast network. Scale-up, scale-out, and scale-in address different infrastructure problems. Scale-up creates tightly coupled accelerator domains. Scale-out connects systems across the AI factory. Scale-in supports storage, security, access, data movement, and operations.
The strategic question is not which fabric has the largest headline bandwidth. It is whether every class of data crosses the appropriate network without leaving expensive accelerators waiting. As AI factories grow, that distinction will increasingly determine performance, power efficiency, and return on investment.
Infrastructure teams should revisit these decisions whenever models, accelerator counts, rack densities, network speeds, storage patterns, or operating requirements change across the evolving AI factory at scale.