PCIe Gen5

Overview#

PCIe Gen5 doubles per-lane bandwidth from Gen4 to 32 GT/s. The headline figure most teams care about is x16 bidirectional throughput: 128 GB/s, up from 64 GB/s on Gen4. For AI servers this matters because every GPU's host link, every NVMe SSD's controller link, and every NIC's host attachment shares the PCIe budget.

Volume deployment came with AMD EPYC Genoa, Intel Sapphire Rapids and NVIDIA Grace — all 2022-2023 server platforms. Modern AI host systems (DGX H100, HGX H200, GB200 trays, AMD MI300X UBBs) all assume PCIe Gen5 connectivity.

Specifications#

Where Gen5 Matters in AI Systems#

The first place Gen5 matters is GPU-to-host bandwidth. SXM GPUs use NVLink for GPU-to-GPU and PCIe for host attachment; data loading, gradient checkpoint paging, and CPU offload all share the PCIe link. At Gen4 (64 GB/s) this is often the bottleneck on optimiser-offload-heavy workloads.

The second is NIC attachment. ConnectX-7 and BlueField-3 DPUs use PCIe Gen5 x16 to keep up with InfiniBand NDR and Ethernet 400 GbE. Mismatched generations (Gen4 host + Gen5 NIC) leave bandwidth on the table.

The third is NVMe storage. Gen5 SSDs can saturate ~14 GB/s sequential reads, useful for large dataset shuffling.

Pitfalls#

• Gen5 trace lengths are short — retimers are often required on motherboards with non-trivial GPU-to-CPU path lengths.
• Signal integrity at 32 GT/s is fussy; poor cabling or connectors throttle silently to Gen4.
• Higher per-lane power than Gen4 means thermal margins on Gen5 switches and retimers require attention.
• Mixed Gen4/Gen5 fleets can confuse capacity planning — always confirm negotiated link speed in production telemetry.

Software Notes#

PCIe is transparent to most software. The relevant telemetry — link speed, link width, retransmit counts — is available via lspci, nvidia-smi and the relevant vendor tools. Production monitoring should alarm on degraded link speeds or width.