AI training clusters depend on RDMA over Converged Ethernet (RoCEv2) for GPU-to-GPU communication across nodes. The network interface card (NIC) — such as NVIDIA ConnectX-7/ConnectX-8 or Intel E810 — is the physical gateway, and its PCB design has evolved dramatically from 100G to 400G speeds. This article traces that evolution and the PCB engineering implications.

NIC Speed Evolution and PCB Impact

Each speed doubling forces a step-function change in PCB materials and design methodology. The jump from 50 Gbps PAM4 to 112 Gbps PAM4 is particularly significant — the Nyquist frequency doubles from 12.5 GHz to 28 GHz, where standard FR-4 is essentially opaque.

Material Selection by Speed Grade

At 28 GHz Nyquist (112 Gbps PAM4), insertion loss must be below −1.0 dB/inch to achieve reasonable trace lengths:

• 100G (25 Gbps NRZ): Mid-loss materials acceptable — Megtron 4 (Df 0.008), IT-968 (Df 0.008), or even high-performance FR-4 for short traces. Trace length up to 12" feasible.

• 200G (50 Gbps PAM4): Low-loss required — Megtron 6 (Df 0.002), IT-968G (Df 0.0025). Trace lengths limited to 8–10".

• 400G (112 Gbps PAM4): Ultra-low-loss mandatory — Megtron 7 (Df 0.001), Tachyon-100G (Df 0.0012). Trace lengths 4–6" maximum without redrivers.

• 800G (224 Gbps PAM4): Premium materials only — Megtron 8 (Df <0.001), PTFE-based laminates (Rogers RO3003, Df 0.001). Trace lengths below 3" unless redrivers are used.

PCB Stackup for 400G NICs

A ConnectX-7 class 400G NIC uses 18–22 layers:

• L1: Top — ASIC BGA, DDR4/DDR5 memory, QSFP-DD/OSFP connectors

• L2–L3: GND planes

• L4–L6: High-speed SerDes routing to front-panel connectors (8× 112G lanes per QSFP-DD port)

• L7–L8: PCIe 5.0 ×16 host interface routing

• L9–L10: PWR planes (Vcore, Vddr, Vio)

• L11–L13: DDR5 memory routing

• L14–L16: Additional SerDes + management routing

• L17–L18: PWR + GND

• L19–L20: Bottom — decoupling, boot flash, BMC

QSFP-DD and OSFP Connector Design

400G NICs use QSFP-DD (Quad Small Form Factor Pluggable Double Density) or OSFP (Octal SFP) connectors. Each supports 8 lanes at 56 Gbps or 112 Gbps PAM4:

• QSFP-DD: 76-pin connector, 8 high-speed lanes, backward-compatible with QSFP28. Cage dimensions: 18.35 mm × 86.4 mm.

• OSFP: 60-pin connector, 8 high-speed lanes, slightly larger than QSFP-DD. Supports up to 800G with 8× 112G PAM4. Better thermal performance due to integrated heatsink.

• Via optimization: The cage connector footprint requires a dense via field. All high-speed vias must be backdrilled. The press-fit or SMT mounting style determines hole size and plating requirements.

• Crosstalk: With 8 lanes in a 20 mm wide connector, crosstalk is aggressive. Connector pin assignment follows G-S-S-G-S-S-G-S-S-G pattern. PCB routing must maintain this isolation.

PCIe 5.0 Host Interface

A 400G NIC requires at least PCIe 5.0 ×16 (64 GB/s) to avoid being host-bus-limited. Key PCB considerations:

• Re-timer requirement: PCIe 5.0 ×16 from the edge connector to the ASIC typically exceeds 10 inches, requiring a PCIe retimer (e.g., Astera Labs Aries, 16-lane) on the NIC PCB

• Retimer placement: At the physical midpoint of the channel. Pre-retimer trace loss + post-retimer trace loss should each stay below −15 dB at 16 GHz

• AC coupling: 220 nF capacitors on each TX pair. Placement critical — within 200 mil of the transmitter

Power and Thermal

• Power consumption: A 400G NIC consumes 25–35W typical, up to 50W for DPUs (BlueField-3). Roughly 60% is ASIC core power, 20% SerDes, 20% memory + I/O

• Cooling: Passive heatsink on the ASIC, often with a vapor chamber base. The QSFP-DD/OSFP cage includes an integrated heatsink for the optical module

• PCB copper: 2 oz on power planes; 1 oz RTF on signal layers. Thermal vias under the ASIC for Z-direction heat conduction