NVLink
High-bandwidth interconnect between GPUs, letting them share memory directly.
Overview
NVLink is a direct GPU-to-GPU interconnect that carries far more bandwidth than the PCIe bus it supplements, and it lets one GPU read another's memory. It solves the problem that training a large model across several cards is bounded by how fast gradients can be exchanged, not by each card's own arithmetic. It exists because as models outgrew one device, the interconnect became the thing worth engineering.
Technology Graph
How NVLink connects to the rest of the stack.
Used by
Applied to
- Pretraining technique Techniques NVLink used for Pretraining High ✓ source