V100
The first NVIDIA GPU with tensor cores, and the one early transformers trained on.
Overview
The V100 introduced tensor cores to the datacentre line, which is what turned mixed-precision training from a technique into a default. It solves the throughput problem that made training a large transformer impractical on the previous generation. It exists as the hardware that the original transformer-scaling work ran on, which is why early papers quote V100-hours rather than GPU-hours.
Technology Graph
How V100 connects to the rest of the stack.
Uses and depends on
- Tensor Core hardware Hardware V100 uses Tensor Core High ✓ source