A100
NVIDIA's datacentre GPU that made large-model training routine.
Overview
The A100 is the GPU that large language model training standardised on: tensor cores, high-bandwidth memory and multi-instance partitioning so one card can serve several workloads. It solves the problem that a model only trains if it fits in memory and can be fed data fast enough, and it did so at a price and support level that made training a frontier model an engineering project rather than a research one. It exists as the baseline every later accelerator is compared against.
Technology Graph
How A100 connects to the rest of the stack.
Uses and depends on
Applied to
- Pretraining technique Techniques A100 used for Pretraining High ✓ source