BF16
Sixteen-bit float that keeps FP32's exponent range for training stability.
Resumen
BF16 is a sixteen-bit format that trades mantissa precision for the same exponent range as FP32. It solves the problem that FP16 overflows and underflows during training, which makes it fragile without loss scaling. It exists because for gradient accumulation the range matters more than the precision, and it has become the default training format.
Grafo de tecnologías
Cómo se conecta BF16 con el resto del stack.
Aplicado a
- Pretraining technique Techniques BF16 used for Pretraining High ✓ source