FP8
Eight-bit floating point, used to train and serve models faster.
Resumen
FP8 is an eight-bit floating point format with a small mantissa and a wide exponent range, designed so that gradients and activations can survive the reduced precision. It solves the problem that lower precision means faster arithmetic and less memory movement, but naive eight-bit fixed point destroys the dynamic range training needs. It exists because the formats and the scaling tricks were worked out well enough to use at scale on current hardware.
Grafo de tecnologías
Cómo se conecta FP8 con el resto del stack.
Aplicado a
- Inference concept Infrastructure FP8 used for Inference High ✓ source