FP8
Eight-bit floating point, used to train and serve models faster.
Aperçu
FP8 is an eight-bit floating point format with a small mantissa and a wide exponent range, designed so that gradients and activations can survive the reduced precision. It solves the problem that lower precision means faster arithmetic and less memory movement, but naive eight-bit fixed point destroys the dynamic range training needs. It exists because the formats and the scaling tricks were worked out well enough to use at scale on current hardware.
Graphe des technologies
Comment FP8 se rattache au reste de la pile.
Appliqué à
- Inference concept Infrastructure FP8 used for Inference High ✓ source