FP8
Eight-bit floating point, used to train and serve models faster.
Überblick
FP8 is an eight-bit floating point format with a small mantissa and a wide exponent range, designed so that gradients and activations can survive the reduced precision. It solves the problem that lower precision means faster arithmetic and less memory movement, but naive eight-bit fixed point destroys the dynamic range training needs. It exists because the formats and the scaling tricks were worked out well enough to use at scale on current hardware.
Technologiegraph
Wie FP8 mit dem übrigen Stack zusammenhängt.
Angewendet auf
- Inference concept Infrastructure FP8 used for Inference High ✓ source