FP8
Eight-bit floating point, used to train and serve models faster.
Overview
FP8 is an eight-bit floating point format with a small mantissa and a wide exponent range, designed so that gradients and activations can survive the reduced precision. It solves the problem that lower precision means faster arithmetic and less memory movement, but naive eight-bit fixed point destroys the dynamic range training needs. It exists because the formats and the scaling tricks were worked out well enough to use at scale on current hardware.
Technology Graph
How FP8 connects to the rest of the stack.
Applied to
- Inference concept Infrastructure FP8 used for Inference High ✓ source