FP8
Eight-bit floating point, used to train and serve models faster.
Visão geral
FP8 is an eight-bit floating point format with a small mantissa and a wide exponent range, designed so that gradients and activations can survive the reduced precision. It solves the problem that lower precision means faster arithmetic and less memory movement, but naive eight-bit fixed point destroys the dynamic range training needs. It exists because the formats and the scaling tricks were worked out well enough to use at scale on current hardware.
Grafo de tecnologias
Como FP8 se liga ao resto do stack.
Aplicado a
- Inference concept Infrastructure FP8 used for Inference High ✓ source