FlashAttention
Attention computed in tiles to avoid materialising the full matrix.
Обзор
FlashAttention computes attention in blocks that fit in fast on-chip memory, never writing the full attention matrix to high-bandwidth memory. It solves the problem that attention's cost is dominated by memory traffic rather than arithmetic, so a mathematically identical computation can be several times faster when scheduled differently. It exists as the clearest case of a systems change producing a large speed-up with no change to the model.
Граф технологий
Как FlashAttention связана с остальным стеком.
Применяется к
- Inference concept Infrastructure FlashAttention used for Inference High ✓ source
Инструменты
ПО, которое реализует FlashAttention или интегрируется с ним.