Attention
Mechanism that weights each part of an input by its relevance to every other part.
Aperçu
Attention is the mechanism that lets a model weigh how much each position in a sequence matters to every other position, instead of compressing the whole input into one fixed vector. It solves the information bottleneck that made early encoder-decoder models lose detail on long inputs. It exists because relevance is not uniform: when predicting a word, only a few earlier words actually matter, and attention lets the model learn which. Scaled dot-product attention is the specific formulation used throughout modern language models.
Graphe des technologies
Comment Attention se rattache au reste de la pile.
Utilise et dépend de
- Transformer architecture Algorithms Transformer based on Attention High ✓ source
Utilisé par
- Image Generation application Applications Attention used by Image Generation High ✓ source
- LLaVA model Multimodal Attention used by LLaVA High ✓ source
- Multimodal AI application Multimodal Attention used by Multimodal AI High ✓ source
- Speech Recognition application Applications Attention used by Speech Recognition High ✓ source
Appliqué à
- Softmax concept Algorithms Softmax used for Attention High ✓ source
Outils
Logiciels qui implémentent Attention ou s'y intègrent.