Attention
Mechanism that weights each part of an input by its relevance to every other part.
Überblick
Attention is the mechanism that lets a model weigh how much each position in a sequence matters to every other position, instead of compressing the whole input into one fixed vector. It solves the information bottleneck that made early encoder-decoder models lose detail on long inputs. It exists because relevance is not uniform: when predicting a word, only a few earlier words actually matter, and attention lets the model learn which. Scaled dot-product attention is the specific formulation used throughout modern language models.
Technologiegraph
Wie Attention mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Transformer architecture Algorithms Transformer based on Attention High ✓ source
Genutzt von
- Image Generation application Applications Attention used by Image Generation High ✓ source
- LLaVA model Multimodal Attention used by LLaVA High ✓ source
- Multimodal AI application Multimodal Attention used by Multimodal AI High ✓ source
- Speech Recognition application Applications Attention used by Speech Recognition High ✓ source
Angewendet auf
- Softmax concept Algorithms Softmax used for Attention High ✓ source
Werkzeuge
Software, die Attention implementiert oder integriert.