Attention
Mechanism that weights each part of an input by its relevance to every other part.
Overview
Attention is the mechanism that lets a model weigh how much each position in a sequence matters to every other position, instead of compressing the whole input into one fixed vector. It solves the information bottleneck that made early encoder-decoder models lose detail on long inputs. It exists because relevance is not uniform: when predicting a word, only a few earlier words actually matter, and attention lets the model learn which. Scaled dot-product attention is the specific formulation used throughout modern language models.
Technology Graph
How Attention connects to the rest of the stack.
Uses and depends on
- Transformer architecture Algorithms Transformer based on Attention High ✓ source
Used by
- Image Generation application Applications Attention used by Image Generation High ✓ source
- LLaVA model Multimodal Attention used by LLaVA High ✓ source
- Multimodal AI application Multimodal Attention used by Multimodal AI High ✓ source
- Speech Recognition application Applications Attention used by Speech Recognition High ✓ source
Applied to
- Softmax concept Algorithms Softmax used for Attention High ✓ source
Tools
Software that implements or integrates with Attention.