Attention
Mechanism that weights each part of an input by its relevance to every other part.
Visão geral
Attention is the mechanism that lets a model weigh how much each position in a sequence matters to every other position, instead of compressing the whole input into one fixed vector. It solves the information bottleneck that made early encoder-decoder models lose detail on long inputs. It exists because relevance is not uniform: when predicting a word, only a few earlier words actually matter, and attention lets the model learn which. Scaled dot-product attention is the specific formulation used throughout modern language models.
Grafo de tecnologias
Como Attention se liga ao resto do stack.
Usa e depende de
- Transformer architecture Algorithms Transformer based on Attention High ✓ source
Usado por
- Image Generation application Applications Attention used by Image Generation High ✓ source
- LLaVA model Multimodal Attention used by LLaVA High ✓ source
- Multimodal AI application Multimodal Attention used by Multimodal AI High ✓ source
- Speech Recognition application Applications Attention used by Speech Recognition High ✓ source
Aplicado a
- Softmax concept Algorithms Softmax used for Attention High ✓ source
Ferramentas
Software que implementa ou se integra com Attention.