Attention
Mechanism that weights each part of an input by its relevance to every other part.
Обзор
Attention is the mechanism that lets a model weigh how much each position in a sequence matters to every other position, instead of compressing the whole input into one fixed vector. It solves the information bottleneck that made early encoder-decoder models lose detail on long inputs. It exists because relevance is not uniform: when predicting a word, only a few earlier words actually matter, and attention lets the model learn which. Scaled dot-product attention is the specific formulation used throughout modern language models.
Граф технологий
Как Attention связана с остальным стеком.
Использует и зависит от
- Transformer architecture Algorithms Transformer based on Attention High ✓ source
Используется в
- Image Generation application Applications Attention used by Image Generation High ✓ source
- LLaVA model Multimodal Attention used by LLaVA High ✓ source
- Multimodal AI application Multimodal Attention used by Multimodal AI High ✓ source
- Speech Recognition application Applications Attention used by Speech Recognition High ✓ source
Применяется к
- Softmax concept Algorithms Softmax used for Attention High ✓ source
Инструменты
ПО, которое реализует Attention или интегрируется с ним.