Multi-Head Attention
Running several attention operations in parallel over different projections.
Обзор
Multi-head attention splits the representation into several subspaces, runs attention independently in each, and concatenates the results. It solves the problem that a single attention distribution has to average over incompatible kinds of relationship - syntactic, semantic, positional - in one weighting. It exists because several narrow attentions outperform one wide one at the same cost.
Граф технологий
Как Multi-Head Attention связана с остальным стеком.
Classification
- Transformer architecture Algorithms Multi-Head Attention part of Transformer High ✓ source
Инструменты
ПО, которое реализует Multi-Head Attention или интегрируется с ним.