Self-Attention
Weighting every position by every other position in the same sequence.
Обзор
Self-attention computes, for each position, a weighted sum of all positions in the sequence, where the weights come from comparing their query and key vectors. It solves the problem that a recurrent model has to compress everything it has seen into one state before moving on. It exists as the mechanism that removed that bottleneck, and it is what makes a transformer's cost grow with the square of the sequence length.
Граф технологий
Как Self-Attention связана с остальным стеком.
Classification
- Transformer architecture Algorithms Self-Attention part of Transformer High ✓ source
Инструменты
ПО, которое реализует Self-Attention или интегрируется с ним.