Positional Encoding
How a transformer, which has no inherent order, learns where tokens are.
Обзор
Positional encoding injects information about a token's position, because attention itself is permutation-invariant and would otherwise see a bag of words. It solves the problem that order carries most of the meaning in language and none of it is present in the mechanism. It exists as one of the few places where the original architecture has been substantially revised, since how position is represented decides how well a model extrapolates to sequences longer than it was trained on.
Граф технологий
Как Positional Encoding связана с остальным стеком.
Classification
- Transformer architecture Algorithms Positional Encoding part of Transformer High ✓ source
Инструменты
ПО, которое реализует Positional Encoding или интегрируется с ним.