RoPE
Rotary position embedding, encoding position as a rotation of the query and key.
Обзор
RoPE encodes a token's position by rotating its query and key vectors by an angle proportional to the position, so the attention score depends on relative distance rather than absolute index. It solves the problem that learned absolute position embeddings do not transfer to sequences longer than those seen in training. It exists because relative geometry extrapolates better, and it is what most current models use.
Граф технологий
Как RoPE связана с остальным стеком.
Применяется к
- Transformer architecture Algorithms RoPE used for Transformer High ✓ source
Инструменты
ПО, которое реализует RoPE или интегрируется с ним.