Feed-Forward Network
The position-wise fully connected layer inside each transformer block.
Resumen
The feed-forward network is the two-layer fully connected block applied to each position inside a transformer, usually expanding to several times the model dimension and back. It solves the problem that attention only mixes information between positions and performs no transformation of its own. It exists as the majority of a transformer's parameters, which is why mixture-of-experts replaces this layer rather than the attention.
Grafo de tecnologías
Cómo se conecta Feed-Forward Network con el resto del stack.
Classification
- Transformer architecture Algorithms Feed-Forward Network part of Transformer High ✓ source