Layer Normalization
Normalising activations across features within each sample.
Resumen
Layer normalisation rescales the activations of a layer using their mean and variance within the same sample, so it does not depend on other items in the batch. It solves the problem that batch normalisation behaves inconsistently when sequence lengths vary, which is always true of text. It exists as a stabiliser that works per example, and its placement inside the block has been revised repeatedly since.
Grafo de tecnologías
Cómo se conecta Layer Normalization con el resto del stack.
Aplicado a
- Deep Learning concept Algorithms Layer Normalization used for Deep Learning High ✓ source
- Transformer architecture Algorithms Layer Normalization used for Transformer High ✓ source
- Vanishing Gradient concept Algorithms Layer Normalization solves Vanishing Gradient High ✓ source