Layer Normalization
Normalising activations across features within each sample.
Visão geral
Layer normalisation rescales the activations of a layer using their mean and variance within the same sample, so it does not depend on other items in the batch. It solves the problem that batch normalisation behaves inconsistently when sequence lengths vary, which is always true of text. It exists as a stabiliser that works per example, and its placement inside the block has been revised repeatedly since.
Grafo de tecnologias
Como Layer Normalization se liga ao resto do stack.
Aplicado a
- Deep Learning concept Algorithms Layer Normalization used for Deep Learning High ✓ source
- Transformer architecture Algorithms Layer Normalization used for Transformer High ✓ source
- Vanishing Gradient concept Algorithms Layer Normalization solves Vanishing Gradient High ✓ source