Layer Normalization
Normalising activations across features within each sample.
Overview
Layer normalisation rescales the activations of a layer using their mean and variance within the same sample, so it does not depend on other items in the batch. It solves the problem that batch normalisation behaves inconsistently when sequence lengths vary, which is always true of text. It exists as a stabiliser that works per example, and its placement inside the block has been revised repeatedly since.
Technology Graph
How Layer Normalization connects to the rest of the stack.
Applied to
- Deep Learning concept Algorithms Layer Normalization used for Deep Learning High ✓ source
- Transformer architecture Algorithms Layer Normalization used for Transformer High ✓ source
- Vanishing Gradient concept Algorithms Layer Normalization solves Vanishing Gradient High ✓ source