Layer Normalization
Normalising activations across features within each sample.
Überblick
Layer normalisation rescales the activations of a layer using their mean and variance within the same sample, so it does not depend on other items in the batch. It solves the problem that batch normalisation behaves inconsistently when sequence lengths vary, which is always true of text. It exists as a stabiliser that works per example, and its placement inside the block has been revised repeatedly since.
Technologiegraph
Wie Layer Normalization mit dem übrigen Stack zusammenhängt.
Angewendet auf
- Deep Learning concept Algorithms Layer Normalization used for Deep Learning High ✓ source
- Transformer architecture Algorithms Layer Normalization used for Transformer High ✓ source
- Vanishing Gradient concept Algorithms Layer Normalization solves Vanishing Gradient High ✓ source