Layer Normalization
Normalising activations across features within each sample.
Aperçu
Layer normalisation rescales the activations of a layer using their mean and variance within the same sample, so it does not depend on other items in the batch. It solves the problem that batch normalisation behaves inconsistently when sequence lengths vary, which is always true of text. It exists as a stabiliser that works per example, and its placement inside the block has been revised repeatedly since.
Graphe des technologies
Comment Layer Normalization se rattache au reste de la pile.
Appliqué à
- Deep Learning concept Algorithms Layer Normalization used for Deep Learning High ✓ source
- Transformer architecture Algorithms Layer Normalization used for Transformer High ✓ source
- Vanishing Gradient concept Algorithms Layer Normalization solves Vanishing Gradient High ✓ source