Vanishing Gradient
Gradients shrinking toward zero as they propagate through many layers.
Überblick
The vanishing gradient problem is that a gradient multiplied by many small derivatives decays toward zero before reaching the early layers, so they stop learning. It solves nothing; it is the reason very deep networks could not be trained for years. It exists as the problem that residual connections, gating and normalisation were each introduced to address.
Technologiegraph
Wie Vanishing Gradient mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Backpropagation technique Algorithms Vanishing Gradient based on Backpropagation High ✓ source
Angewendet auf
- Layer Normalization concept Algorithms Layer Normalization solves Vanishing Gradient High ✓ source
- Residual Connection concept Algorithms Residual Connection solves Vanishing Gradient High ✓ source