Residual Connection
Adding a layer's input to its output so gradients can pass through unchanged.
Overview
A residual connection adds the input of a block to its output, so the block learns a modification rather than a replacement. It solves the problem that gradients vanish as depth grows, which stops very deep networks training at all. It exists as the change that made depth practical, and it is why a transformer block is drawn as an addition rather than a sequence.
Technology Graph
How Residual Connection connects to the rest of the stack.
Applied to
- Deep Learning concept Algorithms Residual Connection used for Deep Learning High ✓ source
- Vanishing Gradient concept Algorithms Residual Connection solves Vanishing Gradient High ✓ source
Classification
- Transformer architecture Algorithms Residual Connection part of Transformer High ✓ source