GELU
Activation function used in transformer feed-forward layers.
Überblick
GELU multiplies its input by the probability that a standard normal is less than it, giving a smooth curve that is not monotonic and dips slightly below zero. It solves the problem that a rectifier has a hard corner at zero, which turns out to train slightly worse in deep transformers. It exists as the default activation in most current models, chosen empirically over ReLU.
Technologiegraph
Wie GELU mit dem übrigen Stack zusammenhängt.
Angewendet auf
- Deep Learning concept Algorithms GELU used for Deep Learning High ✓ source