Learning Rate
How far each gradient step moves the weights.
Überblick
The learning rate sets the size of the step taken against the gradient. It solves nothing by itself, but it is the single most consequential training setting: too large and the loss diverges, too small and training never finishes. It exists as the reason schedules and warmups exist, since the best value changes over the course of a run.
Technologiegraph
Wie Learning Rate mit dem übrigen Stack zusammenhängt.
Genutzt von
- Gradient Descent technique Algorithms Learning Rate used by Gradient Descent High ✓ source
Angewendet auf
- Fine-tuning technique Techniques Learning Rate used for Fine-tuning High ✓ source
- Pretraining technique Techniques Learning Rate used for Pretraining High ✓ source