Scaling Laws
Empirical relationship between model size, data, compute and loss.
Überblick
Scaling laws describe how a model's loss falls as parameters, data and compute are increased, and the trade-offs between them. They solve the problem of deciding how to spend a fixed budget before spending it: whether to train a larger model on less data or a smaller one for longer. They exist because the relationship is smooth and predictable enough over many orders of magnitude to be extrapolated, which is what made frontier training a planning exercise rather than a gamble.
Technologiegraph
Wie Scaling Laws mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Transformer architecture Algorithms Scaling Laws based on Transformer High ✓ source
Angewendet auf
- Pretraining technique Techniques Scaling Laws used for Pretraining High ✓ source