Distillation
Training a small model to reproduce the behaviour of a larger one.
Überblick
Distillation trains a smaller student model to match the outputs, or the full probability distribution, of a larger teacher. It solves the cost of serving a large model in production by moving its behaviour into something cheaper to run. It exists because the teacher's distribution over wrong answers carries information a hard label does not, so the student can learn more from each example than the label alone reveals.
Technologiegraph
Wie Distillation mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- Synthetic Data technique Techniques Distillation uses Synthetic Data High ✓ source