Distillation
Training a small model to reproduce the behaviour of a larger one.
Resumen
Distillation trains a smaller student model to match the outputs, or the full probability distribution, of a larger teacher. It solves the cost of serving a large model in production by moving its behaviour into something cheaper to run. It exists because the teacher's distribution over wrong answers carries information a hard label does not, so the student can learn more from each example than the label alone reveals.
Grafo de tecnologías
Cómo se conecta Distillation con el resto del stack.
Usa y depende de
- Synthetic Data technique Techniques Distillation uses Synthetic Data High ✓ source