Distillation

Training a small model to reproduce the behaviour of a larger one.

technique Techniques 4 relationships

Aperçu

Distillation trains a smaller student model to match the outputs, or the full probability distribution, of a larger teacher. It solves the cost of serving a large model in production by moving its behaviour into something cheaper to run. It exists because the teacher's distribution over wrong answers carries information a hard label does not, so the student can learn more from each example than the label alone reveals.

Graphe des technologies

Comment Distillation se rattache au reste de la pile.

Utilise et dépend de

Synthetic Data technique Techniques Distillation uses Synthetic Data High ✓ source

Appliqué à

BERT model Models Distillation used for BERT High ✓ source
Inference concept Infrastructure Distillation used for Inference High ✓ source
LLM concept Models Distillation used for LLM Medium ✓ source

Position de la technologie

AI
↓
Techniques
↓
technique
↓
Distillation

Technologies liées

BERT model
Inference concept
Synthetic Data technique
LLM concept

Ressources officielles

Aucun lien officiel enregistré pour l'instant.

Dernière mise à jour: 2026-10-08
Dernière vérification: 2026-10-08

Sources citées

Chaque relation de cette page porte sa source.

arxiv.org 4 claims