Distillation

Training a small model to reproduce the behaviour of a larger one.

technique Techniques 4 relationships

Overview

Distillation trains a smaller student model to match the outputs, or the full probability distribution, of a larger teacher. It solves the cost of serving a large model in production by moving its behaviour into something cheaper to run. It exists because the teacher's distribution over wrong answers carries information a hard label does not, so the student can learn more from each example than the label alone reveals.

Technology Graph

How Distillation connects to the rest of the stack.

Uses and depends on

Synthetic Data technique Techniques Distillation uses Synthetic Data High ✓ source

Applied to

BERT model Models Distillation used for BERT High ✓ source
Inference concept Infrastructure Distillation used for Inference High ✓ source
LLM concept Models Distillation used for LLM Medium ✓ source

Technology Position

AI
↓
Techniques
↓
technique
↓
Distillation

Related Technologies

BERT model
Inference concept
Synthetic Data technique
LLM concept

Official resources

No official links recorded yet.

Last updated: 2026-10-08
Last verified: 2026-10-08

Sources cited

Every relationship on this page carries a citation.

arxiv.org 4 claims