Distillation

Training a small model to reproduce the behaviour of a larger one.

technique Técnicas 4 relationships

Visão geral

Distillation trains a smaller student model to match the outputs, or the full probability distribution, of a larger teacher. It solves the cost of serving a large model in production by moving its behaviour into something cheaper to run. It exists because the teacher's distribution over wrong answers carries information a hard label does not, so the student can learn more from each example than the label alone reveals.

Grafo de tecnologias

Como Distillation se liga ao resto do stack.

Usa e depende de

Synthetic Data technique Techniques Distillation uses Synthetic Data High ✓ source

Aplicado a

BERT model Models Distillation used for BERT High ✓ source
Inference concept Infrastructure Distillation used for Inference High ✓ source
LLM concept Models Distillation used for LLM Medium ✓ source

Posição da tecnologia

AI
↓
Técnicas
↓
technique
↓
Distillation

Tecnologias relacionadas

BERT model
Inference concept
Synthetic Data technique
LLM concept

Recursos oficiais

Ainda não há links oficiais registrados.

Última atualização: 2026-10-08
Última verificação: 2026-10-08

Fontes citadas

Cada relação nesta página inclui a sua fonte.

arxiv.org 4 claims