Distillation

Training a small model to reproduce the behaviour of a larger one.

technique Методы 4 relationships

Обзор

Distillation trains a smaller student model to match the outputs, or the full probability distribution, of a larger teacher. It solves the cost of serving a large model in production by moving its behaviour into something cheaper to run. It exists because the teacher's distribution over wrong answers carries information a hard label does not, so the student can learn more from each example than the label alone reveals.

Граф технологий

Как Distillation связана с остальным стеком.

Использует и зависит от

Synthetic Data technique Techniques Distillation uses Synthetic Data High ✓ source

Применяется к

BERT model Models Distillation used for BERT High ✓ source
Inference concept Infrastructure Distillation used for Inference High ✓ source
LLM concept Models Distillation used for LLM Medium ✓ source

Место технологии

AI
↓
Методы
↓
technique
↓
Distillation

Связанные технологии

BERT model
Inference concept
Synthetic Data technique
LLM concept

Официальные ресурсы

Официальные ссылки пока не добавлены.

Последнее обновление: 2026-10-08
Последняя проверка: 2026-10-08

Источники

Каждая связь на этой странице снабжена ссылкой на источник.

arxiv.org 4 claims