MoE
Mixture of Experts: route each token to a few expert subnetworks instead of all parameters.
Обзор
A mixture-of-experts model contains many expert subnetworks and a router that sends each token to only a few of them. It solves the cost problem of scaling a dense model, where every parameter is used for every token. It exists because total capacity and compute per token can then be increased separately, so a model can hold far more knowledge without a proportional increase in inference cost.
Граф технологий
Как MoE связана с остальным стеком.
Использует и зависит от
- Neural Network concept Algorithms MoE based on Neural Network High ✓ source