DeepSpeed
Training optimisation library for large models across many GPUs.
Aperçu
DeepSpeed implements the memory and parallelism techniques that make training a large model fit across many GPUs: sharding optimiser state, offloading to CPU memory, and partitioning the model itself. It solves the problem that a model too large for one device has to be split, and splitting it naively is both slow and error-prone. It exists because the techniques are subtle enough that most teams need them as a library rather than as an exercise.
Graphe des technologies
Comment DeepSpeed se rattache au reste de la pile.
Utilise et dépend de
Appliqué à
- Fine-tuning technique Techniques DeepSpeed used for Fine-tuning High ✓ source
- Pretraining technique Techniques DeepSpeed used for Pretraining High ✓ source