DeepSpeed
Training optimisation library for large models across many GPUs.
Обзор
DeepSpeed implements the memory and parallelism techniques that make training a large model fit across many GPUs: sharding optimiser state, offloading to CPU memory, and partitioning the model itself. It solves the problem that a model too large for one device has to be split, and splitting it naively is both slow and error-prone. It exists because the techniques are subtle enough that most teams need them as a library rather than as an exercise.
Граф технологий
Как DeepSpeed связана с остальным стеком.
Использует и зависит от
Применяется к
- Fine-tuning technique Techniques DeepSpeed used for Fine-tuning High ✓ source
- Pretraining technique Techniques DeepSpeed used for Pretraining High ✓ source