DeepSpeed
Training optimisation library for large models across many GPUs.
Überblick
DeepSpeed implements the memory and parallelism techniques that make training a large model fit across many GPUs: sharding optimiser state, offloading to CPU memory, and partitioning the model itself. It solves the problem that a model too large for one device has to be split, and splitting it naively is both slow and error-prone. It exists because the techniques are subtle enough that most teams need them as a library rather than as an exercise.
Technologiegraph
Wie DeepSpeed mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
Angewendet auf
- Fine-tuning technique Techniques DeepSpeed used for Fine-tuning High ✓ source
- Pretraining technique Techniques DeepSpeed used for Pretraining High ✓ source