DeepSpeed
Training optimisation library for large models across many GPUs.
Overview
DeepSpeed implements the memory and parallelism techniques that make training a large model fit across many GPUs: sharding optimiser state, offloading to CPU memory, and partitioning the model itself. It solves the problem that a model too large for one device has to be split, and splitting it naively is both slow and error-prone. It exists because the techniques are subtle enough that most teams need them as a library rather than as an exercise.
Technology Graph
How DeepSpeed connects to the rest of the stack.
Uses and depends on
Applied to
- Fine-tuning technique Techniques DeepSpeed used for Fine-tuning High ✓ source
- Pretraining technique Techniques DeepSpeed used for Pretraining High ✓ source