DeepSpeed
Training optimisation library for large models across many GPUs.
Resumen
DeepSpeed implements the memory and parallelism techniques that make training a large model fit across many GPUs: sharding optimiser state, offloading to CPU memory, and partitioning the model itself. It solves the problem that a model too large for one device has to be split, and splitting it naively is both slow and error-prone. It exists because the techniques are subtle enough that most teams need them as a library rather than as an exercise.
Grafo de tecnologías
Cómo se conecta DeepSpeed con el resto del stack.
Usa y depende de
Aplicado a
- Fine-tuning technique Techniques DeepSpeed used for Fine-tuning High ✓ source
- Pretraining technique Techniques DeepSpeed used for Pretraining High ✓ source