Pretraining
The initial large-scale training that produces a base model.
Overview
Pretraining trains a model on a very large general corpus to predict text, producing a base model that has learned language and some world structure but does not follow instructions. It solves the problem that task-specific data is scarce while unlabelled text is abundant. It exists as the expensive stage: everything later, fine-tuning and alignment included, starts from a model that already knows language, which is why so few organisations do it.
Technology Graph
How Pretraining connects to the rest of the stack.
Uses and depends on
- Transformer architecture Algorithms Pretraining based on Transformer High ✓ source
Applied to
- A100 hardware Hardware A100 used for Pretraining High ✓ source
- BF16 concept Algorithms BF16 used for Pretraining High ✓ source
- Backpropagation technique Algorithms Backpropagation used for Pretraining High ✓ source
- Colossal Clean Crawled Corpus dataset Datasets Colossal Clean Crawled Corpus used for Pretraining High ✓ source
- Common Crawl dataset Datasets Common Crawl used for Pretraining High ✓ source
- DeepSpeed framework Frameworks DeepSpeed used for Pretraining High ✓ source
- Dropout technique Techniques Dropout used for Pretraining High ✓ source
- FineWeb dataset Datasets FineWeb used for Pretraining High ✓ source
- Gradient Descent technique Algorithms Gradient Descent used for Pretraining High ✓ source
- LAION-5B dataset Datasets LAION-5B used for Pretraining High ✓ source
- LLM concept Models Pretraining used for LLM High ✓ source
- Learning Rate concept Algorithms Learning Rate used for Pretraining High ✓ source
- MLflow tool MLOps MLflow used for Pretraining High ✓ source
- NVLink hardware Hardware NVLink used for Pretraining High ✓ source
- Ray framework Infrastructure Ray used for Pretraining High ✓ source
- Regularization concept Algorithms Regularization used for Pretraining High ✓ source
- Scaling Laws concept Algorithms Scaling Laws used for Pretraining High ✓ source
- The Pile dataset Datasets The Pile used for Pretraining High ✓ source
- Weights & Biases tool MLOps Weights & Biases used for Pretraining High ✓ source