预训练
生成基础模型的大规模初始训练。
概览
预训练在一个非常庞大的通用语料库上训练模型来预测文本，产生一个已经学会了语言和某些世界结构但不遵循指令的基础模型。它解决了任务特定数据稀缺而未标注文本丰富的问题。它作为昂贵的阶段存在：后续的一切，包括微调和对齐，都从一个已经懂得语言的模型开始，这就是为什么很少有组织去做这件事。
技术图谱
预训练 如何与技术栈的其他部分相连。
使用与依赖
- Transformer architecture Algorithms 预训练 based on Transformer High ✓ source
应用于
- A100 hardware Hardware A100 used for 预训练 High ✓ source
- BF16 concept Algorithms BF16 used for 预训练 High ✓ source
- Backpropagation technique Algorithms Backpropagation used for 预训练 High ✓ source
- Colossal Clean Crawled Corpus dataset Datasets Colossal Clean Crawled Corpus used for 预训练 High ✓ source
- Common Crawl dataset Datasets Common Crawl used for 预训练 High ✓ source
- DeepSpeed framework Frameworks DeepSpeed used for 预训练 High ✓ source
- Dropout technique Techniques Dropout used for 预训练 High ✓ source
- FineWeb dataset Datasets FineWeb used for 预训练 High ✓ source
- Gradient Descent technique Algorithms Gradient Descent used for 预训练 High ✓ source
- LAION-5B dataset Datasets LAION-5B used for 预训练 High ✓ source
- LLM concept Models 预训练 used for LLM High ✓ source
- Learning Rate concept Algorithms Learning Rate used for 预训练 High ✓ source
- MLflow tool MLOps MLflow used for 预训练 High ✓ source
- NVLink hardware Hardware NVLink used for 预训练 High ✓ source
- Ray framework Infrastructure Ray used for 预训练 High ✓ source
- Regularization concept Algorithms Regularization used for 预训练 High ✓ source
- Scaling Laws concept Algorithms Scaling Laws used for 预训练 High ✓ source
- The Pile dataset Datasets The Pile used for 预训练 High ✓ source
- Weights & Biases tool MLOps Weights & Biases used for 预训练 High ✓ source