The Pile
800GB of text assembled from 22 diverse sources.
Resumen
The Pile combines twenty-two sources - code, academic papers, books, legal text, forum posts, web pages - into one corpus with declared weights. It solves the problem that a corpus scraped only from the web is dominated by whatever the web is dominated by, which is not what every task needs. It exists as the argument that composition is a design decision with measurable downstream effects.
Grafo de tecnologías
Cómo se conecta The Pile con el resto del stack.
Aplicado a
- LLM concept Models The Pile used for LLM High ✓ source
- Pretraining technique Techniques The Pile used for Pretraining High ✓ source