The Pile
800GB of text assembled from 22 diverse sources.
Aperçu
The Pile combines twenty-two sources - code, academic papers, books, legal text, forum posts, web pages - into one corpus with declared weights. It solves the problem that a corpus scraped only from the web is dominated by whatever the web is dominated by, which is not what every task needs. It exists as the argument that composition is a design decision with measurable downstream effects.
Graphe des technologies
Comment The Pile se rattache au reste de la pile.
Appliqué à
- LLM concept Models The Pile used for LLM High ✓ source
- Pretraining technique Techniques The Pile used for Pretraining High ✓ source