The Pile
800GB of text assembled from 22 diverse sources.
Overview
The Pile combines twenty-two sources - code, academic papers, books, legal text, forum posts, web pages - into one corpus with declared weights. It solves the problem that a corpus scraped only from the web is dominated by whatever the web is dominated by, which is not what every task needs. It exists as the argument that composition is a design decision with measurable downstream effects.
Technology Graph
How The Pile connects to the rest of the stack.
Applied to
- LLM concept Models The Pile used for LLM High ✓ source
- Pretraining technique Techniques The Pile used for Pretraining High ✓ source