The Pile
800GB of text assembled from 22 diverse sources.
Überblick
The Pile combines twenty-two sources - code, academic papers, books, legal text, forum posts, web pages - into one corpus with declared weights. It solves the problem that a corpus scraped only from the web is dominated by whatever the web is dominated by, which is not what every task needs. It exists as the argument that composition is a design decision with measurable downstream effects.
Technologiegraph
Wie The Pile mit dem übrigen Stack zusammenhängt.
Angewendet auf
- LLM concept Models The Pile used for LLM High ✓ source
- Pretraining technique Techniques The Pile used for Pretraining High ✓ source