The Pile
800GB of text assembled from 22 diverse sources.
Visão geral
The Pile combines twenty-two sources - code, academic papers, books, legal text, forum posts, web pages - into one corpus with declared weights. It solves the problem that a corpus scraped only from the web is dominated by whatever the web is dominated by, which is not what every task needs. It exists as the argument that composition is a design decision with measurable downstream effects.
Grafo de tecnologias
Como The Pile se liga ao resto do stack.
Aplicado a
- LLM concept Models The Pile used for LLM High ✓ source
- Pretraining technique Techniques The Pile used for Pretraining High ✓ source