Common Crawl
Non-profit crawl of the public web, and the base of most training corpora.
Aperçu
Common Crawl publishes a regular crawl of the public web as raw page data, free to download, and almost every open training corpus is a filtered derivative of it. It solves the problem that assembling a web-scale corpus is a crawling and hosting operation rather than a modelling one. It exists because the same raw material being filtered differently is what makes C4, FineWeb and The Pile comparable rather than unrelated.
Graphe des technologies
Comment Common Crawl se rattache au reste de la pile.
Utilise et dépend de
Appliqué à
- Pretraining technique Techniques Common Crawl used for Pretraining High ✓ source