Common Crawl
Non-profit crawl of the public web, and the base of most training corpora.
Resumen
Common Crawl publishes a regular crawl of the public web as raw page data, free to download, and almost every open training corpus is a filtered derivative of it. It solves the problem that assembling a web-scale corpus is a crawling and hosting operation rather than a modelling one. It exists because the same raw material being filtered differently is what makes C4, FineWeb and The Pile comparable rather than unrelated.
Grafo de tecnologías
Cómo se conecta Common Crawl con el resto del stack.
Usa y depende de
Aplicado a
- Pretraining technique Techniques Common Crawl used for Pretraining High ✓ source