Common Crawl
Non-profit crawl of the public web, and the base of most training corpora.
Visão geral
Common Crawl publishes a regular crawl of the public web as raw page data, free to download, and almost every open training corpus is a filtered derivative of it. It solves the problem that assembling a web-scale corpus is a crawling and hosting operation rather than a modelling one. It exists because the same raw material being filtered differently is what makes C4, FineWeb and The Pile comparable rather than unrelated.
Grafo de tecnologias
Como Common Crawl se liga ao resto do stack.
Usa e depende de
Aplicado a
- Pretraining technique Techniques Common Crawl used for Pretraining High ✓ source