Common Crawl
Non-profit crawl of the public web, and the base of most training corpora.
Überblick
Common Crawl publishes a regular crawl of the public web as raw page data, free to download, and almost every open training corpus is a filtered derivative of it. It solves the problem that assembling a web-scale corpus is a crawling and hosting operation rather than a modelling one. It exists because the same raw material being filtered differently is what makes C4, FineWeb and The Pile comparable rather than unrelated.
Technologiegraph
Wie Common Crawl mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
Angewendet auf
- Pretraining technique Techniques Common Crawl used for Pretraining High ✓ source