Common Crawl
Non-profit crawl of the public web, and the base of most training corpora.
Обзор
Common Crawl publishes a regular crawl of the public web as raw page data, free to download, and almost every open training corpus is a filtered derivative of it. It solves the problem that assembling a web-scale corpus is a crawling and hosting operation rather than a modelling one. It exists because the same raw material being filtered differently is what makes C4, FineWeb and The Pile comparable rather than unrelated.
Граф технологий
Как Common Crawl связана с остальным стеком.
Использует и зависит от
Применяется к
- Pretraining technique Techniques Common Crawl used for Pretraining High ✓ source