Common Crawl
Non-profit crawl of the public web, and the base of most training corpora.
Overview
Common Crawl publishes a regular crawl of the public web as raw page data, free to download, and almost every open training corpus is a filtered derivative of it. It solves the problem that assembling a web-scale corpus is a crawling and hosting operation rather than a modelling one. It exists because the same raw material being filtered differently is what makes C4, FineWeb and The Pile comparable rather than unrelated.
Technology Graph
How Common Crawl connects to the rest of the stack.
Uses and depends on
Applied to
- Pretraining technique Techniques Common Crawl used for Pretraining High ✓ source