Colossal Clean Crawled Corpus
Heavily filtered Common Crawl used to train T5.
Overview
C4 is a cleaned and filtered derivative of a single Common Crawl snapshot, built for training and released with the T5 paper. It solves the problem that raw crawl data contains boilerplate, navigation and machine-generated text at a volume that damages training. It exists as the corpus whose filtering rules became the baseline every later web dataset is measured against.
Technology Graph
How Colossal Clean Crawled Corpus connects to the rest of the stack.
Uses and depends on
- Common Crawl dataset Datasets Colossal Clean Crawled Corpus based on Common Crawl High ✓ source
Applied to
- Pretraining technique Techniques Colossal Clean Crawled Corpus used for Pretraining High ✓ source
Alternatives
Comparisons
Side-by-side breakdowns with the alternatives above.