Datasets
Corpora and collections used for training and evaluation, with their licensing and provenance, which determines what can legally be built on them.
7 technologies
- Common Crawl dataset Non-profit crawl of the public web, and the base of most training corpora.
- Colossal Clean Crawled Corpus dataset Heavily filtered Common Crawl used to train T5.
- MS MARCO dataset Real search queries with passages and answers, the standard retrieval benchmark.
- LAION-5B dataset Five billion image-text pairs collected from the public web.
- The Pile dataset 800GB of text assembled from 22 diverse sources.
- FineWeb dataset Filtered and deduplicated web corpus with a published processing pipeline.
- SQuAD dataset Questions written by people about Wikipedia articles with answer spans.