Benchmarks
The specific tests behind published claims, what each one measures and what it does not. Useful for telling a real capability improvement from a good score.
11 technologies
- MMLU benchmark Multiple-choice benchmark across 57 academic subjects.
- LMArena benchmark Public arena where anonymous model pairs are ranked by human votes.
- MTEB benchmark Benchmark suite ranking embedding models across many tasks.
- SWE-bench benchmark Real GitHub issues that a model must resolve by editing a repository.
- BEIR benchmark Heterogeneous retrieval benchmark for zero-shot evaluation.
- HumanEval benchmark Python programming problems scored by running the generated code.
- GSM8K benchmark Grade-school mathematics word problems requiring multi-step reasoning.
- MT-Bench benchmark Multi-turn conversation quality judged by a model acting as judge.
- GPQA benchmark Graduate-level science questions written to resist search and recall.
- TruthfulQA benchmark Questions where a plausible-sounding answer is false.
- HellaSwag benchmark Commonsense sentence completion that is trivial for people and hard for models.