LM Evaluation Harness
Standard harness for running language model benchmarks.
Aperçu
The LM Evaluation Harness runs a model against a large set of academic benchmarks with one interface and reports the results. It solves the problem that a benchmark score is only comparable if both models were evaluated identically, and most reported scores were not. It exists because shared evaluation code removes the degrees of freedom that make two numbers look comparable when they are not.
Graphe des technologies
Comment LM Evaluation Harness se rattache au reste de la pile.
Appliqué à
- HellaSwag benchmark Benchmarks LM Evaluation Harness used for HellaSwag High ✓ source
- MMLU benchmark Benchmarks LM Evaluation Harness used for MMLU High ✓ source