LM Evaluation Harness
Standard harness for running language model benchmarks.
Overview
The LM Evaluation Harness runs a model against a large set of academic benchmarks with one interface and reports the results. It solves the problem that a benchmark score is only comparable if both models were evaluated identically, and most reported scores were not. It exists because shared evaluation code removes the degrees of freedom that make two numbers look comparable when they are not.
Technology Graph
How LM Evaluation Harness connects to the rest of the stack.
Applied to
- HellaSwag benchmark Benchmarks LM Evaluation Harness used for HellaSwag High ✓ source
- MMLU benchmark Benchmarks LM Evaluation Harness used for MMLU High ✓ source