LM Evaluation Harness
Standard harness for running language model benchmarks.
Überblick
The LM Evaluation Harness runs a model against a large set of academic benchmarks with one interface and reports the results. It solves the problem that a benchmark score is only comparable if both models were evaluated identically, and most reported scores were not. It exists because shared evaluation code removes the degrees of freedom that make two numbers look comparable when they are not.
Technologiegraph
Wie LM Evaluation Harness mit dem übrigen Stack zusammenhängt.
Angewendet auf
- HellaSwag benchmark Benchmarks LM Evaluation Harness used for HellaSwag High ✓ source
- MMLU benchmark Benchmarks LM Evaluation Harness used for MMLU High ✓ source