LM Evaluation Harness
Standard harness for running language model benchmarks.
Visão geral
The LM Evaluation Harness runs a model against a large set of academic benchmarks with one interface and reports the results. It solves the problem that a benchmark score is only comparable if both models were evaluated identically, and most reported scores were not. It exists because shared evaluation code removes the degrees of freedom that make two numbers look comparable when they are not.
Grafo de tecnologias
Como LM Evaluation Harness se liga ao resto do stack.
Aplicado a
- HellaSwag benchmark Benchmarks LM Evaluation Harness used for HellaSwag High ✓ source
- MMLU benchmark Benchmarks LM Evaluation Harness used for MMLU High ✓ source