LM Evaluation Harness
Standard harness for running language model benchmarks.
Resumen
The LM Evaluation Harness runs a model against a large set of academic benchmarks with one interface and reports the results. It solves the problem that a benchmark score is only comparable if both models were evaluated identically, and most reported scores were not. It exists because shared evaluation code removes the degrees of freedom that make two numbers look comparable when they are not.
Grafo de tecnologías
Cómo se conecta LM Evaluation Harness con el resto del stack.
Aplicado a
- HellaSwag benchmark Benchmarks LM Evaluation Harness used for HellaSwag High ✓ source
- MMLU benchmark Benchmarks LM Evaluation Harness used for MMLU High ✓ source