HumanEval
Python programming problems scored by running the generated code.
Overview
HumanEval asks a model to complete Python functions from their signatures and docstrings, then runs unit tests against what it produced. It solves the problem that most code benchmarks compare generated text to a reference string, which penalises a correct solution written differently. It exists because execution is the only honest oracle for code, and it is the reason pass@k became the standard way to report coding ability.
Technology Graph
How HumanEval connects to the rest of the stack.
Applied to
- Code Generation application Applications HumanEval used for Code Generation High ✓ source
- LLM concept Models HumanEval used for LLM High ✓ source