HumanEval
Python programming problems scored by running the generated code.
Überblick
HumanEval asks a model to complete Python functions from their signatures and docstrings, then runs unit tests against what it produced. It solves the problem that most code benchmarks compare generated text to a reference string, which penalises a correct solution written differently. It exists because execution is the only honest oracle for code, and it is the reason pass@k became the standard way to report coding ability.
Technologiegraph
Wie HumanEval mit dem übrigen Stack zusammenhängt.
Angewendet auf
- Code Generation application Applications HumanEval used for Code Generation High ✓ source
- LLM concept Models HumanEval used for LLM High ✓ source