HumanEval
Python programming problems scored by running the generated code.
Visão geral
HumanEval asks a model to complete Python functions from their signatures and docstrings, then runs unit tests against what it produced. It solves the problem that most code benchmarks compare generated text to a reference string, which penalises a correct solution written differently. It exists because execution is the only honest oracle for code, and it is the reason pass@k became the standard way to report coding ability.
Grafo de tecnologias
Como HumanEval se liga ao resto do stack.
Aplicado a
- Code Generation application Applications HumanEval used for Code Generation High ✓ source
- LLM concept Models HumanEval used for LLM High ✓ source