HumanEval
Python programming problems scored by running the generated code.
Resumen
HumanEval asks a model to complete Python functions from their signatures and docstrings, then runs unit tests against what it produced. It solves the problem that most code benchmarks compare generated text to a reference string, which penalises a correct solution written differently. It exists because execution is the only honest oracle for code, and it is the reason pass@k became the standard way to report coding ability.
Grafo de tecnologías
Cómo se conecta HumanEval con el resto del stack.
Aplicado a
- Code Generation application Applications HumanEval used for Code Generation High ✓ source
- LLM concept Models HumanEval used for LLM High ✓ source