HumanEval
Python programming problems scored by running the generated code.
Обзор
HumanEval asks a model to complete Python functions from their signatures and docstrings, then runs unit tests against what it produced. It solves the problem that most code benchmarks compare generated text to a reference string, which penalises a correct solution written differently. It exists because execution is the only honest oracle for code, and it is the reason pass@k became the standard way to report coding ability.
Граф технологий
Как HumanEval связана с остальным стеком.
Применяется к
- Code Generation application Applications HumanEval used for Code Generation High ✓ source
- LLM concept Models HumanEval used for LLM High ✓ source