AWhat changed
Evaluated GPT-family models fine-tuned on code and introduced HumanEval, a benchmark of Python programming problems tested for functional correctness.
Scientific scope
Evaluated GPT-family models fine-tuned on code and introduced HumanEval, a benchmark of Python programming problems tested for functional correctness. Scope limitation: Codex and HumanEval form one paper-level package. Benchmark pass rates do not show reliable software engineering across real repositories.
Evidence
The retained primary-source abstract, bibliographic record and stated review scope support this provisional assessment. No independent experimental replication by WRS is claimed. Codex and HumanEval form one paper-level package. Benchmark pass rates do not show reliable software engineering across real repositories.
Contributors
- The named authors listed in the linked research record · SCIENCE: Complete source byline retained in the research record; human research direction, method development and evaluation. Individual labor shares are not measured.
- Chen and colleagues: Codex code generation and HumanEval — original paper
- Chen and colleagues: Codex code generation and HumanEval — company research page
Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.
Full scientific record · Browse contribution grades · Request a correction