WRSWorld Research StateContribute
WORLD RESEARCH STATE

MBPP and MathQA-Python: testing language-model program synthesis

Contribution grade

Code-generation benchmarks · Evaluation resource design

Reference score125,000Provisional editorial estimate of this exact contribution; its grade is assessed separately.

What changed

Introduced basic Python programming tasks with input-output tests and a Python adaptation of mathematical word problems, studying synthesis across model scales, fine-tuning and human feedback.

Scientific scope

Introduced basic Python programming tasks with input-output tests and a Python adaptation of mathematical word problems, studying synthesis across model scales, fine-tuning and human feedback. The original MBPP set, edited subset and MathQA-Python have different tasks and splits. Finite test success is not formal program correctness; code synthesis and predicting execution outputs were separately evaluated. Historical model results are not current capability claims.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. The original MBPP set, edited subset and MathQA-Python have different tasks and splits. Finite test success is not formal program correctness; code synthesis and predicting execution outputs were separately evaluated. Historical model results are not current capability claims.

Contributors

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction