AWhat changed
Built an evaluation framework from real repository issues and corresponding fixes, testing whether language models can edit code to resolve them.
Scientific scope
Built an evaluation framework from real repository issues and corresponding fixes, testing whether language models can edit code to resolve them. Year follows the original 2023 preprint. Benchmark performance is version-, environment- and test-dependent; this record excludes current leaderboard scores and later curated subsets.
Evidence
The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Year follows the original 2023 preprint. Benchmark performance is version-, environment- and test-dependent; this record excludes current leaderboard scores and later curated subsets.
Contributors
- Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- SWE-bench original evaluation project
Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.
Full scientific record · Browse contribution grades · Request a correction