AWhat changed
An agent may master its training game levels yet struggle with a new layout. CoinRun makes this gap measurable by generating many levels and comparing performance on training sets of different sizes with zero-shot test performance. Cobbe and colleagues use the benchmark to study network size, regularization and visual augmentation, and show that substantial overfitting can persist even with many training levels. The contribution is the 2018 evaluation resource and its controlled comparisons; earlier generalization studies and the algorithms being tested retain separate credit.
Scientific scope
An agent may master its training game levels yet struggle with a new layout. CoinRun makes this gap measurable by generating many levels and comparing performance on training sets of different sizes with zero-shot test performance. Cobbe and colleagues use the benchmark to study network size, regularization and visual augmentation, and show that substantial overfitting can persist even with many training levels. The contribution is the 2018 evaluation resource and its controlled comparisons; earlier generalization studies and the algorithms being tested retain separate credit. The main tests use harder levels than the mixed-difficulty training distribution, and the CoinRun observations include painted velocity information. Reported comparisons use finite seeds and budgets; dropout receives longer training than the L2 comparison. Combining interventions does not always help. Random level generation is not literally infinite or proof of zero overlap. The 2019 Procgen suite is a later, differently authored extension and is not included in this 2018 score. No model receives scientific-discovery credit merely for being evaluated.
Evidence
The linked original works support the scoped statement and contribution attribution. WRS has not independently replayed experiments or reconstructed a full mathematical proof. The main tests use harder levels than the mixed-difficulty training distribution, and the CoinRun observations include painted velocity information. Reported comparisons use finite seeds and budgets; dropout receives longer training than the L2 comparison. Combining interventions does not always help. Random level generation is not literally infinite or proof of zero overlap. The 2019 Procgen suite is a later, differently authored extension and is not included in this 2018 score. No model receives scientific-discovery credit merely for being evaluated.
Contributors
- The named authors in the research record · SCIENCE: Source-listed human researchers and engineers for the exact method or resource. The evaluated learning system is not an independently attested scientific author. Earlier methods, funding and data-collection roles remain separate. Individual effort shares and unresolved person identities are not inferred.
Provisional WRS assessment, edition 2026-10-23.6. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.
Full scientific record · Browse contribution grades · Request a correction