BWhat changed
Assembled difficult multimodal academic questions with known short or multiple-choice answers, staged expert review and public/private evaluation splits to probe frontier-model accuracy and calibration.
Scientific scope
Assembled difficult multimodal academic questions with known short or multiple-choice answers, staged expert review and public/private evaluation splits to probe frontier-model accuracy and calibration. This assessment concerns the initial January 2025 evaluation resource with 3,000 questions, rather than later 2,500-question, Rolling or Diamond versions. Its 19 organizers and 644 dataset-contributor positions have different documented roles; names are not resolved person identities or equal effort shares. Filtering and closed-ended scoring do not certify every answer, general intelligence or permanent resistance to contamination.
Evidence
The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. This assessment concerns the initial January 2025 evaluation resource with 3,000 questions, rather than later 2,500-question, Rolling or Diamond versions. Its 19 organizers and 644 dataset-contributor positions have different documented roles; names are not resolved person identities or equal effort shares. Filtering and closed-ended scoring do not certify every answer, general intelligence or permanent resistance to contamination.
Contributors
- Source-listed authors and technical contributors · SCIENCE: The source distinguishes 19 organizers and 644 dataset-contributor positions. Shared resource points are associations, not equal work or 663 benchmark inventors.
- Humanity's Last Exam — original January2025 version
- Humanity's Last Exam — official benchmark project
Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.
Full scientific record · Browse contribution grades · Request a correction