AWhat changed
An assistant might remember one fact yet miss a later correction, confuse dates or answer a question whose premise is false. LongMemEval creates a structured evaluation of these abilities. Its questions require extracting information, reasoning across sessions and time, handling updates and abstaining when evidence is absent. Wu and colleagues combine curated questions with long interaction histories and study memory indexing, retrieval and reading choices. The contribution is the evaluation resource and its scoped design analysis, rather than an achievement by any model tested on it.
Scientific scope
An assistant might remember one fact yet miss a later correction, confuse dates or answer a question whose premise is false. LongMemEval creates a structured evaluation of these abilities. Its questions require extracting information, reasoning across sessions and time, handling updates and abstaining when evidence is absent. Wu and colleagues combine curated questions with long interaction histories and study memory indexing, retrieval and reading choices. The contribution is the evaluation resource and its scoped design analysis, rather than an achievement by any model tested on it. The histories include simulated and edited material, and the evaluation uses specified answer rubrics and model judging. A smaller commercial-assistant pilot uses much shorter histories and historical August 2024 configurations. More retrieved text does not always help, and compression can remove useful details. Benchmark performance is not a guarantee for real users or proof of a scientific statement. The source’s illustrative work-hour arithmetic contains a typo; that example is not copied as a measured result.
Evidence
The linked original works support the scoped statement and contribution attribution. WRS has not independently replayed experiments or reconstructed a full mathematical proof. The histories include simulated and edited material, and the evaluation uses specified answer rubrics and model judging. A smaller commercial-assistant pilot uses much shorter histories and historical August 2024 configurations. More retrieved text does not always help, and compression can remove useful details. Benchmark performance is not a guarantee for real users or proof of a scientific statement. The source’s illustrative work-hour arithmetic contains a typo; that example is not copied as a measured result.
Contributors
- The named authors in the research record · SCIENCE: Source-listed human researchers and engineers for the exact method or resource. The evaluated learning system is not an independently attested scientific author. Earlier methods, funding and data-collection roles remain separate. Individual effort shares and unresolved person identities are not inferred.
Provisional WRS assessment, edition 2026-10-23.9. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.
Full scientific record · Browse contribution grades · Request a correction