AWhat changed
A language model can keep a long conversation in its prompt, but attending to that growing text becomes costly. Titans explores another route: a small neural memory learns key–value associations while the sequence arrives. Larger prediction errors produce stronger memory updates, while momentum and adaptive decay help balance recent information with older content. Behrouz, Zhong and Mirrokni combine this changing memory with attention in three different architectures. The contribution is the specified test-time memory design and its reported language and long-context experiments, building on fast weights and earlier test-time training.
Scientific scope
A language model can keep a long conversation in its prompt, but attending to that growing text becomes costly. Titans explores another route: a small neural memory learns key–value associations while the sequence arrives. Larger prediction errors produce stronger memory updates, while momentum and adaptive decay help balance recent information with older content. Behrouz, Zhong and Mirrokni combine this changing memory with attention in three different architectures. The contribution is the specified test-time memory design and its reported language and long-context experiments, building on fast weights and earlier test-time training. Results concern the cited December 2024 version and its finite training and benchmark settings. Memory updates cost computation, deeper memories can reduce throughput, and parallel training evaluates gradients at chunk boundaries. Long-context benchmark accuracy still declines with length. Fine-tuned and few-shot comparisons are different conditions. This is neither perfect unlimited recall nor a proof that every task improves. Biological experiments and an unexamined expressivity proof are outside this assessed contribution.
Evidence
The linked original works support the scoped statement and contribution attribution. WRS has not independently replayed experiments or reconstructed a full mathematical proof. Results concern the cited December 2024 version and its finite training and benchmark settings. Memory updates cost computation, deeper memories can reduce throughput, and parallel training evaluates gradients at chunk boundaries. Long-context benchmark accuracy still declines with length. Fine-tuned and few-shot comparisons are different conditions. This is neither perfect unlimited recall nor a proof that every task improves. Biological experiments and an unexamined expressivity proof are outside this assessed contribution.
Contributors
- The named authors in the research record · SCIENCE: Source-listed human researchers and engineers for the exact method or resource. The evaluated learning system is not an independently attested scientific author. Earlier methods, funding and data-collection roles remain separate. Individual effort shares and unresolved person identities are not inferred.
Provisional WRS assessment, edition 2026-10-23.9. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.
Full scientific record · Browse contribution grades · Request a correction