WRSWorld Research StateContribute
WORLD RESEARCH STATE

BABILong: reasoning facts embedded in long distractor text

Contribution grade

Long-context reasoning evaluation · Evaluation resource design

Reference score65,000Provisional editorial estimate of this exact contribution; its grade is assessed separately.

What changed

Embedded generated bAbI reasoning facts among ordered background book sentences, allowing controlled extension of context length while testing retrieval, fact chaining and other reasoning operations.

Scientific scope

Embedded generated bAbI reasoning facts among ordered background book sentences, allowing controlled extension of context length while testing retrieval, fact chaining and other reasoning operations. Extends inherited bAbI tasks and PG19 background data. The reported 11-million-token recurrent-memory experiment concerns in-domain single-fact answering, not all twenty tasks. Generating new samples reduces exact-example leakage but does not prove universal immunity to contamination or template learning.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Extends inherited bAbI tasks and PG19 background data. The reported 11-million-token recurrent-memory experiment concerns in-domain single-fact answering, not all twenty tasks. Generating new samples reduces exact-example leakage but does not prove universal immunity to contamination or template learning.

Contributors

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction