Press, Smith and Lewis: attention with linear biases

What changed

Biased attention scores with head-specific linear distance penalties and evaluated language-model extrapolation beyond training sequence lengths.

Scientific scope

Biased attention scores with head-specific linear distance penalties and evaluated language-model extrapolation beyond training sequence lengths. Year follows ICLR 2022; first preprint was 2021. Evidence concerns measured length ranges and language-model perplexity, not guaranteed unlimited context or preservation of every downstream capability.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Year follows ICLR 2022; first preprint was 2021. Evidence concerns measured length ranges and language-model perplexity, not guaranteed unlimited context or preservation of every downstream capability.

Contributors

• Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.

• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction