Ainslie and colleagues: grouped-query attention and checkpoint uptraining

What changed

Shared key-value attention heads within query-head groups and described uptraining multi-head checkpoints to obtain a speed-quality compromise.

Scientific scope

Shared key-value attention heads within query-head groups and described uptraining multi-head checkpoints to obtain a speed-quality compromise. Grouped-query attention interpolates between multi-head and multi-query attention. The reported compute fraction and quality tradeoffs describe tested checkpoints, not all architectures or workloads.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Grouped-query attention interpolates between multi-head and multi-query attention. The reported compute fraction and quality tradeoffs describe tested checkpoints, not all architectures or workloads.

Contributors

• Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• EMNLP 2023: GQA original proceedings record

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction