Gemma Team: Gemma 2 distillation-trained small language models

What changed

Used teacher-distribution training for smaller models together with local/global attention and grouped-query attention in an open-weight language-model family.

Scientific scope

Used teacher-distribution training for smaller models together with local/global attention and grouped-query attention in an open-weight language-model family. Distillation and attention components are inherited. The 2B and 9B models use distillation while the 27B model is trained from scratch; one family-level contribution.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Distillation and attention components are inherited. The 2B and 9B models use distillation while the 27B model is trained from scratch; one family-level contribution.

Contributors

• Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.

• Gemma 2: Improving Open Language Models at a Practical Size

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction