TinyBERT: transformer-layer distillation at pretraining and task adaptation

What changed

Distilled transformer representations and predictions in general-domain and task-specific learning stages to produce compact BERT students.

Scientific scope

Distilled transformer representations and predictions in general-domain and task-specific learning stages to produce compact BERT students. Inherits knowledge distillation and BERT. Original paper's measured GLUE/inference tradeoffs are task/hardware-specific and do not imply universal retention or first teacher-student learning.

Evidence

The linked original sources describe the stated contribution and its evidence limits. WRS has not independently rerun the experiments or certified a proof. Inherits knowledge distillation and BERT. Original paper's measured GLUE/inference tradeoffs are task/hardware-specific and do not imply universal retention or first teacher-student learning.

Contributors

• Source-listed authors and technical contributors · SCIENCE: Source-listed authors or explicitly credited technical contributors for the cited versions. Family entries retain source-specific attribution; a combined list is not one original-paper byline or equal individual work. Names alone do not establish resolved identities.

• TinyBERT: Distilling BERT for Natural Language Understanding

Provisional WRS assessment, edition 2026-10-16.1. Independent expert calibration is pending. This grade does not certify correctness or assign ownership.

Full scientific record · Browse contribution grades · Request a correction