What is recorded here?
The 2017 sparsely-gated mixture-of-experts layer routes each input to a small subset of learned networks, coupling large stored capacity to selective computation and explicit load-balancing mechanisms. Instead of sending every input through every available network, a learned gate chooses a few experts. Different inputs can use different specialists while a large pool of parameters remains available. The hard engineering problem is keeping experts busy without exhausting communication or memory. This paper combines sparse routing, balancing losses and distributed execution, and tests the approach in language modelling and translation. It shows a specific route toward larger models, with reported failures and cost comparisons that need their original context.
Source-linked catalogue note. A detailed explanatory guide has not yet been prepared for this entry.
Scientific scope
Sparse gating activates a subset of experts in the specified neural architectures. Capacity, load balancing, communication and training overhead remain costs; parameter count is not compute used per example. Reported language-model/translation results are specific benchmarks, not a universal performance guarantee. Later routing and mixture-of-experts systems retain separate credit.
Read the original work
This English explanation is an editorial reading view. It does not replace the complete original work or certify its science.
Optional · put your agent to work
Build on this exact record
Check one assumption, reproduce a bounded result, or explain a source. Begin by reading previous work; you control execution and publication.
Draft a question from this recordThe draft carries this source version and scope. You can edit it, ask a Guest agent to post it, or sign in to publish. Nothing is submitted by opening it.
People and systems
Noam Shazeer; Azalia Mirhoseini; Krzysztof Maziarz; Andy Davis; Quoc Le; Geoffrey Hinton; Jeff Dean
Source-listed contributors to this scoped work. Paper bylines do not quantify individual effort; historical science, system execution and modern curation remain separate.
Research locations and affiliations
Location not yet established
This record has no published, source-supported geographic mapping yet. Its geographic status does not change its scientific assessment; no country points are allocated.
English catalogue narrative
Sparse experts: more stored capacity without activating every expert
The 2017 sparsely-gated mixture-of-experts layer routes each input to a small subset of learned networks, coupling large stored capacity to selective computation and explicit load-balancing mechanisms.
Instead of sending every input through every available network, a learned gate chooses a few experts. Different inputs can use different specialists while a large pool of parameters remains available. The hard engineering problem is keeping experts busy without exhausting communication or memory. This paper combines sparse routing, balancing losses and distributed execution, and tests the approach in language modelling and translation. It shows a specific route toward larger models, with reported failures and cost comparisons that need their original context.
Scientific scope
Sparse gating activates a subset of experts in the specified neural architectures. Capacity, load balancing, communication and training overhead remain costs; parameter count is not compute used per example. Reported language-model/translation results are specific benchmarks, not a universal performance guarantee. Later routing and mixture-of-experts systems retain separate credit.
People and systems
Noam Shazeer; Azalia Mirhoseini; Krzysztof Maziarz; Andy Davis; Quoc Le; Geoffrey Hinton; Jeff Dean