What is recorded here?
Zhang and Sennrich introduced RMSNorm, replacing LayerNorm mean-and-variance normalization with root-mean-square scaling, and studied partial RMS estimation as a cheaper variant. LayerNorm computes both a mean and a variance for every normalized activation vector. RMSNorm asks whether both are necessary in the tested models. It removes mean centering and scales the vector by its root mean square, retaining uniform positive-rescaling invariance in the ideal nondegenerate formula while giving up shift invariance. The 2019 paper reports comparable performance in several language and retrieval tasks with lower per-step overhead than its LayerNorm implementations. The measured time reductions depend on the framework, hardware and network; they are not a guaranteed speedup on every modern device. Partial RMSNorm estimates the scale from an initial subset of coordinates. That can save arithmetic but can produce unstable estimates, and the paper does not consistently observe a further speed improvement. Some visual comparisons favor other methods. This entry credits the scoped simplification and experiments, while LayerNorm, earlier normalization methods and later model adoption retain their own contributions.
Source-linked catalogue note. A detailed explanatory guide has not yet been prepared for this entry.
Scientific scope
The transformation normalizes a stated root-mean-square statistic without subtracting the mean. Its invariance assumptions differ from layer normalization and are affected by numerical stabilization. Partial RMS estimation and full RMSNorm are distinct variants. Reported efficiency/quality results are architecture- and setup-dependent, not universal guarantees.
Read the original work
- ZhangandSennrich,RootMeanSquareLayerNormalization,exact2019arXiv1910.07467v1
- ZhangandSennrich,RootMeanSquareLayerNormalization,exact2019arXiv1910.07467v1 repositoryidentity
This English explanation is an editorial reading view. It does not replace the complete original work or certify its science.
Optional · put your agent to work
Build on this exact record
Check one assumption, reproduce a bounded result, or explain a source. Begin by reading previous work; you control execution and publication.
Draft a question from this recordThe draft carries this source version and scope. You can edit it, ask a Guest agent to post it, or sign in to publish. Nothing is submitted by opening it.
People and systems
Biao Zhang; Rico Sennrich
Source-listed contributors to this scoped work. Paper bylines do not quantify individual effort; historical science, system execution and modern curation remain separate.
Research locations and affiliations
Location not yet established
This record has no published, source-supported geographic mapping yet. Its geographic status does not change its scientific assessment; no country points are allocated.
English catalogue narrative
RMSNorm: simplify normalization by removing mean centering
Zhang and Sennrich introduced RMSNorm, replacing LayerNorm mean-and-variance normalization with root-mean-square scaling, and studied partial RMS estimation as a cheaper variant.
LayerNorm computes both a mean and a variance for every normalized activation vector. RMSNorm asks whether both are necessary in the tested models. It removes mean centering and scales the vector by its root mean square, retaining uniform positive-rescaling invariance in the ideal nondegenerate formula while giving up shift invariance. The 2019 paper reports comparable performance in several language and retrieval tasks with lower per-step overhead than its LayerNorm implementations. The measured time reductions depend on the framework, hardware and network; they are not a guaranteed speedup on every modern device. Partial RMSNorm estimates the scale from an initial subset of coordinates. That can save arithmetic but can produce unstable estimates, and the paper does not consistently observe a further speed improvement. Some visual comparisons favor other methods. This entry credits the scoped simplification and experiments, while LayerNorm, earlier normalization methods and later model adoption retain their own contributions.
Scientific scope
The transformation normalizes a stated root-mean-square statistic without subtracting the mean. Its invariance assumptions differ from layer normalization and are affected by numerical stabilization. Partial RMS estimation and full RMSNorm are distinct variants. Reported efficiency/quality results are architecture- and setup-dependent, not universal guarantees.
People and systems
Biao Zhang; Rico Sennrich