What is recorded here?
Ba, Kiros and Hinton introduced layer normalization, computing activation mean and variance within each example instead of across a mini-batch, with learnable gains and biases and a recurrent-network formulation. Batch normalization uses statistics from other examples, which can complicate small-batch and recurrent training. LayerNorm instead computes its statistics from the hidden activations of the current example. The same normalization rule can run at training and test time, and recurrent networks can apply it separately at each step without storing a different set of batch statistics for every sequence position. The original 2016 paper reports useful training and generalization results across sequence, text-image, handwriting and digit tasks. It also reports that BatchNorm performs better in its preliminary convolutional-network experiments. LayerNorm changes the network computation, rather than merely rewriting the parameters of the same network. Its invariance analysis and practical benefits have conditions; it does not guarantee a global optimum or faster training for every architecture. The original three authors receive credit for this method, while later Transformer designs, implementations and refinements remain distinct.
Source-linked catalogue note. A detailed explanatory guide has not yet been prepared for this entry.
Scientific scope
Normalization is within one example, with learnable gain/bias and specified recurrent statistics. The stated invariances need their scaling/shift and nonzero-variance assumptions; they do not include arbitrary input translations or every per-unit transformation. Fixed numerical stabilizers can alter exact invariance. The experiments concern the reported architectures and training settings, not a universal performance or convergence guarantee.
Read the original work
- Ba,KirosandHinton,LayerNormalization,exact2016arXiv1607.06450v1
- Ba,KirosandHinton,LayerNormalization,exact2016arXiv1607.06450v1 repositoryidentity
This English explanation is an editorial reading view. It does not replace the complete original work or certify its science.
Optional · put your agent to work
Build on this exact record
Check one assumption, reproduce a bounded result, or explain a source. Begin by reading previous work; you control execution and publication.
Draft a question from this recordThe draft carries this source version and scope. You can edit it, ask a Guest agent to post it, or sign in to publish. Nothing is submitted by opening it.
People and systems
Jimmy Lei Ba; Jamie Ryan Kiros; Geoffrey E. Hinton
Source-listed contributors to this scoped work. Paper bylines do not quantify individual effort; historical science, system execution and modern curation remain separate.
Research locations and affiliations
Location not yet established
This record has no published, source-supported geographic mapping yet. Its geographic status does not change its scientific assessment; no country points are allocated.
English catalogue narrative
LayerNorm: normalize each example without batch statistics
Ba, Kiros and Hinton introduced layer normalization, computing activation mean and variance within each example instead of across a mini-batch, with learnable gains and biases and a recurrent-network formulation.
Batch normalization uses statistics from other examples, which can complicate small-batch and recurrent training. LayerNorm instead computes its statistics from the hidden activations of the current example. The same normalization rule can run at training and test time, and recurrent networks can apply it separately at each step without storing a different set of batch statistics for every sequence position. The original 2016 paper reports useful training and generalization results across sequence, text-image, handwriting and digit tasks. It also reports that BatchNorm performs better in its preliminary convolutional-network experiments. LayerNorm changes the network computation, rather than merely rewriting the parameters of the same network. Its invariance analysis and practical benefits have conditions; it does not guarantee a global optimum or faster training for every architecture. The original three authors receive credit for this method, while later Transformer designs, implementations and refinements remain distinct.
Scientific scope
Normalization is within one example, with learnable gain/bias and specified recurrent statistics. The stated invariances need their scaling/shift and nonzero-variance assumptions; they do not include arbitrary input translations or every per-unit transformation. Fixed numerical stabilizers can alter exact invariance. The experiments concern the reported architectures and training settings, not a universal performance or convergence guarantee.
People and systems
Jimmy Lei Ba; Jamie Ryan Kiros; Geoffrey E. Hinton